Eloi Vanderbéken recently found a backdoor on some common routers, which is described on his GitHub here. Basically, a process that listens on the 32764 TCP port runs, sometimes accessible from the WAN interface. We scanned the v4 Internet to look for the routers that have this backdoor wild open, and gathered some statistics about them. We will also present a way to permanently remove this backdoor on Linksys WAG200G routers.
Note that despite this backdoor allows a free access to many hosts on the Internet, no patch is available as it is not maintained anymore. So we thought about some tricks combined with our tools to imagine how to fix that worldwide.
This backdoor doesn't have any kind of authentication and allows various remote commands, like:
remote root shell
NVRAM configuration dump: Wifi and/or PPPoE credentials can be extracted for instance
file copy
Let's see how many routers are still exposed to this vulnerability, and propose a way to remove this backdoor.
Looking for the backdoor on the Internet
We first used masscan to look for hosts with TCP port 32764 open. We ended up with about 1 million IPv4s [1]. The scan took about 50 hours on a low-end Linux virtual server.
Then, we had to determine whether this was really the backdoor exposed, or some other false positive.
Eloi's POC shows a clear way to do this:
Send a packet to the host device
Wait for an answer with 0x53634D4D (or 0x4D4D6353, according to the endianness of the device, see below)
In such a case, the backdoor is here and accessible.
In order to check the IPs previously discovered, we couldn't use masscan or a similar tool (as they don't have any "plugin" feature). Moreover, sequentially establishing a connection to each IP to verify that the backdoor is present would take ages. For instance, with a 1 second timeout, the worst case scenario is 1 million seconds (about 12 days) and even if half the hosts would answer "directly", it would still run for 6 days. Quick process-based parallelism could help and might divide this time by 10/20. It still remains a lot and is not the good way to do this.
We thus decided to quickly code a scanner based on asynchronous sockets, that will check the availability of the backdoor. The advantage of asynchronous sockets are that lots of them (about 30k in our tests) can be managed at the same time, thus managing 30k hosts in parallel. This kind of parallelism couldn't be achieved with a classical process (or thread)-based parallelism.
This asynchronous model is somehow the same used by masscan and zmap, apart that they bypass sockets to directly emit packets (and thus manage more hosts simultaneously).
Using a classical Linux system, a first implementation using the select function and a five seconds timeout would perform at best ~1000 tests/second. The limitation is mainly due to the fact that the FD_SETSIZE value, on most Linux systems, is set by default to 1024. This means that the maximum file descriptor identifier that select can handle is 1024 (and thus the number of descriptors is inferior or equal to this limit). In the end, this limits the number of sockets that can be opened at the same time, thus the overall scan performance.
Fortunately, other models that do not have this limitation exist. epoll is one of them. After adapting our code, our system was able to test about 6k IP/s (using 30k sockets simultaneously). That is less than what masscan and/or zmap can do (in terms of packets/s), but it gave good enough performance for what we needed to do.
In the end, we found about 6500 routers running this backdoor.
Sources for these scanners are not ready to be published yet (trust me), but stay tuned :)
Note about big vs. little endian
The people that coded this backdoor didn't care about the endianness of the underlying CPU. That's why the signature that is received can have two different values.
To exploit the backdoor, one has to first determine the endianness of the remote router. This is done by checking the received signature: 0x53634D4D means little-endian, 0x4D4D6353 big-endian.
The nerve of a good patch: information
As we are looking to provide a patch for the firmware, we needed to look at what was around, which hardware, where it is, and so on.
Hardware devices
We tried to identify the different hardware devices. It is not obvious at first, and there are several ways to do this by hand:
Look for this information in the web interface;
Parse the various configuration scripts;
grep for "netgear", "linksys" and other manufacturers;
Look for files with a specific names (like WAG160N.ico)
It is a bit hard to automate that process. Fortunately for us, a "version" field can be obtained from the backdoor. This field seems to be consistent across the same hardware. Unfortunately, the mapping between this field version and the real hardware still has to be done by hand. This process is not perfect but we haven't seen so far two different hardwares with the same "version" field.
Moreover, Cisco WAP4410 access points don't support this query but can be identified through the "sys_desc" variable (in its internal configuration).
The final repartition of hardware models is the following:
The unlucky ones are the United States, followed by the Netherlands, with China very close.
Update (23/01/2014): as some people pointed out, these statistics were based on the million IPs masscan found, and not on the 6500 vulnerables.
Here are the real statistics:
The real unlucky ones are the United Kingdom, the United States (still there) and Italy!
By the way, other folks have done a similar study: http://skizzlesec.com/2014/01/22/linksys-netgear-backdoor-by-the-numbers/ (thanks Céline for the link)
Reconstructing a new filesystem
In order to provide a new clean filesystem, we needed to dump one from a router. So, we firstly used the backdoor to retrieve such a filesystem. Moreover, analyzing it makes it easier to understand how the router works and is configured.
We made our experiments on our Linksys WAG200G. These techniques might or might not work on others. On this particular router (and maybe others), a telnet daemon executable is available (/usr/sbin/utelnetd). It can be launched through the backdoor and directly drops a root shell.
The backdoor allows us to execute command as a root user and to copy some files. Unfortunately, on many routers, there is no SSH daemon that can be started or even a netcat utility for easy and quick transfers. Moreover, the only partition that is writable is generally a RAMFS mounted on /tmp.
So let's see what are our options here.
Option 1 : download through the web server
Lots of routers have a web server that is used for their configuration. This web server (as every process running on the router) is running as the root user. This is potentially a good way to download files (and the rootfs) from the router.
For instance, on the Linksys WAG200G (and others), /www/ppp_log is a symbolic link to /tmp/ppp_log (for the web servers to show the PPP log). Thus, the root FS can be download like this:
Get a root shell thanks to the backdoor
# cd /tmp # mkdir ppp_log # cd ppp_log && cat /dev/mtdblock/0 >./rootfs
When it's done, on your computer: wget http://IP/ppp_log/rootfs
The MTD partition to use can be identified thanks to various methods. /proc/mtd is a first try. In our case:
# cat /proc/mtd dev: size erasesize name mtd0: 002d0000 00010000 "mtd0" mtd1: 000b0000 00010000 "mtd1" mtd2: 00020000 00010000 "mtd2" mtd3: 00010000 00010000 "mtd3" mtd4: 00010000 00010000 "mtd4" mtd5: 00040000 00010000 "mtd5"
The name does not give a lot of information. But, from the size of the partitions, we can guess that mtd0 is the rootfs (~2.8Mb). Moreover, a symlink in /dev validate this assumption:
# ls -la /dev [...] root -> mtdblock/0
Option 2 : MIPS cross compilation, netcat and upload
Another way to dump the rootfs is to cross compile a tool like netcat for the targeted architecture. Remember, we have little and big endian MIPS systems. Thus, we potentially need a cross compilation chain for these two systems.
By analyzing some binaries on our test router, it appears that the uClibc library and tool chains have been used by the manufacturer, with versions from 2005. Thus, we downloaded one of the version of uClibc that was released this year. After some difficulties to find the old versions of binutils, gcc (and others) and getting a working GCC 3.x compiler, our MIPSLE cross compiler was ready.
Then, we grabbed the netcat sources and compiled them. There are multiple ways that can be used to upload our freshly compiled binary to the router. The first one is to use the write "feature" of the backdoor:
$ ./poc.py --ip IP --send_file ./nc --remote-filename nc
This has been implemented in Eloi's POC here: https://raw.github.com/elvanderb/TCP-32764/master/poc.py .
The issue with this feature is that it seems to crash with "big" files.
Another technique is to use echo -n -e to transfer our binary. It works but is a bit slow. Also, the connection is sometimes closed by the router, so we have to restart where it stopped. Just do:
$ ./poc.py --ip IP --send_file2 ./nc --remote-filename nc
The MIPSEL netcat binary can be downloaded here: https://github.com/quarkslab/linksys-wag200G/blob/master/binaries/nc?raw=true.
Once netcat has been uploaded, simply launch on the router:
# cat /dev/mtdblock/0 |nc -n -v -l -p 4444
And, on your computer:
$ nc -n -v IP 4444 >./rootfs.img
Patch me if you can (yes we can)
Please note that everything that is described here is experimental and should be done only if you exactly know what you are doing. We can't be held responsible for any damage you will do to your beloved routers!
Note: the tools mentioned here are available on GitHub: https://github.com/quarkslab/linksys-wag200G.
Now that we have the original squashfs image, we can extract it. It is a well-known SquashFS, so let's grab the latest version (4.0 at the time of writing this article) and "unsquash" it:
$ unsquashfs original.img Parallel unsquashfs: Using 8 processors gzip uncompress failed with error code -3 read_block: failed to read block @0x1c139c read_fragment_table: failed to read fragment table block FATAL ERROR:failed to read fragment table $ unsquashfs -s original.img Found a valid little endian SQUASHFS 2:0 superblock on ./original.img [...] Block size 32768
This is the same issue that Eloi pointed out in his slides. What he shows is that he had to force LZMA to be used for the extraction. With the fixes he provided (exercise also left to the reader), we can extract the SquashFS:
# unsquashfs-lzma original.img 290 inodes (449 blocks) to write created 189 files created 28 directories created 69 symlinks created 32 devices created 0 fifos
What actually happened is that, back in 2005, the developer of this firmware modified the SquashFS 2.0 tools to use LZMA (and not gzip). Even if Eloi's "chainsaw" solution worked for extraction, it will not allow us to make a new image with the 2.0 format. So, back with the chainsaw, grab the squashfs 2.2 release from sourceforge, the LZMA 4.65 SDK, and make squashfs use it with this patch: https://github.com/quarkslab/linksys-wag200G/blob/master/src/squashfs-lzma.patch. The final sources can be downloaded here: https://github.com/quarkslab/linksys-wag200G/tree/master/src/squashfs2.2-r2-lzma.
With our new and freshly compiled SquashFS LZMA-enhanced 2.2 version back from the dead, we can now reconstruct the Linksys rootfs image. It is important to respect the endianness and the block size of the original image (or your router won't boot anymore).
$ ./squashfs2.2-r2-lzma/squashfs-tools/mksquashfs rootfs/ rootfs.img -2.0 -b 32768 Creating little endian 2.0 filesystem on modified-bis.img, block size 32768. Little endian filesystem, data block size 32768, compressed data, compressed metadata, compressed fragments [...] $ unsquashfs -s rootfs.img Found a valid little endian SQUASHFS 2:0 superblock on rootfs.img. Creation or last append time Wed Jan 22 10:38:29 2014 Filesystem size 1829.09 Kbytes (1.79 Mbytes) Block size 32768 [...]
Now, let's begin with the nice part. To test that our image works, we'll upload and flash it to the router. This step is critical, because if it fails, you'll end up with a router trying to boot from a corrupted root filesystem.
First, we use our previously compiled netcat binary to upload the newly created image (or use any other method of your choice):
On the router side: .. code:
# nc -n -v -l -p 4444 >/tmp/rootfs.img
On the computer side: .. code:
$ cat rootfs.img |nc -n -v IP 4444
When it's finished, have a little prayer and, on the router side: .. code:
# cat /tmp/rootfs.img >/dev/mtdblock/0
Then, plug off and on your router! If everything went well, your router should have rebooted just like before. If not, then you need to reflash your router another way, using the serial console or any JTAG port available (this is not covered here).
Now, we can simply permanently remove the backdoor from the root filesystem:
# cd /path/to/original/fs # rm usr/sbin/scfgmgr # Edit usr/etc/rcS and remove the following line /usr/sbin/scfgmgr
Then, rebuild your image as above, upload it, flash your router and the backdoor should be gone forever! It's up to you to build an SSH daemon to keep a root access on your router if you still want to play with it.
Linksys WAG200G patch procedure
For those who would just like to patch their routers, here are the steps. Please note that this has only been tested on our Linksys WAG200G! It is really not recommanded to use it with other hardware. And, we repeat it, we cannot be held responsible for any harm on your routers! Use this at your own risk.
Download the fixed image here: https://github.com/quarkslab/linksys-wag200G/blob/master/rootfs/nobackdoor.img?raw=true Control sums are in the README of the github repository)
Get Eloi's POC: https://raw.github.com/elvanderb/TCP-32764/master/poc.py .
Upload the image:
$ ./poc.py --ip IP --send_file2 nobackdoor.img --remote-filename rootfs
Then get a root shell on your router:
$ ./poc.py --ip IP --shell
Check that the file sizes are the same:
# ls -l /tmp/rootfs # it should be 1875968 bytes
To be sure, just redownload the uploaded image thanks to the web server and check that they are the same:
# mkdir /tmp/ppp_log # ln -s /tmp/rootfs /tmp/ppp_log And, on your computer: $ wget http://IP/ppp_log/rootfs $ diff ./rootfs /path/to/linksysWAG200G.nobackdoor.rootfs
Then flash your router:
# cat /tmp/rootfs >/dev/mtdblock/0
And reboot it!
Conclusion
This article showed some statistics about the presence of the backdoor found by Eloi Vanderbéken and how to fix one of the hardware affected. Feel free to comment any mistakes here, and/or provide similar images and/or fixes for other routers :)
Acknowledge
Eloi Vanderbéken for his discovery and original POC
Fred Raynal, Fernand Lone-Sang, @pod2g, Serge Guelton and Kévin Szkudlaspki for their corrections
| [1] | These 1 million IPv4s do not necessarily contain a backdoor as explained later in the section. |