Saturday, December 14, 2013

Slow Internet streaming - Bug in Router Firmware

My Wi-Fi has been puking on me for quite some time now. Streaming services were spending more time loading content, video conferencing was becoming painful, everything seemed generally slow. So, I tried some easy fixes like avoiding neighbor's wireless channels, juggling security combos, tweaking b/g/n modes and mounting the router on a Roomba for better coverage :) I even plugged in a brand new NetGear router - WNR3500LV2 - in search for a quick fix. But none of those steps made a material difference. My speeds were 0.5-1Mbps Downstream (D) & 10Mbps Upstream (U). Finally, I decided to take a closer look. What I found was quite interesting and the resultant fix boosted my Wi-Fi speed from 0.5D to 30D (60X!). Read on.

I have Comcast cable internet. My ethernet consistently gives me 60D/10U. I have my own routers behind the standard Comcast router which I use to power multiple networks (VLans) with custom configs. Please note that 
I do not use Wi-Fi on the standard Comcast router. And this issue affects Wi-Fi traffic only.

Now, my Wi-Fi traffic had downstream speed less than upstream. 
This was suspicious as it is usually the opposite. Upon investigation, I found that the Comcast downstream packets have incorrect DSCP settings. A quick Google search revealed that DSCP is a layer-3 QoS mechanism and is supposed to improve QoS for real-time data streaming applications such as VOIP using prioritization algorithms. The Comcast settings seem to have created a critical bug that is likely crippling thousands of routers across US. It is an edge case interaction caused by bad DSCP information coming from Comcast and poor condition handling by the WMM driver on certain linksys/cisco/netgear routers and possibly others as well. This issue can appear on both stock and modded routers. IPv6 traffic seems to work just fine. Since the only packets with this bad information are the ones that are being downloaded, upload traffic is not affected nearly as badly.

I think the simplest way to fix this problem is to use 
Wi-Fi from a Comcast approved router. Else, you are really limited by capabilities of your personal router's firmware. If it allows you to alter packets (most do not), then you are good. Else, you will have to update your router with an open source firmware. DD-WRT & TomatoUSB are two popular options. I already had DD-WRT but it didn't give me a clean way of modifying stuff. I didn't want to telnet into it - maintainability etc. So, I tried TomatoUSB and I was able to update the router configuration with the following scripts,

INIT
insmod xt_DSCP.ko

WANUP

iptables -t mangle -A PREROUTING -i vlan2 -j DSCP --set-dscp 0

And boom, the DSCP flag was disabled and my Wi-Fi speed jumped to 30Mbps! So, here is what you need if you experience similar issues,
  1. Test your Internet speed using the Speedtest.net apps. If downstream is lower than upstream, that's a big signal. You may use Wireshark to confirm the problem.
  2. Search for a suitable firmware for your router. DD-WRT & TomatoUSB are two popular options for home routers. Follow the instructions and load it. WARNING: You will void warranty and this step could brick your router.
  3. Modify the packets to disable DSCP. For example, you may use the above scripts. Note, your interface may be different from "vlan2".