Nov 20 2009

HFSC traffic shaping for QOS on DD-WRT: New test

After reading more about HFSC, I modified the original script I posted here.

The main changes are:

  • Used a different syntax to specify the different classes.
  • Changed P2P ports to include most high-numbered ports
  • Updated iptables rules to only use destination port, not source port. This is because we are only filtering outgoing packets, there is an assumption no services are being hosted on the connection, and the high port numbers in P2P ports matches most all connections because the source port typically is a very high number.


#!/bin/ash

# Modified to run under DD-WRT
# http://www.morph3ous.net/2009/11/20/beta-hfsc-traffic-shaping-for-qos-on-dd-wrt/

# Go to Administration and then commands
# Paste script in and click on the save firewall button
# Reboot router and test
#
###### Script originally from
# Maciej Bliziński, http://automatthias.wordpress.com/
#
# References:
# http://www.voip-info.org/wiki/view/QoS+Linux+with+HFSC
# http://www.nslu2-linux.org/wiki/HowTo/EnableTrafficShaping
# http://www.cs.cmu.edu/~hzhang/HFSC/main.html
######

# Specify the uplink as 85 or 90 percent of your actual upload speed
# Uplink and downlink speeds
# removed throttling downlink
UPLINK=425

###### From original script, but code has been disabled further below
# IP addresses of the VoIP phones,
# if none, set VOIPIPS=""
VOIPIPS=""
######

# Interactive class: SSH Terminal, DNS, RDP
INTERACTIVEPORTS="22 23 53 3389"

# VoIP telephony
#VOIPPORTS="5060:5100 10000:11000 5000:5059 8000:8016 5004 1720 1731"
VOIPPORTS=""

# WWW, jabber and IRC
BROWSINGPORTS="80 443 8080"

# The lowest priority traffic: IMAP, SMTP, FTP, IMAP, IMAP/S, high port numbers likely to be P2P
P2PPORTS="110 25 21 143 993 1024:65535"

# Device that connects you to the Internet
#DEV=$(nvram get wan_ifname)
DEV=ppp0

# clean up in case re-running
# Reset everything to a known state (cleared)
tc qdisc del dev $DEV root > /dev/null 2>&1
tc qdisc del dev $DEV ingress > /dev/null 2>&1

# Flush and delete tables
iptables -t mangle --delete POSTROUTING -o $DEV -j THESHAPER > /dev/null 2>&1
iptables -t mangle --flush THESHAPER 2> /dev/null > /dev/null
iptables -t mangle --delete-chain THESHAPER 2> /dev/null > /dev/null

# start setting up QOS
# Traffic classes:
# 1:2 Interactive (SSH, DNS, ACK, Quake)
# 1:3 Low latency (VoIP) < - currently being used for Netflix movie streaming from Roku box
# 1:4 Browsing (HTTP, HTTPs)
# 1:5 Default
# 1:6 Low priority (p2p, pop3, smtp, etc)

# add HFSC root qdisc
tc qdisc add dev $DEV root handle 1: hfsc default 5

# add main rate limit class
tc class add dev $DEV parent 1: classid 1:1 hfsc \
sc rate ${UPLINK}kbit ul rate ${UPLINK}kbit

# Interactive traffic
tc class add dev $DEV parent 1:1 classid 1:2 hfsc sc umax 1500b dmax 30ms rate $((5*$UPLINK/10))kbit ul rate ${UPLINK}kbit

# Netflix
tc class add dev $DEV parent 1:1 classid 1:3 hfsc sc umax 1500b dmax 75ms rate $((2*$UPLINK/10))kbit ul rate ${UPLINK}kbit

# Browsing
tc class add dev $DEV parent 1:1 classid 1:4 hfsc sc rate $((1*$UPLINK/10))kbit ul rate ${UPLINK}kbit

# Default traffic

tc class add dev $DEV parent 1:1 classid 1:5 hfsc sc rate $((1*$UPLINK/20))kbit ul rate ${UPLINK}kbit

# Low priority/Bulk
tc class add dev $DEV parent 1:1 classid 1:6 hfsc sc rate 500bit ul rate ${UPLINK}kbit

# add THESHAPER chain to the mangle table in iptables

iptables -t mangle --new-chain THESHAPER
iptables -t mangle --insert POSTROUTING -o $DEV -j THESHAPER

# had to change all iptables rules below to use mark instead of classify. tc filters then pick these up and move to the proper queue
# in the future this should all be done with tc filters as this is somewhat of a hack and there is presumably at least slightly more overhead

# put packets marked by iptables in the right queues using tc filters
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 2 fw flowid 1:2
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 3 fw flowid 1:3
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 4 fw flowid 1:4
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 5 fw flowid 1:5
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 6 fw flowid 1:6

## Note that iptables rules are being appended to the end of the chain. It appears that in the mangle
## table when using mark, all rules are processed
## put more specific rules lower down in the script otherwise more general rules below them may
## re-mark them and cause unexpected behavior
### UNLESS ##
## You add another rule with -j RETURN
## I'm doing this with some rules because I want them to be processed first. It also
## helps keep the packet statistics cleaner when issuing the iptables -t mangle -L -v command (otherwise some traffic is double-counted)

# To speed up downloads while an upload is going on, put short ACK
# packets in the interactive class:
iptables -t mangle -A THESHAPER -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK ACK -m length --length :64 -j MARK --set-mark 2
iptables -t mangle -A THESHAPER -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK ACK -m length --length :64 -j RETURN

# put large (512+) icmp packets in browsing category
iptables -t mangle -A THESHAPER -p icmp -m length --length 512: -j MARK --set-mark 4
iptables -t mangle -A THESHAPER -p icmp -m length --length 512: -j RETURN

# ICMP (ip protocol 1) in the interactive class
iptables -t mangle -A THESHAPER -p icmp -m length --length :512 -j MARK --set-mark 2
iptables -t mangle -A THESHAPER -p icmp -m length --length :512 -j RETURN

# Send traffic with destination to Backblaze IPs and put into the Default Traffic class
# higher priority than P2P, but lower than browsing. Got IPs from arin.net
iptables -t mangle -A THESHAPER -p tcp -d 70.42.234.64/26 -j MARK --set-mark 5
iptables -t mangle -A THESHAPER -p tcp -d 70.42.234.64/26 -j RETURN

iptables -t mangle -A THESHAPER -p tcp -d 770.42.186.64/26 -j MARK --set-mark 5
iptables -t mangle -A THESHAPER -p tcp -d 770.42.186.64/26 -j RETURN
# for BackBlaze as well, but this IP not verified
iptables -t mangle -A THESHAPER -p tcp -d 204.11.104.0/22 -j MARK --set-mark 5
iptables -t mangle -A THESHAPER -p tcp -d 204.11.104.0/22 -j RETURN

# Prioritize Netflix streaming from Roku by IP address and put into the VoIP class
# note that device's IP must either be set manually or a static DHCP reservation set on the router
iptables -t mangle -A THESHAPER -p tcp --dport 80 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p tcp --dport 80 -s 192.168.1.15 -j RETURN
iptables -t mangle -A THESHAPER -p tcp --dport 443 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p tcp --dport 443 -s 192.168.1.15 -j RETURN

setclassbyport() {
port=$1
CLASS=$2
#iptables -t mangle -A THESHAPER -p udp --sport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p udp --dport $port -j MARK --set-mark $CLASS
#iptables -t mangle -A THESHAPER -p tcp --sport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p tcp --dport $port -j MARK --set-mark $CLASS
}

for port in $INTERACTIVEPORTS; do setclassbyport $port 2; done
for port in $VOIPPORTS; do setclassbyport $port 3; done
for port in $BROWSINGPORTS; do setclassbyport $port 4; done
for port in $P2PPORTS; do setclassbyport $port 6; done

for VOIP in $VOIPIPS
do
iptables -t mangle -A THESHAPER --src $VOIP -j MARK --set-mark 3
iptables -t mangle -A THESHAPER --dst $VOIP -j MARK --set-mark 3
done


Nov 15 2009

HFSC traffic shaping for QOS on DD-WRT

The situation:
To have a backup to the backups of my computer offsite, I decided to trial BackBlaze. Their willingness to share parts of their storage architecture as well as the way encryption is employed attracted me to the company.

The software has its own integrated bandwidth management. With its default settings, BackBlaze did a pretty good job of keeping my connection responsive, but the transfer rate was nowhere near as high as it could be. I wanted to tell BackBlaze to go as fast as it could and move the bandwidth management logic to my router.

The goal is to max my upload while not completely killing the speed of downloads and latency of the connection. This can be tricky on asymmetrical connections like DSL and Cable Modems that are common here in the US.

The problems with the builtin DD-WRT (an excellent 3rd party firmware) QOS:

  • HTB QOS as implemented in the firware did not seem limit latency enough. This may just be an HTB issue and not one with DD-WRT.
  • There is no option to prioritize outgoing TCP ACK packets. Due to the nature of TCP/IP, your download is significantly slowed if your upload is clogged and ACK packets are not being sent out.
  • The HFSC implementation is flawed and not very usable.

To make everything work, I would have to use a custom script on the router. Unfortunately, HFSC is not exactly well documented. I was lucky to be able to borrow from the work of others and adapt it to work on DD-WRT.

Thanks to these websites for collectively providing me enough information to get this accomplished:

  • Infotage.net – The black art of traffic shaping provided me with links to a lot of the other sites listed here.
  • HFSC Scheduling with Linux appears to be a definitive site of the theory of HFSC. I’m going to need to re-read this one in order to improve my setup.
  • HFSC and VoIP – Maciej Bliziński is where I got the script that I modified for my own purposes. Most of the script’s settings are left unchanged. I will be revising why certain classes are the way they are. Since I am not using VoIP, I may be able to better optimize everything.
  • QoS Linux with HFSC – voip-info.org has a script that if I paid more attention to, I would have wasted much less time. I was setting the outbound interface as $(nvram get wan_ifname) as other DD-WRT example scripts showed. This came out to eth1. I wasted a lot of time on this problem. Use ppp0 is using a DSL modem with PPPoE.
  • http://wsched.sourceforge.net/examples.html provided example commands that helped me troubleshoot. The problem is that DD-WRT does not give any error messages. (Probably to save space to fit in the tiny amount of memory on these routers.)
  • Packet-Shaping HOWTO provided example commands that helped me create and troubleshoot tc classes and filters.

I feel there is no point in throttling your downstream using your router. If the packet has already traveled over the last mile to your router, why reject it and cause it to be re-transmitted later. I am mainly interested in managing outgoing traffic.

Here is the modified script I am using based very, very closely upon this script. It is modified to run on DD-WRT, but will likely work on other linux-based routers.:


#!/bin/ash

# Modified to run under DD-WRT
# http://www.morph3ous.net/2009/11/15/hfsc-traffic-shaping-for-qos-on-dd-wrt

# Go to Administration and then commands
# Paste script in and click on the save firewall button
# Reboot router and test
#
###### Script originally from
# Maciej Bliziński, http://automatthias.wordpress.com/
#
# References:
# http://www.voip-info.org/wiki/view/QoS+Linux+with+HFSC
# http://www.nslu2-linux.org/wiki/HowTo/EnableTrafficShaping
# http://www.cs.cmu.edu/~hzhang/HFSC/main.html
######

# Specify the uplink as 85 or 90 percent of your actual upload speed
# Uplink and downlink speeds
# removed throttling downlink
UPLINK=425

###### From original script, but code has been disabled further below
# IP addresses of the VoIP phones,
# if none, set VOIPIPS=""
VOIPIPS=""
######

# Interactive class: SSH Terminal, DNS, RDP
INTERACTIVEPORTS="22 23 53 3389"

# VoIP telephony
#VOIPPORTS="5060:5100 10000:11000 5000:5059 8000:8016 5004 1720 1731"
VOIPPORTS=""

# WWW, jabber and IRC
BROWSINGPORTS="80 443 8080"

# The lowest priority traffic: eDonkey, Bittorrent, etc.
P2PPORTS="110 25 21 143 445 137:139 4662 4664 6881:6999"

# Device that connects you to the Internet
#DEV=$(nvram get wan_ifname)
DEV=ppp0

# clean up in case re-running
# Reset everything to a known state (cleared)
tc qdisc del dev $DEV root > /dev/null 2>&1
tc qdisc del dev $DEV ingress > /dev/null 2>&1

# Flush and delete tables
iptables -t mangle --delete POSTROUTING -o $DEV -j THESHAPER > /dev/null 2>&1
iptables -t mangle --flush THESHAPER 2> /dev/null > /dev/null
iptables -t mangle --delete-chain THESHAPER 2> /dev/null > /dev/null

# start setting up QOS
# Traffic classes:
# 1:2 Interactive (SSH, DNS, ACK, Quake)
# 1:3 Low latency (VoIP) < - currently being used for Netflix movie streaming from Roku box
# 1:4 Browsing (HTTP, HTTPs)
# 1:5 Default
# 1:6 Low priority (p2p, pop3, smtp, etc)

# add HFSC root qdisc
tc qdisc add dev $DEV root handle 1: hfsc default 5

# add main rate limit class
tc class add dev $DEV parent 1: classid 1:1 hfsc \
sc rate ${UPLINK}kbit ul rate ${UPLINK}kbit

# Interactive traffic: guarantee realtime full uplink for 50ms, then
# 5/10 of the uplink

tc class add dev $DEV parent 1:1 classid 1:2 hfsc \
rt m1 ${UPLINK}kbit d 50ms m2 $((5*$UPLINK/10))kbit \
ls m1 ${UPLINK}kbit d 50ms m2 $((7*$UPLINK/10))kbit \
ul rate ${UPLINK}kbit

# VoIP: guarantee full uplink for 200ms, then 3/10
#tc class add dev $DEV parent 1:1 classid 1:3 hfsc \
# sc m1 ${UPLINK}kbit d 200ms m2 $((3*$UPLINK/10))kbit \
# ul rate ${UPLINK}kbit

# Netflix: guarantee 1/2 uplink for 200ms, then 3/10
tc class add dev $DEV parent 1:1 classid 1:3 hfsc \
sc m1 $((1*$UPLINK/2))kbit d 200ms m2 $((3*$UPLINK/10))kbit \
ul rate ${UPLINK}kbit

# Browsing: Don't guarantee anything for the first second, then
# guarantee 1/10
tc class add dev $DEV parent 1:1 classid 1:4 hfsc \
sc m1 0 d 1s m2 $((1*$UPLINK/10))kbit \
ul rate ${UPLINK}kbit

# Default traffic: don't guarantee anything for the first two seconds,
# then guarantee 1/20

tc class add dev $DEV parent 1:1 classid 1:5 hfsc \
sc m1 0 d 2s m2 $((1*$UPLINK/20))kbit \
ul rate ${UPLINK}kbit

# Default traffic: don't guarantee anything for the first 10 seconds,
# then guarantee 1/20
tc class add dev $DEV parent 1:1 classid 1:6 hfsc \
sc m1 0 d 10s m2 $((1*${UPLINK}/20))kbit \
ul rate ${UPLINK}kbit

# add THESHAPER chain to the mangle table in iptables

iptables -t mangle --new-chain THESHAPER
iptables -t mangle --insert POSTROUTING -o $DEV -j THESHAPER

# had to change all iptables rules below to use mark instead of classify. tc filters then pick these up and move to the proper queue
# in the future this should all be done with tc filters as this is somewhat of a hack and there is presumably at least slightly more overhead

# put packets marked by iptables in the right queues using tc filters
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 2 fw flowid 1:2
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 3 fw flowid 1:3
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 4 fw flowid 1:4
# none needed for 1:5 as traffic lands there by default
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 6 fw flowid 1:6
# To speed up downloads while an upload is going on, put short ACK
# packets in the interactive class:

iptables -t mangle -A THESHAPER \
-p tcp \
-m tcp --tcp-flags FIN,SYN,RST,ACK ACK \
-m length --length :64 \
#-j CLASSIFY --set-class 1:2
-j MARK --set-mark 2

# put large (512+) icmp packets in browsing category
iptables -t mangle -A THESHAPER \
-p icmp \
-m length --length 512: \
#-j CLASSIFY --set-class 1:4
-j MARK --set-mark 4

# ICMP (ip protocol 1) in the interactive class
iptables -t mangle -A THESHAPER \
-p icmp \
-m length --length :512 \
#-j CLASSIFY --set-class 1:2
-j MARK --set-mark 2

setclassbyport() {
port=$1
CLASS=$2
iptables -t mangle -A THESHAPER -p udp --sport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p udp --dport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p tcp --sport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p tcp --dport $port -j MARK --set-mark $CLASS
}

for port in $INTERACTIVEPORTS; do setclassbyport $port 2; done
for port in $VOIPPORTS; do setclassbyport $port 3; done
for port in $BROWSINGPORTS; do setclassbyport $port 4; done
for port in $P2PPORTS; do setclassbyport $port 6; done

for VOIP in $VOIPIPS
do
iptables -t mangle -A THESHAPER --src $VOIP -j MARK --set-mark 3
iptables -t mangle -A THESHAPER --dst $VOIP -j MARK --set-mark 3
done

# Prioritize Netflix streaming from Roku by IP address and put into the VoIP class
# note that device's IP must either be set manually or a static DHCP reservation set on the router
iptables -t mangle -A THESHAPER -p tcp --dport 80 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p tcp --dport 443 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p udp --dport 80 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p udp --dport 443 -s 192.168.1.15 -j MARK --set-mark 3

Updated script I’m testing here.


Oct 9 2006

JohnChow.com: The Internet’s Biggest Google Whores

JohnChow.com: The Internet’s Biggest Google Whores:

The following is a list of the Internet’s eight biggest Google AdSense publishers. The information was compiled from interviews and articles found on the Internet. Whenever possible, I list the source of the information.

All I can say is “WOW.” It takes a lot of pageviews to earn that kind of money! I need to think up the next killer idea…


Sep 14 2006

BroadbandReports.com: 25 Cents to Stream a DVD Quality Film – 80 Cents for HD, a nickel for iPod TV…

From BroadbandReports.com: 25 Cents to Stream a DVD Quality Film – 80 Cents for HD, a nickel for iPod TV…:

Dave Burstein crunches the numbers behind offering video via broadband and concludes that it costs a quarter to stream a DVD quality movie, 80 cents for an HD film, or a nickel for iPod or AOL TV shows. A much more reasonable analysis than the recent UK report that claimed it cost ISPs $39 to stream a two hour HD film. This compared to broadcast over the air video, which costs a few pennies per hour to distribute.

“For providing managed servers and internet bandwidth, several content delivery networks are bidding $10,000 to $12,000 per continuous gigabit per month. That’s enough for 700 1.5 megabit streams, almost DVD quality if pre-encoded in the latest MPEG-4, Flash, or Windows Media. Amazon’s choice of 2.5 megabit encoding may be raising the bar. It’s also enough for over 3,000 300 Kbps streams, appropriate for iPods or the quarter screen video AOL and ABC are distributing supported by ads.”

Burstein also comments on how players like Apple and Amazon will threaten TelcoTV’s already fragile projected profit margins.


Sep 6 2006

CNET News.com: Will Vista stall Net traffic

CNet News.com reports:

Vista will also support the current IPv4. The side effect, according to Mockapetris, is that a Vista PC will make two DNS requests, one for each IP version, instead of just one.

“It is going to try a DNS lookup for the IPv6 address and then a DNS lookup for the IPv4 address,” Mockapetris said. “It just uses more DNS, and until we increase the supply, things are going to go slower.”

The article is a bit too sensational for my tastes. The clear and simple answer to the question “Will Vista stall net traffic?” is no.
Continue reading


Aug 27 2006

Survived a digging!

This site got dugg yesterday in response to my post about the State of Florida’s People First job portal shutting out alternate browsers. The post rose to the front page of Digg and remained there for several hours. It continued to be included in the rss feed after it was no longer on the front page.

The changes that I implemented in my SuEXEC, Fastcgi and cPanel post handled the high traffic generated by Digg with ease and the site remained pretty responsive throughout.

I hope that the 8,000+ people who read the post , and the many who were kind enough to leave comments will show the State that it is indeed an important issue. It is an issue easily fixed.

Thank you Digg users for helping to bring this issue to the public light.


Aug 27 2006

Email to state – PeopleFirst@dms.myflorida.com

I followed the the advice of a user of digg, Sillywampa, on Digg’s entry about my earlier post on Florida’s People First Job Portal and emailled PeopleFirst@dms.myflorida.com about this issue. Here is what I wrote:
Continue reading


Aug 25 2006

State of Florida job portal shuts out alternative browsers

Update: I wrote a post about an email I sent to the State of Florida and the response I received. This was suggested by Sillywampa on Digg. Thanks Sillywampa for the email address.

The State of Florida’s People First job portal is still running an outdated script that locks out users of alternative browsers from applying to state jobs. The only supported browsers are Internet Explorer and Netscape Navigator.
Continue reading


Aug 9 2006

TheStar.com: Wikipedia handles Colbert

The wisdom of the crowd is definitely a powerful force.

TheStar.com – Wikipedia handles Colbert:

…Colbert stated, if enough people alter the page on Wikipedia, it’s fact.

It should have been the biggest threat to the institution of Wikipedia to date. It was a condemnation of the site’s credibility. And it didn’t come from the Times, or from some retired political refugee from the Kennedy era.

It came from someone cool. Colbert, after all, has some serious credibility among the crowd who use and contribute to Wikipedia.

Within a day, Colbert’s report had hit some of the Net’s geek nerve centres, Slashdot.org and Fark.com. Wikipedia’s protection log was recording ongoing attempts at vandalism.

There was just one problem with the prank: Colbert was ultimately proven wrong.

Although the servers were tested, soon after Colbert began ranting about wikiality on his show, the encyclopedia had locked down the page on elephants and Colbert’s biography.


Aug 9 2006

CNET News.com: Protecting yourself from search engines

CNET News.com has written an article about protecting yourself from search engines. The piece is a fluff piece; however, does have some useful tips. I recommend that you read it.

This comes after AOL released the search records of users to the public.

FAQ: Protecting yourself from search engines | CNET News.com:

AOLs publication of the search histories of more than 650,000 of its users should reinforce an important point: What you type in online may not be as private as you think.

Search engines place a multibillion-dollar infrastructure at the hands of any random user who stops by their Web site. The price you pay, however, is that the company may hold on to your search queries–which can provide a glimpse into your life–forever.

To offer some suggestions about preserving your privacy while using search engines, CNET News.com has prepared the following list of frequently asked questions.