Nov 20 2009

HFSC traffic shaping for QOS on DD-WRT: New test

After reading more about HFSC, I modified the original script I posted here.

The main changes are:

  • Used a different syntax to specify the different classes.
  • Changed P2P ports to include most high-numbered ports
  • Updated iptables rules to only use destination port, not source port. This is because we are only filtering outgoing packets, there is an assumption no services are being hosted on the connection, and the high port numbers in P2P ports matches most all connections because the source port typically is a very high number.


#!/bin/ash

# Modified to run under DD-WRT
# http://www.morph3ous.net/2009/11/20/beta-hfsc-traffic-shaping-for-qos-on-dd-wrt/

# Go to Administration and then commands
# Paste script in and click on the save firewall button
# Reboot router and test
#
###### Script originally from
# Maciej Bliziński, http://automatthias.wordpress.com/
#
# References:
# http://www.voip-info.org/wiki/view/QoS+Linux+with+HFSC
# http://www.nslu2-linux.org/wiki/HowTo/EnableTrafficShaping
# http://www.cs.cmu.edu/~hzhang/HFSC/main.html
######

# Specify the uplink as 85 or 90 percent of your actual upload speed
# Uplink and downlink speeds
# removed throttling downlink
UPLINK=425

###### From original script, but code has been disabled further below
# IP addresses of the VoIP phones,
# if none, set VOIPIPS=""
VOIPIPS=""
######

# Interactive class: SSH Terminal, DNS, RDP
INTERACTIVEPORTS="22 23 53 3389"

# VoIP telephony
#VOIPPORTS="5060:5100 10000:11000 5000:5059 8000:8016 5004 1720 1731"
VOIPPORTS=""

# WWW, jabber and IRC
BROWSINGPORTS="80 443 8080"

# The lowest priority traffic: IMAP, SMTP, FTP, IMAP, IMAP/S, high port numbers likely to be P2P
P2PPORTS="110 25 21 143 993 1024:65535"

# Device that connects you to the Internet
#DEV=$(nvram get wan_ifname)
DEV=ppp0

# clean up in case re-running
# Reset everything to a known state (cleared)
tc qdisc del dev $DEV root > /dev/null 2>&1
tc qdisc del dev $DEV ingress > /dev/null 2>&1

# Flush and delete tables
iptables -t mangle --delete POSTROUTING -o $DEV -j THESHAPER > /dev/null 2>&1
iptables -t mangle --flush THESHAPER 2> /dev/null > /dev/null
iptables -t mangle --delete-chain THESHAPER 2> /dev/null > /dev/null

# start setting up QOS
# Traffic classes:
# 1:2 Interactive (SSH, DNS, ACK, Quake)
# 1:3 Low latency (VoIP) < - currently being used for Netflix movie streaming from Roku box
# 1:4 Browsing (HTTP, HTTPs)
# 1:5 Default
# 1:6 Low priority (p2p, pop3, smtp, etc)

# add HFSC root qdisc
tc qdisc add dev $DEV root handle 1: hfsc default 5

# add main rate limit class
tc class add dev $DEV parent 1: classid 1:1 hfsc \
sc rate ${UPLINK}kbit ul rate ${UPLINK}kbit

# Interactive traffic
tc class add dev $DEV parent 1:1 classid 1:2 hfsc sc umax 1500b dmax 30ms rate $((5*$UPLINK/10))kbit ul rate ${UPLINK}kbit

# Netflix
tc class add dev $DEV parent 1:1 classid 1:3 hfsc sc umax 1500b dmax 75ms rate $((2*$UPLINK/10))kbit ul rate ${UPLINK}kbit

# Browsing
tc class add dev $DEV parent 1:1 classid 1:4 hfsc sc rate $((1*$UPLINK/10))kbit ul rate ${UPLINK}kbit

# Default traffic

tc class add dev $DEV parent 1:1 classid 1:5 hfsc sc rate $((1*$UPLINK/20))kbit ul rate ${UPLINK}kbit

# Low priority/Bulk
tc class add dev $DEV parent 1:1 classid 1:6 hfsc sc rate 500bit ul rate ${UPLINK}kbit

# add THESHAPER chain to the mangle table in iptables

iptables -t mangle --new-chain THESHAPER
iptables -t mangle --insert POSTROUTING -o $DEV -j THESHAPER

# had to change all iptables rules below to use mark instead of classify. tc filters then pick these up and move to the proper queue
# in the future this should all be done with tc filters as this is somewhat of a hack and there is presumably at least slightly more overhead

# put packets marked by iptables in the right queues using tc filters
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 2 fw flowid 1:2
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 3 fw flowid 1:3
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 4 fw flowid 1:4
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 5 fw flowid 1:5
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 6 fw flowid 1:6

## Note that iptables rules are being appended to the end of the chain. It appears that in the mangle
## table when using mark, all rules are processed
## put more specific rules lower down in the script otherwise more general rules below them may
## re-mark them and cause unexpected behavior
### UNLESS ##
## You add another rule with -j RETURN
## I'm doing this with some rules because I want them to be processed first. It also
## helps keep the packet statistics cleaner when issuing the iptables -t mangle -L -v command (otherwise some traffic is double-counted)

# To speed up downloads while an upload is going on, put short ACK
# packets in the interactive class:
iptables -t mangle -A THESHAPER -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK ACK -m length --length :64 -j MARK --set-mark 2
iptables -t mangle -A THESHAPER -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK ACK -m length --length :64 -j RETURN

# put large (512+) icmp packets in browsing category
iptables -t mangle -A THESHAPER -p icmp -m length --length 512: -j MARK --set-mark 4
iptables -t mangle -A THESHAPER -p icmp -m length --length 512: -j RETURN

# ICMP (ip protocol 1) in the interactive class
iptables -t mangle -A THESHAPER -p icmp -m length --length :512 -j MARK --set-mark 2
iptables -t mangle -A THESHAPER -p icmp -m length --length :512 -j RETURN

# Send traffic with destination to Backblaze IPs and put into the Default Traffic class
# higher priority than P2P, but lower than browsing. Got IPs from arin.net
iptables -t mangle -A THESHAPER -p tcp -d 70.42.234.64/26 -j MARK --set-mark 5
iptables -t mangle -A THESHAPER -p tcp -d 70.42.234.64/26 -j RETURN

iptables -t mangle -A THESHAPER -p tcp -d 770.42.186.64/26 -j MARK --set-mark 5
iptables -t mangle -A THESHAPER -p tcp -d 770.42.186.64/26 -j RETURN
# for BackBlaze as well, but this IP not verified
iptables -t mangle -A THESHAPER -p tcp -d 204.11.104.0/22 -j MARK --set-mark 5
iptables -t mangle -A THESHAPER -p tcp -d 204.11.104.0/22 -j RETURN

# Prioritize Netflix streaming from Roku by IP address and put into the VoIP class
# note that device's IP must either be set manually or a static DHCP reservation set on the router
iptables -t mangle -A THESHAPER -p tcp --dport 80 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p tcp --dport 80 -s 192.168.1.15 -j RETURN
iptables -t mangle -A THESHAPER -p tcp --dport 443 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p tcp --dport 443 -s 192.168.1.15 -j RETURN

setclassbyport() {
port=$1
CLASS=$2
#iptables -t mangle -A THESHAPER -p udp --sport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p udp --dport $port -j MARK --set-mark $CLASS
#iptables -t mangle -A THESHAPER -p tcp --sport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p tcp --dport $port -j MARK --set-mark $CLASS
}

for port in $INTERACTIVEPORTS; do setclassbyport $port 2; done
for port in $VOIPPORTS; do setclassbyport $port 3; done
for port in $BROWSINGPORTS; do setclassbyport $port 4; done
for port in $P2PPORTS; do setclassbyport $port 6; done

for VOIP in $VOIPIPS
do
iptables -t mangle -A THESHAPER --src $VOIP -j MARK --set-mark 3
iptables -t mangle -A THESHAPER --dst $VOIP -j MARK --set-mark 3
done


Nov 15 2009

HFSC traffic shaping for QOS on DD-WRT

The situation:
To have a backup to the backups of my computer offsite, I decided to trial BackBlaze. Their willingness to share parts of their storage architecture as well as the way encryption is employed attracted me to the company.

The software has its own integrated bandwidth management. With its default settings, BackBlaze did a pretty good job of keeping my connection responsive, but the transfer rate was nowhere near as high as it could be. I wanted to tell BackBlaze to go as fast as it could and move the bandwidth management logic to my router.

The goal is to max my upload while not completely killing the speed of downloads and latency of the connection. This can be tricky on asymmetrical connections like DSL and Cable Modems that are common here in the US.

The problems with the builtin DD-WRT (an excellent 3rd party firmware) QOS:

  • HTB QOS as implemented in the firware did not seem limit latency enough. This may just be an HTB issue and not one with DD-WRT.
  • There is no option to prioritize outgoing TCP ACK packets. Due to the nature of TCP/IP, your download is significantly slowed if your upload is clogged and ACK packets are not being sent out.
  • The HFSC implementation is flawed and not very usable.

To make everything work, I would have to use a custom script on the router. Unfortunately, HFSC is not exactly well documented. I was lucky to be able to borrow from the work of others and adapt it to work on DD-WRT.

Thanks to these websites for collectively providing me enough information to get this accomplished:

  • Infotage.net – The black art of traffic shaping provided me with links to a lot of the other sites listed here.
  • HFSC Scheduling with Linux appears to be a definitive site of the theory of HFSC. I’m going to need to re-read this one in order to improve my setup.
  • HFSC and VoIP – Maciej Bliziński is where I got the script that I modified for my own purposes. Most of the script’s settings are left unchanged. I will be revising why certain classes are the way they are. Since I am not using VoIP, I may be able to better optimize everything.
  • QoS Linux with HFSC – voip-info.org has a script that if I paid more attention to, I would have wasted much less time. I was setting the outbound interface as $(nvram get wan_ifname) as other DD-WRT example scripts showed. This came out to eth1. I wasted a lot of time on this problem. Use ppp0 is using a DSL modem with PPPoE.
  • http://wsched.sourceforge.net/examples.html provided example commands that helped me troubleshoot. The problem is that DD-WRT does not give any error messages. (Probably to save space to fit in the tiny amount of memory on these routers.)
  • Packet-Shaping HOWTO provided example commands that helped me create and troubleshoot tc classes and filters.

I feel there is no point in throttling your downstream using your router. If the packet has already traveled over the last mile to your router, why reject it and cause it to be re-transmitted later. I am mainly interested in managing outgoing traffic.

Here is the modified script I am using based very, very closely upon this script. It is modified to run on DD-WRT, but will likely work on other linux-based routers.:


#!/bin/ash

# Modified to run under DD-WRT
# http://www.morph3ous.net/2009/11/15/hfsc-traffic-shaping-for-qos-on-dd-wrt

# Go to Administration and then commands
# Paste script in and click on the save firewall button
# Reboot router and test
#
###### Script originally from
# Maciej Bliziński, http://automatthias.wordpress.com/
#
# References:
# http://www.voip-info.org/wiki/view/QoS+Linux+with+HFSC
# http://www.nslu2-linux.org/wiki/HowTo/EnableTrafficShaping
# http://www.cs.cmu.edu/~hzhang/HFSC/main.html
######

# Specify the uplink as 85 or 90 percent of your actual upload speed
# Uplink and downlink speeds
# removed throttling downlink
UPLINK=425

###### From original script, but code has been disabled further below
# IP addresses of the VoIP phones,
# if none, set VOIPIPS=""
VOIPIPS=""
######

# Interactive class: SSH Terminal, DNS, RDP
INTERACTIVEPORTS="22 23 53 3389"

# VoIP telephony
#VOIPPORTS="5060:5100 10000:11000 5000:5059 8000:8016 5004 1720 1731"
VOIPPORTS=""

# WWW, jabber and IRC
BROWSINGPORTS="80 443 8080"

# The lowest priority traffic: eDonkey, Bittorrent, etc.
P2PPORTS="110 25 21 143 445 137:139 4662 4664 6881:6999"

# Device that connects you to the Internet
#DEV=$(nvram get wan_ifname)
DEV=ppp0

# clean up in case re-running
# Reset everything to a known state (cleared)
tc qdisc del dev $DEV root > /dev/null 2>&1
tc qdisc del dev $DEV ingress > /dev/null 2>&1

# Flush and delete tables
iptables -t mangle --delete POSTROUTING -o $DEV -j THESHAPER > /dev/null 2>&1
iptables -t mangle --flush THESHAPER 2> /dev/null > /dev/null
iptables -t mangle --delete-chain THESHAPER 2> /dev/null > /dev/null

# start setting up QOS
# Traffic classes:
# 1:2 Interactive (SSH, DNS, ACK, Quake)
# 1:3 Low latency (VoIP) < - currently being used for Netflix movie streaming from Roku box
# 1:4 Browsing (HTTP, HTTPs)
# 1:5 Default
# 1:6 Low priority (p2p, pop3, smtp, etc)

# add HFSC root qdisc
tc qdisc add dev $DEV root handle 1: hfsc default 5

# add main rate limit class
tc class add dev $DEV parent 1: classid 1:1 hfsc \
sc rate ${UPLINK}kbit ul rate ${UPLINK}kbit

# Interactive traffic: guarantee realtime full uplink for 50ms, then
# 5/10 of the uplink

tc class add dev $DEV parent 1:1 classid 1:2 hfsc \
rt m1 ${UPLINK}kbit d 50ms m2 $((5*$UPLINK/10))kbit \
ls m1 ${UPLINK}kbit d 50ms m2 $((7*$UPLINK/10))kbit \
ul rate ${UPLINK}kbit

# VoIP: guarantee full uplink for 200ms, then 3/10
#tc class add dev $DEV parent 1:1 classid 1:3 hfsc \
# sc m1 ${UPLINK}kbit d 200ms m2 $((3*$UPLINK/10))kbit \
# ul rate ${UPLINK}kbit

# Netflix: guarantee 1/2 uplink for 200ms, then 3/10
tc class add dev $DEV parent 1:1 classid 1:3 hfsc \
sc m1 $((1*$UPLINK/2))kbit d 200ms m2 $((3*$UPLINK/10))kbit \
ul rate ${UPLINK}kbit

# Browsing: Don't guarantee anything for the first second, then
# guarantee 1/10
tc class add dev $DEV parent 1:1 classid 1:4 hfsc \
sc m1 0 d 1s m2 $((1*$UPLINK/10))kbit \
ul rate ${UPLINK}kbit

# Default traffic: don't guarantee anything for the first two seconds,
# then guarantee 1/20

tc class add dev $DEV parent 1:1 classid 1:5 hfsc \
sc m1 0 d 2s m2 $((1*$UPLINK/20))kbit \
ul rate ${UPLINK}kbit

# Default traffic: don't guarantee anything for the first 10 seconds,
# then guarantee 1/20
tc class add dev $DEV parent 1:1 classid 1:6 hfsc \
sc m1 0 d 10s m2 $((1*${UPLINK}/20))kbit \
ul rate ${UPLINK}kbit

# add THESHAPER chain to the mangle table in iptables

iptables -t mangle --new-chain THESHAPER
iptables -t mangle --insert POSTROUTING -o $DEV -j THESHAPER

# had to change all iptables rules below to use mark instead of classify. tc filters then pick these up and move to the proper queue
# in the future this should all be done with tc filters as this is somewhat of a hack and there is presumably at least slightly more overhead

# put packets marked by iptables in the right queues using tc filters
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 2 fw flowid 1:2
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 3 fw flowid 1:3
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 4 fw flowid 1:4
# none needed for 1:5 as traffic lands there by default
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 6 fw flowid 1:6
# To speed up downloads while an upload is going on, put short ACK
# packets in the interactive class:

iptables -t mangle -A THESHAPER \
-p tcp \
-m tcp --tcp-flags FIN,SYN,RST,ACK ACK \
-m length --length :64 \
#-j CLASSIFY --set-class 1:2
-j MARK --set-mark 2

# put large (512+) icmp packets in browsing category
iptables -t mangle -A THESHAPER \
-p icmp \
-m length --length 512: \
#-j CLASSIFY --set-class 1:4
-j MARK --set-mark 4

# ICMP (ip protocol 1) in the interactive class
iptables -t mangle -A THESHAPER \
-p icmp \
-m length --length :512 \
#-j CLASSIFY --set-class 1:2
-j MARK --set-mark 2

setclassbyport() {
port=$1
CLASS=$2
iptables -t mangle -A THESHAPER -p udp --sport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p udp --dport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p tcp --sport $port -j MARK --set-mark $CLASS
iptables -t mangle -A THESHAPER -p tcp --dport $port -j MARK --set-mark $CLASS
}

for port in $INTERACTIVEPORTS; do setclassbyport $port 2; done
for port in $VOIPPORTS; do setclassbyport $port 3; done
for port in $BROWSINGPORTS; do setclassbyport $port 4; done
for port in $P2PPORTS; do setclassbyport $port 6; done

for VOIP in $VOIPIPS
do
iptables -t mangle -A THESHAPER --src $VOIP -j MARK --set-mark 3
iptables -t mangle -A THESHAPER --dst $VOIP -j MARK --set-mark 3
done

# Prioritize Netflix streaming from Roku by IP address and put into the VoIP class
# note that device's IP must either be set manually or a static DHCP reservation set on the router
iptables -t mangle -A THESHAPER -p tcp --dport 80 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p tcp --dport 443 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p udp --dport 80 -s 192.168.1.15 -j MARK --set-mark 3
iptables -t mangle -A THESHAPER -p udp --dport 443 -s 192.168.1.15 -j MARK --set-mark 3

Updated script I’m testing here.


Oct 3 2009

Great article about using ZFS for a home NAS

The following post is from someone who has used ZFS on his home fileserver for a year. It is very well-written and I recommend it to anyone considering doing this.

http://breden.org.uk/2009/05/01/home-fileserver-a-year-in-zfs/

Click here for a listing of all his posts on ZFS


Oct 2 2009

Article about low CableCARD adoption

http://www.dslreports.com/shownews/Cable-Industry-Shucks-Guess-Nobody-Wants-CableCARDs-104768


Sep 26 2009

CableCARD problems again after all these years!?

Please note the last update at the bottom of this post. The issue was once again solved by a firmware update from Toshiba.

It took more than a month of back and forth to get the CableCARD working in the first place. Once the CableCARD was set up properly and the firmware for the Toshiba TV had been corrected everything was great… for a little more than 4 years. Then came Tuesday, September 8 2009 and everything changed.

If you’ve read my prior posts about Comcast and using a CableCARD you’ll see that it was a huge ordeal to get the CableCARD working in the first place.

The CableCARD is now not working properly. Many channels are working, but there are a large number that do not. When flipping to one of the channels the is not working the TV shows a black screen, then it goes to channel 3 and a message comes up stating the CableCARD is acquiring the channel information. Then once this information has been acquired, the TV changes to channel 1, but sometimes another channel.

I went through all of the HD channels to note which ones do not work. Here is the list:

  • SunSports/FSN HD
  • ESPNHD
  • ESPN2 HD
  • TNT HD
  • A&E HD
  • Food HD
  • HGTV HD
  • USA HD
  • TBS HD
  • History HD
  • Discovery HD
  • Animal Planet HD
  • Sci-Fi HD

These are all channels we received before and are entitled to receive. I did not go through all of the standard definition channels, but I did note that ESPN and ESPN2 standard definition also exhibit the same behavior. This is not just limited to HD.

The problem began occurring on Tuesday, September 8 2009.

The same problems we had before getting this to work are now haunting us again. You would think that in 4 years Comcast would figure out how to get a CableCARD fixed and to properly escalate the issue to an engineer when needed.

I wanted to go through the standard channels rather than crying foul and going to higher levels as I had before. The idea was to give Comcast the benefit of the doubt. That was a mistake. Here’s the sequence of events that so far has not solved the problem:

  • Called Comcast on Tuesday, September 8, 2009. We did some basic troubleshooting and they “hit” the card several times to no avail. We then scheduled a technician to come on Saturday, September 12, 2009.

    Saturday came and went and no one showed up. We called and whoever we had spoken to on Tuesday never even scheduled the appointment. We insisted someone be sent on Sunday.

  • Sunday, a technician arrived, but he was never told this was a CableCARD issue and did not have one in his van. He had them hit the card several times and replaced some splitters in the house that were not faulty.

    The technician was really trying his best and wanted to do something, but the company had kind of screwed him over. He helped us schedule a technician to come on Tuesday, September 15, 2009.

  • The technician, a Technical Operations Supervisor, Ryan came on September 15, 2009. He had a CableCARD, but this was just luck as his notes did not specify this was a CableCARD issue. He tried to get it working by trying a new CableCARD as well as hitting the card several times. This did not work.

    He promised to forward our information to an engineer to have the issue resolved. We were to be called by the supervisor shortly.

It is Saturday, September 26, 2009 and the issue has yet to be resolved and we have not heard a thing. This is unacceptable. Why does our city keep renewing your franchise agreement?

Update – Tuesday, September 28, 2009:
We emailled the technician and the engineer, Lazaro, came today. He tried around 8 CableCARDs. He also spoke with the DAC about our setup and everything checked out. There was a recent channel re-alignment that is likely the genesis of this problem. When we originally tried to get the TV working with a CableCARD years ago, there were firmware issues related to frequency response that had to be fixed. We are now getting in touch with Toshiba to check for an update.

We called Toshiba and new firmware is available and is to be shipped to us. Hopefully this will correct the issue.

Update – Monday, October 5, 2009:
The firmware arrived. It is Seine version 1.6.5. I installed it following the instructions Toshiba sent to the letter. The CableCARD is now working perfectly.


Sep 9 2009

Apple has a great sense of humor

Due to a power failure, my NAS device was powered down. When the connection failed, the icon used to represent the NAS changed.

Normally it looks like a rack-mounted XServe, but when the connection was lost it showed up as an old-school CRT monitor with a Windows blue screen of death (BSOD).

Here’s a screenshot:
Mac icon of Windows BSOD

This is how the NAS normally appears:
The normal icon for the NAS

To see how to customize the icon used on Mac OS for a UPNP-advertised device, please read the earlier post: Using MDNS to advertise OpenSolaris NAS to Mac computers


Sep 5 2009

Automatic email alerting of zpool problems and weekly zpool scrub

Since I’m using consumer drives that are prone to failure (one already failed within 2 days, but ZFS prevented any data loss).

I am using the script located here:
http://www.sun.com/bigadmin/scripts/submittedScripts/zpadmin.txt.

This will script allows for regular monitoring and scrubs of the pool with email alerting.

These next bunch of steps may not be necessary for everyone. I’m essentially setting up sendmail to use a smarthost. Even if you do need to use a smarthost, there may be an easier way by just modifying one line in the script.

Set smarthost for sendmail email delivery since ATT blocks outgoing port 25 on home DSL connections.

Steps learned from:

Edited sendmail.mc
cd /usr/lib/mail/cf
nano sendmail.mc

This line only needed if relay not using port 25:
define(`RELAY_MAILER_ARGS', `TCP $h 587')dnl

Added:
define(`RELAY_MAILER',`esmtp')dnl
define(`SMART_HOST', `relay:[SmartHostAddressGoesHere]')dnl

Commented out by adding # the followng line:
#define(`confFALLBACK_SMARTHOST', `mailhost$?m.$m$.')dnl

Then at the command line:
m4 /usr/lib/mail/m4/cf.m4 sendmail.mc > /etc/mail/sendmail.cf

Restarted sendmail
svcadm restart sendmail

Then I had to get the script working:

Steps taken:

  • Created a directory called scripts
  • downloaded the script
  • renamed it
  • set it to be executable
  • edited settings
  • scheduled with cron

mkdir /scripts
cd /scripts
wget http://www.sun.com/bigadmin/scripts/submittedScripts/zpadmin.txt
mv zpadmin.txt zpadmin.pl
chmod ug+x zpadmin.pl

Edit script. Need to at minimum change:
my $log_path=
Set mine to "/var/log/zpool_adm_log"
my $mailto =
I had to set my $mailhost = "127.0.0.1" might be due to some sort of misconfiguration on my end. This also means that some of the steps above for configuring the smarthost could have been avoided. I still think they are worthwhile because sendmail may be needed to send other sorts of alerts in the future.

Add to cron:
crontab -e

Crontab uses vi, so we need to enter:
G goes to last line of file
o opens new line after line you’re on and gets you into edit mode

Enter the following. Feel free to omit the commented lines if you’d like.
# ZPOOL MONITOR ->
0 * * * * /scripts/zpadmin.pl
# ZPOOL WEEKLY SCRUB ->
0 5 * * 0 /scripts/zpadmin.pl -scrub

Press Esc to get out of edit mode
Then :wq which tells vi to write the file and then quit.

Now your pools will be checked every hour and will be scrubbed once a week on Sunday at 5AM.

See cron on Wikipedia for more information.


Sep 5 2009

Using MDNS to advertise OpenSolaris NAS to Mac computers

This step is not necessary, but it seems like a good idea if you have Mac computers.

Create the XML files that tell MDNS what to advertise. (Found this information from here, here, and here.

Create /etc/avahi/services/smb.service with the following content:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_device-info._tcp</type>
<port>548</port>
<txt-record>model=RackMac</txt-record>
</service>
<service>
<type>_smb._tcp</type>
<port>139</port>
</service>
</service-group>

The service that contains <txt-record>model=RackMac</txt-record> is only there so that on Mac computers the NAS shows up as an XServe server.

Enable mdns and the avahi-bridge-dsd service:
svcadm enable dns/multicast
svcadm enable avahi-bridge-dsd

I had to reboot the NAS after doing this before MDNS started working and properly advertising.

This is how the NAS appears on the Macs:
NAS normal icon


Aug 30 2009

Setting up a home NAS using OpenSolaris and ZFS

I got my inspiration for this project from reading the following posts on Adam Retter’s blog:

I particularly liked that he made every attempt to make the NAS energy efficient. The hardware he chose made sense, so I essentially bought the same parts he used.

Part suppliers used:

NewEgg has been a great supplier. I use them often and have always been provided great service. One of my hard drives went bad within two days and they made the RMA process painless. Lucky for me ZFS saved the day and I did not lose any data. OrbitMicro provided excellent service as well.

This post does not cover the parts used or steps for assembling the NAS. Please use the steps on Adam Retter’s blog. The posts are linked above.

Here are the steps that I took to set up OpenSolaris and turn it into a storage server for the house.

Downloaded and burned the OpenSolaris iso. I used version 2009.06 which was current at the time of writing. Downloads are available from http://www.opensolaris.com/get/

Tweaked BIOS settings:

  • Disabled all Serial ports
  • Enabled IDE Busmastering
  • Enabled hyperthreading

Booted up the NAS with the OpenSolaris CD in the drive and double-clicked on Install OpenSolaris. The install took some time, and then booted into the Gnome-based environment.

Downloaded the gani driver for my NICs. The built-in rge driver appears to have problems. To install it, I followed the directions in the README.txt file that was distributed with it.

Commands below assume that you are in a root shell. The easiest way seems to be using this command:

  • pfexec bash

This device is going to be a server and I did not want to use up resources having the graphical environment loaded, so I took the following steps:

  • Opened the terminal
  • Typed svcadm disable gdm to disable the graphical environment (you can always type the same command substituting enabled to bring back the GUI)

Then I logged in to the console

Now I disabled the graphical boot and sped up the boot process

  • nano /rpool/boot/grub/menu.lst
  • Changed timeout 30 to timeout 2
  • Removed:
    splashimage /boot/solaris.xpm
    foreground d25f00
    background 115d93
    ,console=graphics

Then I wiped out a lot of the system configuration to properly set up networking:

  • sys-unconfig

Now the system booted into a text-based setup where I set up networking. There are ways to do this all by hand, but the setup makes it easier.

I followed the directions in the Building my DIY NAS post on Adam Retter’s blog to:

  • set up my zfs pool
  • installed the solaris smb software
  • set up users*
  • created zfs file systems

* I set up the users a little differently. After using groupadd to add the groups, I modified the command used to create the users. I essentially removed creating a home directory for the user and changed their shell to /bin/false so they could not log in to the NAS at the console or using SSH:
useradd -c "Name" -g group -G vusers -s /bin/false name

If you have problems with permissions, it my be related to Solaris ACLs and inheritance. These links might help with that:

Performance tweaks I saw listed here and there on the internet.

  • nano /etc/system
  • added the following lines to the bottom of the file
    set pcplusmp:apic_intr_policy=1
    set zfs:zfs_txg_synctime=1

Then I rebooted the NAS. My understanding is the first line changes how interrupts are handled and may increase network IO. The second seems to tell zfs to sync changes to disk more often.

Before these changes I could not use gzip-2 compression on the pool. (Remember this is a very underpowered dual-core Intel Atom processor.) From time to time kernel CPU percentage would spike and file transfers would freeze. Now while writing a file, the kernel CPU % jumps up and down more frequently, but never goes as high.

Maybe someone else could chime in and give us further insight on the performance tuning.

Additional steps taken:
Automatic email alerting of zpool problems and weekly zpool scrub
Using MDNS to advertise OpenSolaris NAS to Mac computers

Please see the OpenSolaris and Storage categories for any future posts about this that may not be linked here.


May 3 2009

JAVA – Apache Tomcat – servlets links

These are links that I have found useful in setting up and learning Apache Tomcat, learning about servlets, research into open-source java-based CMS, and development frameworks built on top of java.

Tomcat:
Java and Tomcat on Mac OS X

Servlets:
Definition of a servlet

Open Source JAVA-based CMS:
Alfresco
OpenCMS
Magnolia
dotCMS – built on an older version of LifeRay
LifeRay
Apache Lenya
List of open source JAVA-based CMS from Wikipedia

Development frameworks built on top of JAVA:
APPFuse
Scala
Hibernate
Spring

I hope that someone finds these useful. This is by no means exhaustive and is just the result of the beginning of my research into using JAVA as the basis for an enterprise CMS.