Setting up a home NAS using OpenSolaris and ZFS

I got my inspiration for this project from reading the following posts on Adam Retter’s blog:

• In need of Open Storage
• Choosing Software and Hardware for my DIY NAS
• Building my DIY NAS

I particularly liked that he made every attempt to make the NAS energy efficient. The hardware he chose made sense, so I essentially bought the same parts he used.

Part suppliers used:

• NewEgg
• OrbitMicro

NewEgg has been a great supplier. I use them often and have always been provided great service. One of my hard drives went bad within two days and they made the RMA process painless. Lucky for me ZFS saved the day and I did not lose any data. OrbitMicro provided excellent service as well.

This post does not cover the parts used or steps for assembling the NAS. Please use the steps on Adam Retter’s blog. The posts are linked above.

Here are the steps that I took to set up OpenSolaris and turn it into a storage server for the house.

Downloaded and burned the OpenSolaris iso. I used version 2009.06 which was current at the time of writing. Downloads are available from http://www.opensolaris.com/get/

Tweaked BIOS settings:

• Disabled all Serial ports
• Enabled IDE Busmastering
• Enabled hyperthreading

Booted up the NAS with the OpenSolaris CD in the drive and double-clicked on Install OpenSolaris. The install took some time, and then booted into the Gnome-based environment.

Downloaded the gani driver for my NICs. The built-in rge driver appears to have problems. To install it, I followed the directions in the README.txt file that was distributed with it.

Commands below assume that you are in a root shell. The easiest way seems to be using this command:

• pfexec bash

This device is going to be a server and I did not want to use up resources having the graphical environment loaded, so I took the following steps:

• Opened the terminal
• Typed svcadm disable gdm to disable the graphical environment (you can always type the same command substituting enabled to bring back the GUI)

Then I logged in to the console

Now I disabled the graphical boot and sped up the boot process

• nano /rpool/boot/grub/menu.lst
• Changed timeout 30 to timeout 2
• Removed:
  splashimage /boot/solaris.xpm
foreground d25f00
background 115d93
,console=graphics

Then I wiped out a lot of the system configuration to properly set up networking:

• sys-unconfig

Now the system booted into a text-based setup where I set up networking. There are ways to do this all by hand, but the setup makes it easier.

I followed the directions in the Building my DIY NAS post on Adam Retter’s blog to:

• set up my zfs pool
• installed the solaris smb software
• set up users*
• created zfs file systems

* I set up the users a little differently. After using groupadd to add the groups, I modified the command used to create the users. I essentially removed creating a home directory for the user and changed their shell to /bin/false so they could not log in to the NAS at the console or using SSH:
useradd -c "Name" -g group -G vusers -s /bin/false name

If you have problems with permissions, it my be related to Solaris ACLs and inheritance. These links might help with that:

• http://wiki.genunix.org/wiki/index.php/CIFS_Service_Troubleshooting#Access_Control_List_Inheritance_Issues
• http://gibbs.acu.edu/2008/07/20/my-cifs-on-zfs-acl/
• http://cuddletech.com/blog/pivot/entry.php?id=939

Performance tweaks I saw listed here and there on the internet.

• nano /etc/system
• added the following lines to the bottom of the file
  set pcplusmp:apic_intr_policy=1
set zfs:zfs_txg_synctime=1

Then I rebooted the NAS. My understanding is the first line changes how interrupts are handled and may increase network IO. The second seems to tell zfs to sync changes to disk more often.

Before these changes I could not use gzip-2 compression on the pool. (Remember this is a very underpowered dual-core Intel Atom processor.) From time to time kernel CPU percentage would spike and file transfers would freeze. Now while writing a file, the kernel CPU % jumps up and down more frequently, but never goes as high.

Maybe someone else could chime in and give us further insight on the performance tuning.

Additional steps taken:
Automatic email alerting of zpool problems and weekly zpool scrub
Using MDNS to advertise OpenSolaris NAS to Mac computers

Please see the OpenSolaris and Storage categories for any future posts about this that may not be linked here.