Story
I have above average, but still relatively modest home computing needs.
In the past, I've had overly complicated systems and network designs,
and have done a fair share of GNU/Linux machine builds and various
software and service setups. Now I want something functional that does
not require a lot of maintenance. As the saying goes: "GNU/Linux is
free ... if you do not value your time." I know I will have to do
some work, but I want to minimize this, especially relative to the
amount of effort I have put into past projects. Fortunately, over the
last 8 years since I last was building machines, the GNU/Linux world is
a friendly place.
Most of the previous projects were about tinkering around with hardware
and software, more for the learning experience, so there was never a
great plan or looking at the problem from first principles. There was a
lot of doing things just to do them or because that seemed like a neat
thing to learn (e.g., DNS, NFS, NIS). Now I want something that serves
my needs with little hassle, and I would probably just pay for a
solution but there is one major problem: no one offers what I need.
There are lots of pieces available, but they are all isolated systems
which I would have to cobble together.
I began by enumerating my requirements:
- ZoneMinder - I have some IP-based cameras for a modicum of security and genrally keeping an eye on things when I am away.
- OpenVPN - I need to be able to remote desktop into my home machine, access files, security camera feeds, Insteon-controlled lights and other things all without having to open up machine/service to the world-wide Internet. Instead of worrying about mapping this port and that port and ensuring all the software is properly patched, I prefer to just make sure OpenVPN is secure.
- OpenSSH - A backup to OpenVPN and alternative way to get into my system if needed. Not everything is able to run an OpenVPN client. (N.B. I use DynDNS free service to map a domain name to by Time-Warner-delivered DHCP address.)
- GIT Server - I manage all my notes, files and software projects in GIT, so want a master server that my machines can sync with. This is the main way I keep files in sync between desktop, laptop and work machines, so it is an essential service.
- Web Server - Not essential, but I have some plans for a series of internal web pages. For instance: an collage of camera feeds to display on main TV/Monitor or tablets for monitoring.
- Shared Storage - For photos, music files, videos, software that want to be accessible from any computer and any part of the house, including my Home Theater System.
- File Backups - Backing up data from laptops, servers, desktops, etc., ideally including use of Apple's Time Machine for some Apple laptops.
- Dual Monitor Dev Server - I spend a lot of time writing software and using graphical design tools, so I need screen real estate and compute cycles to support this. A laptop does no provide a comforatble coding space for me, nor is it good for using tools like Gimp or Blender.
I currently have all this, but it is on a single machine that is over 8
years old. The more recent addition of ZoneMinder and the motion
detection processing it does for the cameras has saturated this machine,
and the single point of failure for all these systems is very worrying.
This also runs in my office/spare bedroom, so the temperature and noise
issues makes this not useful for a bedroom and not so comfortable to work
in. I want to relocate most of this to a converted closet and want it to
be quieter and cooler. I'll need to reconfigure the closet and provide
better cooling, which is a related, but separate project called the Server Closet Project.
All these shortcomings of my current setup lead to a second set of
non-functional requirements:
- Redundancy - Being able to easily recover from a software or hardware failure.
- Resilency - Some amount of uninterruptable power, mostly to protect against short outages (say 30 minutes).
- Cool and Quiet - My home office is notoriously hot and noisy, and my emerging server/network/home theater converted closet is only moderately ventilated.
- No 3rd party dependency - I do not trust the cloud with my data, do not want an external point of failure and do not want to be tied into a particular vendor. There are bandwidth issue too that make dealing with lots of data in the cloud too slow and/or annoying when you cannot afford commericial grade upload bandwidth.
Lots of people have home security/camera systems, lots of people use
"cloud-based" file storage, everyone needs backup space, and many people
want shared storage for media files. This stuff should be part of your
house just like plumbing, electrical, sprnkler systems, etc. The fact
that these requirements are not easily met, and not standard is just a
testament to the immature state computer technology as a true consumer
product. Since I cannot transport myself to the future where people have
finally solved these problems, I'll have to provide my own solution.
The Beginnings
Since I want redundancy, and to not put everything on a single machine,
the first question to tackle is exactly how many machines and what will
be their function. I decided on 4 separate machines as follows:
- Main Server - Runs the critical services and functions. Lives in the server closet. Must be cool and quiet, but also be able to process 4 camera feeds with ZoneMinder. Should be on a 30-minute UPS. This will run ZoneMinder (including MySQL), SSH, OpenVPN, Apache, and be the Git Server.
- Storage Server - A network attached storage solution for shared files and backups. This will be located on the other end of the house from the Main Server, which is the best I can afford instead of off-site disaster protection. Needs to make data shareable across at least GNU/Linux and Max OS X. I will use FreeNAS for the software. If it supports Windoze, fine, if not, I'll call it a security feature.
- Desktop Server - For software development and graphical design software as well as a more comfortable computing space. Using a laptop on the couch is not so ergonomically friendly. Will need enough power to run things like Eclipse, Gimp, Blender.
- Backup to Main Server - Identical in hardware and software to the Main Server to provide a fast path to recover from failure of that machine. This is a luxury that I might not get to. It does not have to be any automatic failover, but maybe that would be nice. An alternative plan would be to have the Desktop Server configured and capable of taking over for the Main Server in an emergency.
Next up was to spec out the hardware I will need for these, which is
where a lot of research is needed. I've not been in touch with the
personal computer hardware world for 8 years, so a lot of catch up
research ws needed. There's the build vs. buy decision as well, with my
preferring to buy if the right solution exists for a reasonable price.
However, I found that buy options were about $1,300, while build were
about $700 with overall better components, so I decided to build. Also,
to keep my head from spinning on too many hardware options and
configurations, I tried to make all the servers as similar as possible.
What was to be the same for all servers:
- CPU - The hardest choice for finding something that would work for all machines. I wanted low power consumption, but reasonable performance, which generally costs a little more. Intel has a line of processors geared toward low power consumption that are quite capable. I probably could have gone with much less CPU power on the NAS, but the consistency and re-purposability of the machine made me lean toward using the same CPU everywhere.
- Motherboard - I decided to go with the Mini ITX form factor due having a lot of small, quiet case options. I needed at least 4 SATA channels, could support 16GB RAM and an LGA 1155 socket to match the CPU. I have had a lot of past success with ASUS motherboards. Ideally it would support SATA 6GB/s, but the 3GB/s will also suffice since latency is likely the biggest factor in performance given how I will be using it.
- Power Supply - Research told me that SeaSonic was a very good quality brand. Something in the 400W to 500W should be fine and something that is as quiet as possible, even fanless would be best. The exact item chosen needs to be compatible with the case and the 24 pin for power the motherboard needs (ATX 12V). The cases I chose allowed me to use a normal ATX form factor with the Mini ITX motherboard, which was very nice.
- USB stick - Seems that a big evolution is a BIOS that can boot from a USB stick and the ability to put the OS on a USB stick instead of the hard drive. I like this for many reasons. No moving parts, easy to set up the OS on another machine and quickly swap it out. Need at least 4GB for this, and 8GB would be better.
- USB header adapter - If the USB stick is going to contain the OS, then it really belongs inside the computer, not sticking out of a port in the back/front. They make little adapters to plug into the motherboard that allow you to internally mount a USB stick.
The main differences between the servers:
- Case - For the NAS machine, I want a small (Mini ITX) case that supports swappable drives. For the Main Server, and even for the Desktop server, I want something with a small form factor and maybe something a little nicer looking for the Desktop Server. I found some LIAN cases that met these criteria and nicely can suppport regular ATX power supplies, which is very nice since there does not seem to be too many options for Mini ITX power supplies.
- Disk Drives - The Desktop and Main Server can get buy with a paltry 1TB of disk space, while the NAS server wants gobs of disk space. The case chosen supports 4 hot swappable drives, and there are good "green" drives from Western Digital which max out at 2TB. So, 8TB of storage it is for the NAS box, though effective size will be less after RAID takes its toll.
- Memory - The NAS and Main Server can easily get by with 8GB of RAM, while the Desktop machine will benfit from more. The motherboard has 2 slots for DDR3 memory and the motherboard chosen supports a maximum of 16GB.
Viewing the backup server as a luxury for now, I priced out building 3 servers at about $2,300 from newegg.com. Before sinking all that money, I wanted to build one version first to make sure there were no gotchas. The Main Server is the biggest need, so that came first and was priced around $650.
My final thoughts after buying the first server's parts that could alter my future plans:
- I may decide to use RAID on the main server and need another disk drive (or maybe repurpose some existing ones).
- I may decide to use a less capable CPU for the NAS.
- I may decide to get a motherboard supporting SATA 6GB/s for the NAS.
- I may need more capable graphics for the Desktop Server to power two monitors as high resolution (and consequently more capable power supply and/or CPU).
- Rethinking running the OS from USB stick after reading this article: Large USB Flash Drive Performance. This is dated information and newer drives and USB 3.0 might combine to make this information obsolete..
- More and more I am liking the idea of having the Desktop Server being the emergency backup for the Main Server.
Main Server Build
Here's the final parts list for this first server build:
Qty. | Description | Source | Unit Price | Total Price |
---|---|---|---|---|
1 | Intel Core i3-2120T Sandy Bridge 2.6GHz LGA 1155 Processor | newegg.com | $134.99 | $134.99 |
1 | ASUS P8H61-I R2.0 LGA 1155 Intel H61 Mini ITX Motherboard | newegg.com | $85.99 | $85.99 |
1 | Western Digital WD Green WD10EARX 1TB 3.5" Hard Drive | newegg.com | $79.99 | $79.99 |
1 | LIAN LI PC-Q25B Black Mini-ITX Computer Case | newegg.com | $119.99 | $119.99 |
1 | SeaSonic SS-520FL 520W ATX12V Power Supply | newegg.com | $149.99 | $149.99 |
2 | CORSAIR 4GB 240-Pin DDR3 SDRAM DDR3 1600 Memory | newegg.com | $21.99 | $43.98 |
1 | CORSAIR Flash Voyager 8GB USB 3.0 Flash Drive | newegg.com | $14.99 | $14.99 |
1 | Koutech USB 2.0 Header-Pin to Dual Type-A Adapter | amazon.com | $14.35 | $14.35 |
Total | $644.27 |
All the parts came in quick order from newegg, and as I opened them up and
aquainted myself, I got a bit more confidence that the world had not
changed much in the last 8 years. Everything started seeming familiar. I
did still read some of the documentation closely just in case though.
The case was very nice, hard drive trivial to install: just add plastic
rail. Motherboard, CPU and memory installed like the old days for the most
part. On-board everything else (graphics, sounds, NIC) made is super
simple. Power supply was packaged like it was the crown jewels: in a cloth
bag. Everything went easy until the final step of inserting the power
supply into the case. It did not fit.
The outbound wires from the power supply were jutting into the hard drive
cage. Only needed an inch more clearance, but no way to get it in without
doing damage. I though of just moving the hard drive from the cage and
removing the cage, but it was riveted in, so would have been messy to
remove. I thought of buying a new, smaller power supply and reusing this
elsewhere. I did some research and quickly found others running into this
issue. In fact, Lian, the case manaufacturer, actually makes a solution for
this: The Lian Li PE-01B Extender. Found this on Amazon for
$15.
While waiting for for the extender delivery, I just left the power supply
propped up on a box so I coukd boot up the machine and begin the install
and configuring. When it arrived, it did the job perfectly, so I was glad
that I could solve it for $15, but not so happy that Lian advertised that
it could take a full-sized ATX power supply. Note that the entire machine
configuration described below was done before this extender was installed.
The main "gotchas" found on the hardware side for this build were:
- The on-board USB pins only were USB 2.0 and not USB 3.0. In fact, there were only two USB 3.0 portds on the back, but the other 4 on thew back and the two extra board pins were all USB 2.0.
- Motherboard SATA channels only 3.0Gb/s (only 4 channels too)
- Power supply did not fit in case.
- Fanless power suply requires a specific side to face up, but case orientation makes it mount sideways. I put the "up" side as close to the case's top fan as I could and I hope there are no issues.
- I originally forgot to connect the special 4 pin ATX12V connector that powers the CPU. Machine will not boot without it. This extra power connector was something I did not remember from the olden days.
Then went and configured the server for the services I needed:
- Prepare bootable USB drive with Ubuntu Server 12.01.1 (64bit)
- Boot server with USB drive and do Ubuntu install
- Configuring Email (for sending only)
- Zoneminder install and configuration
- GIT Rpository setup and data migration
- OpenVPN install and configuration
- Changing router settings to forward ports and assign fixed IP address
Revisions to Original Plan
Here's how I now see the other machine builds after the passage of some
time and with the experience of this Main Server.
NAS Machine:
- Using lower power CPU (which is also cheaper)
- Must have 4 SATA 6.0Gb/s channels (more better)
- Must have at least 6 total SATA channels (4 for storage and 2 for OS)
- Use my two existing 120 GB drives for RAIDed boot/os partitions
- Use two existing external drives exported for Time Machine backups (maybe)
Desktop Machine:
- Reusing existing full ATX case
- It should still be relatively quiet
- Buy quieter power supply
- Buy quieter case fans
- Full size motherboard Opens up possibility of more than 16GB memory (maybe 24 GB)
- I am not as concerned about power consumption of CPU (cheaper)
- I will reuse the two 23" LCD screens (for now)
- I will reuse keyword and mouse
- I will configure this as the backup to Main Server
- Maybe install VM software to run Windows XP for MS Office stuff?
NAS Server Build
As I was shopping around newegg.com for the NAS Server and the new
perspective I had, I was not able to easily find a Mini ITX motherboard
that had 6 SATA ports with at least 4 of them being 6 Gb/s. I found the
ASRock
A75M-ITX which had (4) 6Gb/s SATA ports, but no others, so would
not support 5 or 6 drives. It is also a Motherboard brand I had no
experience with. I did find the ASUS
P8H77-I which has 6 SATA ports, but only (2) are 6 Gb/s.
After more research, I found what looks to be as very good match in the ASUS
E35M1-I Deluxe that has 5 internal 6 Gb/s SATA ports and an additional
one on the back panel. Very promising since having 5 SATA ports matches
precisely with the Lian Li PC-Q25B case's nuber of hot swappable drives. I
would then probably buy 5 drives instead of 4, and forego a separate array
for the OS.
Unfortunately, after further research, I started to get very worried about
the ASUS E35M1-I Deluxe motherboard. Newegg reviews show a lot of people
fretting over memory compatability issues and some having heat issues. A
lot of the positive recommendations I ran across for this board are for its
previous (non-Deluxe) version, which seems not quite the same. Maybe this
board tried to be too ambitious, and I never like that in hardware. With
all this, I changed my thinking about the importance of having SATA 6
Gb/s. Decided to go with the ASUS P8H77-I motherboard which has 6 SATA
ports, though only two of them are 6 Gb/s. I expect to have less trouble
with this.
After revising the hard drive decision to use 5 disks, I briefly considered
going to 3 TB drives instead of 2 TB, but the cost difference was about $40
each (times 5) and I decided that I should probably still have lots of
space anyway. Further research into hard drives turned up that for RAID/NAS
environments, the Western Digital "Red" drives are much better than the
"Green" ones. Power, error recovery, vibration resistance are all things
that make the "Red" drives better and the "Green" ones inferior. So I went
with the: Western
Digital Red WD20EFRX 2TB which I found at Amazon.com for about $8
chepaer than Newegg.com.
Next up is the CPU, with emphasis on low power and not on performance (and
LGA 1155 socket). From previous research, the Intel Celeron G530T was a
good low power, mild performance CPU and seemed to fit the requirements..
Here's the final parts list for the NAS server:
Qty. | Description | Source | Unit Price | Total Price |
---|---|---|---|---|
1 | LIAN LI PC-Q25B Black Mini-ITX Computer Case | newegg.com | $129.99 | $129.99 |
1 | SeaSonic SS-520FL 520W ATX12V / EPS12V Power Supply | newegg.com | $149.99 | $149.99 |
2 | CORSAIR 4GB 240-Pin DDR3 SDRAM 1600 Memory | newegg.com | $25.99 | $51.98 |
1 | Intel Celeron G530 Sandy Bridge 2.4GHz Dual-Core Processor | newegg.com | $48.99 | $48.99 |
1 | ASUS P8H77-I LGA 1155 Mini ITX Intel Motherboard | newegg.com | $109.99 | $109.99 |
5 | Western Digital Red 2 TB NAS Hard Drive - WD20EFRX | amazon.com | $116.99 | $584.95 |
1 | Lian Li PE-01B Power Supply Unit Extender | amazon.com | $16.99 | $16.99 |
Total | $1,092.88 |
Desktop Server Build
Next up was specing out the desktop server and I have now learned that the
starting point is with the CPU. Because I was going to drive multipler
monitors and want reasonably good graphics function, I would be using a
dedicated graphics card, so integrated graphics in the CPU was not a
necessity. After much research, I found that the Intel Core-i5 3350P was an
excellent balance of performance, low power and reasonable price. The
3350P also lacks on-board video, which made it perfect for me.
I looked for specific motherboard recommendations around the CPU, but found
nothing so fell back to my preferred vendor: Asus. The ASUS P8Z77-V LE
seemed to be quite capable and would support up to 32GB of DDR3 1600 RAM,
so I added four 8 GB CORSAIR Vengeance memory modules.
The graphics card was tricky. I want to drive at least 2 monitors, each at
at least 1920x1200. This is what my old monitors are currently operating
at and I prefer to have a card that has at least 2 DVI connectors for this.
It was unclear when the graphic cards stated their maximum resolution
whether this was total or per monitor. I surmised that it had to be
per-monitor since my 8 year old graphic card has no problems doing
3840x1200 across two monitors and many decent cards did not list that high
a maximum resolution.
One thing that has seemed to not change at all in the last 8 years is that
for GNU/Linux you still defintiely want to buy an NVidia card and steer
clear of ATI. The GeForce GTX 650 seemed like the right blend of
performance and cost, though I did decide to go with the 2 GB version. The
Asus version only had one DVI port (and 1 HDMI and 1 VGA), but the Gigbyte
version had 2 DVI ports (and one each of the others.)
My oldr drives are mostly IDE, and they are 8 years old, so I decided to
get new SATA drives for this server, and also to get a pair of drives to
RAID-0 them for some redundancy. I settled on Western Digital WD Green 1TB
(WD10EZRX).
I went with the same fanless power supply I used in the other two server
boxes. A bit pricey, but excellent reviews and best efficiency and
quietness. On the topic of quiet, though I am going to re-use my existing
big full ATX cases, the fans in there are a big noise contributor to the
room. Thus, I decided to replace these fans with some ones that are much
quieter, so got three Nexus SP802512L-03 80mm fans. Not having access to
the case when I ordered these, I ordered 3 fans to replace the 3 that I
knew are in there (I kept good notes), though I think the case has brackets
to house 4 fans.
Overall, this desktop server wound up being a lot more capable (and costly)
than I originally planned. A more powerful processor, more memory and RAID-ed
disks were the big contributors to the elevated price. Here's the final
Desktop Server parts list:
Qty. | Description | Source | Unit Price | Total Price |
---|---|---|---|---|
1 | Intel Core i5-3350P Ivy Bridge 3.1GHz 69W Quad-Core Processor | newegg.com | $179.99 | $179.99 |
1 | ASUS P8Z77-V LE LGA 1155 ATX Intel Motherboard | newegg.com | $139.99 | $139.99 |
4 | CORSAIR Vengeance LP 8GB 240-Pin DDR3 SDRAM DDR3 1600 Memory | newegg.com | $49.99 | $199.96 |
1 | GIGABYTE GV-N650OC-2GI GeForce GTX 650 2GB 128-bit Video Card | newegg.com | $139.99 | $139.99 |
2 | Western Digital Green WD10EZRX 1TB SATA 6.0Gb/s 3.5" Hard Drive | newegg.com | $79.99 | $159.98 |
1 | Seasonic SS-560KM, 560W ATX12V Power Supply | newegg.com | $139.99 | $139.99 |
3 | Nexus SP802512L-03 80mm Case cooler | newegg.com | $8.49 | $25.47 |
Total | $985.37 |