Choosing hard drivesFollow

So here's an odd question. We're looking for some hard drives for the lab. We've run across some slightly unusual requirements, so we're bit at a loss of where to start. Basically we don't care so much about size, a few hundred gigabytes is fine, but instead we're looking to buy drives that have a couple of features.

1) The ability to quickly write large numbers of files. A lot of the files we generate are smaller text files, 1-10kb in size, but we generate them by the millions. Having the ability to read/write those quickly is a plus.

2) Long term stability. Something like a 2-year warranty is out. Hopefully will find something that will last longer than that without having to be pitched. Data would be archived other places than these disks, so that isn't so much the concern, but keeping them running over a time-frame like 5-6 years without mechanical difficulties, corruption, etc. would be nice. Having to re-analyze lost data isn't fun.

Any thoughts? I've never shopped for drives like this before, so wasn't even sure how to start comparing things.

are the files being written sequentially, or in mass batches all at once? Also, when you say "long term stability" do you need to access those same files continually throughout those same 5 years, or can you offload those to backup as they are processed.

You have a couple of options here, and there are going to be tradeoffs in terms of speed and price whichever route you go.

At the top of the heap, you have things like the Texas Memory systems RamSAN http://www.ramsan.com/ Which essentially employes a boatload of solid state drives in a fiberchannel Storage Array Network (SAN) attached array. It isn't going to get any faster with publically available technology at the moment then that or one of the FusionIO type systems. You are looking at a minimum of $20,000 to get into one of their small servers, plus a fiberchannel attached server ~$7,000ish (something like a HP dl580 G8 server), and possibly a fiberchannel switch ~$2,000 - ~8,000, depending on options. You're going to want a multiple core server with as many processors and cores as you can possibly cram in it, and a fair amount of very high speed ram. Solid State operating system drives in the server too.

If the cost of a small luxury car is not in the budget, you can build a fairly fast disk array on a desktop class system. Something based on a fusion-IO ioDrive ioFX for example (http://www.solidstateworks.com/) would run about $2,000 for the drive + machine costs ($2,000 - $3,000 depending on feature set). A server chassis would probably still give you better throughput with multiple processor cores than a desktop would.

If you need to go low budget, simple solid state drives would probalby work. you want the fastest one available, with at least a 6GB / second SATA III controller. A Single Level Cache or SLC is going to be faster than the MLC chips, but much more expensive.

You also want to think about backup windows and hardware redundancy. With the really expensive SAN approach, drive failures are accounted for in the chassis, and can usually be replaced without even shutting down. An enterprise class backup solution of some sort like ArcServ R16 with the open file agent could be configured to back your files up nightly, weekely, or monthly, etc without impacting operations on the ones that were open. In the desktop solution, you're more looking at either running drives in a raid 1 or raid 5 array for hardware redundancy, and then installing a secondary drive array to push backups to as needed. then off to tape.

Without knowing an approximate budget, I can't really give you a good idea of the best possible approach, but that should at least give you a few things to think over. There are "Build your own SAN" options out there, and lots of drives is going to tend to be better suited for lots of little file writes. For small files in all cases a solid state drive is going to be better for throughput these days. In some cases for large files, a 15,000 RPM rotational drive in a raid 0 array might have a faster write rate.

If possible, bring a few different solutions in and then run benchmarking on them using hard drive specific benchmark utilities such as attodisk. Here are a few individual drive results. http://www.harddrivebenchmark.net/high_end_drives.html

Ah. THe westernd Digital "green" series drives are the slowest pieces of junk on the entire planet. they are in fact slower than a 5200 RPM hard drive. They try to make up for it with cache, which for your process does you almost no good at all. Solid state drives would be a major improvement. FusionIO cards should be a massive improvement. The fiberchannel SAN should be able to handle multiple computers, and the T5500 is essentially a server motherboard more than a workstation, so you can add a fiberchanne. card to it pretty easily.

I'd bring in some SSD's or a couple fusionIO cards forst to test and see if that improves the situation enough, and if not keep those in play as OS drives and bring in a fast SAN to write things to. If nothing else, you shouldn't be able to find anything slower than a western digital green drive. Floppy drives might be an improvement.