RAID Storage ideas

[narikin]

I appreciate all your replies but I feel your getting a little obsessed by speed. My main wish is data safety over speed and as kindly explained by narikin RAID 5 or 6 seems like the best deal to me.

Sorry for the very basic question but never even considered NAS before today, (thought it was for xbox/TV etc) If I opt for a NAS system why do they come with their own memory and processor and what effect does the processor speed have on the unit?

Yes Gazwas, we got a little sidetracked. apologies.

NAS are completely self contained little computers, with processors, power supplies, etc. and as they don't really do anything but control a bunch of disks, and connect with the Ethernet, they are very happy with low power processors. I have never had one break, or noticed it as being 'slow'.

You don't sound uber-technical (and nor am I despite my modest Raid experience) so just be sure you are happy setting up a NAS. It's really not that complicated, unless you want to do something more fancy, like setting up your NAS to be accessible from outside the home (eg for a traveling photographer who needs to see files sometimes from far away). I have mine set up for that, and to act as time machine backup for girlfriends Powerbook. works perfectly. I am extremely happy with my Qnap.

If its just internal home network use, and RAID protection of your data, then its kind of perfect. Raid 5 is the de-facto standard for most people. Raid 6 is for the paranoid.

Good luck.

 

[Lars]

If its just internal home network use, and RAID protection of your data, then its kind of perfect. Raid 5 is the de-facto standard for most people. Raid 6 is for the paranoid.

Good luck.

With today's disk sizes, rebuild time for a RAID5 array after a disk failure is long enough that the probability of another disk failure and total data loss becomes significant. Throw some reasonable probability estimates into the equation and you can find a probability in the single-digit percents for a total loss at any given year for a 4-disk RAID5. Move up to more disks and the probability increases.

On top of that is the possibility of other hardware failure, like RAID card or the NAS box, the only way to protect against that is to purchase an identical backup box at the time you buy your RAID hardware, so you can move the disk array over to identical hardware. Don't expect to be able to read the disks anywhere else.

Furthermore is the uncertainty around if your RAID hardware is actually able to use a new disk that you purchase down the road, and successfully rebuild the array. This needs to be tested _before_ you fill up the array with vital data. Unless you get paid by the hour as an IT pro, I think life is too short for weeks of disk testing.

Yeah, good luck indeed. My recommendation is frequent backups if you consider RAID5. And if you go above 4 drives, use RAID6 if you want to sleep well. Or get enterprise-quality drives for 3x the cost.

The most important rule of thumb here is worth reiterating: your data isn't safe unless it is stored at two geographically separate locations.

 

[narikin]

With today's disk sizes, rebuild time for a RAID5 array after a disk failure is long enough that the probability of another disk failure and total data loss becomes significant. Throw some reasonable probability estimates into the equation and you can find a probability in the single-digit percents for a total loss at any given year for a 4-disk RAID5. Move up to more disks and the probability increases.

On top of that is the possibility of other hardware failure, like RAID card or the NAS box, the only way to protect against that is to purchase an identical backup box at the time you buy your RAID hardware, so you can move the disk array over to identical hardware. Don't expect to be able to read the disks anywhere else.

Furthermore is the uncertainty around if your RAID hardware is actually able to use a new disk that you purchase down the road, and successfully rebuild the array. This needs to be tested _before_ you fill up the array with vital data. Unless you get paid by the hour as an IT pro, I think life is too short for weeks of disk testing.

Yeah, good luck indeed. My recommendation is frequent backups if you consider RAID5. And if you go above 4 drives, use RAID6 if you want to sleep well. Or get enterprise-quality drives for 3x the cost.

The most important rule of thumb here is worth reiterating: your data isn't safe unless it is stored at two geographically separate locations.

amazing how anyone sleeps at night isn't it?

I had a 3Tb disk fail in my 6 disk Areca array last month. Insert a new disk, and it rebuilt in 5 hours. (Raid 6)

once again: Raid is not backup.
backup is backup (having a separate copy of your data stored elsewhere)
raid is raid (protection against disk failure, speed, and multi Tb ability)

 

[Jack]

My .02 on the entire issue:

RAID5: With current disk prices and sizes, R5 has outlived it's usefulness --- as others have indicated, it can be more hassle to keep an R5 up and take more time to rebuild a single drive than it takes to copy an entire archive.

RAID6: Ditto R5, except it rebuilds about 2x to 4x faster due to the dual redundancy, but a bit more costly due to drive overhead.

DROBO: While a RAID-5 or 6 in principle and depending on model, it is an automated R5/6 and can used mixed disks so this is one unit I think is viable. It's downsides are I/O speeds are slow and the time it takes to rebuild the array or an added drive. It takes far less time to wipe and re-copy an entire archive than for DROBO to rebuild the failed or added drive. On the upside, DROBO remains accessible while this is going on, but is not considered "safe," and really needs to be backed up or be the back-up array in the first place. In use, it is really more a sophisticated JBOD than a true RAID device.

JBOD: Slow, but very easy and most economical. Can use bare drives or boxes of various sizes in almost any connection interface, allowing you to add to your storage requirements as needed and as cost-effectively as possible. If access speed is non-critical, this is an excellent and inexpensive option. But it is not redundant, so everything needs to be backed up.

RAID0: Fast and easy to maintain via system software, but theoretically less safe than any single drive and not redundant, so back-up is mandatory. Theoretically the most expensive option because you need 100% redundancy of drives for back-up, but the back-up drive array does not need to match the original drive array.

RAID-0>1, What I do: Since drives (and simple enclosures) are so cheap, I now keep my main archive on a 4-drive R-0 array inside my MacPro. That is on-site copied hourly to external 2-drive 4TB eSATA R-0 units, which are also very fast access and write. This effectively makes this a RAID0-1 or RAID10 configuration. The archive is again backed-up offsite to matching size JBOD devices after every major shoot, and at least monthly. In this fashion, I always at least two back up copies of critical data onsite, and have my critical image inventory intact even if a total physical loss strikes at either storage location. As my storage requirements grow, I simply add a pair of the 2-drive R-0 boxes (they're cheap at around $110 from OWC) one for on-site and the second for of-site.

 

[Googaliser]

I love the speed of RAID0 but would never use again for large arrays (not even with SSDs). An obvious, but in my case overlooked point is the following:
If each drive has a 5% chance of failing in a particular year, then in a 6 drive RAID0 array, there is a 30% chance the array fails in a year - its additive.

RAID 5 is scary. Its really nail-biting watching it rebuild since this puts a lot of pressure on the drives and could trigger another failure (especially if they are from the same batch). I have had 2 drives fail simultaneously in a RAID 6 with SMART errors on a 3rd drive - see above for nervousness.

I now use RAID 1 for primary and RAID 6 for back-up (offsite). An Areca 1880 controller assures this is still fast enough for my needs. I use different batches of disks throughout.

My 2c

 

[narikin]

My .02 on the entire issue:

RAID5: With current disk prices and sizes, R5 has outlived it's usefulness --- as others have indicated, it can be more hassle to keep an R5 up and take more time to rebuild a single drive than it takes to copy an entire archive.

RAID6: Ditto R5, except it rebuilds about 2x to 4x faster due to the dual redundancy, but a bit more costly due to drive overhead.

DROBO: While a RAID-5 or 6 in principle and depending on model, it is an automated R5/6 and can used mixed disks so this is one unit I think is viable. It's downsides are I/O speeds are slow and the time it takes to rebuild the array or an added drive. It takes far less time to wipe and re-copy an entire archive than for DROBO to rebuild the failed or added drive. On the upside, DROBO remains accessible while this is going on, but is not considered "safe," and really needs to be backed up or be the back-up array in the first place. In use, it is really more a sophisticated JBOD than a true RAID device.
.

Sorry, not my experience, or the tens (hundreds?) of thousands of people out there with Raid 5 / Raid 6 boxes.

Why is a Raid 5 array rebuilding a single lost drive working harder than your backup drive rebuilding a lost drive? Why is the raid going to break down in a rebuild and your backup drive is not? Can anyone point me to a professional IT storage study done that shows this to be a fact? If not, then its just here-say, and needless scare-mongering, that will stop people from protecting themselves properly.

also FWIW, my experience of Drobo is dreadful. they are slick looking but slow as heck. underpowered pretty boxes. I posted the transfer rates of my Raid 6 array above, as hard facts for anyone to see. Compare that to a Drobo and weep. They cost about the same.

 

[Ben Rubinstein]

As a matter of interest as I'm about to buy two 8tb RAID boxes, what companies should I be looking at? I was quoted for a Lacie 4big Quadra Enterprise Class 8TB RAID 5 protected system and a Promax SataMax FX4 8TB RAID 5 protected system. I asked for quotes on Raid 6 boxes as well but I need two different types as one of these will be backup for the other and I know Murphy far too well to buy two of the same make when one is backup...

Suggestions on what types or where I can learn about this stuff enough to make my own choice?

 

[archivue]

i'm in a process to buy a back up solution... finally, i won't go with a RAID...
i will buy internal HD and put them in a box... and i will make a clone on an other one, but an other model of HD... and store them in two locations.
I think i will buy HD that are sold for Server, they cost nearly the double price, but they are supposed to have a longer life time !

Not a good idea ?

 

[Jorgen Udvang]

I've been following this thread for a while, and the more I read, the happier I am for not having chosen a RAID solution. The complications and costs seem to outweigh the advantages by a solid margin. If anybody are interested, here's my setup:

I use Probox units (similar to Drobo, but cheaper and easier to find, at least here) with 4 x 2TB in each, two storage disks and two backups. Backup run automatically every second day on the current disk set and weekly on the older archives.

Every week, I back up the most recent data (usually the last six months) to a 1TB Samsung portable 2.5" and take it to an identical setup at a different location where I mostly spend my weekends anyway. If I do work while staying there, which I usually do, I follow the same routine when going back.

I use Carbon Copy Cloner (Mac only, I believe) for all backups, and since the program only backs up data that have been changed or added, the process is rather swift. I have set CCC to not delete data on the target disk if deleted on the source disk. That way, if data that have previously been backed up, they will automatically come back to life again after one weekly cycle should I delete something by mistake. If I don't know that I have deleted it, I won't even notice.

This system might not be the ultimate for speed, but so far, I've found it to be more or less fail safe. It also means that I can work at any of two or more locations and always have multiple backup of any changes.

Just my 2c

 

[narikin]

I've been following this thread for a while, and the more I read, the happier I am for not having chosen a RAID solution. The complications and costs seem to outweigh the advantages by a solid margin. )

So the scaremongers here have convinced you that 99% of the world, and hundreds of thousands of users over decades are also wrong, and that Raid 5 is dangerous. This is just plain risky, but it is your work and your right to ignore the happy majority, and the entire data storage industry, who say otherwise. It is not complicated it is same or cheaper than your setup, faster and safer, assuming you complement it with a good backup regime. (like you have already)

I use Probox units (similar to Drobo, but cheaper and easier to find, at least here) with 4 x 2TB in each, two storage disks and two backups. Backup run automatically every second day on the current disk set and weekly on the older archives.

If I understand this the way it is written, then if you have an important shoot, put it onto your storage system (no Raid) and a single disk crashes between that shoot and "every second day" - then what? You have lost your shoot. Not Great.

One of your disks will crash at some point. let me repeat: It. Will. Crash. And you will loose work, because you only have protection "every second day". Why are you risking this, when simple protection for the most likely event, the inevitable event of disk failure, is easily adopted?

 

[Jorgen Udvang]

No, I will not lose my work. After each shoot, I run two backups: one to the online backup disk and one to the portable, both before i format or re-use the memory cards. That takes me less than a minute to initialize.

Probably nothing wrong with RAID, but after having worked with computers since the early seventies, 20 of those years as my main profession, I have learned to appreciate simple standards, where I have control, not the computer. If 5 billion people mean otherwise, I don't care. My system works for me.

 

[Jack]

I chose a RAID-0 copied to another R-0 for my onsite back-up precisely so I'd be certain to know immediately if any single disk in either array failed. I have heard horror stories where a single drive in a JBOD back-up failed and it was never discovered until it was needed for a rebuild, and of course then it was too late. My offsite array is JBOD, and so I check it completely every time I write new data to it to make sure all disks are available.

Anybody doing a RAID-5 for back-up as their only solution, should store an IDENTICAL RAID-5 box and controller AND a host of the exact same drives for both units as that is the only way to insure recovery if any part of the original array fails. And of course the second you price that out, you will find that simple boxes doing mirrored RAID-0's or RAID-0 to JBOD are way cheaper, very fast and very secure when used IN TOTAL.

 

[Lars]

So the scaremongers here have convinced you that 99% of the world, and hundreds of thousands of users over decades are also wrong, and that Raid 5 is dangerous. This is just plain risky, but it is your work and your right to ignore the happy majority, and the entire data storage industry, who say otherwise. It is not complicated it is same or cheaper than your setup, faster and safer, assuming you complement it with a good backup regime. (like you have already)?

Statistics and references please. Or are you just pulling those numbers out of your hat? How many RAID5 users have given up on RAID5 over the years because of multiple failure risks? Without the full picture "hundreds of thousands" is just an arbitrary number.

Because of RAID5 evangelists like you convincing me that RAID5 is a good idea, I lost data a few years ago. I had a proper RAID5 set up, one disk failed, card wouldn't accept a new drive, replacement card was not available for purchase, and while I was pulling vital data off to another storage another drive in the array failed.

This cost me irreplaceable images. So please stop repeating that "RAID5 is safe" crap. If you want to be blind to risk, fine, but please don't spread that opinion to others in an unreserved fashion.

 

[weinschela]

Sorry I am late to this, but no RAID system is a substitute for back ups, in separate Hd's. Losing data because a Raid failed and there is no other backup copy is not good, but is avoidable. Here is what I do: i have two computers in two different locations. The photos are separated into current and archive drives. The drives are synched to a Wiebe tough tech portable RAID (2 750g 2 1/2 inch 7200 rpm drives). Then the Wiebe is synched to the second computer (Chronosync). I end up with 3 copies, 2 in different physical locations and one portable. There is obviously for now a 750G limitation but this works for me. Ymmv.

 

[ustein]

My RAID (OWC) has a noisy fan).

 

[stngoldberg]

I have three LaCie external drives of varying capacity.All have failed within the last year-all have been replaced after going through agonizing communication with the vendor. Fortunately, I back up three times after every shoot, and have not lost any images permanently. My question to this group is what about ICloud? Is anyone planning on depending on that source when it becomes available?

Stanley

 

[ustein]

>Is anyone planning on depending on that source when it becomes available?

1. Not saure how safe it is
2. Uploading 20GB will be a pain
3. Cost of 500GB or more?

Not at this time.

 

[weinschela]

The OP in this thread if my memory is correct, was looking for a solution for MF. At today's technology and today's prices, the cloud does not seem to offer a solution. Uploading gigabytes is too sloooow, and storage for hundreds of gigabytes will not be free. Some cloud services like Sugarsync (highly recommended by the way for documents) already will store jpgs from phones automatically. But that is very different from giga or terabytes of raw files from MF or other larg(er) sensors.