Dual 60TB, how do I begin to do that?

[Ben Rubinstein]

A project I've been working on has just changed drastically in size. Basically over two years I'll be in charge of a unit photographing 1 million pages using a 40 megapixel back. Even at the RAW compressed size that's a staggeringly huge 53 Terabytes of data and I'll need redundancy, off site redundancy by choice. If I want to use 16bit uncompressed Tiff we're talking about some 200TB!

How do I even begin to do that? A huge Raid setup with networking to the offsite? Mutiple smaller units networked? I'm not a computer engineer, this project had originated as using a 28 megapixel back over about 5 more years as is being currently forecast so the original estimates were way, way off.

 

[kit laughlin]

Curious to see the answers to this one. Good luck, KL

 

[cjlacz]

That's a lot of space. Something like this with the extension cabinet would hold 24 drives, or 72TB, -6TB for raid 6 redundancy. You'd need a second setup for backup of course. That's probably on the affordable side without building your own boxes.

If it was me, I'd probably look at higher end options, especially for better performance. Synology has some options, but looking at real enterprise storage might be the way to go. That's out of my range of knowledge though.

I'd try to find a solution that you can grow with. Either adding disks, expansion units etc. Storage gets cheaper at a pretty rapid rate and I'd try to add storage as you need it. Creating a system that isn't going to be filled up for at least two years isn't economical.

Good luck!

 

[bensonga]

This raised interesting questions in my mind about the level of detail (megapixels) needed to record the actual information contained on documents (even rare documents, which presumably will not be destroyed after they are photographed).

As a test, I took a quick photograph of a 8.5x11 page of small text with a 12 megapixel DSLR with a 50mm macro lens attached (no special lighting). I had no problem seeing very fine detail in the image....frankly, better than my own eyes would see.

Of course, if these documents contain very fine detail and/or images, that will likely up the required resolution.

I know....off topic, not relevant, not helpful.....it just got me to thinking....one thing leads to another.

Gary

 

[Ben Rubinstein]

OK just been off the phone to the shop, this is what they are suggesting: http://www.promax.com/Solutions/ProjectSTOR7Archive5/ a raid enclosure with LTO tape backup (for offsite storage) for starters plus a few of these: http://www.promax.com/Solutions/SataMAXFX8/

 

[docmoore]

a staggeringly huge 53 Terabytes of data and I'll need redundancy, off site redundancy by choice. If I want to use 16bit uncompressed Tiff we're talking about some 200TB!

How do I even begin to do that?

Ben,

Might be instructive to spend a bit of time with the digital cinema crowd...these numbers whilst large are fairly common there.

I assume that your primary could be large drives whether RAID is needed depends on who and how quickly the data needs to be accessed and or retrieved. Begs the whole DAM question and how to catalog it all.

A similar question was posed on the RED forum and the backup redundancy part of it seemed to be Quantum LTO5 tapes...each at 1.5 - 3 TB ( with compression. ) Tapes are selling in bulk for $50 - 60USD. HP or Quantum drives seem to be favored and these run $2500-4000 with software and card to interface.

Link:
http://reduser.net/forum/showthread...-an-LTO-system&p=817462&viewfull=1#post817462

So, the overall costs are not in hardware...more in manpower hours to accomplish it all.

Would love to hear how you approach this and how it evolves.

Bob

 

[Ben Rubinstein]

The DAM question is one I've not really started thinking about yet. Proper keywording and naming convensions applied preshoot so that the tethered files are automatically given the information is only the start of it. When dealing with tape storage you need your filing system finalised before you even start as you can't change anything without having to rewrite a cupboard full of tapes.