Site Sponsors
Results 1 to 7 of 7

Thread: Dual 60TB, how do I begin to do that?

  1. #1
    Super Duper
    Senior Member

    Join Date
    May 2008
    Location
    Manchester/Jerusalem
    Posts
    2,652
    Post Thanks / Like
    Images
    290

    Dual 60TB, how do I begin to do that?

    A project I've been working on has just changed drastically in size. Basically over two years I'll be in charge of a unit photographing 1 million pages using a 40 megapixel back. Even at the RAW compressed size that's a staggeringly huge 53 Terabytes of data and I'll need redundancy, off site redundancy by choice. If I want to use 16bit uncompressed Tiff we're talking about some 200TB!

    How do I even begin to do that? A huge Raid setup with networking to the offsite? Mutiple smaller units networked? I'm not a computer engineer, this project had originated as using a 28 megapixel back over about 5 more years as is being currently forecast so the original estimates were way, way off.
    I am not a painter, nor an artist. Therefore I can see straight, and that may be my undoing. - Alfred Stieglitz

    Website: http://www.timelessjewishart.com

  2. #2
    Subscriber Member kit laughlin's Avatar
    Join Date
    Nov 2007
    Location
    Two suitcases and the latest MBA
    Posts
    1,334
    Post Thanks / Like

    Re: Dual 60TB, how do I begin to do that?

    Curious to see the answers to this one. Good luck, KL

  3. #3
    Senior Member
    Join Date
    Jan 2008
    Location
    Tokyo, Japan
    Posts
    280
    Post Thanks / Like
    Images
    34

    Re: Dual 60TB, how do I begin to do that?

    That's a lot of space. Something like this with the extension cabinet would hold 24 drives, or 72TB, -6TB for raid 6 redundancy. You'd need a second setup for backup of course. That's probably on the affordable side without building your own boxes.

    If it was me, I'd probably look at higher end options, especially for better performance. Synology has some options, but looking at real enterprise storage might be the way to go. That's out of my range of knowledge though.

    I'd try to find a solution that you can grow with. Either adding disks, expansion units etc. Storage gets cheaper at a pretty rapid rate and I'd try to add storage as you need it. Creating a system that isn't going to be filled up for at least two years isn't economical.

    Good luck!
    Charles - flickr

  4. #4
    Senior Member bensonga's Avatar
    Join Date
    May 2008
    Location
    Alaska
    Posts
    2,416
    Post Thanks / Like
    Images
    819

    Re: Dual 60TB, how do I begin to do that?

    This raised interesting questions in my mind about the level of detail (megapixels) needed to record the actual information contained on documents (even rare documents, which presumably will not be destroyed after they are photographed).

    As a test, I took a quick photograph of a 8.5x11 page of small text with a 12 megapixel DSLR with a 50mm macro lens attached (no special lighting). I had no problem seeing very fine detail in the image....frankly, better than my own eyes would see.

    Of course, if these documents contain very fine detail and/or images, that will likely up the required resolution.

    I know....off topic, not relevant, not helpful.....it just got me to thinking....one thing leads to another.

    Gary

  5. #5
    Super Duper
    Senior Member

    Join Date
    May 2008
    Location
    Manchester/Jerusalem
    Posts
    2,652
    Post Thanks / Like
    Images
    290

    Re: Dual 60TB, how do I begin to do that?

    OK just been off the phone to the shop, this is what they are suggesting: http://www.promax.com/Solutions/ProjectSTOR7Archive5/ a raid enclosure with LTO tape backup (for offsite storage) for starters plus a few of these: http://www.promax.com/Solutions/SataMAXFX8/
    I am not a painter, nor an artist. Therefore I can see straight, and that may be my undoing. - Alfred Stieglitz

    Website: http://www.timelessjewishart.com

  6. #6
    Subscriber and Workshop Member
    Join Date
    Dec 2007
    Posts
    2,802
    Post Thanks / Like

    Re: Dual 60TB, how do I begin to do that?

    Quote Originally Posted by Ben Rubinstein View Post
    a staggeringly huge 53 Terabytes of data and I'll need redundancy, off site redundancy by choice. If I want to use 16bit uncompressed Tiff we're talking about some 200TB!

    How do I even begin to do that?
    Ben,

    Might be instructive to spend a bit of time with the digital cinema crowd...these numbers whilst large are fairly common there.

    I assume that your primary could be large drives whether RAID is needed depends on who and how quickly the data needs to be accessed and or retrieved. Begs the whole DAM question and how to catalog it all.

    A similar question was posed on the RED forum and the backup redundancy part of it seemed to be Quantum LTO5 tapes...each at 1.5 - 3 TB ( with compression. ) Tapes are selling in bulk for $50 - 60USD. HP or Quantum drives seem to be favored and these run $2500-4000 with software and card to interface.

    Link:
    http://reduser.net/forum/showthread....l=1#post817462

    So, the overall costs are not in hardware...more in manpower hours to accomplish it all.

    Would love to hear how you approach this and how it evolves.

    Bob

  7. #7
    Super Duper
    Senior Member

    Join Date
    May 2008
    Location
    Manchester/Jerusalem
    Posts
    2,652
    Post Thanks / Like
    Images
    290

    Re: Dual 60TB, how do I begin to do that?

    The DAM question is one I've not really started thinking about yet. Proper keywording and naming convensions applied preshoot so that the tethered files are automatically given the information is only the start of it. When dealing with tape storage you need your filing system finalised before you even start as you can't change anything without having to rewrite a cupboard full of tapes.
    I am not a painter, nor an artist. Therefore I can see straight, and that may be my undoing. - Alfred Stieglitz

    Website: http://www.timelessjewishart.com

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •