Site Sponsors
Results 1 to 31 of 31

Thread: Image backup solution for MF

  1. #1
    Senior Member
    Join Date
    Feb 2012
    Location
    Los Angeles
    Posts
    253
    Post Thanks / Like

    Image backup solution for MF

    As we all know MF takes of quite a bit of space if you shoot regularly..

    Curious as to what methods of file storage some of you use.. RAID, NAS, archival drives, etc... I am about to have my 20TB of drives filled (Down to the last 1TB left) and am looking into getting a new storage solution..

    The Synology NAS looks nice, and I like that files can be accessed anywhere. Are they realistically fast enough to edit off of?

    RAID seems to be the best solution for direct connection to a computer (running a maxed rMBP) but what raid setup is ideal? 0,1,5, etc..?

    I currently run paired drives, which I carbon copy clone and store the clone off site. This is less ideal as I am not backing drives up as regularly as I'd like.

    I like the idea of instantly having the photos copied to more than one drive and easily managed.

    If this is the wrong section to ask in, I'd like to ask a moderator to move it to the right section. Thank you!
    Rick Rose
    www.RickrPhoto.com
    HasselPhase

  2. #2
    Senior Member
    Join Date
    Apr 2013
    Posts
    398
    Post Thanks / Like

    Re: Image backup solution for MF

    Hi, the post-production facility that is in the same building where I have my studio uses LTO tape media. LTO 6's have a 2.5 TB (native) and 6.25 TB (Compressed) capacity per tape. It is reasonably fast and quite reliable. Each LTO Tape is $110. But the recorder/player is not cheap.

    I personally use a hard drive docking station and just keep buying hard drives as I need them That way I do not need to keep buying external hard drives (and deal with their associated cables and power supplies)
    Last edited by Ken_R; 27th January 2015 at 05:35.

  3. #3
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Hi,


    NAS is the right solution.
    Tapes can be very dangerous and are suitable for organizations where there's a person whose job inculdes managing the tapes, and even then when you need them they may very well turn out bad.

    Synology is the best NAS firm IMO.

    NAS contains RAID inside - you seem to be confused about this - don't use RAID inside your computer, use a proper NAS box - NAS has tremendous advtantages in protecting your data over a RAID inside your desktop machine.
    The best RAID configuration for SOHO is 10 - where every drive is mirrored separately.
    That way you avoid the long loop of parity rebuilding in RAID 5/6 which can and will create many problems after a drive fails - you rebuild only from one parity drive.
    If you use a good Gigabit switch (like SG-200) then the access speed will be more than sufficient for photo editing - I use my NAS for all my data needs, and it's very good.
    You can also hook up your NAS directly to your computer without network and that way you get the protection advantages and high speed but no access from other machines.
    It is also possible to put it on a 10Gbe switch for higher speed access.

    You must put your NAS on a dedicated UPS.
    You must connect the UPS communication line to the NAS and define to the NAS to shut down when there is a power failure.
    You should put your NAS behind a firewall router and define the firewall properly.
    Allowing remote access to your NAS makes it easier for hackers to ransom your data - think if you really need that access.
    NAS's are specifically targeted by ransom hackers and you want all the protection you can get.

    You can't rely only on one NAS - you must have at least one backup for it.
    Also it would be a very good idea to have an offsite backup in case of fire or burglary.
    Offsite backup is slower than onsite backup because of your internet uplink speed.
    If you have only one backup, have it offsite.
    Don't rely on manual backups because nobody has the discipline to do them every day or week for years.
    I think one onsite and one offsite backup is a good plan - but most would say it is overkill.
    The Synology NAS operating system contains all the backup programs you might need, you just need to activate them. Set it up to upload the difference every night - best use site-to-site VPN through firewalls on both ends - that way the remote access of the backup will not be used for malice.
    The backup NAS can be RAID 5 to save money.
    If your internet uplink is sufficient, then you can also use cloud backup - I would recommend this only as a second backup.


    You MUST use only proper drives for your NAS.
    I recommend, in this order:
    1. WD RE
    2. WD SE
    3. WD Red
    Note that RE and SE are too noisy for your study or living room - they must be placed in a closet or something.

    If you're planning to use your old drives, then don't even bother with the NAS, you will only create problems for yourself.
    I went through this last year and it was a very expensive and time consuming ordeal, but now I feel my data is finally properly protected against accident and attacks.

    I know this may seem a bit extreme - but once you go NAS then you put all your eggs in one basket - you want to make sure this basket is well protected. I've known photographers who put their entire catalogs into NAS or desktop RAID without properly understanding it, and lost all of it. Keeping it in regular drives docked in your workstation, means you run a real risk of losing a large chunk of your catalog. Looking at your work it seems worth the effort to keep it safe.
    Last edited by shlomi; 27th January 2015 at 07:22.
    Likes 2 Member(s) liked this post

  4. #4
    Senior Member
    Join Date
    Feb 2012
    Location
    Los Angeles
    Posts
    253
    Post Thanks / Like

    Re: Image backup solution for MF

    Thanks for the reply Shlomi,

    I am familiar with RAID (was looking into the OWC Thunderbay IV) and was referring to a direct thunderbolt connection to my mac, a NAS sounds like the better option.

    I am strongly considering the Synology DS1815+ enclosure to start with. I would add a secondary one when funds allow (just upgraded my digital back) and have it do the offsite sync.. that sounds like a relatively low maintenance regular backup with a good amount of redundancy.

    Are the WD RE drives that much better than the Red? I saw there's 6TB offerings for the Red, but only 4 for the RE. Obviously redundancy and reliability are always the better option.

    Would I be able to set up 4 drives to start with the DS1815+ in a Raid 10? I am all for data backup. I currently backup all my important Tiffs to Cloud Dropbox (have the 1tb) and then store archival drives off site in addition to what I have here. I mainly want an easier and more reassuring way of keeping everything safe.
    Rick Rose
    www.RickrPhoto.com
    HasselPhase

  5. #5
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    You might want to consider DS2015, which has 10GBe which you might want later.
    1815 is also very good.

    As far as drives, the error probability calculations are mindboggling, but to simplify it, the chance for an error in RE is 10 times lower than SE, which is 10 times lower than Red. I think it's worth the money and noise. Essentially Red is a pretty good consumer drive, while SE and RE are enterprise grade.

    You can set up 4 drives in RAID 10 which will leave you with only 2 drives for data.
    If you are not setting up a backup NAS from the start, then I would say RAID 10 is crucial.

    IMO the only realiable backup is one that you set up once and then it runs automatically in perpetuity. I have that with Synology. Uploading to dropbox is not a sufficient solution IMO. Synology can backup automatically to cloud services such as Glacier.

  6. #6
    Member
    Join Date
    Dec 2013
    Location
    Pittsburgh
    Posts
    49
    Post Thanks / Like

    Re: Image backup solution for MF

    Shlomi is right on the mark, in my opinion. I have used Synology NAS products exclusively for years as the repository for all my images, and they have been absolutely wonderful. The firmware is kept up to date, and even the DS1010+ I bought many years ago now supports 4TB and larger drives. I have 2-5 bay units, one backs up to the other using the built in folder sync feature. I find the built in Synology Hybrid Raid (1 disk fault tolerance) is sufficient for my needs, with the second NAS backup. I have had 1 disk in each box fail and lost nothing.

    To the answer of "is it fast enough to edit from?", in my experience, absolutely. You could install an SSD locally, work through a session in COP, and then move the folder to the NAS for safe keeping. But I've been
    entirely happy with the speed for direct access. Amazon is offering "unlimited photo storage" with Prime accounts now, I haven't explored that as another repository option.
    Aaron

    AaronKPhoto : Flickr : IQ140, 645DF, various lenses

  7. #7
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by AaronK View Post
    I have had 1 disk in each box fail.
    What model drives were they?
    Did you have a UPS with comms?

  8. #8
    Workshop Member Wayne Fox's Avatar
    Join Date
    Nov 2009
    Location
    Draper, Utah
    Posts
    871
    Post Thanks / Like
    Images
    134

    Re: Image backup solution for MF

    NAS has it's place, but network speeds are substantially slower than direct connect, especially direct connect TB raid.

    Some discussion about this in this thread not long ago (along with some other subjects). http://www.getdpi.com/forum/medium-f...aphy-only.html

    I opt for ThunderBolt raid 0's that are cloned to others, raid 5 just doesn't seem to be worth the loss of storage. Great for when you need to have 0 downtime, but rebuilds are extremely slow and odds are pretty high a second drive will fail in the process.
    wayne
    My gallery

  9. #9
    Member
    Join Date
    Dec 2013
    Location
    Pittsburgh
    Posts
    49
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by shlomi View Post
    What model drives were they?
    Did you have a UPS with comms?
    Both drives that failed were older Seagate drives. I have been running Western Digital drives and not had any issues. On one of the replacements I decided to get a Hitachi drive, to see how they go; so far no issues there either.

    WD20EVDS-63T3B0
    WD20EVDS-63T3B0
    HDN724040ALE640
    WD20EVDS-63T3B0
    WD20EVDS-63T3B0

    The other box has these drives, all of various vintages:

    WD20EVDS-63T3B0
    WD20EVDS-63T3B0
    ST31500341AS
    WD20EVDS-63T3B0
    WD40EFRX-68WT0N0

    Looks like I have one more Seagate to fail Not really its fault, using it like this.

    I do use a UPS with USB comms to the primary unit. No comm on the backup, but still running a UPS.

    Note that the mix of drive sizes I'm using has made it impossible to fully expand the array to cover all the potential storage - I don't mind, as my approach is that as the older drives fail I'll replace with larger ones and eventually increase the size of the array. I still have over 1 TB of free space so I'm not worrying about that for now.
    Aaron

    AaronKPhoto : Flickr : IQ140, 645DF, various lenses

  10. #10
    Member
    Join Date
    Dec 2013
    Location
    Pittsburgh
    Posts
    49
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by Wayne Fox View Post
    NAS has it's place, but network speeds are substantially slower than direct connect, especially direct connect TB raid.

    Some discussion about this in this thread not long ago (along with some other subjects). http://www.getdpi.com/forum/medium-f...aphy-only.html

    I opt for ThunderBolt raid 0's that are cloned to others, raid 5 just doesn't seem to be worth the loss of storage. Great for when you need to have 0 downtime, but rebuilds are extremely slow and odds are pretty high a second drive will fail in the process.
    Direct connect will be faster, but I have Photoshop using a (directly connected) SSD for its scratch disk, so once the image opens I don't use the NAS again until I save back to it. A typical 500 MB file opens or saves back in less than 10 seconds, even with the pedestrian drives I'm using. Gigabit ethernet isn't too shabby, but as you and I agree, not as fast as a direct connection.
    Aaron

    AaronKPhoto : Flickr : IQ140, 645DF, various lenses

  11. #11
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by AaronK View Post
    Both drives that failed were older Seagate drives. I have been running Western Digital drives and not had any issues. On one of the replacements I decided to get a Hitachi drive, to see how they go; so far no issues there either.

    WD20EVDS-63T3B0
    WD20EVDS-63T3B0
    HDN724040ALE640
    WD20EVDS-63T3B0
    WD20EVDS-63T3B0

    The other box has these drives, all of various vintages:

    WD20EVDS-63T3B0
    WD20EVDS-63T3B0
    ST31500341AS
    WD20EVDS-63T3B0
    WD40EFRX-68WT0N0

    Looks like I have one more Seagate to fail Not really its fault, using it like this.

    I do use a UPS with USB comms to the primary unit. No comm on the backup, but still running a UPS.

    Note that the mix of drive sizes I'm using has made it impossible to fully expand the array to cover all the potential storage - I don't mind, as my approach is that as the older drives fail I'll replace with larger ones and eventually increase the size of the array. I still have over 1 TB of free space so I'm not worrying about that for now.
    I'm using WD enterprise drives and I have the hope they will run 10 years with no problems.

    I've had many problems with Seagate drives, in fact I can say that pretty much every seagate drive I've had had problems. I swore many times to never again use Seagate, but after the last time I will remember.

    I've also had problems with WD green drives, and even with blacks.
    That is what prompted me to the NAS solution with enterprise drives.
    I do have a pretty good feel about when to expect a disaster using normal drives in normal setting - I am interested in more information about fails under NAS that is properly structured and maintained.

    You say your backup NAS is under the same UPS without comms. That means that it is not protected against power outages. Also it means it is close to the main NAS meaning it has similar vulnerabilities to fire and burglary. I would recommend placing the backup NAS as far away from the main one as possible, on a separate UPS. UPS costs $100 and you need a 20m ethernet cable and you're done.

    I sold all my older drives on ebay and am running 100% new drives - enterprise on production NAS and Reds on backup NAS's. I believe any rebuild puts unnecessary stress on your NAS and introduces a chance for data loss, so you want to avoid them as much as possible.

  12. #12
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by Wayne Fox View Post
    NAS has it's place, but network speeds are substantially slower than direct connect, especially direct connect TB raid.

    Some discussion about this in this thread not long ago (along with some other subjects). http://www.getdpi.com/forum/medium-f...aphy-only.html

    I opt for ThunderBolt raid 0's that are cloned to others, raid 5 just doesn't seem to be worth the loss of storage. Great for when you need to have 0 downtime, but rebuilds are extremely slow and odds are pretty high a second drive will fail in the process.
    After having done extensive research on the subject and having substanbtial professional technical background, I can say that NAS does have its place, and that place it is front and center. It is the only truly viable solution, and implemented properly it can reduce data loss probability to very close to zero.

    I've seen that discussion thread you referred to, but since it is about Macs, I can't comment on it.

    I can tell you this:

    - If super speed is so important to you, you can get that with NAS in two ways - direct connect or 10GBe network. Both are vastly superior to RAID with no NAS box.

    - Using RAID 0 is asking for trouble. It is significantly less safe than just drives in your desktop.

    - RAID 5 is not an approriate configuration for your production NAS. It is no wonder that you've excperienced disasters with that, and had you done the research you would know that this is to be expected. Using RAID 10 the risk of rebuild problems are all but eliminated.

    - It is also critical which NAS firm you are using - LaCie for instance is not an acceptable one. IMO and after extenbsive research, the only acceptable ones are Synology and QNAP. Drobo a distant third. Also critical to use enterprise drives and connected UPS. At least one backup NAS should exist doubling your production NAS.

    The software on a proper NAS box offers many data safety and security options, and one should make himself well acquainted with those options and make sure all of them are optimized for his needs.

    I will no go on again about a firewall, but it is another component that can't be skipped.

    If you use ALL of these measures and don't skip on ANY of the critical components, you can expect an extremely low probability of data loss. If an error does occur, you are still running, and rebuiliding time is very short (RAID 10 not 5). In the extremely unlikely event of rebuild fail or total box fail, you still have your backup NAS (there you can run RAID 5).

    I would like very much to hear from someone who'se done EVERYTHING right and still experenced data loss. So far all the testimonies I've encountered were about people who did at least one thing wrong.

  13. #13
    Member
    Join Date
    Dec 2013
    Location
    Pittsburgh
    Posts
    49
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by shlomi View Post
    I'm using WD enterprise drives and I have the hope they will run 10 years with no problems.

    I've had many problems with Seagate drives, in fact I can say that pretty much every seagate drive I've had had problems. I swore many times to never again use Seagate, but after the last time I will remember.

    I've also had problems with WD green drives, and even with blacks.
    That is what prompted me to the NAS solution with enterprise drives.
    I do have a pretty good feel about when to expect a disaster using normal drives in normal setting - I am interested in more information about fails under NAS that is properly structured and maintained.

    You say your backup NAS is under the same UPS without comms. That means that it is not protected against power outages. Also it means it is close to the main NAS meaning it has similar vulnerabilities to fire and burglary. I would recommend placing the backup NAS as far away from the main one as possible, on a separate UPS. UPS costs $100 and you need a 20m ethernet cable and you're done.

    I sold all my older drives on ebay and am running 100% new drives - enterprise on production NAS and Reds on backup NAS's. I believe any rebuild puts unnecessary stress on your NAS and introduces a chance for data loss, so you want to avoid them as much as possible.
    Actually, my backup is on a separate UPS and physically separated from the primary. I use a wifi extender to connect back to the main system, folder sync is all I can use the backup for.
    Aaron

    AaronKPhoto : Flickr : IQ140, 645DF, various lenses

  14. #14
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by AaronK View Post
    Actually, my backup is on a separate UPS and physically separated from the primary. I use a wifi extender to connect back to the main system, folder sync is all I can use the backup for.
    If it's a separate UPS, then why not USB connect it to the unit?
    With wifi connection you are kind of wasting the backup unit.
    I guess it is difficult to run wires through some houses, but if you do then you can have everything doubled.
    You don't need anything other than folder sync to back up all of your data.

  15. #15
    Member
    Join Date
    Dec 2013
    Location
    Pittsburgh
    Posts
    49
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by shlomi View Post
    If it's a separate UPS, then why not USB connect it to the unit?
    With wifi connection you are kind of wasting the backup unit.
    I guess it is difficult to run wires through some houses, but if you do then you can have everything doubled.
    You don't need anything other than folder sync to back up all of your data.
    Folder sync works fine over wifi. Once set the Synology OS prevents using the folders on the backup NAS (they are read only), so it's doing everything it's supposed to.

    As for the USB UPS, it's because it's old and I haven't bothered.
    Aaron

    AaronKPhoto : Flickr : IQ140, 645DF, various lenses

  16. #16
    Member Mr.Gale's Avatar
    Join Date
    Mar 2011
    Location
    Saratoga, CA
    Posts
    245
    Post Thanks / Like
    Images
    4

    Re: Image backup solution for MF

    I have a 20TB Drobo for in house back-up and I use Backblaze (https://www.backblaze.com/) for cloud backup. It takes Backblaze a long time for the initial backup but it runs in the background. The good thing is it is only $50 a year for unlimited storage.
    Mr.Gale

  17. #17
    Workshop Member Wayne Fox's Avatar
    Join Date
    Nov 2009
    Location
    Draper, Utah
    Posts
    871
    Post Thanks / Like
    Images
    134

    Re: Image backup solution for MF

    Quote Originally Posted by shlomi View Post
    After having done extensive research on the subject and having substanbtial professional technical background, I can say that NAS does have its place, and that place it is front and center. It is the only truly viable solution, and implemented properly it can reduce data loss probability to very close to zero.

    I've seen that discussion thread you referred to, but since it is about Macs, I can't comment on it.

    I can tell you this:

    - If super speed is so important to you, you can get that with NAS in two ways - direct connect or 10GBe network. Both are vastly superior to RAID with no NAS box.

    - Using RAID 0 is asking for trouble. It is significantly less safe than just drives in your desktop.

    - RAID 5 is not an approriate configuration for your production NAS. It is no wonder that you've excperienced disasters with that, and had you done the research you would know that this is to be expected. Using RAID 10 the risk of rebuild problems are all but eliminated.

    - It is also critical which NAS firm you are using - LaCie for instance is not an acceptable one. IMO and after extenbsive research, the only acceptable ones are Synology and QNAP. Drobo a distant third. Also critical to use enterprise drives and connected UPS. At least one backup NAS should exist doubling your production NAS.

    The software on a proper NAS box offers many data safety and security options, and one should make himself well acquainted with those options and make sure all of them are optimized for his needs.

    I will no go on again about a firewall, but it is another component that can't be skipped.

    If you use ALL of these measures and don't skip on ANY of the critical components, you can expect an extremely low probability of data loss. If an error does occur, you are still running, and rebuiliding time is very short (RAID 10 not 5). In the extremely unlikely event of rebuild fail or total box fail, you still have your backup NAS (there you can run RAID 5).

    I would like very much to hear from someone who'se done EVERYTHING right and still experenced data loss. So far all the testimonies I've encountered were about people who did at least one thing wrong.
    I guess my question is what's the difference between NAS and a direct connect box ... to me it's simply the method of connection and that's it. The NAS itself still has to be configured and by nature doesn't really offer any more security than direct connect devices other than it could be in a different location. Its still a raid 5, 6 or 10 device of some type, just like a direct connect raid would be and all of those issues remain. Yes perhaps better made than many and with some nice utilities, but it's still just a raid that's connected to a network instead of directly to a computer.

    the challenge with any backup solution is maintaining the reliability of the backup and having enough redundancy to not fail in a catastrophic situation. This usually means a series of backups all on systems that are designed to be as reliable as possible, knowing that at some point in time some will fail and can be rebuilt before others fail.

    To me Raid 10 really doesn't offer much more than cloning 2 raid 0's other than it's automatic and instant so failure means 0 downtime. It does have risks in that writing something mistakenly to the drive will get immediately copied to the mirror backup, where as a nightly clone process which archives changed or deleted files offers a way to get something back ... maybe not that important but perhaps occasionally useful. Of course the trade off there is failure of the main raid before the daily backup could mean loss of data. One thought is a raid 10 that is cloned nightly to another raid would be something to consider ... offering advantages of each method.

    Any reliable backup strategy is more about how data is stored redundantly and how the hardware is monitored and maintained because the data will need to be rewritten to new devices (hard drives etc) over time. This really does mean more than one redundant copy.

    In some circumstances a well maintained and properly backed up NAS by a sysadmin is a great option and great solution, but in a single user environment I just can't see it adds any value. I have a synology and a Drobo NAS at my store, I do it for collaboration and the need to share image files with various output devices, but oddly the 7 year old windows server I was trying to replace when I purchased those two devices is still a more reliable file sharing device than either NAS which seem to have trouble showing up for users or seem to randomly drop from the network - one issue is I'm still struggling with a network that must keep XP, 7, 8 and OS X all sharing files ( i probably still have a vista box running somehwere too).. No solution so far other than keeping that aging windows server running for files that absolutely must be available to any machine on the entire network.
    wayne
    My gallery

  18. #18
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by Wayne Fox View Post
    I guess my question is what's the difference between NAS and a direct connect box ... to me it's simply the method of connection and that's it. The NAS itself still has to be configured and by nature doesn't really offer any more security than direct connect devices other than it could be in a different location. Its still a raid 5, 6 or 10 device of some type, just like a direct connect raid would be and all of those issues remain. Yes perhaps better made than many and with some nice utilities, but it's still just a raid that's connected to a network instead of directly to a computer.
    You don't know what the differences are, but you assume they are insignificant - but they aren't. There are a lot of different types of boxes which offer different types of features. These features are not trinkets, they are crucial to perserve your data alive. You can take a Synology box and direct connect it and it still has most of its advantages.

    These advatages include:
    - Good quality heatsink that keeps active drives at 34c
    - Software to monitor drive temperature
    - Redundant fans
    - Software to monitor the fans
    - Software to monitor the entire system and alert you via email about any problem
    - Software to periodically check each drive fix and notify autromatically
    - Different ready to use backup plans
    - Different ready to use remote access plans
    - Control all access types to avoid abuse
    And there are dozens more features that exist in an excellent data OS such as Synology in a way that is stable and very easy to use. These features along with the robust build make the difference between different boxes.

    Quote Originally Posted by Wayne Fox View Post
    the challenge with any backup solution is maintaining the reliability of the backup and having enough redundancy to not fail in a catastrophic situation. This usually means a series of backups all on systems that are designed to be as reliable as possible, knowing that at some point in time some will fail and can be rebuilt before others fail.
    In general yes, but a properly constructed main system will have very little chance of disaster. You do need 2-3 boxes each containing your entire data. If the main box is RAID 10 with enterprise drives, then a main total failure is extremely unlikely. Talk to me in a few years and I can give you more concrete data on how much enterprise drives fail in a proper enclosure. You do realize that a regular drive in a regular enclosure will have totally different statistics.

    Quote Originally Posted by Wayne Fox View Post
    To me Raid 10 really doesn't offer much more than cloning 2 raid 0's other than it's automatic and instant so failure means 0 downtime. It does have risks in that writing something mistakenly to the drive will get immediately copied to the mirror backup, where as a nightly clone process which archives changed or deleted files offers a way to get something back ... maybe not that important but perhaps occasionally useful. Of course the trade off there is failure of the main raid before the daily backup could mean loss of data. One thought is a raid 10 that is cloned nightly to another raid would be something to consider ... offering advantages of each method.
    You are looking at it wrong. RAID 10 is not comparable to two RAID 5 - it is comparable to one RAID 5. You still must have two enclosures at least. RAID 10 is the safest configuration from the small ones - RAID 50 or 60 are safer but these are not applicable to you. RAID 10 is safer, because when you rebuild you rebuild from one drive only. In RAID 5 or 6 when you rebuild, then you "resilver" the entire NAS. Meaning that every bit in the NAS gets read and written. This creates very heavy stress on the drives and often generates new problems and the famous double failure which loses you all your data. This does not happen with RAID 10 which rebuilds one drive from one drive - and not taking six days to do it. Again RAID 10 serves as one box - if you consider it as having two copies of your data, then your math is wrong. It's one copy with really good safety. You still need the second independent copy and there you can skimp on the cost and use RAID 5 or 6.

    Quote Originally Posted by Wayne Fox View Post
    Any reliable backup strategy is more about how data is stored redundantly and how the hardware is monitored and maintained because the data will need to be rewritten to new devices (hard drives etc) over time. This really does mean more than one redundant copy.
    Yes.

    Quote Originally Posted by Wayne Fox View Post
    In some circumstances a well maintained and properly backed up NAS by a sysadmin is a great option and great solution
    You don't need a sysadmin for that. You can do everything yourself. Follow my instructions here, invest a few thousand dollars and a couple of weeks of your time, and you're 99.99% protected. Just don't skip on some parts that you think are not important. The Synology system does the system administration for you. You just need to set it up once, and once a week look in on it to see all is well. It will let you know anyway when anything is amiss. That is hugely different from a Windows server which does require you to perform the sysadmin tasks. The Synology automatic administraion may not be sufficient for an actual enterprise, but it is very sufficient for SOHO.

    Quote Originally Posted by Wayne Fox View Post
    but in a single user environment I just can't see it adds any value.
    The value depends on how important your data is to you - how much is it worth to you that it is not lost. Differently from a camera, insurance money can't rebuild your data. Your camera system costs $50k - is it really so much to create a data preservation system that costs $8k but gives you real protection? It makes sense to me.

    Quote Originally Posted by Wayne Fox View Post
    I have a synology and a Drobo NAS at my store, I do it for collaboration and the need to share image files with various output devices, but oddly the 7 year old windows server I was trying to replace when I purchased those two devices is still a more reliable file sharing device than either NAS which seem to have trouble showing up for users or seem to randomly drop from the network - one issue is I'm still struggling with a network that must keep XP, 7, 8 and OS X all sharing files ( i probably still have a vista box running somehwere too).. No solution so far other than keeping that aging windows server running for files that absolutely must be available to any machine on the entire network.
    You present it as if having Synology means that you will have low stability. That is not really the case. How did you implement your Synology and your Drobo? Are they RAID 5? Are they Red or Enterpise drives? Are they on UPS with comms for each one? Why you are having trouble seeing the Synology - I do not know. I have zero problems accessing mine from different systems, not all Windows. If it is configured correctly, your NAS should have no problem appearing on your network 100% of the time. If you are using Macs, then this could be the problem - I don't know what goes on in the Apple universe, but I know they don't encourage connectivity to machines that were not sold by them.
    Last edited by shlomi; 5th February 2015 at 22:50.
    Likes 1 Member(s) liked this post

  19. #19
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by AaronK View Post
    Folder sync works fine over wifi. Once set the Synology OS prevents using the folders on the backup NAS (they are read only), so it's doing everything it's supposed to.

    As for the USB UPS, it's because it's old and I haven't bothered.
    I've never tried it - if the bandwidth is sufficient to transmit all your changes overnight then it's fine. There is a security risk running your data over the air, but that's just me. I've blocked my wireless network from accessing any of my data, for security reasons.

    No USB means that a power outage longer than the battery can hold out which is typically 10-20 minutes - your drives will suffer unexpected shutdown.

  20. #20
    Workshop Member Wayne Fox's Avatar
    Join Date
    Nov 2009
    Location
    Draper, Utah
    Posts
    871
    Post Thanks / Like
    Images
    134

    Re: Image backup solution for MF

    Thatís a long reply, and seems to overcomplicate things.

    A NAS is simply a high end raid box with itís own processor and server firmware built in. You can buy raid boxes with all of the same features. what makes the NAS different is it is network accessible without a host computer.

    All of the comments about NAS (drive type, temperature control etc etc) apply equally to any raid device, be it NAS or direct connect. You can buy cheap raid cabinets, you can buy cheap JBOD cabinents and use software solutions to create raids, or you can buy better designed and more feature full raid cabinets (dual power supplies, faster firmware parity algorithms, disk health monitoring, etc). But if you have no reason to share the information on the network then I canít see what a NAS would add. And adding the network component in any strategy if unnecessary can also introduce issues that could be avoided with a simpler direct connect strategy, and just adds costs which may mean going cheaper in the overall strategy and not setting up enough redundancy and maintenance investments.

    Protecting data is a strategy. Itís as much about procedure and methodology as it is about hardware. Choice of raid is about several factors ... speed, reliability, up time, rebuild times, etc. But raid has nothing to do with backup. There are many ways to create and build a backup strategy, but adding a NAS to the equation is mainly beneficial if 1) 1 device can backup multiple users or 2) the NAS can be kept at another physical location and then the backup can be handle via network. But many make the mistake of thinking the NAS is the backup just like they like having a raid 5 or raid 6 cabinet is ďbackup. So they donít bother to actually implement a true backup strategy and backup the NAS or Raid.

    The issues of raid 5ís failing during rebuilding is becoming more commonplace, so do you opt for raid 6? Or are you better of building a series of redundant less expensive raids. Ultimately the question is if I loose device A, where is my data protected and how safe is it while I repair/rebuild/replace device A. This usually means at least triple redundancy with at least one copy at a different physical location. Thatís the starting point. ... I have 4 tiers of redundancy but I donít have a lot invested in the devices themselves. but Iím pretty confident that if one of the devices in the chain fails, I can replace it and reclone it before the other 3 would fail. Probably 3 would be enough, but adding the 4th was easy because I already had an available cabinet.

    I think NAS is great and useful, but the idea that it deserves a special place in a backup strategy for a single user system just doesnít make sense to me.
    wayne
    My gallery

  21. #21
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    I'm not overcomlicating things - data architecture is actually very complicated, and I believe oversimplifying it introduces significant risks.

    Data safety is a very significant issue today that is not being taken seriously by most. I've witnessed many data losses by people I know which could have been avoided. Data safety was not an important issue ten years ago, and it will take some time for people to realize how important it has become and what to do about it. Cloud storage is probably the long term solution, but not really here yet.

    You can build your own NAS box from components you buy at radio shack and software you write yourself. If you believe it will be equal to what the best company in the market offers - I can only wish you luck. The operating system is the most important component of the NAS, and I believe it took them thousands of man days to get where they are today.

    Even if you have only one PC you still want to put your NAS on the network. That way it can sync with backup units without interrupting your PC. Also pretty much everybody has more than one data consuming device in their house or business and can benefit from a NAS.

    Regarding RAID 6 - it doesn't solve the resilvering problem. It just means that if the first resilver fails, you go to an even more complicated process of secondary resilvering from all the remaining drives. RAID 6 is not significantly safer than RAID 5. RAID 10 is.

    If you really have three full copies of your data which are fully synced every day, then you can say that each one can be less safe. However if they are direct connect to your PC then I don't see how a proper sync can work well. Also you want your main data to never fail if possible. Main fail means you go into recovery mode which is very unpleasant and can interrupt your business seriously. I agree it is not really necessary to make the main extra safe if you have 2 others, but for the convenience it is worth it if you can afford it. You still need 3 copies with one off site, which I believe almost nobody does today.

    I don't understand everything but this I understand very well. You don't have to take my word for it, and if you prefer making your own boxes, using RAID 5/6, or whatever you feel is best, I wish you all the best. I posted this information in the hope that people will read it and learn from it, not to win an argument.
    Last edited by shlomi; 6th February 2015 at 13:56.

  22. #22
    Workshop Member Wayne Fox's Avatar
    Join Date
    Nov 2009
    Location
    Draper, Utah
    Posts
    871
    Post Thanks / Like
    Images
    134

    Re: Image backup solution for MF

    I don't disagree with most of your points. I think few realize how serious and how challenging it is to employ an effective and affordable backup strategy. I believe most confuse the idea of raids and backup when indeed raids are more about data transfer rates or whether data needs to be accessible when failures occur. I also think most don't invest enough resources and effort keeping their backups healthy.

    I just don't agree that NAS brings that much special to the party when we're talking backup data for many users. In might be under some circumstances, but most photographers who are lurking on forums like this aren't sitting in corporate environments with extensive network resources. In fact most are sitting there with a wireless connection to a small home or work network so they can get on the internet and maybe share some files with some other user out there. (maybe). Many are just sitting at home and the only reason they have the network at all is to connect to the internet and to print their work.

    YOU said "Even if you have only one PC you still want to put your NAS on the network. " Why should I invest the extra money to make my large storage device a network device instead of just hooking it up to my PC? Why is hooking a network cable to my main storage device better than hooking a USB 3 or Thunderbolt cable?

    As far as synchronizing data there are many strategies each with tradeoffs. The more you can invest in hardware the fewer the tradeoffs might be. My synchronizing is completely automatic, full clones of raid cabinets happening at regular intervals. When they trigger I see my hard drives going to town, but I see no real hit in my productivity. Each hour my main raid 0 mounts my secondary raid 0, archives any changes into a folder and clones over any new things then unmount the secondary raid. This usually takes less than a minute. Each night, my secondary raid mounts my third raid, archives then clones, then unmounts. Once a week I go to the fireproof vault in the cement room under my porch, get my 4th level raid, clone it over to the third level raid, put it back in the vault ... and this is the weakness to my system. An offsite storage accessible via network would be great be it a NAS or any other file server, or even cloud based. But I'm not sure how I can get 10 TB's of data transferred so I'm "synced". But in my case, about 2 minutes to change some symlinks on my startup drive and any of those other drives fully replaces the original main drive. So I have a pretty easy recovery solution. Not for all and maybe a little technical for many (although the only reason I do that is I don't keep my home folders in my user folder on my mac's startup drive and most users don't do this)

    Certainly in any backup strategy, a NAS device should be considered. But NAS is about replacing expensive windows and unix server boxes to manage data storage with a simpler solution, and it's not a backup device or strategy ... it can be part of one, but before paying the extra cost it needs to be determined if the added feature of network accessibility is important and if it is your primary storage device and is only accessible on the network will the lower throughput be a problem. (and that can only be answered by each user based on their file types and workflows). the problem I have is most NAS makers tout them as these amazing bulletproof devices that will solve all your backup problems ... when the NAS itself can be just as problematic as a PC hooked to a bunch of hard drives. (ask Seth Resnick about that one and his Drobo experiences).

    Anyway, as you mentioned, time to move on. certainly enough information here to at least get readers to realize it's worth investing some research and thought into.
    wayne
    My gallery
    Likes 2 Member(s) liked this post

  23. #23
    Senior Member
    Join Date
    Feb 2012
    Location
    Los Angeles
    Posts
    253
    Post Thanks / Like

    Re: Image backup solution for MF

    An update.. Just got the Synology DS1815+ and am loving it so far... (the NAS features are proving useful to me personally..)

    I only got 4 drives with it and have it setup as a SHR for the time being. After reading more online I am thinking about biting the bullet and buying 4 more drives, and restarting with a RAID 10 from the get go.

    At this time a secondary NAS isn't an option, but I can see myself buying more drives and doing the USB 3 SATA reader and carbon copy cloning the drives, and storing a set off site. I am also thinking that some sort of online cold storage service would be ideal as well... Any suggestions on a provider to have the Synology automatically backup to? I have around 8 TB of data that would take a while to initially sync..
    Rick Rose
    www.RickrPhoto.com
    HasselPhase

  24. #24
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Amazon S3 is very good for $0.03/GB/mo
    Amazon Glacier is slower for $0.01/GB/mo

  25. #25
    Senior Member
    Join Date
    Feb 2012
    Location
    Los Angeles
    Posts
    253
    Post Thanks / Like

    Re: Image backup solution for MF

    That'd add up to $240 a month? (8TB = 8000 GB x $0.03)
    Or $80 with the alternative..

    What about something like Backblaze with unlimited data for a yearly price?
    Rick Rose
    www.RickrPhoto.com
    HasselPhase

  26. #26
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Synology connects to:
    - S3
    - Glacier
    - Azure
    - CHT
    - SFR
    - ElephantDrive
    - HiDrive
    - Symform

    I do not know most of these services.
    I don't see the service you mentioned in this list.
    I prefer Amazon since it is important to me to be sure the company I trust my data to will remain in business for a long time.
    I don't know the other companies other than Microsoft and SFR.
    I don't trust Microsoft very much, because they have a history of retiring services for no good reason (hotmail, messenger...)
    SFR are in France and cellular carriers have a habit of merging or being bought.

    If anyone knows much about any of the other connectable services, I would be interested to know more.
    For sure Amazon is not cheap for a large chunk of data.

  27. #27
    Senior Member
    Join Date
    Sep 2011
    Posts
    1,068
    Post Thanks / Like
    Images
    83

    Re: Image backup solution for MF

    Quote Originally Posted by Wayne Fox View Post
    NAS has it's place, but network speeds are substantially slower than direct connect, especially direct connect TB raid. A network connection has quite a bit more going on that a SCSI/TB/SATA direct connection.

    Some discussion about this in this thread not long ago (along with some other subjects). http://www.getdpi.com/forum/medium-f...aphy-only.html

    I opt for ThunderBolt raid 0's that are cloned to others, raid 5 just doesn't seem to be worth the loss of storage. Great for when you need to have 0 downtime, but rebuilds are extremely slow and odds are pretty high a second drive will fail in the process.
    ++++1

    NAS or any kind of network connection really does not cut it for editing images. You need some kind of direct attached storage.

    You will need a back-up either way, so not sure RAID5 is worth the performance hit. If you get an 8-drive kit, and stripe 4 and mirror against the second set of 4 (RAID10), you'll probably see benchmark results approaching a single SSD.

    After a very painful,very stupid, and very expensive experience, I know don't take any chances. My setup is:

    1. SSD RAID 0 stripe - Purely for performance. I have a pretty crazy setup and it flies.

    2. HDD RAID 10 - This is storage. Stuff gets mirrored off SSD array within a few hours. So those hours are the most I'll use.

    3. Backup - HDDs. Just pop them into a swappable bay. I have 3 sets. Two with me, one offsite.

    Good luck.
    Likes 1 Member(s) liked this post

  28. #28
    Member
    Join Date
    Sep 2013
    Location
    Munich, Germany
    Posts
    219
    Post Thanks / Like

    Re: Image backup solution for MF

    I think one very important thing is to have some backups that are NOT connected to the internet / the computer most of the time.
    The newest form of Internet crime I've heard of, is for hackers to take your data hostage. They seem to target the smaller companies, who don't have large IT departments, but value their data. So the criminals will make a customized virus / trojan that is undetectable by virus scanners (because it is made only for one company). This virus will then crypt all the data the affected computer has access to. Then the bad guys ask for money to get a decryption key to your own data.
    The novelty of this, is that they really target someone. It is worth their extra customization work, because people would pay perhaps 10k$ to get their data back - although it is questionable if they really decrypt your data, once they get the money.
    They do not target the larger companies because those have big IT departments and in principle very solid backups. Private people are also not a target, since they don't value their data enough. But a small photo-studio ? I could see that happening.
    Only way to protect yourself is to have an offline backup, that cannot be easily accessed by the crypting virus.

  29. #29
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Ransomware is a threat, but there is a lot you can do to minimize it.

    1. You put your NAS behind a firewall appliance that allows only the services you actually use.
    2. You install a backup NAS which is dormant most of the time and goes live only to receive backups, also behind a firewall.
    3. You connect these two NASs with site-to-site VPN.
    4. Using the firewalls, you don't allow either NAS any access to or from the cloud other than the VPN.
    5. You install the latest firmware and check all the security recommended setting - Synology has a security wizards that really closes a lot of holes.

    You still have a possibility of getting hit if a virus inhabits your computer inside your LAN, and then attacks the NAS from there.
    For sure a totally offline copy is the best, but if you reach around 10TB which many do now, it becomes quite difficult and you will not very likely keep up to date copies of all of it.

    Recent ransomware events like you said written especially for Synology, took around 3000 units around the world. They Demanded $100,000 from Synology that they refused to pay. The data was lost. I do not have the data on that infestation but I believe the units taken were not properly isolated from the internet. I know one person who was affected and had internet access enabled on his NAS. That way the hackers can use port scanning to look for an opening.

    I am quite sure no hacker would bother writing a worm especially for a specific photo studio.

  30. #30
    Senior Member Chris Giles's Avatar
    Join Date
    Jan 2012
    Location
    Brighton
    Posts
    342
    Post Thanks / Like

    Re: Image backup solution for MF

    I've not read the whole thread but I have about 16tb of data that I have to manage. I have a drive for incoming work, another for outgoing work, a 2tb hybrid SSD drive for recent storage ( which file syncs to a seperate external drive ) - then several hgst hard drives that hold the raws I've processed. They aren't massively important.

    (Also, the working raw files are synced with Dropbox too).

    Any finished work is stored in JPEG on a 4tb hgst drive and also backed up on disc bluray.

    I'll never, ever use raid or a nas system. Unless it's a mirrored c drive.
    Chris Giles Photography

  31. #31
    Senior Member
    Join Date
    Apr 2010
    Posts
    255
    Post Thanks / Like

    Re: Image backup solution for MF

    Quote Originally Posted by jagsiva View Post
    NAS or any kind of network connection really does not cut it for editing images. You need some kind of direct attached storage.
    That is a very subjective statement.
    NAS is cutting it very well for me and others.
    I have been using NAS for almost all my work for over six months.
    I have no problems doing post processing with it.
    I have no problem shooting directly to it.
    When I am with a client, sometimes I redirect to local SSD to avoid the 2 seconds delay before the preview comes up on C1.
    That does not bother me one bit when I'm working alone.
    If I shoot an animation with dozens of shots, that is the only case when I feel I really need to redirect to local SSD.
    I really don't like shooting direct to SSD, because if the drive fails mid work, I will never get those shots back.
    It's almost always impossible to recover from SSD.


    Quote Originally Posted by jagsiva View Post
    After a very painful,very stupid, and very expensive experience, I know don't take any chances. My setup is:
    What was the mistake and what was the damage?
    I'm very interested in data loss stories, to know what to avoid.


    Quote Originally Posted by jagsiva View Post
    1. SSD RAID 0 stripe - Purely for performance. I have a pretty crazy setup and it flies.

    2. HDD RAID 10 - This is storage. Stuff gets mirrored off SSD array within a few hours. So those hours are the most I'll use.

    3. Backup - HDDs. Just pop them into a swappable bay. I have 3 sets. Two with me, one offsite.
    If I understand you correctly and (1) and (2) are off the same desktop,
    then if that desktop experience a catastrophe, you are left only with your manual backups.
    That is OK for you as an amateur, but for a profssional it's too big of a gap in data safety.
    I know the company you work for takes better care of its data...

    It's fine for an amateur to prefer speed over safety and security, but I think most professionals would go the other way.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •