New Hi-Res cameras - file size issues

[RobbieAB]

I repeat, nothing in what I have said goes against buying 4TB drives if they are the sweet spot. Last time I checked, 4TB drives could hold 1TB of data…

What I AM saying is unnecessary is provisioning 30-40TB of usable storage!

As far as RAID is concerned, RAID is not a backup. The main benefit of RAID is it maintains availability of a system in the event of disk failure. From the original post, I don’t get the impression availability is a significant concern. In terms of backups, RAID should not be a factor.

Even RAID 1, the simplest mirror RAID option, requires additional code or hardware to implement. This is extra places for bugs or failures to happen. If availability is not a concern, and you have a suitable, robust, tested, backup system in place, what benefit are you gaining? Even if RAID 1 is worthwhile, you don’t need a NAS to implement it!

To go back to the numbers from the original post, ~1050 images, with the largest being 600MB, gives 600GB as a rough upper limit. This is tiny by modern digital photography standards. I am questioning the suitability of setups designed for photographers shooting that many pictures in a day, that much data in a week. Yes, they would work, but in terms of capacity they are significant overkill.

 

[docholliday]

And I repeat, as I said above: to each their own. If it works for you and you feel all warm and cuddly with your solution, sweet, cool, and awesome.

 

[Derek Zeanah]

If you’re going to get critical of 50TB in one box…I’ll just tell you that box is labeled NAS3 and leave it at that.

I think OP has a good range of solid options now.

 

[RobbieAB]

@Derek Zeanah I am not critical of that much storage if you are using it (or at least filling it! Sounds like my problem with your 50TB would be “why not 100?!”

But I am not sure that same capacity requirement applies for OP.

 

[gurtch]

To all the generous and knowledgable folks here: thank you so much. It will take me a while to absorb the wonderful responses and plan a course of action. Although I am an Amateur, today I got a large print order, which will provide funds to re-do my filing and back up setup.

Sincerely , and thanks again

Dave Gurtcheff

Beach Haven, NJ

 

[gurtch]

To all the generous and knowledgable folks here: thank you so much. It will take me a while to absorb the wonderful responses and plan a course of action. Although I am an Amateur, today I got a large print order, which will provide funds to re-do my filing and back up setup.

Sincerely , and thanks again

Dave Gurtcheff

Beach Haven, NJ

 

[docholliday]

To all the generous and knowledgable folks here: thank you so much. It will take me a while to absorb the wonderful responses and plan a course of action. Although I am an Amateur, today I got a large print order, which will provide funds to re-do my filing and back up setup.

Sincerely , and thanks again

Dave Gurtcheff

Beach Haven, NJ

You're very welcome Dave! One extra bit of advice: whatever you plan on doing, look ahead to the future when you plan. It's one thing for a lot of my clients where we have plans to update storage/hardware every "x" years, but it's another when you don't do it for a living. Spending a bit extra now and giving yourself the extra room will save you a lot of headache in the future from having to do it all over again.

The 400TB cluster I just upgraded was for a client who wasn't a photographer - not a single image on it. It was a database platform, and the plan is that in 2 years, we'll probably roll it out for new hardware and double the capacity again! In that design, the space isn't the big deal. It's that the database platform will have new features and requirements in 2 years. For archiving and backing up long-term, like an image catalog, you don't need to do those upgrades because of requirements, it's just a limitation of space. That is where it pays to buy as big as you can now so you don't have to redo it all again for a while!

 

[gurtch]

Well folks, after all the knowledgeable and generous suggestions, I decided to keep it as simple as I could, keeping in mind I am an amateur and not shooting as much as I once did. I shoot less, but the files are bigger, so I still produce a lot of GBs. Here is my setup: My PC is about 8 years old. I had it custom built by Puget Systems. About a year ago I had it upgraded to 32MB RAM, Windows 10 Pro, two internal drives, both SSDs. The C: drive is 500GB (nominal) and holds the operating system and all my programs. It has 251GB free of 465GB. The other drive (D is 1 TB (nominal) and holds all my files. It has 285 GB free of 932 GB. My photo files, as I produce them, both raw and Photoshop processed, are stored temporarily on my D: drive and simultaneously stored to two external USB drives. I also have a third USB drive where my back up program once a week stores copies of my photo files from the external drives, but also backs up my internal C: and D; drives.
So this is what I purchased during "CYBER MONDAY" sales: The two external drives for storing files permanantly are 5TB. The third drive to receive backups from those two drives plus my C: and D: drives is a powered desktop drive of 14TB.
I realize the capacities are over kill, but look at these prices from B&H:
Small USB external drives:
1TB = $55
2TB = $58
4TB = $100
5TB = $110
Powered USB external desktop drives:
8TB = $190
10TB = $210
14TB = $250
I will not install all these new drives until the present drives are filled.
I just cannot thank everyone enough who responded .
Dave Gurtcheff
Beach Haven NJ
www.modernpictorials.com
PS: in my original post I asked about compression to external drives, those files already losslessly compressed by the camera. What I was referring to is my present backup program is called AShampoo Pro 15. When setting up a backup plan, it asks if I want the files encrypted (I chose "no") and compressed (I think the default may have been "yes") but yesterday I changed that to "no"
Dave

 

[docholliday]

You should be good to go for a while and will hopefully not have to think about all this again for some time!

As far as compression goes, it's always a tradeoff. Compression gives you a small savings on the backup medium, but can also slow down the copy as compression has to happen. It can also speed up the copy if you have enough CPU as it allows more data to move across the pipe to overcome the speed limit of the storage medium. Even though the images are already lossless compressed, the backup compression can still work, and possibly quite well - as each lossless compressed image file will have repeating patterns that the backup compression can take advantage of.

The only way to tell is to try a sampling backup of each method. Pick about 50GB of images, and try your backup each way. Look at the time it took to complete each backup, and the subsequent backup sizes.

 

[gurtch]

Thanks again Doc. Good idea.
Dave

 

[gurtch]

Hey guys: What would you reccommend for back up software? I first used Acronis True Image after reading several reviews. For some reason, I gave that up (can't remember why) and switched to AShmpoo Pro 15. It is easy to set up a backup plan, and it sends an email report every week. Right now, for some reason, it is mis behaving. I told it to back up an external drive with 353GB to another external drive with 1.2 TB free space. It has been running for about 18 hours and still not finished!
Dave

 

[Aviv1887]

Hey guys: What would you reccommend for back up software? I first used Acronis True Image after reading several reviews. For some reason, I gave that up (can't remember why) and switched to AShmpoo Pro 15. It is easy to set up a backup plan, and it sends an email report every week. Right now, for some reason, it is mis behaving. I told it to back up an external drive with 353GB to another external drive with 1.2 TB free space. It has been running for about 18 hours and still not finished!
Dave

Look into ChronoSync. Easy and reliable.

 

[MGrayson]

Carbon Copy Cloner is my long-time companion.

 

[anyone]

Is there any program for mac which you could recommend to test new external drives for hardware failures? Thank you!

 

[docholliday]

Hey guys: What would you reccommend for back up software? I first used Acronis True Image after reading several reviews. For some reason, I gave that up (can't remember why) and switched to AShmpoo Pro 15. It is easy to set up a backup plan, and it sends an email report every week. Right now, for some reason, it is mis behaving. I told it to back up an external drive with 353GB to another external drive with 1.2 TB free space. It has been running for about 18 hours and still not finished!
Dave

I use Veritas Backup Exec (the old Symantec Backup Exec), but that's probably way too much for what you need. Acronis works well, but is not very tolerant of removeable drives. It works best when sent to NAS or SANs, localized clusters on SATA/SAS, or network shares. USB/FW/TB dropouts will make it lose sync and that usually results in those "200 hour remaining" errors. Those dropouts are usually from power save issues, so disabling the USB/FW/TB power saving crap will sometimes resolve it, but even then, USB will still cause little loss of sync issues. It's acceptable in the spec as error correction will make it unnoticeable.

I'm assuming you're talking about Windows platform here... no clue about Mac - the only interaction with Macs I have is removing them from the network and replacing them with non-Mac boxes.

 

[Godfrey]

I've been maintaining my photographic work and archives the same way since 2004. The working library which contains both new work and completed work is on one external drive, a pair of external archive drives back that up whenever I do any work in the working library. The backup program I use is ChronoSync, running on macOS.

Over the past 17 years, I've had six or seven hard drives fail. Despite that, I've never lost a single file. My house has never burned down, nor has an earthquake, hurricane, tsunami, or radical power surge destroyed my computer (for the last of these, I have a very nice UPS (battery backup system) with very good spike protection powering my systems). While the 4 terabytes of photographic work represents a huge amount of time, money, and effort on my part, I'm just not paranoid enough to warrant the business of managing a cloud storage facility as well ... I don't believe it's necessary for my level of photographic involvement.

I agree that CD and DVD storage is flakey and prone to failure. I tried it for a short time a decade or so ago and was not impressed. Spinning hard drives are a pretty mature technology now, I'll stick with them for the foreseeable future.

In the end, I suspect that the archival grade prints and printed photo books I make and store will outlive all of the electronic media combined once I'm not around to maintain the system. And since they're the finished end products that fully express what I was looking to create, I'm happy if they're all that survive me. Them, and the photos I upload to the Library Of Congress as "published" and copyrighted works, will be what's left when all else is a mass of junk in an estate sale...

Yeah, sorry for the cheery thoughts.

G

 

[pflower]

I've been reading this thread with interest as I have been thinking about updating my backup procedures for a while. At present I have 6 6TB Sata drives. 2 hold my original catalogs which I then backup weekly using SuperDuper. I then have 2 additional drives which I back up monthly so that I have 2 backups of both my main drives. At present I put these together using 2 very cheap (£30) Inatek docking stations - 1 of which has been running 24 hrs a day for 2 years without problems. But I am looking to see if there is a slightly more elegant solution.

From my reading and YouTube research, I have decided that I neither want nor need a RAID system or an NAS system. But the Synology NAS systems do look useful, but I would only really want to use then as directly attached drives to my Mac via Thunderbolt or USBC. However although a 4 bay system from Synology is not ludicrously expensive (c£500) I wonder if anyone can recommend an alternative that would suit as, what I understand the terminology to be, a DAS system. I have also started the laborious process of uploading what is currently about 9TB of photos to Backblaze.

So any thoughts on storage system that would suit me? Or should I just go with Synology even though I won't use it to its full potential.

 

[rdeloe]

Synology makes excellent NAS devices, but I wouldn't use a NAS for direct connection.

It sounds like what you need is a good enclosure. I wouldn't reject RAID out of hand. One of my (many) backup devices is a simple two drive RAID enclosure that looks like one big hard drive to the computer. Mine is made by Stardock, which offers these in many different forms. The only thing I don't like is the fan is loud. This is not a concern for me because I only turn it on to run backups, but it would be a problem were I to leave it on all the time. Setup is very simple: insert drives, move a switch to the desired RAID option, and you're done. ]

I also use a docking station for bare drives, but it's purely for backup. I four drives that I rotate through the docking station. I keep two at home and two at my work. Once a month or so I swap work and home drives. Cheap but strong plastic drive boxes make transportation of bare drives safe and secure.

 

[docholliday]

The number one killer of spinning rust is load/unload cycles (power on-off) with the second being physical shock with the heads not parked off the ramp. That is followed by high thermals and finally run time. I've seen/owned drives with 100K+ hours on them with 0 errors. It's because the start/stop count was around 100. The drives run 24/7 in solid enclosures and are never moved. A properly unpowered drive can handle a surprising amount of shock, but when the head isn't locked, even a tilt of the drive can cause a head to smack the spinning platter!

I've also had to look at/work on issues where drives had died with less than 500 hours on them from being plugged/unplugged often, been bumped during their plug/unplugging (causing head skates during ramp unloading), or been overheated by using a non-cooled USB dock. Almost all the dead drives I've had to deal with have been USB externals, being killed during transport during an "oops" moment or by the user moving the drive while running.

The benefit of a NAS isn't just the constant connection. It's also that a centralized, reliable location exist for those backup with the drives being protected and properly cooled. You don't have to run RAID in a NAS. You can run BIG (where drives are just added up, 4TB + 2TB=6TB), JBOD (where each drive is just a drive), or striped (RAID 0 is not a RAID). WIth the NAS running, you can set your software to perform consistent, timed backups in an incremental mode which will only backup changed files, increasing your usable storage time.

You could also get a multi-drive direct TB/USB bay to connect and do the same thing without a network. But get one with adequate cooling, not a sealed plastic/rubber box that is designed to "protect" the drive. Those tiny USB drives are meant for short bursts of data transfer, not for extended runs that make the drive hot. The key is also to not be powering up/down and plugging/unplugging the drives all the time.

 

[gurtch]

Hey guys: up until now the posts have been theoretical, I have never lost a file. But now I really need to set up my new drives. I just got two print orders. One was for 4 images (big, 20"x30" in 30"x36 mats and frames). The second order was for two 13"x19"prints in 18"x24" mats and frames. In spite of my triple redundancy with external drives, I "lost" both files for the two 13"x19" prints and also lost a file for one of the big prints. Somehow I must have accidentally deleted three files. My AShampoo Pro 15 may or may not have filed it before I "lost" the files, but I have not yet figured how to "restore" files from that drive. I think my new file system will not include backup software. I will drag and drop files so the files will have names, and I will not need to use some propriatery restoration procedure. Sorry, end of rant. Embarrassing as all heck.
Dave