The GetDPI Photography Forum

Great to see you here. Join our insightful photographic forum today and start tapping into a huge wealth of photographic knowledge. Completing our simple registration process will allow you to gain access to exclusive content, add your own topics and posts, share your work and connect with other members through your own private inbox! And don’t forget to say hi!

Image backup solution for MF

shlomi

Member
I'm not overcomlicating things - data architecture is actually very complicated, and I believe oversimplifying it introduces significant risks.

Data safety is a very significant issue today that is not being taken seriously by most. I've witnessed many data losses by people I know which could have been avoided. Data safety was not an important issue ten years ago, and it will take some time for people to realize how important it has become and what to do about it. Cloud storage is probably the long term solution, but not really here yet.

You can build your own NAS box from components you buy at radio shack and software you write yourself. If you believe it will be equal to what the best company in the market offers - I can only wish you luck. The operating system is the most important component of the NAS, and I believe it took them thousands of man days to get where they are today.

Even if you have only one PC you still want to put your NAS on the network. That way it can sync with backup units without interrupting your PC. Also pretty much everybody has more than one data consuming device in their house or business and can benefit from a NAS.

Regarding RAID 6 - it doesn't solve the resilvering problem. It just means that if the first resilver fails, you go to an even more complicated process of secondary resilvering from all the remaining drives. RAID 6 is not significantly safer than RAID 5. RAID 10 is.

If you really have three full copies of your data which are fully synced every day, then you can say that each one can be less safe. However if they are direct connect to your PC then I don't see how a proper sync can work well. Also you want your main data to never fail if possible. Main fail means you go into recovery mode which is very unpleasant and can interrupt your business seriously. I agree it is not really necessary to make the main extra safe if you have 2 others, but for the convenience it is worth it if you can afford it. You still need 3 copies with one off site, which I believe almost nobody does today.

I don't understand everything but this I understand very well. You don't have to take my word for it, and if you prefer making your own boxes, using RAID 5/6, or whatever you feel is best, I wish you all the best. I posted this information in the hope that people will read it and learn from it, not to win an argument.
 
Last edited:

Wayne Fox

Workshop Member
I don't disagree with most of your points. I think few realize how serious and how challenging it is to employ an effective and affordable backup strategy. I believe most confuse the idea of raids and backup when indeed raids are more about data transfer rates or whether data needs to be accessible when failures occur. I also think most don't invest enough resources and effort keeping their backups healthy.

I just don't agree that NAS brings that much special to the party when we're talking backup data for many users. In might be under some circumstances, but most photographers who are lurking on forums like this aren't sitting in corporate environments with extensive network resources. In fact most are sitting there with a wireless connection to a small home or work network so they can get on the internet and maybe share some files with some other user out there. (maybe). Many are just sitting at home and the only reason they have the network at all is to connect to the internet and to print their work.

YOU said "Even if you have only one PC you still want to put your NAS on the network. " Why should I invest the extra money to make my large storage device a network device instead of just hooking it up to my PC? Why is hooking a network cable to my main storage device better than hooking a USB 3 or Thunderbolt cable?

As far as synchronizing data there are many strategies each with tradeoffs. The more you can invest in hardware the fewer the tradeoffs might be. My synchronizing is completely automatic, full clones of raid cabinets happening at regular intervals. When they trigger I see my hard drives going to town, but I see no real hit in my productivity. Each hour my main raid 0 mounts my secondary raid 0, archives any changes into a folder and clones over any new things then unmount the secondary raid. This usually takes less than a minute. Each night, my secondary raid mounts my third raid, archives then clones, then unmounts. Once a week I go to the fireproof vault in the cement room under my porch, get my 4th level raid, clone it over to the third level raid, put it back in the vault ... and this is the weakness to my system. An offsite storage accessible via network would be great be it a NAS or any other file server, or even cloud based. But I'm not sure how I can get 10 TB's of data transferred so I'm "synced". But in my case, about 2 minutes to change some symlinks on my startup drive and any of those other drives fully replaces the original main drive. So I have a pretty easy recovery solution. Not for all and maybe a little technical for many (although the only reason I do that is I don't keep my home folders in my user folder on my mac's startup drive and most users don't do this)

Certainly in any backup strategy, a NAS device should be considered. But NAS is about replacing expensive windows and unix server boxes to manage data storage with a simpler solution, and it's not a backup device or strategy ... it can be part of one, but before paying the extra cost it needs to be determined if the added feature of network accessibility is important and if it is your primary storage device and is only accessible on the network will the lower throughput be a problem. (and that can only be answered by each user based on their file types and workflows). the problem I have is most NAS makers tout them as these amazing bulletproof devices that will solve all your backup problems ... when the NAS itself can be just as problematic as a PC hooked to a bunch of hard drives. (ask Seth Resnick about that one and his Drobo experiences).

Anyway, as you mentioned, time to move on. certainly enough information here to at least get readers to realize it's worth investing some research and thought into.
 

Zerimar

Member
An update.. Just got the Synology DS1815+ and am loving it so far... (the NAS features are proving useful to me personally..)

I only got 4 drives with it and have it setup as a SHR for the time being. After reading more online I am thinking about biting the bullet and buying 4 more drives, and restarting with a RAID 10 from the get go.

At this time a secondary NAS isn't an option, but I can see myself buying more drives and doing the USB 3 SATA reader and carbon copy cloning the drives, and storing a set off site. I am also thinking that some sort of online cold storage service would be ideal as well... Any suggestions on a provider to have the Synology automatically backup to? I have around 8 TB of data that would take a while to initially sync..
 

Zerimar

Member
That'd add up to $240 a month? (8TB = 8000 GB x $0.03)
Or $80 with the alternative..

What about something like Backblaze with unlimited data for a yearly price?
 

shlomi

Member
Synology connects to:
- S3
- Glacier
- Azure
- CHT
- SFR
- ElephantDrive
- HiDrive
- Symform

I do not know most of these services.
I don't see the service you mentioned in this list.
I prefer Amazon since it is important to me to be sure the company I trust my data to will remain in business for a long time.
I don't know the other companies other than Microsoft and SFR.
I don't trust Microsoft very much, because they have a history of retiring services for no good reason (hotmail, messenger...)
SFR are in France and cellular carriers have a habit of merging or being bought.

If anyone knows much about any of the other connectable services, I would be interested to know more.
For sure Amazon is not cheap for a large chunk of data.
 

jagsiva

Active member
NAS has it's place, but network speeds are substantially slower than direct connect, especially direct connect TB raid. A network connection has quite a bit more going on that a SCSI/TB/SATA direct connection.

Some discussion about this in this thread not long ago (along with some other subjects). http://www.getdpi.com/forum/medium-...-question-medium-format-photography-only.html

I opt for ThunderBolt raid 0's that are cloned to others, raid 5 just doesn't seem to be worth the loss of storage. Great for when you need to have 0 downtime, but rebuilds are extremely slow and odds are pretty high a second drive will fail in the process.
++++1

NAS or any kind of network connection really does not cut it for editing images. You need some kind of direct attached storage.

You will need a back-up either way, so not sure RAID5 is worth the performance hit. If you get an 8-drive kit, and stripe 4 and mirror against the second set of 4 (RAID10), you'll probably see benchmark results approaching a single SSD.

After a very painful,very stupid, and very expensive experience, I know don't take any chances. My setup is:

1. SSD RAID 0 stripe - Purely for performance. I have a pretty crazy setup and it flies.

2. HDD RAID 10 - This is storage. Stuff gets mirrored off SSD array within a few hours. So those hours are the most I'll use.

3. Backup - HDDs. Just pop them into a swappable bay. I have 3 sets. Two with me, one offsite.

Good luck.
 

miska

Member
I think one very important thing is to have some backups that are NOT connected to the internet / the computer most of the time.
The newest form of Internet crime I've heard of, is for hackers to take your data hostage. They seem to target the smaller companies, who don't have large IT departments, but value their data. So the criminals will make a customized virus / trojan that is undetectable by virus scanners (because it is made only for one company). This virus will then crypt all the data the affected computer has access to. Then the bad guys ask for money to get a decryption key to your own data.
The novelty of this, is that they really target someone. It is worth their extra customization work, because people would pay perhaps 10k$ to get their data back - although it is questionable if they really decrypt your data, once they get the money.
They do not target the larger companies because those have big IT departments and in principle very solid backups. Private people are also not a target, since they don't value their data enough. But a small photo-studio ? I could see that happening.
Only way to protect yourself is to have an offline backup, that cannot be easily accessed by the crypting virus.
 

shlomi

Member
Ransomware is a threat, but there is a lot you can do to minimize it.

1. You put your NAS behind a firewall appliance that allows only the services you actually use.
2. You install a backup NAS which is dormant most of the time and goes live only to receive backups, also behind a firewall.
3. You connect these two NASs with site-to-site VPN.
4. Using the firewalls, you don't allow either NAS any access to or from the cloud other than the VPN.
5. You install the latest firmware and check all the security recommended setting - Synology has a security wizards that really closes a lot of holes.

You still have a possibility of getting hit if a virus inhabits your computer inside your LAN, and then attacks the NAS from there.
For sure a totally offline copy is the best, but if you reach around 10TB which many do now, it becomes quite difficult and you will not very likely keep up to date copies of all of it.

Recent ransomware events like you said written especially for Synology, took around 3000 units around the world. They Demanded $100,000 from Synology that they refused to pay. The data was lost. I do not have the data on that infestation but I believe the units taken were not properly isolated from the internet. I know one person who was affected and had internet access enabled on his NAS. That way the hackers can use port scanning to look for an opening.

I am quite sure no hacker would bother writing a worm especially for a specific photo studio.
 

Chris Giles

New member
I've not read the whole thread but I have about 16tb of data that I have to manage. I have a drive for incoming work, another for outgoing work, a 2tb hybrid SSD drive for recent storage ( which file syncs to a seperate external drive ) - then several hgst hard drives that hold the raws I've processed. They aren't massively important.

(Also, the working raw files are synced with Dropbox too).

Any finished work is stored in JPEG on a 4tb hgst drive and also backed up on disc bluray.

I'll never, ever use raid or a nas system. Unless it's a mirrored c drive.
 

shlomi

Member
NAS or any kind of network connection really does not cut it for editing images. You need some kind of direct attached storage.
That is a very subjective statement.
NAS is cutting it very well for me and others.
I have been using NAS for almost all my work for over six months.
I have no problems doing post processing with it.
I have no problem shooting directly to it.
When I am with a client, sometimes I redirect to local SSD to avoid the 2 seconds delay before the preview comes up on C1.
That does not bother me one bit when I'm working alone.
If I shoot an animation with dozens of shots, that is the only case when I feel I really need to redirect to local SSD.
I really don't like shooting direct to SSD, because if the drive fails mid work, I will never get those shots back.
It's almost always impossible to recover from SSD.


After a very painful,very stupid, and very expensive experience, I know don't take any chances. My setup is:
What was the mistake and what was the damage?
I'm very interested in data loss stories, to know what to avoid.


1. SSD RAID 0 stripe - Purely for performance. I have a pretty crazy setup and it flies.

2. HDD RAID 10 - This is storage. Stuff gets mirrored off SSD array within a few hours. So those hours are the most I'll use.

3. Backup - HDDs. Just pop them into a swappable bay. I have 3 sets. Two with me, one offsite.
If I understand you correctly and (1) and (2) are off the same desktop,
then if that desktop experience a catastrophe, you are left only with your manual backups.
That is OK for you as an amateur, but for a profssional it's too big of a gap in data safety.
I know the company you work for takes better care of its data...

It's fine for an amateur to prefer speed over safety and security, but I think most professionals would go the other way.
 
Top