Site Sponsors
Results 1 to 25 of 25

Thread: Getting Organised - your suggestions appreciated

  1. #1
    Senior Member
    Join Date
    Nov 2007
    Location
    Melbourne, Australia
    Posts
    2,338
    Post Thanks / Like
    Images
    52

    Getting Organised - your suggestions appreciated

    I would appreciate some help from anyone regarding what I am calling my "getting organised" Project.

    I have hard disks all over the place dating back to early 2000's - when I got back into photography after a 20 year lay-off.

    As you can imagine technology has jumped leaps and bounds over this decade. Fortunately, I have all my files processed and unprocessed in various places - often copied for back-up..but NO SYSTEMATIC COLLECTION and true Multiple alternate bac-up strategy.

    I woudl like to implement the following:

    1. Organize ALL my files in one place - so that I can then back this one place up
    2. Systematically organise for back-up of all new work I do.
    3. Have a copy of my entire OS system and applications on a bootable / portable drive - so that I can make use of the three different macs I have as if they are the same mac - if I have to.

    I have DROBO with 4 terabytes of disk - this is plenty.
    I have looked into Carbon Copy Cloner and Super dooper- frankly they seem to do the same thing

    I have used Lightroom/Aperture but am no expert in cataloging systematically in either so I dont - execpt for historical libraries - now all over the place and therefore useless as they reference drives all over the place...I use Phocus/C1Pro/Sinar Exposure and CS4 with a bunch of plug ins regularly. I print with Epson and I use imageprint RIP do make things work nicely. This is the entirety of my Mac stuff.

    I have come to the point where updating software in one machine makes me scared and frightened - because of all teh possible negative repercussions I have in my disjointed system. I am now at the situtatioun where I am shooting hardly anythign because I am confused abotu hwere to put what and why! -

    So I would very much appreciate your thoughts on what you woudl do if you were me - or what you do because you are you!

    Thank you in desperate anticipation.

    This is a long weekend here and I would like to start with making a couple of bootable copies of my OS and Applications of my portable laptop and my Home Desktop. I don't even know how I can choose to start a machine from a bootable copy of my System and Applications if I have to.

    Supplementary issue is I run VMWare Fusion as a Wndows replicator for office programmes as well - is it possible to include the fusion/windows and Applications in the back-up procedure?

    Then I would like to start the process of copying every file I have from every hard disk I have accumulated over the years into one place ( DROBO ?) the DROBO is currently sitting in a cupboard gathering dust.

    Cheers and looking forward to your suggestions!

  2. #2
    Senior Member darr's Avatar
    Join Date
    Jan 2009
    Location
    North Florida
    Posts
    980
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    I use Lightroom for cataloging and some pp; my desktop PC (Win) has four hard drives and I backup to a (Raid) Netgear ReadyNas+ with four 1T hds. I do a further backup online with my sell-ables, projects, and client work through photoshelter. Other than filling up hard drives and moving data around, this is my system. I could not do it at all without organizing through Lightroom first. I can recommend a good book for you to help with organization and Digital Asset Management, it is what got me going: The DAM Book.
    "Creativity takes courage." ~ Henri Matisse
    Darlene Almeda, photoscapes.com

  3. #3
    Subscriber and Workshop Member
    Join Date
    Dec 2007
    Posts
    2,790
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    Peter,

    You luckily have not had major file or OS loss so this is a fairly easy implementation.

    Carbon copy cloner will make a great disk copy that is bootable of your main OS drive with all of the files on it. In system preferences you can pick a boot drive...should your main drive fail the system should find your backup clone and boot from it.

    Apple Time Machine is now fairly stable .... get a drive that is big enough to mirror your working system and use it with TM. You can step back in time as needed to recover should that be a necessity.

    Snow Leopard? Forget the whiners...I was one but it works and is rock stable. Enough time has passed so that the majority of issues have been worked out. Printing with IP 8.0.2 is perfect without the SL hell issues although the Print from Application is still a bit rough.

    Drobo running as main photo vault should be great but is no guarantee that you will not lose data without an offsite backup...get a blueray drive and invest in a decent coffee maker. Multiple copies of every file of value...yes cull the out of focus uninteresting and poor exposures that you will never bother to print or use. Set Time Machine privileges to ignore your photo files or it will never shut off and you will need two or three Drobos.

    The DAM book is great ... if you like inputting huge amounts of metadata for searches...if not change the structure to archival by date...with secondary reference points for subject.

    One great MAC program is ImageIngester....it will handle multiple camera and card inputs and writes extremely robust and fast duplicate copies to multiple drives...remembers camera and card as preferences you set so you can drop a card in and all else is done for you.

    DAM book makes a great case for having archival files and working files....

    This project will take you a long time...but once sorted will make it easy to find what you want when you want it. At least for your photo files.

    Best of luck.

    Bob

  4. #4
    Senior Member
    Join Date
    Nov 2007
    Location
    Melbourne, Australia
    Posts
    2,338
    Post Thanks / Like
    Images
    52

    Re: Getting Organised - your suggestions appreciated

    Thanks Guys - much appreciated! I will check out the DAM book as per recommendations.

  5. #5
    Senior Member
    Join Date
    May 2008
    Location
    California/Thailand
    Posts
    1,206
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    Hi Peter,

    I'm a big fan of SuperDuper!. CCC does essentially the same thing, but I feel SD! is a cleaner process (more polished), has a great method of providing updates (practically auto), has scheduling, incrementals, etc. (some of which the newest version of CCC may have - I haven't followed the recent versions). SD! is cheap and the developer is a nice guy. I use it on several Macs and backup to many volumes from each.

    Good luck with your process.

  6. #6
    Senior Member
    Join Date
    Nov 2007
    Location
    Melbourne, Australia
    Posts
    2,338
    Post Thanks / Like
    Images
    52

    Re: Getting Organised - your suggestions appreciated

    Thanks Dale - yes SD looks very interesting.

  7. #7
    Workshop Member glenerrolrd's Avatar
    Join Date
    Nov 2007
    Location
    Jupiter FL/Atlanta GA
    Posts
    2,279
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    Peter

    I have been working on this for a full year and have had great advice from Jack,Guy and number of others. You will get some advice about how each person organizes their work,favorite hardware /software etc but your requirements are YOUR REQUIREMENTS.
    Its most important to really understand them before you undertake any across the board reorganization. A few things I learned along the way.

    1. How frequently will you access you archive ? Stock photographers will tell you that you need to invest in extensive Metadata. This takes a lot of effort to do well . They need to randomly access hundreds of thousands of images and allow others to do the same. Event photographers can organize by client ,event etc. After a year do they ever go back and if they do how hard is it? This can be a pay me now or pay me later decision..requiring balance between up front effort and the responsive to access later.

    The result here should be your folder structure and the associated image numbering scheme.

    As previously recommended you can do worse than YEAR/SUBJECT/DATE. I looked at a years images ...25K and found that a folder structure of about 20 folders inside a year covered all my major subjects . e.g . 2010/Paris 2010/NYC 2010/Baseball 2010/surfing 2010/weddings 2010/ portraits . I maintain integrity back to the capture by having subfolders that go down to a specific card. I can an have can and have changed my folder structure .

  8. #8
    Sr. Administrator Jack's Avatar
    Join Date
    Oct 2007
    Location
    Los Altos, CA
    Posts
    10,486
    Post Thanks / Like
    Images
    1031

    Re: Getting Organised - your suggestions appreciated

    A little late here, but figure I'll throw my .02 in.

    1) I use, and have used since forever, Year/Subject/Date/Camera, as appropriate. For me it works like a charm. But I do not shoot stock so metadata subject tags in a library are not important to me. (PS: I *HATED* LR's library structure.)

    2) I use CCC for back-ups. Right now, it and Super Duper are essentially peas in a pod. I schedule regular back-ups per my storage schema.

    3) Main images are stored on a RAID-0 array onboard for fast access and writes. Back-ups are stored on an external drive, and scheduled via CCC for daily back-ups, though I can do a manual at any point in time by running it on CCC. (This external used to be a DROBO, but since it was just for back-up, I recently opted for a less expensive but faster JBOD/RAID box.) I then store a redundant third copy off-site. These drives are naked, and written to using a Voyager Q. I have CCC scheduled to write new files to the current offsite drive any time it is reconnected. I update these at minimum of once per month, and always after every large shoot.

    4) I use C1 as my main image editor now, and it has a built-in importer which allows for file renaming and subfolder creation on import, all of which fit perfectly with my current schema.
    Jack
    home: www.getdpi.com

    "Perfection is not attainable. But if we chase perfection, we can catch excellence."

  9. #9
    Workshop Member glenerrolrd's Avatar
    Join Date
    Nov 2007
    Location
    Jupiter FL/Atlanta GA
    Posts
    2,279
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    I meant to follow thru with points 2,3 but Starbucks WiFI cut me off....a frequent problem when I exceed my coffee limit.

    2. Repository hosting.....where do you want to keep your photography database and does it all need to be on line at once. The issue here is is your DB growing faster than the improvements in technology. Hard to believe but my image sizes are 4 times what they were with the M8 s ..so one copy of 25K images is a now a TB . The other issue is how fast does it need to be. If you are a CS power user its never fast enough but LR is pretty fast and I didn t have any trouble with C1. I follow Diglloyd advice and I have a Mercury Elite Pro with 4x2TB Hitachi 2gb drives set up in a RAID5 connected thru eSata . So my main repository is 6GB of usable storgage ...I partitioned 4gb for the photography DB.

    The issue here though is do you need it all on line at once. In LR you can maintain 1:1 previews so that you can search and find anything without the raws being on line. This is explained in Seth Resnick s book on LR. The raws can then be partitioned into drives by year . I use this approach when I travel .....I relocate my favorite folders to a portable drive and point LR to them . LR also lets you distribute your repository over any number of physical drives ..so you can daisy chain firewire drives as well.

    3. Backups ......with LR I don t really get the issue ? Import to the folder(in your main photography repository) and create a backup.(that is off line). Periodically create another backup that is stored offsite. Annually I archive the entire year and store offsite. The raws don t change . Now the catalog is a different issues..this is where all your process settings are stored . I back this up everytime I use it . In recovery you lose everything back to the last clean backup.

    The issue with backups is much more complex if you are creating processed copies of the raw file. Like a TIFF from a C1 conversion or a PSD from CS setup for printing. This is a game changer ....because now your raw repository is changing. So those folders I have on Wimbeldon in 2003 raw files can change. For me this requires some work ...but for now I am organized. I don t lose anything , I can find stuff and I am not blocked from changing my workflow .

    Those where the big 3 decisions ... I went with simple for the volume and difficult for the exceptions.

  10. #10
    Administrator, Instructor Guy Mancuso's Avatar
    Join Date
    Oct 2007
    Location
    Arizona
    Posts
    23,623
    Post Thanks / Like
    Images
    2555

    Re: Getting Organised - your suggestions appreciated

    I'm the weirdo on this but a firm believer in the raws and final tiffs shall never meet under the same drive. I use a Drobo and they are both on there but i do use separate drives to just back up all the raws and it sits outside my system in a secure place. I honestly hate these catalog programs like Aperture and LR . I use date and job name on every folder to find what I need. It really comes down to date from clients requests.

    Now my reason for this is very simple, you just heard the words very simple. I immediately rename the folder on importing of images to a Raw image drive to work from and also copy the raws right to the Drobo . Now before I start anything after this my files are on three different spots or drives . The are still on the card and stay there until I use the card again by than the files are processed and safe. They are also on the Drobo never to be touched again by any human unless there is a failure. I work from the Raw drive and after they have been sitting there a month or so i will delete it but not until it is in the extra drive that I store off site. Final Tiffs are on Drobo and also DVD made for client and also for me. Point being I have been doing this way for a long time , it's a religion. never change your religion on workflow. I don't have to deal with updates , software or anything that can change. All I hear is my library need to be updated when I installed version X . Thats BS and for me not safe.

    Obviously this is Guy's way and may not be someone else's . Key word SIMPLE, you have a hard time f______________ up when you keep it simple.
    Photography is all about experimentation and without it you will never learn art.

    www.guymancusophotography.com

  11. #11
    Workshop Member glenerrolrd's Avatar
    Join Date
    Nov 2007
    Location
    Jupiter FL/Atlanta GA
    Posts
    2,279
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    I don t know Aperture but with Lr the catalog can just mirror your folder structure . It doesn t affect in anyway how you manage your raws (if you reference without moving on import) . I can delete LR and never use it and it will not affect my folder structure.

    The issue is that all the processing including the raw conversion is maintained in the catalog so if you want to see a processed image ..you have to have the catalog and the raw image. In Jack s and Guy s workflow they both create a TIFF (I believe) after raw conversion...so you now have the "negative" as another copy.....and maybe another copy set up for printing. This eliminates the dependence on the software .

    Now I have seen workflows where you lose the linkage to the original capture but this isn t a function of LR .

    The complexity occurs when you want to use multiple raw developers, special print files, lots of plug ins etc. A typical CS workflow doesn t play well with LR.

    You can always set up folders for TIFFS , PRINT JOBS etc and include them where ever you want in your folder structure. But keeping track of back ups etc can be impacted .

    I would start with creating a decent repository with a folder structure to your liking and try to keep the workflow software away from your backups.

  12. #12
    Administrator, Instructor Guy Mancuso's Avatar
    Join Date
    Oct 2007
    Location
    Arizona
    Posts
    23,623
    Post Thanks / Like
    Images
    2555

    Re: Getting Organised - your suggestions appreciated

    Roger I think more my point is just more complexity that is added sometimes can be a time waster but it also depends on what you are doing and how you are using your files. i output 16 bit Tifs on everything and store them. My theme is keep it simple and I never have to second guess myself . i do the same thing every time without fail. It's just built in the brain.
    Photography is all about experimentation and without it you will never learn art.

    www.guymancusophotography.com

  13. #13
    Workshop Member glenerrolrd's Avatar
    Join Date
    Nov 2007
    Location
    Jupiter FL/Atlanta GA
    Posts
    2,279
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    Quote Originally Posted by Guy Mancuso View Post
    Roger I think more my point is just more complexity that is added sometimes can be a time waster but it also depends on what you are doing and how you are using your files. i output 16 bit Tifs on everything and store them. My theme is keep it simple and I never have to second guess myself . i do the same thing every time without fail. It's just built in the brain.
    Guy I agree 100% with building a simple process and then ''its the discipline" that makes it work. Like starting with fresh batteries and cards on ever shoot..no exceptions.

    However the point on LR is that it can easily mirror your folder structure..whatever you use . LR isn t making this part complex its how we are using it .

    The other issue, beyond your requirements, is your familiarity and skills developed over years in working with CS and then C1. This makes LR less attractive..who wants to spend time learning new software to solve problems you don t have? But likewise....I am sure we could get a few pages on whats wrong with CS or C1
    from the perspective of a skilled LR user.

    So again I agree with simple ,discipline etc but its important to understand the requirements and existing skills of the individual.

    Peter s situation looks quite complex because we know he has quite a collection of gear and will benefit from using several raw developers . This in itself makes LR more complex because you need to create a "developed" TIFF .

    Starbuck s is OPEN!

  14. #14
    Senior Member
    Join Date
    Nov 2007
    Location
    Melbourne, Australia
    Posts
    2,338
    Post Thanks / Like
    Images
    52

    Re: Getting Organised - your suggestions appreciated

    First up thanks Roger, Jack and Guy for your inputs.

    Regarding hardware storage strategies - Jack - honestly I have no idea what half the acronyms you used mean ( you may guess from this that I am no computer expert) truth is I have no idea how to set up a raid whatsit - even though I own and run a business with networked servers connecting Melbourne Sydney and now Hong Kong - i have no idea about the hardware side LOL thats what the IT guys are for. I gave up programming in APL when I coudl afford to hire my own programmers to cut the latest binomial option valuation systems and our own in house monte carlo simulations etc etc

    Of course work computers are PC and some prety heavy duty servers - not a mc in sight and funny enough most of my geek It people haev a hreart attack when I mention I liek macs.

    so a good source to read up on raid for dummies would be welcome!

    Regarding storage strategies I like the idea of keeping it simple very much Guy and (historically) have used a raw then copy to photoshop then save to Print file or Jpeg for web display strategy.

    here is my problem I never bothered to datestamp my raw files and I used subject or date stamps for photoshop files and subject or date stamps for my Jpegs! LOL

    Your solution(s) Jack and Guy are simple and clever. I wish I was doing this 10 years ago....

    Roger - I have a problem with Lightroom - I used version One for a long time and catalogued a great deal of stuff and then when version two came out it coincided with my trying Aperture and also with getting into C1/Phocus/Flexcolour and eXposure - not to mention little sidelines in Raw Developer ( before Adobe)

    You are right the number of different MF systems and 35mm systems I was and am playing with exaggerate the complications. So no consistent storage path and NO I dont like Adobe Bridge at all at all.. and now version three of Aperture is out and I still havent learned how to use the storage capabilities of version 1/2 yet..LOL and I am only 1/3rd the way through a lightroom videop tutorial i bought from LL and already version 3 beta is out....hmmm

    teh main issue was when I left Canon as my main weapon of choice I had to start using all these proprietary raw developers ..and it tok too much extra steps to consistently go from prop developer to tiff to photoshop to Jpeg and catalogue and use teh evr evolving ever improving metadata as well..

    You guys have helped a lot - by confirming a couple of things

    1. Use date/camera/subject for raw file saved on to a storage system
    2. Use a copy of raw file to do processed file ( tiff or PSD ) using date/camera/subject saved onto a storage system
    3. Use a copy of Jpeg saved by date/camera/subject saved on a storage system

    Then a have a copy of the whole storage system in a seperate storage system regularly backed up.

    the good news?

    I only have about 6000 images all up over 10 years - because I ditch at the start of the process and am quite ruthless. Of these 6000 roughly half are processed files and or Jpgs and more than 80% are family asociated shots ..

    havent done a commercial gig for a few years now - and when I do it is editorial style fashion not enough for me to have to concern myself with a lot of the power of LR in terms of metadata etc.

    So I am happy with a date / camera/ seperate raw and developed file storage system - I just need to firstly get on top of hardware side and then choose smartst software for cataloging - if required.

    I do loiek LR - and wish it did a better job with raw from Leica and teh MF cameras I use. I am a veru anal processor and for every shot I make I use native manufacturers developer ( every time) and then export to photoshop and do a bunch of stuff on that.

    Anyway I probably have typed too much - but much appreciated.

    I need to learn about raid internal and external now.

    Cheers.

  15. #15
    Workshop Member glenerrolrd's Avatar
    Join Date
    Nov 2007
    Location
    Jupiter FL/Atlanta GA
    Posts
    2,279
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    Peter

    Lr fits my needs but not yours . I shoot approximately 25K images a year split between my M9 s and my D3x s . At current file sizes I blow thru 1TB per copy per year. Lr provides the framework to process this volume and extensive tools for ranking and editing.

    You situation is actually pretty easy. With 6000 images you can set up a folder called LEGACY . If you have time you can establish maybe 10 subfolders based on subject ..which can be anything you want. You can fit 6000 images on a 320GB drive.

    I think my points are valid in terms of requirements first:

    1. Folder Structure/Image Labeling
    2. Data Repository
    3. BackUp

    Diglloyd has a great website ...he is on the margin better than trying to figure it out for yourself.

  16. #16
    Sr. Administrator Jack's Avatar
    Join Date
    Oct 2007
    Location
    Los Altos, CA
    Posts
    10,486
    Post Thanks / Like
    Images
    1031

    Re: Getting Organised - your suggestions appreciated

    Quote Originally Posted by PeterA View Post
    Jack - honestly I have no idea what half the acronyms you used mean ( you may guess from this that I am no computer expert) truth is I have no idea how to set up a raid whatsit - even though I own and run a business with networked servers connecting Melbourne Sydney and now Hong Kong - i have no idea about the hardware side LOL thats what the IT guys are for.
    Hi Pete, sorry...

    RAID-0 is for fast reads and writes of large files to spinning disks which are inherently slow, and much slower than the I/O (Input/Output) on most computers, but has no safe redundancy; if one drive fails the entire array is kaput and all data is lost.

    RAID-1 is drive mirroring, so one drive is an exact copy of the other for full redundancy, but no increase in speed over a single drive.

    RAID-5 is a performance plus redundancy strategy requiring at least 3 drives. If one goes down, the remaining are structured to be able to rebuild the lost data on the failed drive. While good, still not good enough for a stand-alone fool-proof back-up.

    The ONLY fool proof back-up involves an onsite, fully redundant copy of some sort, and an offsite fully redundant copy of some sort. In this way you have everything covered except a nuclear blast or the astronomical odds that your onsite and offsite storage locations are compromised at the same time.

    CCC = Carbon Copy Cloner, a software like Super Duper that does scheduled copies of files, folders and/or volumes automatically so you don't have to run them manually (but you can as needed).

    Some links to RAID and RAID strategies...

    Simple: http://www.integratedsolutions.org/raid_ov.htm

    More detail: http://en.wikipedia.org/wiki/RAID

    How RAID-5 "parity" works: http://www.scottklarr.com/topic/23/h...-really-works/
    ~~~

    So to clarify, I use a 4-drive Raid-0 on my main editing machine (Mac Pro) as primary image storage for quick reads and saves of large files. This gets backed up daily using CCC to an onsite, external drive that is just single drives in a box, called JBOD (Just Bunch Of Drives) -- this is a sort-of RAID-01 concept. Alternatively, one can back up to a DROBO here which is an automated RAID-5 device with redundancy, so a RAID-05. The only downside to DROBO is it's reads and writes are somewhat slower; the upside is it is (much) easier to maintain than typical RAID-5 devices and fully redundant in case a single drive fails. Then the entire database gets additionally backed up to an offsite bare drives using a drive docking device called a Voyager Q (cheap and convenient to transport bare drives in boxes). See: http://eshop.macsales.com/item/Newer...gy/FWU2ES2HDK/
    ~~~

    PS #1: Your files are already date-stamped in the exif based on the time/date as set in your digital camera when you took them. Scans should have a file creation date as of the date they were scanned. Any good image browser should be able to sort by that exif parameter.

    PS #2: I would consider using Year/Subject/Camera/Date instead of Date/Camera/Subject only for simplicity at the times when a given shoot covers more than one day (or even crosses over a set of months as in a multi-week vacation) and/or was made with more than one camera -- easier to lump all of the same subject matter into appropriate sub-groups a year from now...

    Cheers,
    Jack
    home: www.getdpi.com

    "Perfection is not attainable. But if we chase perfection, we can catch excellence."

  17. #17
    Senior Member
    Join Date
    May 2008
    Location
    California/Thailand
    Posts
    1,206
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    Hi Peter,

    Lots of good info here for different approaches. Your storage needs seem modest – probably similar to mine, though I have more image files – I don't shoot numerous jobs like some of the folks here do. So I'll mention what I do that works well on my Macs, in case it might appeal to you.

    I have a drive called "Main_Data" for image files. (Actually I have several, but one will hold thousands of files and is good for our discussion here.) This is separate from my Operating System and Applications drive (aka Macintosh HD or in my case "Main_HD"). I have a folder on the "Main_Data" drive called "Pictures" that holds all images files for that big chunk of files. The folder (called "Pictures") is also aliased in my user account so that when you see a Finder window there is that "Pictures" folder on the left side under User Name, Applications, etc. I also have a folder called "Documents" and a few others which are pointed at the "Main_Data" drive for storage of such data. This allows for easy backups of just data so that I can make duplicates to store elsewhere, etc.

    In the "Pictures" folder are folders labeled as others have mentioned above. In my case I use YYYYMMDD_jobname_camer-or-other-detail, so that 20100315_Yosemite_P25+ contains images from March 15, 2010, etc. This method allows all the files to list in chronological order and is very fast to scan. Sub-folders are used for further organization. Others have mentioned slightly different date/description configurations and I'm not sure of advantages to each, but describe mine with its affects so that you have an example.

    I also use some RAID solutions, and I appreciate how it can get confusing. Jack states that he has moved to "JBOD" which stands for "Just a Bunch Of Drives" and this is mostly what I use. This means that one has an enclosure that holds 2 or more hard drives and each is simply an independent drive, e.g. "Main_Data_1", "Main_Data_2", ect. Other RAID options allow for striping data onto multiple drives which can boost speed (write times), but also has it's risks, in that data can be lost if one of the drives fails. I played with this, but no longer mess with it because I have more "peace" not having to be quite so careful with my backup strategy. So the only striped RAID I use is for Photoshop scratch disk. And even that is not necessary for those who aren't obsessed with saving seconds while working on image files. Much of my processing is done while sipping wine and not worrying about production time, etc.

    So to clarify:

    1. I copy my CF cards to my Mac, typically to my desktop. I name the folder in my convention as above. I drag it to my "Pictures" folder alias (it's in my Finder windows, but also on my Dock) which creates a duplicate because the "Pictures" folder is on the "Main_Data" drive, not the drive containing my Desktop. That means I have two copies now.

    2. I process the images, browse them, criticize myself for screwing up a shot I should have nailed, etc. This is done on the files that are in the "Pictures" folder on the "Main_Data" drive.

    2a. I create an Output folder inside of each "job" folder to keep output stuff separate from RAWs. So all of the Capture One processed files go into an Output folder inside each dated folder.

    3. I run Super Duper! to copy my additions on "Main_Data" to a drive called "Main_Data_B.U." This means that I have three copies of my RAWs, because I still have the one on the Desktop that was untouched, other than to copy it to the "Pictures" folder. It sits there for a while, and when I feel like all is ducky with the other copies I delete the one from my desktop. (Sometimes I don't copy to the Desktop at first, but go strait to the "Pictures" folder, but I kind of like that step of a copy on the Desktop because it's an extra copy in case I'm multi-tasking badly. )

    4. I format the CF card after files are all confirmed valid.

    5. Periodically, I copy the "Main_Data" drive(s) (Main_Data_1, Main_Data_2, etc.) to other externals, as Jack describes, and keep them elsewhere.

    Others would sooner drink motor oil laced with broken glass than use this approach, but I like it. I'm not in a hurry, this is safe and redundant, easy to find files, and allows access by any application. It's actually very fast, but my writing here is verbose. I like that I can look in the "Pictures" folder and scan a huge amount of files in an instant. And backups are a snap.

    My apps of choice are Capture One and Photoshop (w/Bridge for some items), but I also drag stuff in to Preview for a quick glance also. This storage system allows for one to access from any app., but so do other options.


    Dale

    Edit: haha, I was typing so slowly (and doing other stuff) that Roger and Jack added replies above, covering some of the same stuff.

  18. #18
    Senior Member
    Join Date
    May 2008
    Location
    California/Thailand
    Posts
    1,206
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    P.S. Glad that Jack included the descriptions of the various RAID options, as I had neglected to add a comment for Raid-1. I do like Raid-1 in some cases, but don't use it for images. I use it for (non-photo) business needs.

  19. #19
    Workshop Member glenerrolrd's Avatar
    Join Date
    Nov 2007
    Location
    Jupiter FL/Atlanta GA
    Posts
    2,279
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    Jack

    I would differentiate between reliability and secure backup. Unless you post to two separate physical devices .. your recovery is to redo the work since your last backup.

    The key factors in the backup "equation" would be the reliability of the primary repository and the value you place on saving the work performed.

    RAID 5 improves reliability by offering a fairly painless recovery and minimal rework. Frequent Backups reduce the pain of rework by limiting it. (This of course assumes that you immediately create the backups of raws during import. ) The trade off is the amount of time and effort you expend backing up files.

    I am coming around to your solution because I have found the drives to be ridiculously reliable. The only failure I have had in 5 years and almost 100K images was the new drive I just purchased and it failed at start up. So reliability may not justify RAID 5.



    Quote Originally Posted by Jack Flesher View Post
    Hi Pete, sorry...

    RAID-0 is for fast reads and writes of large files to spinning disks which are inherently slow, and much slower than the I/O (Input/Output) on most computers, but has no safe redundancy; if one drive fails the entire array is kaput and all data is lost.

    RAID-1 is drive mirroring, so one drive is an exact copy of the other for full redundancy, but no increase in speed over a single drive.

    RAID-5 is a performance plus redundancy strategy requiring at least 3 drives. If one goes down, the remaining are structured to be able to rebuild the lost data on the failed drive. While good, still not good enough for a stand-alone fool-proof back-up.

    The ONLY fool proof back-up involves an onsite, fully redundant copy of some sort, and an offsite fully redundant copy of some sort. In this way you have everything covered except a nuclear blast or the astronomical odds that your onsite and offsite storage locations are compromised at the same time.

    CCC = Carbon Copy Cloner, a software like Super Duper that does scheduled copies of files, folders and/or volumes automatically so you don't have to run them manually (but you can as needed).

    Some links to RAID and RAID strategies...

    Simple: http://www.integratedsolutions.org/raid_ov.htm

    More detail: http://en.wikipedia.org/wiki/RAID

    How RAID-5 "parity" works: http://www.scottklarr.com/topic/23/h...-really-works/
    ~~~

    So to clarify, I use a 4-drive Raid-0 on my main editing machine (Mac Pro) as primary image storage for quick reads and saves of large files. This gets backed up daily using CCC to an onsite, external drive that is just single drives in a box, called JBOD (Just Bunch Of Drives) -- this is a sort-of RAID-01 concept. Alternatively, one can back up to a DROBO here which is an automated RAID-5 device with redundancy, so a RAID-05. The only downside to DROBO is it's reads and writes are somewhat slower; the upside is it is (much) easier to maintain than typical RAID-5 devices and fully redundant in case a single drive fails. Then the entire database gets additionally backed up to an offsite bare drives using a drive docking device called a Voyager Q (cheap and convenient to transport bare drives in boxes). See: http://eshop.macsales.com/item/Newer...gy/FWU2ES2HDK/
    ~~~

    PS #1: Your files are already date-stamped in the exif based on the time/date as set in your digital camera when you took them. Scans should have a file creation date as of the date they were scanned. Any good image browser should be able to sort by that exif parameter.

    PS #2: I would consider using Year/Subject/Camera/Date instead of Date/Camera/Subject only for simplicity at the times when a given shoot covers more than one day (or even crosses over a set of months as in a multi-week vacation) and/or was made with more than one camera -- easier to lump all of the same subject matter into appropriate sub-groups a year from now...

    Cheers,

  20. #20
    Super Duper
    Senior Member
    Godfrey's Avatar
    Join Date
    Jan 2009
    Location
    Near San Jose, California
    Posts
    7,927
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    I wrote a couple of articles on the topics of image file organization and backup just recently. Rather than paste them in here (as they are a couple of pages long each), I made them into PDF files and posted them to my website for download .

    Go to http://www.gdgphoto.com/articles/

    and click on articles:

    06 - Backup System Notes
    07 - Image File Organization and Naming Notes

    Comments and feedback is always appreciated. :-)

  21. #21
    Senior Member
    Join Date
    Nov 2007
    Posts
    1,513
    Post Thanks / Like

    Re: Getting Organised - your suggestions appreciated

    Rather than get deep into a possible fray, and not to push one way over another, I ran across an interesting resource that some/many folks may find it worth the time to explore. It is a new organization dedicated to digital photography best practices and workflow ( http://www.dpbestflow.org/ ). The premise being that there are some best practices that have emerged over time, as well as different workflows, and folks should construct systems and organizations that fit their use and needs, rather than trying to force a way around how they work. It is pretty much centered around RAW capture and processing, though not pushing one app or tool. I found it rather refreshing in some respects. It has a lot of information on lots of topics that get bantered about here over and over, so some of it will seem familiar, some new. It will take time to go through a lot of it, but it does appear to be a good resource worth bookmarking.

    LJ

  22. #22
    Senior Member
    Join Date
    Nov 2007
    Location
    Melbourne, Australia
    Posts
    2,338
    Post Thanks / Like
    Images
    52

    Re: Getting Organised - your suggestions appreciated

    Jack -- no need for any apology it is my ignorance and thanks again, Guy, Roger Dale and Godfrey thanks. LGlL- I will certainly check that site out!

    A lot of thinking up front Ifigure in order to make the best decisions!

    Just sorting out an issue with my Sinar back ( ughhh ) all this talk of backing up made me run to a camera and make some shots over the last few days..now I haev an inexplicable cant download error from back to Expose - that even has my guru dealer flummoxed!

    sheesh the Hy6 is lovely to use if you like WLF!

    hahahha - life huh?

    Thanks Again everyone.
    Pete

  23. #23
    Senior Member
    Join Date
    Nov 2007
    Location
    Melbourne, Australia
    Posts
    2,338
    Post Thanks / Like
    Images
    52

    Re: Getting Organised - your suggestions appreciated

    *Aside - I have fried one computers ram a few months ago - when we had a severe electrical storm..now checking my other puter I have 7 gigs of ram....

    I am going to make one computer a test bed for stuff and the other puter a working puter..who are your favourite suppliers for ram and storage over there in the US?

    Pete

  24. #24
    Senior Member
    Join Date
    Jun 2009
    Posts
    1,347
    Post Thanks / Like
    Images
    12

    Re: Getting Organised - your suggestions appreciated

    FWIW...:

    I don't use a cataloguing software. What if the software is discontiued one day??

    1.) For the RAW files I simply store folders ("year|month|day|something"). They are stored within my Capture One sessions containg all the settings etc.
    The processed 16bit TIFs (but without further editing) are stored in subfolders in the same directory/session.
    2.) In addition I store a folder with the 16bit TIFs edited (on layers).
    3.) In addition a folder with the 16bit TIFs flattened on one layer - this folder contains subfolders sorted by subject/themes.
    All these folders are stored on an internal RAID (mirrored).
    4.) I back up the entire RAID on 2 external (single) drives (using Super Duper, respectively "Data Backup" now as it is easier to exclude the cache files from the back up).
    5.) In addition I back up the RAID on a third drive that is located elsewhere (I update this drive every 3-6 months depending on the amount of images I have to back up).
    6.) I addition I back up the most important images (the actual RAW file without settings and a 16bit PDF of the finished file) in an online database.

    As to 4.) and 5.) I prefer to use "small" single drives (1TB) for external back up as I replace these drives every 2-3 years (man, those drives cost nearly nothing if you consider the "value" they are storing).

    Obviously I don't shoot high volumes, quite the opposite...

  25. #25
    Subscriber & Workshop Member
    Join Date
    Nov 2007
    Location
    New York, NY
    Posts
    1,178
    Post Thanks / Like
    Images
    414

    Re: Getting Organised - your suggestions appreciated

    Quote Originally Posted by PeterA View Post
    who are your favourite suppliers for ram and storage over there in the US?
    For Mac RAM and some storage solutions - Other World Computing: http://www.macsales.com/

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •