processing large files 80 meg +, need a bit of understanding

[msadat]

Looking at the current trends, it seems like all new cameras are heading up there in terms of files sizes. I know there are people here that already are familiar with processing these type of files.

So how many images do you typically process each time?
If you are using Lightroom, what is the import times?
How is the editing process within Lightroom?
people who have moved to c1 or use both is c1 any faster
what kind of computers (CPU/ram) do you currently use
what about storage and archiving, in-house or cloud (if cloud, have ever tried to restore/retrieve a lot of files with success)

it might be time for me to upgrade my computer or my expectation!

 

[baudolino]

I only get at the file size you mention when using my IQ280. Here I use C1 v12 for as much of the processing as I can, including layers, masks, etc. before exporting a 16 bit TIFF to Photoshop CC 2019. My computer is an iMac Pro with 32GB RAM (5K monitor) with a secondary monitor being the EIZO 4K (can't remember the model). For storage, I use the G-Technology Shuttle SSD 8TB Thunderbolt 3 as my primary drive (read/write speeds about 1600MB/s in RAID 5), with back-ups going to the Lacie 5big Thunderbolt 2 8TB drive (read/write speeds about 1000MB/s in RAID 5) and a Synology NAS 24TB drive over the wifi. In addition, to download and process the most recent shoots, I use the Lacie Bolt SSD drive (2TB, RAID 0) and an Akitio drive with a single 2TB Samsung PRO 860 SSD disc installed (the read/write speeds of these drives exceed 2500MB/s). The setup is very fast and responsive even when I work with multiple layers and 16 bit TIFF files at full resolution.

 

[Paul2660]

With 100MP P1 files the issue quickly becomes not one of processing power, but storage, as a 100MP 16 bit image from C1 with 2 layers and masks will easily be over the 4GB limit of a current tiff. Which means PSB, which I do not like to use. Slower to save on my PC and LR can't see it.

A older 3.8Gz 8 core i7 with 64 GB of ram, and a 4 GB AMD 580 Video card can process out the images all day long and in CC you can work on them to your hearts content, it's just when you attempt to save the file where the issues start at least for me.

LR has a terrible IQ3100 conversion, Adobe only made one attempt and any file that is slightly underexposed will show up with a lot of red noise in the shadows, unlike the C1 conversion (this is on a 16 image not the 14 bit files). So you are pretty limited to C1 for the raw conversion, from there I go to both CC and LR and do further work, mainly CC since it will read PSB files.

The 150 back still has same issue even more now, as Topaz refuses to work on multiple stitched 150MP files, (due to a total pixel size limit). And at times I have very slow brush behavior in CC with the 150MP files, again I feel it's nothing to do with my PC or Mac, it's the overall pixel dimensions of the image and brush just gets bogged down as you are close to maxing out the brush size to do much work on a 150MP image.

I can see the Fuji 100MP files have similar issues also.

Paul C

 

[Satrycon]

pixel dimension limits of the GPU could be a big factor as well as GPU memory

With 100MP P1 files the issue quickly becomes not one of processing power, but storage, as a 100MP 16 bit image from C1 with 2 layers and masks will easily be over the 4GB limit of a current tiff. Which means PSB, which I do not like to use. Slower to save on my PC and LR can't see it.

A older 3.8Gz 8 core i7 with 64 GB of ram, and a 4 GB AMD 580 Video card can process out the images all day long and in CC you can work on them to your hearts content, it's just when you attempt to save the file where the issues start at least for me.

LR has a terrible IQ3100 conversion, Adobe only made one attempt and any file that is slightly underexposed will show up with a lot of red noise in the shadows, unlike the C1 conversion (this is on a 16 image not the 14 bit files). So you are pretty limited to C1 for the raw conversion, from there I go to both CC and LR and do further work, mainly CC since it will read PSB files.

The 150 back still has same issue even more now, as Topaz refuses to work on multiple stitched 150MP files, (due to a total pixel size limit). And at times I have very slow brush behavior in CC with the 150MP files, again I feel it's nothing to do with my PC or Mac, it's the overall pixel dimensions of the image and brush just gets bogged down as you are close to maxing out the brush size to do much work on a 150MP image.

I can see the Fuji 100MP files have similar issues also.

Paul C

 

[msadat]

so how many files do you process at a time? and what sort of time frmaes you get out of this setup?

I only get at the file size you mention when using my IQ280. Here I use C1 v12 for as much of the processing as I can, including layers, masks, etc. before exporting a 16 bit TIFF to Photoshop CC 2019. My computer is an iMac Pro with 32GB RAM (5K monitor) with a secondary monitor being the EIZO 4K (can't remember the model). For storage, I use the G-Technology Shuttle SSD 8TB Thunderbolt 3 as my primary drive (read/write speeds about 1600MB/s in RAID 5), with back-ups going to the Lacie 5big Thunderbolt 2 8TB drive (read/write speeds about 1000MB/s in RAID 5) and a Synology NAS 24TB drive over the wifi. In addition, to download and process the most recent shoots, I use the Lacie Bolt SSD drive (2TB, RAID 0) and an Akitio drive with a single 2TB Samsung PRO 860 SSD disc installed (the read/write speeds of these drives exceed 2500MB/s). The setup is very fast and responsive even when I work with multiple layers and 16 bit TIFF files at full resolution.

 

[baudolino]

so how many files do you process at a time? and what sort of time frmaes you get out of this setup?

I usually work on images one at a time (i.e. export them individually, not in batches - or only in small batches where I may need say 15-20 jpegs at a time).
As for timeframes, copying 40GB of data from my tether notebook (MacBook Pro Retina 2015) takes about 5 minutes. With USB3 this would take about 25 minutes.
(These are just my recollections, don't hold me accountable).
Previews get generated nearly instantly in C1.
I can work on images from any of the drives directly, without any speed penalty.
What other specific time frames interest you?

 

[vjbelle]

The 150 back still has same issue even more now, as Topaz refuses to work on multiple stitched 150MP files, (due to a total pixel size limit). And at times I have very slow brush behavior in CC with the 150MP files, again I feel it's nothing to do with my PC or Mac, it's the overall pixel dimensions of the image and brush just gets bogged down as you are close to maxing out the brush size to do much work on a 150MP image.

I can see the Fuji 100MP files have similar issues also.

Paul C

Paul..... if you are referring to the 22,000 limitation in Giga AI there is light at the end of the tunnel..... I email Topaz asking them to get rid of that limitation and they emailed me back saying that it's in the works, so maybe the next iteration will have that limitation removed.

Victor

 

[Paul2660]

Paul..... if you are referring to the 22,000 limitation in Giga AI there is light at the end of the tunnel..... I email Topaz asking them to get rid of that limitation and they emailed me back saying that it's in the works, so maybe the next iteration will have that limitation removed.

Victor

It's also an issue with Topaz Clarity, which I use a lot. Same limitation on size. You can use Clarity on a single IQ4 image, but if you stitch even 2 files you will get beyond the pixel limitation, and thus you have to down sample to use the tool. I have mentioned this several time to their team, but so far, no response.

Paul C

 

[MrSmith]

With 100MP P1 files the issue quickly becomes not one of processing power, but storage, as a 100MP 16 bit image from C1 with 2 layers and masks will easily be over the 4GB limit of a current tiff. Which means PSB, which I do not like to use. Slower to save on my PC and LR can't see it.

A older 3.8Gz 8 core i7 with 64 GB of ram, and a 4 GB AMD 580 Video card can process out the images all day long and in CC you can work on them to your hearts content, it's just when you attempt to save the file where the issues start at least for me.

You have disabled compression in the preferences? That will make a massive difference to save times.

My retouch/comping PSD’s are between 5and 20gb as I usually work in 16bit so saves while taking a while are not to bad on fast internal ssd’s (mac)

I don’t run a catalogue but just keep current jobs on my Mac and then delete them when finished with (obviously I have multiple back-ups)

 

[vjbelle]

It's also an issue with Topaz Clarity, which I use a lot. Same limitation on size. You can use Clarity on a single IQ4 image, but if you stitch even 2 files you will get beyond the pixel limitation, and thus you have to down sample to use the tool. I have mentioned this several time to their team, but so far, no response.

Paul C

It really is kind of ridiculous..... When I emailed them I gave specific examples where I would never exceed the TIF size limitations but would be beyond the 22,000 pixel limitation they had set. They agreed and said that even though it would not be in the next update that it was in the works. That was one update ago. I suggested that they take a cue from Adobe and try to process all files and if the TIF size limitation is exceeded simply stop and inform the user.

I'm amazed at all the issues you are having with the 4150.... Back/file size issues. I'm staying away for now and am looking forward to the GFX 100 and the 'Rody 90 aperture only'. That lens will be great on both of my platforms.

Victor