RAW Truth

[Terry]

Jono - It has expanded beyond m4/3 to compacts because I've heard the same complaints about the LX3/d-lux4.

 

[jonoslack]

I look at the situation 2 ways. I'm pretty convinced the reason that we see the soft corner problems on the legacy wide angle lenses is related to the complications that the M8/M9 with the short register distance and the need for micro lenses. If the only way to go wide in this format is to make lenses with corrections then I'm OK (the tradeoff being avoiding a hugely expensive sensor). Technically, I could be way off base and I'm not a physicist but I do think physics is involved. I'm specifically talking about the 7-14 here. I'm not sure where the crossover point (what focal length) lens design gets easier.

Hi Terry
I'm sure you're right here - I think it's the RIGHT way to design lenses. The mistake is not to process the corrections into the RAW file BEFORE sending it to the conversion software (as leica do). What would be even better would be if they ditched these proprietary raw files altogether and used DNG (which will support this sort of thing)

 

[jonoslack]

Jono - It has expanded beyond m4/3 to compacts because I've heard the same complaints about the LX3/d-lux4.

Exactly
Which is why I'm no longer using my othewise lovely d-lux4

 

[Guy Mancuso]

The thing many people fail to catch with many of the P&S and even some high priced cameras like an M8 or M9 and probably the X1 is that some vendors process lens corrections in their processing even for RAW files while other vendors provide information in the RAW files that can be used in post processing to make the corrections but if not used you see the lens' flaws (distortion or CA). In the M8 and M9 coded lenses give you better results because Leica can process in corrections based on the known characteristics of the lens. No one will ever know how really good or bad the X1 lens is since you can't remove it from the processed image. If Oly or Panny had processed in the corrections into the RAW file you would have no idea how good or bad the distortion or CA was unless you tested the lens by itself. I think that with known lenses they should process in the correction so that no matter what post processing software you use the images will appear to be optically the same.

To expand on this a little . It is VERY difficult and expensive to build any wide angle without barrel distortion . I would most likely say you can't and it does not matter who makes the lens. There will always be some barrel. In this 4/3rds camera we can't expect near perfect lenses for this so as John said many of these corrections are wrapped up in the DNG or Raw extension they use and only raw programs that can see those corrections take advantage of them. Let's face it one reason they give you there raw processing programs for free or small cost. When you jump to a 3rd party you are on your own but obviously some programs will support those corrections.
Now just a case in point the Hassy 28 and Mamiya 28mm we have the same issues with very expensive lenses but we have dedicated software that does correct for barrel distortion and other lens aberrations. Now case in point Leica is claiming it does not need these in there S2 well I'm the first one to throw up a red flag and say sure it don't need it in a dedicated software package but it is wrapped up in the DNG so other programs can see it just like the 4/3rds lenses obviously the corrections will be far less but they are still there ( I am still waiting for a REAL answer here because I am not buying the claim). All my life or career i have never seen a wide angle lens have no barrel distortion ever. Hassy and Mamiya admit it and have there own software to correct it. So it depends on how some OEM's work which is fine but the bottom line is there is some correction for it either in the raw file extension or in a software package. In this case Aperture is not recognizing those corrections than that is a Apple downfall for not supporting these certain cameras.

My morning mantra. So where the hell is my GF-1 already. LOL

 

[jonoslack]

To expand on this a little . It is VERY difficult and expensive to build any wide angle without barrel distortion . I would most likely say you can't and it does not matter who makes the lens. There will always be some barrel. In this 4/3rds camera we can't expect near perfect lenses for this so as John said many of these corrections are wrapped up in the DNG or Raw extension they use and only raw programs that can see those corrections take advantage of them. Let's face it one reason they give you there raw processing programs for free or small cost. When you jump to a 3rd party you are on your own but obviously some programs will support those corrections.
Now just a case in point the Hassy 28 and Mamiya 28mm we have the same issues with very expensive lenses but we have dedicated software that does correct for barrel distortion and other lens aberrations. Now case in point Leica is claiming it does not need these in there S2 well I'm the first one to throw up a red flag and say sure it don't need it in a dedicated software package but it is wrapped up in the DNG so other programs can see it just like the 4/3rds lenses obviously the corrections will be far less but they are still there ( I am still waiting for a REAL answer here because I am not buying the claim). All my life or career i have never seen a wide angle lens have no barrel distortion ever. Hassy and Mamiya admit it and have there own software to correct it. So it depends on how some OEM's work which is fine but the bottom line is there is some correction for it either in the raw file extension or in a software package. In this case Aperture is not recognizing those corrections than that is a Apple downfall for not supporting these certain cameras.

My morning mantra. So where the hell is my GF-1 already. LOL

Well, I quite agree with you about the facts of the case, but NOT the interpretation.

If the lens correction is in a PROPRIETARY raw file, then, if you want anyone to be able to see your pictures in the future, you are forced to archive material as a .TIFF or .jpg file . . . which is fine if you want to use C1 or another 'converter' and process your files (as long as they support the lens correction information . . do they? I think not at present)

If you want to use a DAM like Lightroom or Aperture and only output files as and when you need them for specific purposes, then it really isn't safe to use these crazy proprietary files. It's a little different with Hassleblad and Mamiya; being professional it's likely that support will always be available (although it might be expensive if either of them go tits up!)

If Olympus and Panasonic were to put the lens corrections into DNG files within the Adobe specification (whatever the current version is). THEN it's okay to blame Apple for not supporting them. It would also be okay (by me at least) if they did the corrections before writing the DNG files (lets face it, who doesn't want barrel distortion corrected:wtf - one could then convert them to useable dng files for archiving.

I think we live in a crazy world where there is a new file specification for every camera which comes out - it's getting more and more unmanageable, and it's only a case of laziness and greed on the part of the manufacturers. (Olympus Studio is by no means cheap). If you think I'm paranoid try finding something that'll read some of the older computer file formats which were used by thousands and thousands of people.

It's ironic that Adobe is, in a way, supporting this madness by being so quick to support the odd file formats.

 

[Guy Mancuso]

If the lens correction is in a PROPRIETARY raw file, then, if you want anyone to be able to see your pictures in the future, you are forced to archive material as a .TIFF or .jpg file . . . which is fine if you want to use C1 or another 'converter' and process your files (as long as they support the lens correction information . . do they? I think not at present)

No not really I can certainly take my Phase files and work with them in ACR or LR . I just don't have a way to correct for barrel or lens corrections.

Not sure I read you correctly though , still half asleep.

The real issue is programs like Aperture just don't update often enough to keep supporting the onslaught of new camera's. Otherwise it's a great program

The digital world has changed but I do agree in principle as to what Adobe tried to accomplish was get a standard like DNG for all these OEM's . But I think many OEM's thought it was self serving as well.

 

[Ranger 9]

I'm not sure where the crossover point (what focal length) lens design gets easier.

Basically, the narrower the angle of view and the smaller the maximum aperture, the easier it is to design.

If you need only a very narrow angle of view, and don't care about having a large aperture, an extremely simple optical design called a "telescope objective" (cemented doublet) can give excellent results. The old Leitz 400mm f/6.8 Telyt, which was famous for its sharpness back in the day, was designed this way: just two pieces of well-chosen glass in a long tube.

The wider, the faster, the zoomier... the harder it gets.

 

[jonoslack]

Not sure I read you correctly though , still half asleep.

The real issue is programs like Aperture just don't update often enough to keep supporting the onslaught of new camera's. Otherwise it's a great program

The digital world has changed but I do agree in principle as to what Adobe tried to accomplish was get a standard like DNG for all these OEM's . But I think many OEM's thought it was self serving as well.

My point is that if you use a DAM (like Lightroom) and you use your GF1 raw files in it (and you don't archive them as TIFF or JPG).

Then, when someone finds your archive of your GF1 raw files in 20 years (like your grandchild perhaps) then the chances of them being able to do ANYTHING with them is pretty much zero.

Other Proprietary formats (Nikon / Canon / Sony / Olympus full 4/3) aren't so bad, because you can convert them to a .DNG file which contains all the instructions.

As for Aperture, I quite agree, but although it's fine having an onslaught of new cameras, it certainly isn't fine to have an onslaught of new file formats. Adobe may be self serving, but there is no other choice at present than DNG.

 

[Guy Mancuso]

Yea Jono but who is to say DNG will still be around 20 years from now also. That may change but Hassy and Phase shooters can convert to DNG and actually i can convert today with corrections I think to a DNG. Have to check that with Doug but I can output to DNG raw file.

DNG was the theory to be the best raw extension issue is not everyone bought into that theory and we have what we have today but yes it maybe the standard today but may not be saying much since not everyone jumped on it.


But back on point with these little 4/3rds camera's those corrections are built in and no getting around it since there is no way they can make a lens with no barrel distortion at these costs

 

[Brian Mosley]

Then what good does it do to start threads calling the lens the most underrated m4/3 lens. Not sure how that gets the right message to the manufacturers.

http://forums.dpreview.com/forums/read.asp?forum=1041&message=33244076

Well, I don't need to 'get the right message to the manufacturers' - if I want to do that, I send them an e-mail

That thread was to express that (prior to the 20mm f1.7 landing) the 17mm f2.8 was my preferred m4/3rds lens, due to the size, fov and speed.

Thanks for the special attention though Terry, a bit creepy, but flattering :ROTFL:

Cheers

Brian

 

[jonoslack]

Yea Jono but who is to say DNG will still be around 20 years from now also.

No guarantee Guy, but if ANY raw files can be converted, then DNG will be possible.

That may change but Hassy and Phase shooters can convert to DNG and actually i can convert today with corrections I think to a DNG. Have to check that with Doug but I can output to DNG raw file.

DNG was the theory to be the best raw extension issue is not everyone bought into that theory and we have what we have today but yes it maybe the standard today but may not be saying much since not everyone jumped on it.

But saying that DNG isn't perfect doesn't suddenly make a format which is exclusive to a single camera from a single manufacturer okay.

I would have said that, if someone comes across a disk full of DNG files in 20 years it's probable that they'll be able to do something with them.
If they come across a disk full of GF-1 raw files it's almost inevitable that they won't be able to.

But back on point with these little 4/3rds camera's those corrections are built in and no getting around it since there is no way they can make a lens with no barrel distortion at these costs

I'm not against the corrections Guy - I'm not against corrections in the S2 or the Hassleblad or anything else. What I'm against is sticking the corrections into some obscure file format.

The solution for the little micro 4/3 cameras (I'll reiterate that it doesn't affect ordinary 4/3) is for the corrections to be baked into the RAW files (just like they are in the jpgs) NOT added as instructions to the raw processor. Either that or to use proper DNG files

 

[barjohn]

While I can see the argument for embedding the corrections in the raw files, the notion that DNG is the guarantee of usability 20 or 30 years from now, is harder to buy into. I think the best bet for you grand children is to print on the new archival papers and pigment inks if the images are important enough. Let's face it technology is changing at such a fast pace that none of us can predict what will work 20 or 30 years from now. When I started in the computer business, computer memory was an acoustic delay line that filled a large room, then came drum memory, followed by fixed drive heads over a magnetic platter, followed by magnetic core memory and so on. I was just reading about developments to use flash memory as an imaging sensor with much higher sensitivity and 200 mega-pixel density. I'm sure our forefathers (maybe some of us there ) thought film archival would be the answer but in 20 or 30 years where will they go to get it processed?

 

[jonoslack]

While I can see the argument for embedding the corrections in the raw files, the notion that DNG is the guarantee of usability 20 or 30 years from now, is harder to buy into. I think the best bet for you grand children is to print on the new archival papers and pigment inks if the images are important enough. Let's face it technology is changing at such a fast pace that none of us can predict what will work 20 or 30 years from now. When I started in the computer business, computer memory was an acoustic delay line that filled a large room, then came drum memory, followed by fixed drive heads over a magnetic platter, followed by magnetic core memory and so on. I was just reading about developments to use flash memory as an imaging sensor with much higher sensitivity and 200 mega-pixel density. I'm sure our forefathers (maybe some of us there ) thought film archival would be the answer but in 20 or 30 years where will they go to get it processed?

HI John
i agree with every word . . . but the fact that nothing is perfect is no reason to accept something which is clearly a catastrophe.

 

[kwalsh]

Terry - you're right, it's in the DNG standard, but converting E-p1 files to DNG using Adobe converter doesn't embed these lens corrections.

I don't think you are using the correct version of the DNG converter. 5.5 does put the lens correction parameters in, at least for my G1, and I'd imagine for the EP-1 and all other m43 cameras as well. The whole point of the new DNG version and converter version was to support the opcodes for these cameras. I suspect you are using an older version of the DNG converter. I don't have an EP-1 though so I can't be sure, apologies if you've already tried.

Also, the m43rds standard includes specifications for how lenses pass corrections to the camera as well as the correction algorithms to apply. Andy Westlake at DPR has in fact verified identical corrections being applied by Lightroom/ACR, Silkypix, and in camera JPEG.

Providing corrections to the RAW file in the camera (specifically distortion and CA) is a truly awful idea. Then you'd be stuck with the camera's demosaicing algorithm. No the corrections should be metadata just as they are being implemented.

I think the hoopla about opcodes somehow being some mysterious evil force that will cause compatibility problems in the future is a bit silly. They are no different than unusual RGGB arrays, different sensor black points between manufacturers, different hot-pixel methods, or any of a long list of other meta-data that is already handled by DNG and RAW converters.

I will completely agree and stand up and shout from the top of every mountain that more camera manufacturers should use DNG as their default RAW file format. However, I'm also cognizant of some manufacturers concern that Adobe's control of the DNG format can stifle innovation (witness how long it took DNG to support lens corrections even though they were in the market place for years prior).

Ken

 

[Terry]

I thought the problem is that Apple doesn't have support for the opcodes in the DNG. So, the information may indeed be there but it isn't used.

 

[barjohn]

Brian,

Not sure what happened to your post asking for a link to the FLASH memory article but here it is: http://gizmodo.com/5377487/flash-memory-sensors-100x-smaller-than-ccds-better-at-low-light

 

[jonoslack]

Hi Ken
Thank you - great post. I'm afraid I'm a bit proneto 'go off on one' about this :deadhorse: and all!

I don't think you are using the correct version of the DNG converter. 5.5 does put the lens correction parameters in, at least for my G1, and I'd imagine for the EP-1 and all other m43 cameras as well. The whole point of the new DNG version and converter version was to support the opcodes for these cameras. I suspect you are using an older version of the DNG converter. I don't have an EP-1 though so I can't be sure, apologies if you've already tried.

I didn't think it did. the older version produced the huge, horrible, demosaiced linear dng files - I thought the next version produced ordinary DNG files, but without the correction?

Maybe I'm wrong - I'd be interested to know, as it definitely does change things.

Also, the m43rds standard includes specifications for how lenses pass corrections to the camera as well as the correction algorithms to apply. Andy Westlake at DPR has in fact verified identical corrections being applied by Lightroom/ACR, Silkypix, and in camera JPEG.

Providing corrections to the RAW file in the camera (specifically distortion and CA) is a truly awful idea. Then you'd be stuck with the camera's demosaicing algorithm. No the corrections should be metadata just as they are being implemented.

Why? Leica are sorting out the vignetting and colour shift with IR for the M8 and M9, and these are written into the RAW file without doing the demosaicing. Or am I missing something here?

I think the hoopla about opcodes somehow being some mysterious evil force that will cause compatibility problems in the future is a bit silly. They are no different than unusual RGGB arrays, different sensor black points between manufacturers, different hot-pixel methods, or any of a long list of other meta-data that is already handled by DNG and RAW converters.

Well - if they are going in to the DNG files, then I quite agree (I was under the impression that they were specific to the proprietary RAW file).

I will completely agree and stand up and shout from the top of every mountain that more camera manufacturers should use DNG as their default RAW file format. However, I'm also cognizant of some manufacturers concern that Adobe's control of the DNG format can stifle innovation (witness how long it took DNG to support lens corrections even though they were in the market place for years prior).

Ken

I understand why the manufacturers don't like it, but it can't be necessary to make a new file format for every camera (surely).

I thought the problem is that Apple doesn't have support for the opcodes in the DNG. So, the information may indeed be there but it isn't used.

Hi Terry
Well, I guess we need to know the truth about this one - I'm not defending Apple for being:
1. terribly slow
2. completely silent
about their updates.

So, the question is:
Do the lens correction opcodes go into DNG files converted from raw?

 

[Vivek]

Not all the distortions from the Oly-D 17/2.8 are correctable.

While the barrel distortion can be minimized in the post processing, nothing can be done about the field distortion this lens has.

Incredibly, Panasonic lenses (all of them) have minimal field distortion compared to the Oly-D.

It is an over priced, poorly constructed lens (compared to the Pana 20) for what it can do.

 

[Brian Mosley]

Brian,

Not sure what happened to your post asking for a link to the FLASH memory article but here it is: http://gizmodo.com/5377487/flash-memory-sensors-100x-smaller-than-ccds-better-at-low-light

Thanks John,

Sorry, I just googled it - didn't want to drift OT (for once! :ROTFL

Cheers

Brian

 

[nostatic]

Jono, I think you're right that part of Apple's tardiness is due to the low-level support for RAW (ie in the OS instead of the app). This feature is part of what makes creating on Apple so easy in some respects...most of the pro apps and all of the iLife apps are able to access quite a bit of data from other programs within each individual program. And that is super...well, unless you want to actually use the raw files from your damn camera.

I believe that correction data is the future, and you'll see more and more of this. Unfortunately, I think it will also provide another excuse for manufacturers to retain their propreitary formats to help ensure compatibility. Then the third party converters will be playing catch-up constantly. Surprisingly Adobe is pretty good about this, and Apple isn't. But Apple is known for doing some odd things. For instance, up until the most recent version of FinalCut you were very limited in what formats of video you could import. Given the myriad frame rates and compression schemes perhaps that isn't that odd...except that iMovie, the consumer app, would ingest damn near anything.

I'm on the fence still. I did a shoot today and pulled all my 5D2 files right into Aperture, did a quick web output so that people could see proofs, and I'll start rating shots and outputting for a piece I'm doing in FinalCut. Very quick and easy. But if I'd also shot raw with my DLux4, I'd have to implement other steps just so that I could have the files to work with. Not quick and easy.

I suppose it comes down to a couple of choices. APerture does support the Canon .cr2 files (but not the sRAW files), and would support files from the forthcoming S90 (I'm assuming). So I can get rid of cameras and keep Aperture, or I can get rid of Aperture and use whatever cameras I want (for today at least).

Such is living digital...