Site Sponsors
Results 1 to 36 of 36

Thread: Strange moire-like pattern - assistance requested

  1. #1
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Strange moire-like pattern - assistance requested

    Hello chums,

    I am a keen exponent of star trail photography - something I have been doing for years with my D800E and am now doing successfully with the 645Z, which handles it very well.

    However, since the transition to the 645Z, a strange phenomenon has been occurring that I am at a loss to explain, let alone solve. I am hoping that the brains trust here can help.

    If you look at the following images, you will see a strange banding artefact. In this picture, it is most pronounced on the right hand side of the star trail area, showing up as a series of vertical stripes (it is also there to a lesser extent on the left hand side of the star trail area):

    [IMG]StarTrailsFromFiles_IMG4165-4619LITVERSIONStep7CropSMALL by Ed Hurst, SpiffingPics (1/4million views -thanks!), on Flickr[/IMG]

    In this image, the artefact shows up in the bottom left of the image as a series of radial bands:

    [IMG]StarTrailsFromFiles_IMG3935-4000Step10SMALL by Ed Hurst, SpiffingPics (1/4million views -thanks!), on Flickr[/IMG]


    In this image, it is once again visible as a series of radial bands to the left, but this time clearly extends beyond the star trail area and can be seen over the beach etc. on the left hand side; it is also there to some degree on the right just over the trees:

    [IMG]StarTrailsFromFiles_IMG3363-3426Step11CropSpotSMALL by Ed Hurst, SpiffingPics (1/4million views -thanks!), on Flickr[/IMG]

    So the issue occurs with these images made up of a large number of stacked files, most especially in the corners with (perhaps) a tendency to happen most in the lower left. The issue is not restricted to the star trail areas themselves (in the images where you cannot see the banding in non-star trail areas, that is because those areas may have been layered in from single [i.e. non-stacked] files).

    Now, here is the most perplexing thing of all: this effect only appears once the stacked images are combined. When I flatten the image (or use 'Merge Visible), the banding artefact appears. Until that point, when viewing the image on screen, even zoomed in close, those areas are very clean. The stacked files combine beautifully with no sign whatever of the artefact. It is only the flattening (or merging) of the layers that produces this effect, which strikes me as very odd indeed. This seems to happen regardless of whether I include all the layers or only a smaller subset of them (I thought it might be PS getting confused by the huge number of layers, but seemingly not). If the effect was simply there on screen once the layers were stacked, then I would assume it's something to do with how the layers are interacting with one another, but how can the layers display perfectly when they are all stacked, but screw up like this once flattened or merged?

    This is very odd indeed and extremely hard to rectify in post. I emphasise that I never had this problem with the D800E, though a friend of mine is having it with similar images shot with his 6D. At first I thought it might be something to do with moire caused by no antialiasing filter (combined with the high res sensor), but then how can the images look perfect before flattening/merging?

    It seems to me that it has to be something to do with how PS flattens or merges files, but I am damned if I can think of what it is or how to solve it.

    Any help or suggestions would make a silly man very happy!

    Thanks in advance to you all,

    Ed

  2. #2
    Member
    Join Date
    Sep 2013
    Location
    Munich, Germany
    Posts
    219
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    I had a very similar effect doing star trails with my Canon 1Ds III. Banding in the star-trail area, that would slowly appear as I added frames to the image (I did not use Photoshop though, I used a software called StarStaX). This software can play a little movie, adding each picture to the final image with a slight delay (so you see the image constructing itself). So with the first image, no banding (and no star trails !) and then as the trails grew, so did the banding.
    As I remember (it was a while ago), I traced the problem to a processing I did to the images before stacking (was it adding clarity ? Or perhaps it was stretching levels / adding contrast ?). Anyway, my way of "solving" the problem was to first stack all the images (RAW exported into TIFFs) basically without any processing, and then doing the levels, clarity, and the rest of the processing on the final (already stacked) image.
    That way, no more banding. Of course, you are losing the ability to process directly the raws - but for me, the banding was just too much.

    Another thing to have a look at would be to do a stack of dark frames (so doing perhaps 5-10 dark frames with the same exposure as you used for a single trail image, and stack those), and see if that pattern is there. If so, you probably need to see if there is a way to better process the dark frames.

    Hope this helps !

  3. #3
    Senior Member ErikKaffehr's Avatar
    Join Date
    Oct 2012
    Location
    Nyköping Sweden
    Posts
    1,193
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi,

    I don't think the problem is aliasing related. Can you see something similar on single images?

    Are you using some lens corrections?

    No firm idea what would cause this.

    Best regards
    Erik

  4. #4
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Thank you both for your input.

    Miska - one question: if the issue is the result of processing the files before layering (clarity, levels, etc.), how can the picture look so good with all layers included before flattening? With the method you describe with StarStaX, the artefact appears as you add the trails by adding layers. But with my method in PS, the trails are fully in place with all layers there with no artefact. It is only when I flatten (or merge visible) that the artefact appears. That's why I can't see how it can be the processing of the individual files. All I want is for the flattened image to look as it does just before flattening. I might try what you suggest if I can find no other method, but it seems to me that the key issue here is something that is happening during the flattening - otherwise how could it display perfectly just prior to that step? Really appreciate the suggestion though :-)

    Erik - I can see no such issue on similar images. I do use automated lens corrections on each file, yes. But, as I say, the artefact is not there with all of the processed and corrected files layered - until the file is flattened. So can't see how the lens correction can be to blame. But open to all suggestions!

  5. #5
    Member
    Join Date
    Mar 2012
    Location
    Hamburg/Beijing
    Posts
    247
    Post Thanks / Like
    Images
    7

    Re: Strange moire-like pattern - assistance requested

    well on the last image its clear to see that its bent like the star trails, i would suspect some flare/ghosting
    strange that it appears only on the sides as the frame is full of star trails

  6. #6
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    It does look like that, yet it wasn't there before the file was flattened...

  7. #7
    Member
    Join Date
    Sep 2013
    Location
    Munich, Germany
    Posts
    219
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    I am really not sure what is happening, so I am inventing here...

    By the way, I was using Lightroom to process my images to TIFF, before sending them to StarStax, so there was an Adobe product involved in the chain (therefore it could be that this "bug" is related to the way Adobe applies their correction). Or then it was something completely unrelated to what you have here. BTW, I did not have ring-like structures, just a square grid.

    So here goes my explanation... It could be a numerical effect, due to handling a lot of layers. So related to the internals of the image processing software. It could be (from a numerical calculation point of view, due to roundoff errors) that applying a modification to an image once (when the images are stacked) and 500 times (or whatever the number of images you have) but with 500 times less amplitude is not exactly the same. That things that you are seeing are produced by PS, but that you usually don't see them because now you are amplifying the effect 500 times.
    And you see them only when you flatten the image, because the roundoff error only happens when the operation is really done (and not in a preview mode). I agree that it is strange, so I don't know if this is it, or something completely unrelated.

    One test you could do, is to somehow apply the filters in PS (clarity, levels,...) in a different order (if you can). Or (and it can quickly become tedious) try to find if one of your adjustments is making trouble (disable them one by one, and see if the effect disappears).
    Then, if nothing else helps: stack all images, flatten, and then start to stretch the curves and so on, and see if that improves.

    The major question is: does your artifact come from the camera (but it is well hidden on each individual frame, so you don't see it usually), or does it only appear through Photoshop processing (so you are seeing a PS internal "flaw", because you are processing many many frames, so this flaw - normally invisible - adds up). The square grid: could be both PS or the camera (since the CMOS is a square grid of pixels, so perhaps a "feature" in the dark frame). The ring-like structure ? More difficult to imaging what is causing this, but perhaps some Newton ring kind of thing (due to a filter in front of your lens), or the lens correction by PS.
    My money is still on a PS bug though.

    Hope this helps. It is really annoying, since once you see the effect, it's hard to forget it :-(

  8. #8
    Senior Member
    Join Date
    May 2014
    Location
    Oxford
    Posts
    759
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    I agree with miska. Try to process the same number (and exposure) of darkframes first and see if you get the similar pattern. If yes, then consider adding the layer into your image (subtraction).

  9. #9
    Super Duper
    Senior Member
    k-hawinkler's Avatar
    Join Date
    Dec 2009
    Location
    The "Land of Enchantment"
    Posts
    3,298
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Could Capture One Pro handle this task?
    There is a free trial period.

    A nice overview.
    https://www.youtube.com/watch?v=m6KqYfjVt50
    With best regards, K-H.

  10. #10
    Senior Member
    Join Date
    Dec 2007
    Location
    Little Rock AR
    Posts
    1,926
    Post Thanks / Like
    Images
    3

    Re: Strange moire-like pattern - assistance requested

    Hello Ed,

    First of all, great shots, love the one shot around the power windmill.

    I assume you are stacking? If so, what tool set are you using to run the stack modes?

    Do you see the pattern in the single non stacked files, as that will point to some issue in the lens or sensor. Or only when you do the combine and flatten. Are you creating a smart object before you run the stack modes?

    I also wonder if the issues is something with 16 bit vs 8 bit. I tend to leave all my star work in 8 bit.

    On the last shot, you can see that the ring are concentric and circular moving out from center. The real key is if you see them in a singe shot from the stack or if they only show up when you stack them.

    I personally have never seen this, but I am only using Nikon and Canon, and Fuji. No Medium format.

    Paul

  11. #11
    Senior Member ErikKaffehr's Avatar
    Join Date
    Oct 2012
    Location
    Nyköping Sweden
    Posts
    1,193
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi,

    The reason I was asking is that lens corrections move pixels. So I was thinking that lens correction plus layer alignment may produce "fake" pixels that show up as low frequency alias. The other idea I had was that dark frame suppression may not work well with lens corrections.

    These both are WILD GUESSES.

    Why you cannot see them in layers but in the flattened image? No idea.

    But, photoshop often shows a simplified image, so screen update is fast and when the image is finalised a new and more complete representation is built.

    Best regards
    Erik

    Quote Originally Posted by Ed Hurst View Post
    Thank you both for your input.



    Erik - I can see no such issue on similar images. I do use automated lens corrections on each file, yes. But, as I say, the artefact is not there with all of the processed and corrected files layered - until the file is flattened. So can't see how the lens correction can be to blame. But open to all suggestions!

  12. #12
    Member
    Join Date
    Aug 2013
    Posts
    34
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi,

    I have infrequently experienced some odd things happening like this in Photoshop when flattening an image. In the cases where I experienced, it was a Photoshop issue rather than anything else.

    Some tests to try would be MERGING a couple of layers at a time, while keeping an eye on out for whether or when this effect appears again. You may be able to isolate it to a single layer then, for further investigation.

    I do also remember times when MERGING layers together produced better/different results than when FLATTENING an image.

    Best,
    Foster

  13. #13
    Super Duper
    Senior Member
    Shashin's Avatar
    Join Date
    May 2011
    Location
    Florida, USA
    Posts
    4,499
    Post Thanks / Like
    Images
    141

    Re: Strange moire-like pattern - assistance requested

    Quote Originally Posted by Ed Hurst View Post
    With the method you describe with StarStaX, the artefact appears as you add the trails by adding layers. But with my method in PS, the trails are fully in place with all layers there with no artefact. It is only when I flatten (or merge visible) that the artefact appears. That's why I can't see how it can be the processing of the individual files. All I want is for the flattened image to look as it does just before flattening. I might try what you suggest if I can find no other method, but it seems to me that the key issue here is something that is happening during the flattening - otherwise how could it display perfectly just prior to that step? Really appreciate the suggestion though :-)
    Flattening is finally doing the math. While the layers are separate, Photoshop might not be doing all the math to save space, especially if you have a lot of layers. Also I assume your images are 16-bit, where the layer preview is only 8-bit. Perhaps when you are flattening, the difference between the 8-bit preview and the actual 16-bit data is revealed resulting in banding.

    I wonder what would happen if you saved the file to jpeg (without flattening or changing to 8 bit)? Would the jpeg copy have the banding?

  14. #14
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Firstly, thanks to all of you for your help, your time and your willingness to help me. I love you guys and I love this place :-)

    It seems that there's a theme running through many of the ideas above - namely that what I see on screen before flattening or merging may not be the full resolution image and that the mathematics / rounding that goes on only occurs during that flattening / merging stage, which would explain why the problem only shows up after that is done. If that's the case, it opens up a range of possibilities for what the root cause of the problem is. I have several things I can now try. It will take me a while to try these things out because some of these shots involve very large numbers of frames being combined (in some cases over a thousand), but I will do it and report back.

    One reason why I am healthily sceptical (not cynical!) about this idea of what I see pre-flattening (or merging) being not full resolution is that if I pixel peep and zoom in close (before flattening or merging), it looks very high resolution - but maybe the maths refer to how layers are combined rather than resolution of the image per se. Anyhow, it's certainly worth running with this idea.

    Let me now reply to each of you as best I can...

    Miska - you're suggesting that there is rounding that is occurring only when the image is flattened. That opens up the possibility that it is indeed the clarity/levels/etc. tweaks made on the individual files that lies at the bottom of this issue - with that only coming to light once the rounding is carried out. I will test this by processing one of these scenes without those adjustments made pre-stacking, though it's a shame if that is the issue, since the clarity and other edits make the stars 'pop' more (and I would rather do those edits to the raw data). Still, if it works, I will live with it as a workaround. I will also do the processing without lens corrections made for the same reason. FYI, I use no filters with these shots, so it can't literally be Newton's rings (and in any case that would show up on individual files, I believe).

    Voidshatter - I will also try what you suggest with dark frames, though that involves shooting a large number of dark frames at the same time as the main shots (to get the same ambient conditions) - which of course will be time consuming. It'll probably be something I try out as and when I can. But if this does fix it, I'd be very interested.

    K-H - I presume you are suggesting the use of Capture One for the stacking task rather than raw conversion? I won't be using C1 for raw conversion unless it natively understands 645Z raw files at some future stage (it doesn't now, and I suspect it never will, since the 645Z is a competing product). Although there are workarounds involving the use of DNG files, these don't bring with them the ability of the software to 'understand' the camera's unique files, lens profiles, etc. which is one of the main purposes of such a raw converter. However, if you mean using it for the stacking task, if that works without this artefact, I would be delighted to use it!

    Paul - Thanks for your positive feedback on the shots - glad you like 'em! I do my stacking from Bridge using Dr. Brown's Services - which launches the files as layers in PS (with the layer blending mode set to 'Lighten'). The artefact I am talking about is not visible in individual, single files and not in the stack until the file is flattened or merged. I am not using Smart Objects in this case. However, I will take your suggestion and try using 8 bit mode throughout and not 16 bit to see if that helps (another idea that makes sense if what I see on screen before flattening or merging is not the full resolution final image, but a version that doesn't have the rounding or mathematics done in full).

    Erik - thanks for the suggestion - as said above, it all could make sense if the pre-flattened image is a simplified image.

    Foster - if I use Merge instead of Flattening, the issue still occurs. The effect builds in intensity as I add layers and is certainly not present in any single file. However, it reaches full intensity quickly long before I have added all of the layers - so using a smaller subset of files - but still enough to get decent trails - doesn't seem to reduce the effect.

    Will - thanks for the input. As mentioned above, if flattening is indeed doing the full maths (and what one sees before that is a lower resolution preview), then that makes sense. I will try JPEG, but surely I need to convert to 8 bit before trying that, as the JPEG option in the Save As dialogue only appears in 8 bit mode (since it's an 8 bit format); or do I have that wrong?

    Thanks once again to all of you - it's wonderful of you to take the time to think about this and help.

    Warmest regards,

    Ed
    Ed Hurst, www.spiffingpics.com
    Likes 1 Member(s) liked this post

  15. #15
    Senior Member
    Join Date
    Dec 2007
    Location
    Little Rock AR
    Posts
    1,926
    Post Thanks / Like
    Images
    3

    Re: Strange moire-like pattern - assistance requested

    Ed:

    If you get a chance, it would be interesting to try this,

    Load all the images into CC from bridge, they will all load into bridge as layers.

    Select all of them, and create a smart object.

    Then run the Max and Mean stack mode from CC on the smart object. See if you see the same concentric rings. In fact just run the Max mode and see if you get the rings when you flatten the smart object.

    I show the process in the link:

    Stacking your way to amazing night photography results | getDPI

    Paul

  16. #16
    Super Duper
    Senior Member
    Shashin's Avatar
    Join Date
    May 2011
    Location
    Florida, USA
    Posts
    4,499
    Post Thanks / Like
    Images
    141

    Re: Strange moire-like pattern - assistance requested

    Quote Originally Posted by Ed Hurst View Post
    Will - thanks for the input. As mentioned above, if flattening is indeed doing the full maths (and what one sees before that is a lower resolution preview), then that makes sense. I will try JPEG, but surely I need to convert to 8 bit before trying that, as the JPEG option in the Save As dialogue only appears in 8 bit mode (since it's an 8 bit format); or do I have that wrong?
    Ed, if you just save your 16-bit unmerged image as a jpeg, photoshop simply makes a copy of your stack without altering your image--it will remain a psd. I am just curious if the resulting jpeg is generated from the 8-bit layer preview, which might mean the banding will not be visible. In that case, perhaps it is coming from the the 16-bit data, but not visible in 8-bit.

    This is purely speculation, both in how photoshop works and what is happening. I have never come across this issue.

  17. #17
    Senior Member
    Join Date
    Mar 2008
    Location
    Brooklyn NY
    Posts
    457
    Post Thanks / Like
    Images
    8

    Re: Strange moire-like pattern - assistance requested

    I think it has to be a preview issue- if you have 100's or 1000's of layers as you say in this case there is no way that Photoshop is doing all the math to correctly represent the results of each individual pixel location- it is taking some average or mean value.

    However if you think of it this way, at each pixel location there are rgb values, for the sake of argument 150,150,150. Sometimes it is maybe one or two values higher or lower depending on how the layer was rendered by the raw processing. I think the vignetting corrections are producing the effect. Because you have a statistically significant number of layers, any difference in the calculation of each pixel location means that from layer to layer, areas that should be be continuous tone gradients from one set of values to another get bumpy- and then when you flatten the layers and the math is done, sometimes the values calculate higher or lower than the mean.

    If this wasn't on a gradient I think you'd not see the issue as you are, and if the gradient was not produced by the vignetting then the shape would be not as it is.

    there is another possibility and that is it is an interaction between the vignetting corrections and whatever microlens pattern is on the CMOS- that radial fresnel pattern looks suspiciously like what might be on chip vignetting corrections similar to what leica engineered for its ccd's. I don't know if that 50mp has any sort of radial micro lenses to compensate for off-axis light loss but if it does, and then introduce vignetting corrections and multiply that across 1000 layers it could show as banding, as a guess?.....

    you are certainly on the edge case scenario so lots to consider.

    perhaps other people with the other manufacturers versions of this chip who also do this kind of stacking could venture a better opinion.

  18. #18
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Thank you once again for the continuing excellent ideas.

    Here is what I will do... My goal is to change one thing at a time so I can isolate what helps (assuming anything does, of course!).

    1. Using the files I already have (which contain lens corrections, clarity/levels adjustments, etc.) I will apply Paul's method - to see if the process of combining the layers is the issue and whether this different method gets around it.
    2. Also using the files I already have, I will take the stacked files and create a JPEG without flattening / merging - as recommended by Will - to see if that produces something akin to the onscreen preview I am seeing before flattening / merging.
    3. I will create a new set of files with no processing applied then stack and flatten as per my current process (to accommodate everyone's different ideas above assuming the processing becomes a problem at the flattening stage).
    4. The same as 3., but with lens corrections applied (no other processing) - to see if that is the problem - as per Robert's (and others') suggestions above.
    5. The same as 3., but with clarity/levels/etc. applied (without lens corrections) - to see if the problem is with this type of processing (as per various suggestions above).
    6. If all of the above fails, I will try shooting as many dark frames as actual frames (under the same conditions as the actual shoot); I am keeping this in reserve because intuitively I feel it's less likely to be the issue and also because, as a 'fix', it's painful as it would require a doubling of the shooting time and I am hoping this isn't what is needed :-)

    All of this will take some time, but once done, I will report back.

    Thanks again to everyone!
    Ed Hurst, www.spiffingpics.com
    Likes 2 Member(s) liked this post

  19. #19
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    I am delighted to say that I have a solution to the problem! This was discovered before completing all of the steps I listed in my last message, so can't claim to have done them all.

    I tried the following:
    1. Reprocessing all of the raw files without any adjustments that would affect tones (clarity, contrast, curves, etc.) and also without any lens corrections.
    2. Reprocessing all of the raw files with the tone adjustments, but not the lens corrections.
    3. Reprocessing all of the raw files with the lens corrections but not the tone adjustments.

    In each case, I then stacked and flattened the files.

    In case 3., the strange artefact occurred. In cases 1 & 2., it did not. So the conclusion I have come to is that the problem arises from lens corrections made during raw conversion. The other tonal adjustments seem to be no problem. As discussed above, the issue is not visible with the files stacked but occurs when the file is flattened (or the layers are merged). So the image one sees on screen before flattening or merging is clearly not the fully processed image (despite appearing very high res) but a partially processed preview. The processing occurs during the flattening/merging and that's when the lens corrections lead to the artefact.

    It's a shame, but not the end of the world (after all, one can apply lens corrections at the end to the flattened file if desired). But the main thing is that we have an answer! That's very exciting for me, as the problem was messing with my head!

    Thanks again everyone above for your wonderful help. Lots of you suggested the ideas that led to the solution :-)

    Yours happily,

    Ed

    P.S. I will post one of the images without the problem over the next day or two when I have time to finish the processing.
    Last edited by Ed Hurst; 22nd January 2015 at 17:31.
    Ed Hurst, www.spiffingpics.com
    Likes 6 Member(s) liked this post

  20. #20
    Member
    Join Date
    Jan 2015
    Posts
    39
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi Ed,

    What you are experiencing is not surprising - in Astrophotography we encounter these sorts of processing issues all the time. It helps to think about what all the processes are doing behind the scenes, then it becomes evident that before you stack and put one pixel on top of the other, you need to be very certain that they depict the exact same part of the picture.

    As you've found out, lens correction algorithms shift the pixels in the x-y plane by some amount calculated by the lens projection function. There is one reason why this doesn't match in every frame: The sub-pixel accuracy of your pointing (i.e. the stability of your setup) is not stable enough. If you were able to get the camera to point to the exact same spot with true sub-pixel accuracy, you could apply the lens correction beforehand and then stack, but the end result will be the same. Expressed in mathematical terms, the sum of the corrections in every frame equals to the correction in the summed frame. This is assuming you have sub-pixel accuracy in your image. If you can process an image with a time consuming algorithm once, just do it once... not 100 times for each frame. :-)

    To give you an idea of the level of precision required to make this happen: the camera can't move more than about 5 microns (5/1000th of a millimeter). You'd need a thermally controlled mount so that uneven heating/cooling of the tripod legs is compensated for, as that causes slight expansion or contraction, you could have no wind at all... you'd essentially be building an astronomical telescope for terrestrial purposes. :-)

    Having said all that, your photography is stunning! Beautiful shots, and from a fellow Sydneysider, you've motivated me to make a go for vivid Sydney next time with the camera!

    Cheeers

    - Balt

  21. #21
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    G'day Balt,

    Thanks for your kind compliment - and also for your very interesting insights to the way in which these things tend to work.

    I am not in the least surprised that small movements make a difference, or am not very surprised that the lens corrections are the culprit given that they change the geometry (and other variables) of the scene. What surprised me, if these factors are at play, is that the stacked images could display without the artefact showing, then for flattening/merging to reveal the problem. This was because I was not previously aware that what one sees on screen before flattening/merging does not fully represent the full file once flattened/merged. So the appearance of the artefact at that stage was what confused me (the image having, I wrongly assumed, appeared in its full splendour without the artefact prior to flattening/merging). Now that I know that flattening/merging actually involves performing various algorithms, and that the preview prior to that is not 'complete', it all makes eminent sense to me. I feel like I have learnt something!

    Would love to chat about Sydney photography and astrophotography some time - perhaps in a Sydney pub ;-)

    Warmest regards,

    Ed


    P.S. To everyone who suggested other solutions that I have not tried yet, I will still give them a go - they might suggest something useful for other reasons!

  22. #22
    Member
    Join Date
    Jan 2015
    Posts
    39
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi Ed,

    no worries, and yes, we should go have a beer someday and have a bit of a chat about things photography. As you might have seen in my other post, I'm looking into (likely) buying a CFV50c to get my Hassy setup (and my mojo) working again. Some others have suggested I look into a 645, you seem to be working with one? Perhaps you can tell me a little about your equipment?

    Another thing you stumbled over is the representation of your image, 8bit or 16bit makes a huge difference. Photoshop to my knowledge in preview only shows you 8bit. Also, when you're stacking lots of images, keep in mind that with 8bits you run out of dynamic range in the stacked image immediately, 16bit is not much better. Ideally you stack into a 24bit or even 32bit format, then use HDR techniques to "recover" i.e. bring into focus those parts of the dynamic range you're interested in.

    Remember this: In unsigned integer world (which is what pixels usually are represented as), 8bit = 256 levels, 16bit = 64k levels, 24bit = 16M levels and 32bit = 4G levels. What does that mean? When you combine 100 frames into an 8bit image, the finest gradation you get between pixels can be 1/256th of the full dynamic range in the image. You can see how that gets much finer the more bits per pixel you have to represent the numbers: 32bpp has 16 million times the dynamic range of an 8bpp image.

    But then again you probably knew all that, don't mean to be a smartarse... :-)

    Cheers

    = Balt

  23. #23
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    I am broadly aware of the implications of 8 versus 16 versus 24 versus 32 bit, but had not really thought about doing this stacking with 24 or 32 bit files. Are you suggesting that I create my files from raw at those bit levels, do the stacking, flatten, then use tone mapping to get back to the 16 or 8 bit file? Not being a smartarse at all - except in the best sense of genuinely being smart ;-)

    I will PM you and we can organise a meet up somewhere... We can talk cameras. I can bring along my 645Z if you wish. I am sure I can learn a lot from you, especially in the area of astrophotography!

    Ed

  24. #24
    Member
    Join Date
    Jan 2015
    Posts
    39
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi Ed,

    both amateur and professional astrophotographers try to stay in the most bits per pixel yielding format for as long as possible. Every time you perform a mathematical operation on an image (and all the processes you're talking about are just that), you are limiting yourself to the "number space" available by how many bits you get to represent each pixel.

    Say if you're just linearly adding two pictures together, if you're using 8bit (256 levels), and say the same pixel has a value of 130 in one picture, and 140 in the other, the total is 270. That's higher than the maximum number (256), therefore will have "saturated".

    Cheers

    - Balt

  25. #25
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    For that reason, I always do everything in 16-bit and then only create an 8 bit version for web use at the very end of my workflow. That's always seemed fine for single shots and panos. However, I am intrigued to hear that there might be benefits to a higher bit level with star trail stacks. Definitely something I will try...

    Thanks!

    Ed

  26. #26
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hmm... ACR seems only to allow for raw conversion to 8 or 16 bit. Are you suggesting that I need to do raw conversion at a higher bit level or that I can take the 16 bit files and convert them to, say, 32 bit individually (which would not add data to any single file, of course) before doing the stacking?

  27. #27
    Senior Member
    Join Date
    May 2014
    Location
    Oxford
    Posts
    759
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Quote Originally Posted by bindermuehle View Post

    Say if you're just linearly adding two pictures together, if you're using 8bit (256 levels), and say the same pixel has a value of 130 in one picture, and 140 in the other, the total is 270. That's higher than the maximum number (256), therefore will have "saturated".

    Cheers

    - Balt
    The RAW file of the Pentax 645Z is 14-bit (16384 levels), whereas the RAW files of the IQ250/CFV-50C are 16-bit (65536 levels).

  28. #28
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    But is that significant in terms of the specific issue being discussed? We are talking about using bit levels higher than any sensor produces - not because the individual files contain those bits of data but because the stacking process might benefit from them (when the differences between files may be most effectively represented with more bits)... Or perhaps I misunderstand?

    In any case, I am yet to be convinced that the difference you refer to is actually real in terms of the final output file. But very happy to learn :-)

  29. #29
    Senior Member ErikKaffehr's Avatar
    Join Date
    Oct 2012
    Location
    Nyköping Sweden
    Posts
    1,193
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi,

    IQ 250 is 14 bit according to Phase One.

    The raw format used by Phase One has been reverse engineered by Anders Torger, and it is 14 bits. So that 16-bit is 99.99% a marketing lie bye Phase One.

    With Hasselblad it is a bit different, their raw format actually stores 16 bits, but 3-4 of those bits are just garbage. It is feasible that the IQ-250 sensor can deliver actual 14, it is probably pretty close to sensors used in Nikon D810 and D750. Both these sensors have around 13.7 EV DR.

    Now, the definition of DR is based on a signal noise ratio of 1, which would not be usable so real world DR may be say 11EV corresponding to 11 bits.

    Just keep in mind, bits and EV are essentially the same. When Phase One says that a sensor has a DR of 13EV it means it the sensor signal can be accurately represented by 13 bits and also that the last bit represents 50% noise and 50% signal.

    Now, there is a natural explanation for those 16 bits. Digital devices normally are either 8 or 16 bit wide. So, if you use 16 bit components the digital channel will be 16-bit wide, that doesn't mean it will pass 16 bit clean information.

    Another way to see it, 16 bits correspond to 96 dB, 14 bits to 84 dB and 12 bits to 72 dB. If you check the Dalsa spec sheet of the FTF9168C 60 MP sensor (the one probably used in some 60 MP backs) it says that typical dynamic range is 73 dB, with the linear part being 70 dB. So it is essentially a 12 bit device. That is 12 bits are sufficient to hold all meaningful data and any other bits represent noise.

    DxO has not measured the IQ-260 but they have measured the IQ-180, and it had a DR of 11.89. Now, DxO mark normally normalizes DR to 8 MP and that value is 13.56 EV. 1.66 EV is coming from normalisation ln(sqrt(10))/ln 2.

    So the IQ-180 is not really sixteen bits, not even fourteen but just twelve bits.



    Best regards
    Erik



    Quote Originally Posted by voidshatter View Post
    The RAW file of the Pentax 645Z is 14-bit (16384 levels), whereas the RAW files of the IQ250/CFV-50C are 16-bit (65536 levels).

  30. #30
    Senior Member ErikKaffehr's Avatar
    Join Date
    Oct 2012
    Location
    Nyköping Sweden
    Posts
    1,193
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi,

    Stacking improves noise levels. Quadrupling the information gives an extra EV in dynamic range, corresponding to one EV.

    By the way, my guess is that the banding you observed was coming from the lens correction being applied to to the image but not to the dark frame shots. But that was just a guess.

    Best regards
    Erik

    Quote Originally Posted by Ed Hurst View Post
    But is that significant in terms of the specific issue being discussed? We are talking about using bit levels higher than any sensor produces - not because the individual files contain those bits of data but because the stacking process might benefit from them (when the differences between files may be most effectively represented with more bits)... Or perhaps I misunderstand?

    In any case, I am yet to be convinced that the difference you refer to is actually real in terms of the final output file. But very happy to learn :-)

  31. #31
    Senior Member Ed Hurst's Avatar
    Join Date
    Sep 2008
    Location
    Sydney
    Posts
    1,161
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Quote Originally Posted by ErikKaffehr View Post
    Hi,

    Stacking improves noise levels. Quadrupling the information gives an extra EV in dynamic range, corresponding to one EV.
    So does this mean that the above idea of converting the 16bit TIFFs to large bit levels before stacking would be beneficial (even though it's not adding any extra data to the individual files)?

    By the way, there was no dark frame used with any of the shots posted in this thread...

    Thanks all,

    Ed

  32. #32
    Senior Member
    Join Date
    May 2014
    Location
    Oxford
    Posts
    759
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Quote Originally Posted by ErikKaffehr View Post
    There is no longer any official document from Phase One saying that the IQ250 is 14-bit. If you check with Raw Digger or other software you can see that the RAW file of the IQ250 is 16-bit (65536 levels):



    Quote Originally Posted by ErikKaffehr View Post

    The raw format used by Phase One has been reverse engineered by Anders Torger, and it is 14 bits. So that 16-bit is 99.99% a marketing lie bye Phase One.

    With Hasselblad it is a bit different, their raw format actually stores 16 bits, but 3-4 of those bits are just garbage. It is feasible that the IQ-250 sensor can deliver actual 14, it is probably pretty close to sensors used in Nikon D810 and D750. Both these sensors have around 13.7 EV DR.
    I need to see hard evidence of this claim. I wish he could post part of the source code so I can verify it with my own code. I agree that the 16-bit (65536 levels) in the IQ250 RAW file might be due to interpolation from 14-bit (16384 levels) but there are talkings about the ADC used in the Sony sensor being 14-bit inside and there is no way Hasselblad can provide "true 16-bit" either.

    Quote Originally Posted by ErikKaffehr View Post
    Now, the definition of DR is based on a signal noise ratio of 1, which would not be usable so real world DR may be say 11EV corresponding to 11 bits.

    Just keep in mind, bits and EV are essentially the same. When Phase One says that a sensor has a DR of 13EV it means it the sensor signal can be accurately represented by 13 bits and also that the last bit represents 50% noise and 50% signal.

    Now, there is a natural explanation for those 16 bits. Digital devices normally are either 8 or 16 bit wide. So, if you use 16 bit components the digital channel will be 16-bit wide, that doesn't mean it will pass 16 bit clean information.

    Another way to see it, 16 bits correspond to 96 dB, 14 bits to 84 dB and 12 bits to 72 dB. If you check the Dalsa spec sheet of the FTF9168C 60 MP sensor (the one probably used in some 60 MP backs) it says that typical dynamic range is 73 dB, with the linear part being 70 dB. So it is essentially a 12 bit device. That is 12 bits are sufficient to hold all meaningful data and any other bits represent noise.

    DxO has not measured the IQ-260 but they have measured the IQ-180, and it had a DR of 11.89. Now, DxO mark normally normalizes DR to 8 MP and that value is 13.56 EV. 1.66 EV is coming from normalisation ln(sqrt(10))/ln 2.

    So the IQ-180 is not really sixteen bits, not even fourteen but just twelve bits.

    Best regards
    Erik
    For this part I agree - the IQ260 just stores garbage in the lower bits, especially in long exposure mode, where the picture must be taken at ISO 140 (actually at ISO 200 native) and the shadow is as noisy as Canon, which is fairly pointless for landscape shots.

  33. #33
    Senior Member ErikKaffehr's Avatar
    Join Date
    Oct 2012
    Location
    Nyköping Sweden
    Posts
    1,193
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi,

    Anders (Torger) was very specific about the Phase One raw format being 14 bits. He was developing code to write IIQ files from HDR conversion, so I am pretty sure he knows what he does. I could find his writing but he doesn't give coding details.

    On the other hand, it is quite possible to pass say 16 bit data if data is coded non linearly. Sony is doing this on most cameras. They seem to send 13 bits worth of data trough an 11 bit wide channel. Lots of noise about that. Sony also has a delta compression that can cause artefacts.

    My issue is mostly that MFD people use the 16 bits as a sales argument, although it is irrelevant.

    The issue of 16-bitness is of course quite irrelevant in the original context. Sorry for the deviation!

    Best regards
    Erik
    Quote Originally Posted by voidshatter View Post
    There is no longer any official document from Phase One saying that the IQ250 is 14-bit. If you check with Raw Digger or other software you can see that the RAW file of the IQ250 is 16-bit (65536 levels):


    I need to see hard evidence of this claim. I wish he could post part of the source code so I can verify it with my own code. I agree that the 16-bit (65536 levels) in the IQ250 RAW file might be due to interpolation from 14-bit (16384 levels) but there are talkings about the ADC used in the Sony sensor being 14-bit inside and there is no way Hasselblad can provide "true 16-bit" either.


    For this part I agree - the IQ260 just stores garbage in the lower bits, especially in long exposure mode, where the picture must be taken at ISO 140 (actually at ISO 200 native) and the shadow is as noisy as Canon, which is fairly pointless for landscape shots.
    Last edited by ErikKaffehr; 25th January 2015 at 12:02.

  34. #34
    Senior Member
    Join Date
    May 2014
    Location
    Oxford
    Posts
    759
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Quote Originally Posted by ErikKaffehr View Post
    Hi,

    Anders (Torger) was very specific about the Phase One raw format being 14 bits. He was developing code to write IIQ files from HDR conversion, so I am pretty sure he knows what he does. I could find his writing but he doesn't give coding details.
    I have seen his writing but that doesn't change the fact that whatever software you use (aside from his software as he claimed) you get 16-bit (65536 levels) from an IQ250 RAW file, whereas you only get 14-bit (16384 levels) from a 645Z / Nikon / Canon RAW file.
    Quote Originally Posted by ErikKaffehr View Post
    On the other hand, it is quite possible to pass say 16 bit data if data is coded non linearly. Sony is doing this on most cameras. They seem to send 13 bits worth of data trough an 11 bit wide channel. Lots of noise about that. Sony also has a delta compression that can cause artefacts.
    I am quite aware of this Sony issue, and to be more precise, it is a general issue for all electronic viewfinder models of all current Sony camera bodies. The actual level precision I have tested is as follows:

    a) 10.7-bit (around 1700 levels) for normal settings:

    no special setting;
    electronic front curtain.

    b) 10.4-bit (around 1400 levels) for any of the following settings:

    silent shooting (electronic shutter) for A7S;
    long exposure noise reduction mode;
    B mode (verified at both 24 seconds and 38 seconds);
    continuous shooting mode (verified at both the 1st and the 2nd frame);
    speed priority cont. shooting mode (verified at both the 1st and the 2nd frame).

    Such lossy compression does not only cause artifact on edges of high contrast (like star trails) but also cripple the shadow recoverability (color precision issue for demosaicing).
    Quote Originally Posted by ErikKaffehr View Post
    My issue is mostly that MFD people use the 16 bits as a sales argument, although it is irrelevant.

    The issue of 16-bitness is of course quite irrelevant in the original context. Sorry for the deviation!

    Best regards
    Erik
    Then why did you specifically target Phase One's (removed) official statement of the IQ250 being only 14-bit? I agree that 16-bit is purely hype, especially meaningless for the CCD sensors (e.g. IQ280/IQ260/H5D-60 etc) as the SNR in the shadow is very poor (as poor as at Canon level), but you still cannot persuade me that the IQ250 is not 16-bit (65536 levels), albeit which might have been interpolated from 14-bit (16384 levels). If the interpolation was true, then this would be true for IQ260/IQ280 as well, and it essentially makes no difference for Hasselblad as they do not have a sensor that can saturate a whole 14 stops of dynamic range at pixel level either.
    Last edited by voidshatter; 25th January 2015 at 12:53. Reason: spelling

  35. #35
    Senior Member ErikKaffehr's Avatar
    Join Date
    Oct 2012
    Location
    Nyköping Sweden
    Posts
    1,193
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Hi,

    I was objecting to this statement of yours:

    "The RAW file of the Pentax 645Z is 14-bit (16384 levels), whereas the RAW files of the IQ250/CFV-50C are 16-bit (65536 levels)."

    I was looking at this info from Phase One


    .

    This is low bits from an IQ-150 image in Raw Digger:


    You see that 3 out of four slots are empty.

    Turning 14 bits to 16 bits is not interpolation but logical shift two. Interesting though that the file contains more than 16000 different values.

    But, you need also regard that the higher levels have gaussian distribution, with sigma being the square root of the signal. This is due to shot noise. So say a level of 1000 will be like a bell shape with standard of deviation 100, so a difference between say 1000 and 1005 will be meaningless.

    Best regards
    Erik

    Quote Originally Posted by voidshatter View Post

    Then why did you specifically target Phase One's (removed) official statement of the IQ250 being only 14-bit? I agree that 16-bit is purely hype, especially meaningless for the CCD sensors (e.g. IQ280/IQ260/H5D-60 etc) as the SNR in the shadow is very poor (as poor as at Canon level), but you still cannot persuade me that the IQ250 is not 16-bit (65536 levels), albeit which might have been interpolated from 14-bit (16384 levels). If the interpolation was true, then this would be true for IQ260/IQ280 as well, and it essentially makes no difference for Hasselblad as they do not have a sensor that can saturate a whole 14 stops of dynamic range at pixel level either.
    Last edited by ErikKaffehr; 25th January 2015 at 23:00.

  36. #36
    Senior Member
    Join Date
    May 2014
    Location
    Oxford
    Posts
    759
    Post Thanks / Like

    Re: Strange moire-like pattern - assistance requested

    Quote Originally Posted by ErikKaffehr View Post
    Hi,

    I was objecting to this statement of yours:

    "The RAW file of the Pentax 645Z is 14-bit (16384 levels), whereas the RAW files of the IQ250/CFV-50C are 16-bit (65536 levels)."

    I was looking at this info from Phase One
    So, did you manage to find where Phase One officially states 14-bit (on the current webpage or document)?

    Quote Originally Posted by ErikKaffehr View Post
    This is low bits from an IQ-150 image in Raw Digger:


    You see that 3 out of four slots are empty.

    Turning 14 bits to 16 bits is not interpolation but logical shift two. Interesting though that the file contains more than 16000 different values.

    But, you need also regard that the higher levels have gaussian distribution, with sigma being the square root of the signal. This is due to shot noise. So say a level of 1000 will be like a bell shape with standard of deviation 100, so a difference between say 1000 and 1005 will be meaningless.

    Best regards
    Erik
    Raw Digger cannot fully support the IIQ format. I have tested the following 3 cases on a single dark frame shot of an IQ250:

    a) Open the RAW file directly with Raw Digger: there are gaps between levels;

    b) Convert the RAW file into DNG with Adobe Camera Raw and then open it with Raw Digger: there are no gaps but the standard deviation of noise is very high (and hence lower performance of usable dynamic range);

    c) Convert the RAW file into DNG with Capture One and then open it with Raw Digger: there are no gaps and the standard deviation of noise is low (and hence better performance of usable dynamic range).







    I bet Phase One knows how to optimize and cook the RAW files better than Raw Digger, Adobe or Anders Torger does.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •