The GetDPI Photography Forum

Great to see you here. Join our insightful photographic forum today and start tapping into a huge wealth of photographic knowledge. Completing our simple registration process will allow you to gain access to exclusive content, add your own topics and posts, share your work and connect with other members through your own private inbox! And don’t forget to say hi!

Strange moire-like pattern - assistance requested

Ed Hurst

Well-known member
Hello chums,

I am a keen exponent of star trail photography - something I have been doing for years with my D800E and am now doing successfully with the 645Z, which handles it very well.

However, since the transition to the 645Z, a strange phenomenon has been occurring that I am at a loss to explain, let alone solve. I am hoping that the brains trust here can help.

If you look at the following images, you will see a strange banding artefact. In this picture, it is most pronounced on the right hand side of the star trail area, showing up as a series of vertical stripes (it is also there to a lesser extent on the left hand side of the star trail area):

[/url]StarTrailsFromFiles_IMG4165-4619LITVERSIONStep7CropSMALL by Ed Hurst, SpiffingPics (1/4million views -thanks!), on Flickr[/IMG]

In this image, the artefact shows up in the bottom left of the image as a series of radial bands:

[/url]StarTrailsFromFiles_IMG3935-4000Step10SMALL by Ed Hurst, SpiffingPics (1/4million views -thanks!), on Flickr[/IMG]


In this image, it is once again visible as a series of radial bands to the left, but this time clearly extends beyond the star trail area and can be seen over the beach etc. on the left hand side; it is also there to some degree on the right just over the trees:

[/url]StarTrailsFromFiles_IMG3363-3426Step11CropSpotSMALL by Ed Hurst, SpiffingPics (1/4million views -thanks!), on Flickr[/IMG]

So the issue occurs with these images made up of a large number of stacked files, most especially in the corners with (perhaps) a tendency to happen most in the lower left. The issue is not restricted to the star trail areas themselves (in the images where you cannot see the banding in non-star trail areas, that is because those areas may have been layered in from single [i.e. non-stacked] files).

Now, here is the most perplexing thing of all: this effect only appears once the stacked images are combined. When I flatten the image (or use 'Merge Visible), the banding artefact appears. Until that point, when viewing the image on screen, even zoomed in close, those areas are very clean. The stacked files combine beautifully with no sign whatever of the artefact. It is only the flattening (or merging) of the layers that produces this effect, which strikes me as very odd indeed. This seems to happen regardless of whether I include all the layers or only a smaller subset of them (I thought it might be PS getting confused by the huge number of layers, but seemingly not). If the effect was simply there on screen once the layers were stacked, then I would assume it's something to do with how the layers are interacting with one another, but how can the layers display perfectly when they are all stacked, but screw up like this once flattened or merged?

This is very odd indeed and extremely hard to rectify in post. I emphasise that I never had this problem with the D800E, though a friend of mine is having it with similar images shot with his 6D. At first I thought it might be something to do with moire caused by no antialiasing filter (combined with the high res sensor), but then how can the images look perfect before flattening/merging?

It seems to me that it has to be something to do with how PS flattens or merges files, but I am damned if I can think of what it is or how to solve it.

Any help or suggestions would make a silly man very happy!

Thanks in advance to you all,

Ed
 

miska

Member
I had a very similar effect doing star trails with my Canon 1Ds III. Banding in the star-trail area, that would slowly appear as I added frames to the image (I did not use Photoshop though, I used a software called StarStaX). This software can play a little movie, adding each picture to the final image with a slight delay (so you see the image constructing itself). So with the first image, no banding (and no star trails !) and then as the trails grew, so did the banding.
As I remember (it was a while ago), I traced the problem to a processing I did to the images before stacking (was it adding clarity ? Or perhaps it was stretching levels / adding contrast ?). Anyway, my way of "solving" the problem was to first stack all the images (RAW exported into TIFFs) basically without any processing, and then doing the levels, clarity, and the rest of the processing on the final (already stacked) image.
That way, no more banding. Of course, you are losing the ability to process directly the raws - but for me, the banding was just too much.

Another thing to have a look at would be to do a stack of dark frames (so doing perhaps 5-10 dark frames with the same exposure as you used for a single trail image, and stack those), and see if that pattern is there. If so, you probably need to see if there is a way to better process the dark frames.

Hope this helps !
 

ErikKaffehr

Well-known member
Hi,

I don't think the problem is aliasing related. Can you see something similar on single images?

Are you using some lens corrections?

No firm idea what would cause this.

Best regards
Erik
 

Ed Hurst

Well-known member
Thank you both for your input.

Miska - one question: if the issue is the result of processing the files before layering (clarity, levels, etc.), how can the picture look so good with all layers included before flattening? With the method you describe with StarStaX, the artefact appears as you add the trails by adding layers. But with my method in PS, the trails are fully in place with all layers there with no artefact. It is only when I flatten (or merge visible) that the artefact appears. That's why I can't see how it can be the processing of the individual files. All I want is for the flattened image to look as it does just before flattening. I might try what you suggest if I can find no other method, but it seems to me that the key issue here is something that is happening during the flattening - otherwise how could it display perfectly just prior to that step? Really appreciate the suggestion though :)

Erik - I can see no such issue on similar images. I do use automated lens corrections on each file, yes. But, as I say, the artefact is not there with all of the processed and corrected files layered - until the file is flattened. So can't see how the lens correction can be to blame. But open to all suggestions!
 

drevil

Well-known member
Staff member
well on the last image its clear to see that its bent like the star trails, i would suspect some flare/ghosting
strange that it appears only on the sides as the frame is full of star trails
 

miska

Member
I am really not sure what is happening, so I am inventing here...

By the way, I was using Lightroom to process my images to TIFF, before sending them to StarStax, so there was an Adobe product involved in the chain (therefore it could be that this "bug" is related to the way Adobe applies their correction). Or then it was something completely unrelated to what you have here. BTW, I did not have ring-like structures, just a square grid.

So here goes my explanation... It could be a numerical effect, due to handling a lot of layers. So related to the internals of the image processing software. It could be (from a numerical calculation point of view, due to roundoff errors) that applying a modification to an image once (when the images are stacked) and 500 times (or whatever the number of images you have) but with 500 times less amplitude is not exactly the same. That things that you are seeing are produced by PS, but that you usually don't see them because now you are amplifying the effect 500 times.
And you see them only when you flatten the image, because the roundoff error only happens when the operation is really done (and not in a preview mode). I agree that it is strange, so I don't know if this is it, or something completely unrelated.

One test you could do, is to somehow apply the filters in PS (clarity, levels,...) in a different order (if you can). Or (and it can quickly become tedious) try to find if one of your adjustments is making trouble (disable them one by one, and see if the effect disappears).
Then, if nothing else helps: stack all images, flatten, and then start to stretch the curves and so on, and see if that improves.

The major question is: does your artifact come from the camera (but it is well hidden on each individual frame, so you don't see it usually), or does it only appear through Photoshop processing (so you are seeing a PS internal "flaw", because you are processing many many frames, so this flaw - normally invisible - adds up). The square grid: could be both PS or the camera (since the CMOS is a square grid of pixels, so perhaps a "feature" in the dark frame). The ring-like structure ? More difficult to imaging what is causing this, but perhaps some Newton ring kind of thing (due to a filter in front of your lens), or the lens correction by PS.
My money is still on a PS bug though.

Hope this helps. It is really annoying, since once you see the effect, it's hard to forget it :-(
 
I agree with miska. Try to process the same number (and exposure) of darkframes first and see if you get the similar pattern. If yes, then consider adding the layer into your image (subtraction).
 

Paul2660

Well-known member
Hello Ed,

First of all, great shots, love the one shot around the power windmill.

I assume you are stacking? If so, what tool set are you using to run the stack modes?

Do you see the pattern in the single non stacked files, as that will point to some issue in the lens or sensor. Or only when you do the combine and flatten. Are you creating a smart object before you run the stack modes?

I also wonder if the issues is something with 16 bit vs 8 bit. I tend to leave all my star work in 8 bit.

On the last shot, you can see that the ring are concentric and circular moving out from center. The real key is if you see them in a singe shot from the stack or if they only show up when you stack them.

I personally have never seen this, but I am only using Nikon and Canon, and Fuji. No Medium format.

Paul
 

ErikKaffehr

Well-known member
Hi,

The reason I was asking is that lens corrections move pixels. So I was thinking that lens correction plus layer alignment may produce "fake" pixels that show up as low frequency alias. The other idea I had was that dark frame suppression may not work well with lens corrections.

These both are WILD GUESSES.

Why you cannot see them in layers but in the flattened image? No idea.

But, photoshop often shows a simplified image, so screen update is fast and when the image is finalised a new and more complete representation is built.

Best regards
Erik

Thank you both for your input.



Erik - I can see no such issue on similar images. I do use automated lens corrections on each file, yes. But, as I say, the artefact is not there with all of the processed and corrected files layered - until the file is flattened. So can't see how the lens correction can be to blame. But open to all suggestions!
 

foster_jb

Member
Hi,

I have infrequently experienced some odd things happening like this in Photoshop when flattening an image. In the cases where I experienced, it was a Photoshop issue rather than anything else.

Some tests to try would be MERGING a couple of layers at a time, while keeping an eye on out for whether or when this effect appears again. You may be able to isolate it to a single layer then, for further investigation.

I do also remember times when MERGING layers together produced better/different results than when FLATTENING an image.

Best,
Foster
 

Shashin

Well-known member
With the method you describe with StarStaX, the artefact appears as you add the trails by adding layers. But with my method in PS, the trails are fully in place with all layers there with no artefact. It is only when I flatten (or merge visible) that the artefact appears. That's why I can't see how it can be the processing of the individual files. All I want is for the flattened image to look as it does just before flattening. I might try what you suggest if I can find no other method, but it seems to me that the key issue here is something that is happening during the flattening - otherwise how could it display perfectly just prior to that step? Really appreciate the suggestion though :)
Flattening is finally doing the math. While the layers are separate, Photoshop might not be doing all the math to save space, especially if you have a lot of layers. Also I assume your images are 16-bit, where the layer preview is only 8-bit. Perhaps when you are flattening, the difference between the 8-bit preview and the actual 16-bit data is revealed resulting in banding.

I wonder what would happen if you saved the file to jpeg (without flattening or changing to 8 bit)? Would the jpeg copy have the banding?
 

Ed Hurst

Well-known member
Firstly, thanks to all of you for your help, your time and your willingness to help me. I love you guys and I love this place :)

It seems that there's a theme running through many of the ideas above - namely that what I see on screen before flattening or merging may not be the full resolution image and that the mathematics / rounding that goes on only occurs during that flattening / merging stage, which would explain why the problem only shows up after that is done. If that's the case, it opens up a range of possibilities for what the root cause of the problem is. I have several things I can now try. It will take me a while to try these things out because some of these shots involve very large numbers of frames being combined (in some cases over a thousand), but I will do it and report back.

One reason why I am healthily sceptical (not cynical!) about this idea of what I see pre-flattening (or merging) being not full resolution is that if I pixel peep and zoom in close (before flattening or merging), it looks very high resolution - but maybe the maths refer to how layers are combined rather than resolution of the image per se. Anyhow, it's certainly worth running with this idea.

Let me now reply to each of you as best I can...

Miska - you're suggesting that there is rounding that is occurring only when the image is flattened. That opens up the possibility that it is indeed the clarity/levels/etc. tweaks made on the individual files that lies at the bottom of this issue - with that only coming to light once the rounding is carried out. I will test this by processing one of these scenes without those adjustments made pre-stacking, though it's a shame if that is the issue, since the clarity and other edits make the stars 'pop' more (and I would rather do those edits to the raw data). Still, if it works, I will live with it as a workaround. I will also do the processing without lens corrections made for the same reason. FYI, I use no filters with these shots, so it can't literally be Newton's rings (and in any case that would show up on individual files, I believe).

Voidshatter - I will also try what you suggest with dark frames, though that involves shooting a large number of dark frames at the same time as the main shots (to get the same ambient conditions) - which of course will be time consuming. It'll probably be something I try out as and when I can. But if this does fix it, I'd be very interested.

K-H - I presume you are suggesting the use of Capture One for the stacking task rather than raw conversion? I won't be using C1 for raw conversion unless it natively understands 645Z raw files at some future stage (it doesn't now, and I suspect it never will, since the 645Z is a competing product). Although there are workarounds involving the use of DNG files, these don't bring with them the ability of the software to 'understand' the camera's unique files, lens profiles, etc. which is one of the main purposes of such a raw converter. However, if you mean using it for the stacking task, if that works without this artefact, I would be delighted to use it!

Paul - Thanks for your positive feedback on the shots - glad you like 'em! I do my stacking from Bridge using Dr. Brown's Services - which launches the files as layers in PS (with the layer blending mode set to 'Lighten'). The artefact I am talking about is not visible in individual, single files and not in the stack until the file is flattened or merged. I am not using Smart Objects in this case. However, I will take your suggestion and try using 8 bit mode throughout and not 16 bit to see if that helps (another idea that makes sense if what I see on screen before flattening or merging is not the full resolution final image, but a version that doesn't have the rounding or mathematics done in full).

Erik - thanks for the suggestion - as said above, it all could make sense if the pre-flattened image is a simplified image.

Foster - if I use Merge instead of Flattening, the issue still occurs. The effect builds in intensity as I add layers and is certainly not present in any single file. However, it reaches full intensity quickly long before I have added all of the layers - so using a smaller subset of files - but still enough to get decent trails - doesn't seem to reduce the effect.

Will - thanks for the input. As mentioned above, if flattening is indeed doing the full maths (and what one sees before that is a lower resolution preview), then that makes sense. I will try JPEG, but surely I need to convert to 8 bit before trying that, as the JPEG option in the Save As dialogue only appears in 8 bit mode (since it's an 8 bit format); or do I have that wrong?

Thanks once again to all of you - it's wonderful of you to take the time to think about this and help.

Warmest regards,

Ed
 

Paul2660

Well-known member
Ed:

If you get a chance, it would be interesting to try this,

Load all the images into CC from bridge, they will all load into bridge as layers.

Select all of them, and create a smart object.

Then run the Max and Mean stack mode from CC on the smart object. See if you see the same concentric rings. In fact just run the Max mode and see if you get the rings when you flatten the smart object.

I show the process in the link:

Stacking your way to amazing night photography results | getDPI

Paul
 

Shashin

Well-known member
Will - thanks for the input. As mentioned above, if flattening is indeed doing the full maths (and what one sees before that is a lower resolution preview), then that makes sense. I will try JPEG, but surely I need to convert to 8 bit before trying that, as the JPEG option in the Save As dialogue only appears in 8 bit mode (since it's an 8 bit format); or do I have that wrong?
Ed, if you just save your 16-bit unmerged image as a jpeg, photoshop simply makes a copy of your stack without altering your image--it will remain a psd. I am just curious if the resulting jpeg is generated from the 8-bit layer preview, which might mean the banding will not be visible. In that case, perhaps it is coming from the the 16-bit data, but not visible in 8-bit.

This is purely speculation, both in how photoshop works and what is happening. I have never come across this issue.
 

robertwright

New member
I think it has to be a preview issue- if you have 100's or 1000's of layers as you say in this case there is no way that Photoshop is doing all the math to correctly represent the results of each individual pixel location- it is taking some average or mean value.

However if you think of it this way, at each pixel location there are rgb values, for the sake of argument 150,150,150. Sometimes it is maybe one or two values higher or lower depending on how the layer was rendered by the raw processing. I think the vignetting corrections are producing the effect. Because you have a statistically significant number of layers, any difference in the calculation of each pixel location means that from layer to layer, areas that should be be continuous tone gradients from one set of values to another get bumpy- and then when you flatten the layers and the math is done, sometimes the values calculate higher or lower than the mean.

If this wasn't on a gradient I think you'd not see the issue as you are, and if the gradient was not produced by the vignetting then the shape would be not as it is.

there is another possibility and that is it is an interaction between the vignetting corrections and whatever microlens pattern is on the CMOS- that radial fresnel pattern looks suspiciously like what might be on chip vignetting corrections similar to what leica engineered for its ccd's. I don't know if that 50mp has any sort of radial micro lenses to compensate for off-axis light loss but if it does, and then introduce vignetting corrections and multiply that across 1000 layers it could show as banding, as a guess?.....

you are certainly on the edge case scenario so lots to consider.

perhaps other people with the other manufacturers versions of this chip who also do this kind of stacking could venture a better opinion.
 

Ed Hurst

Well-known member
Thank you once again for the continuing excellent ideas.

Here is what I will do... My goal is to change one thing at a time so I can isolate what helps (assuming anything does, of course!).

1. Using the files I already have (which contain lens corrections, clarity/levels adjustments, etc.) I will apply Paul's method - to see if the process of combining the layers is the issue and whether this different method gets around it.
2. Also using the files I already have, I will take the stacked files and create a JPEG without flattening / merging - as recommended by Will - to see if that produces something akin to the onscreen preview I am seeing before flattening / merging.
3. I will create a new set of files with no processing applied then stack and flatten as per my current process (to accommodate everyone's different ideas above assuming the processing becomes a problem at the flattening stage).
4. The same as 3., but with lens corrections applied (no other processing) - to see if that is the problem - as per Robert's (and others') suggestions above.
5. The same as 3., but with clarity/levels/etc. applied (without lens corrections) - to see if the problem is with this type of processing (as per various suggestions above).
6. If all of the above fails, I will try shooting as many dark frames as actual frames (under the same conditions as the actual shoot); I am keeping this in reserve because intuitively I feel it's less likely to be the issue and also because, as a 'fix', it's painful as it would require a doubling of the shooting time and I am hoping this isn't what is needed :)

All of this will take some time, but once done, I will report back.

Thanks again to everyone!
 

Ed Hurst

Well-known member
I am delighted to say that I have a solution to the problem! This was discovered before completing all of the steps I listed in my last message, so can't claim to have done them all.

I tried the following:
1. Reprocessing all of the raw files without any adjustments that would affect tones (clarity, contrast, curves, etc.) and also without any lens corrections.
2. Reprocessing all of the raw files with the tone adjustments, but not the lens corrections.
3. Reprocessing all of the raw files with the lens corrections but not the tone adjustments.

In each case, I then stacked and flattened the files.

In case 3., the strange artefact occurred. In cases 1 & 2., it did not. So the conclusion I have come to is that the problem arises from lens corrections made during raw conversion. The other tonal adjustments seem to be no problem. As discussed above, the issue is not visible with the files stacked but occurs when the file is flattened (or the layers are merged). So the image one sees on screen before flattening or merging is clearly not the fully processed image (despite appearing very high res) but a partially processed preview. The processing occurs during the flattening/merging and that's when the lens corrections lead to the artefact.

It's a shame, but not the end of the world (after all, one can apply lens corrections at the end to the flattened file if desired). But the main thing is that we have an answer! That's very exciting for me, as the problem was messing with my head!

Thanks again everyone above for your wonderful help. Lots of you suggested the ideas that led to the solution :)

Yours happily,

Ed

P.S. I will post one of the images without the problem over the next day or two when I have time to finish the processing.
 
Last edited:

bindermuehle

New member
Hi Ed,

What you are experiencing is not surprising - in Astrophotography we encounter these sorts of processing issues all the time. It helps to think about what all the processes are doing behind the scenes, then it becomes evident that before you stack and put one pixel on top of the other, you need to be very certain that they depict the exact same part of the picture.

As you've found out, lens correction algorithms shift the pixels in the x-y plane by some amount calculated by the lens projection function. There is one reason why this doesn't match in every frame: The sub-pixel accuracy of your pointing (i.e. the stability of your setup) is not stable enough. If you were able to get the camera to point to the exact same spot with true sub-pixel accuracy, you could apply the lens correction beforehand and then stack, but the end result will be the same. Expressed in mathematical terms, the sum of the corrections in every frame equals to the correction in the summed frame. This is assuming you have sub-pixel accuracy in your image. If you can process an image with a time consuming algorithm once, just do it once... not 100 times for each frame. :)

To give you an idea of the level of precision required to make this happen: the camera can't move more than about 5 microns (5/1000th of a millimeter). You'd need a thermally controlled mount so that uneven heating/cooling of the tripod legs is compensated for, as that causes slight expansion or contraction, you could have no wind at all... you'd essentially be building an astronomical telescope for terrestrial purposes. :)

Having said all that, your photography is stunning! Beautiful shots, and from a fellow Sydneysider, you've motivated me to make a go for vivid Sydney next time with the camera!

Cheeers

- Balt
 
Top