Sofware for down-sizing avoiding demosaicing ?

[Francois]

Hello,

Here are some thoughts: I don't know if it is anything possible, but it looks like it theoretically should be, so I would appreciate the insights of experimented photographers...

The context

I'm planning on buying a D800 to upgrade Full-Frame. D600 would be perfect for me, except I need a reliable autofocus in low-lights situations (I'm not happy on this point with the D7000, and it shares the same autofocus).

The D800 looks great, but 36 MP is far more than I need in most situations. So I will keep most of my shots in a much smaller format, and I'm looking for the best way to do so.

Two problems

I've read here about how a Bayer sensor works
https://www.cambridgeincolour.com/tu...ra-sensors.htm
So for each pixel, there is a "guess" of 2/3 of the colors, which I think gives a worse result than if we had an actual triple sensor (one for each color) for each pixel, like in a Sigma Foveon.

I've also read here about possible problems in down-sizing
https://www.cambridgeincolour.com/tu...ze-for-web.htm

The idea

I have then two opposite operations, one for getting a bigger image, one for getting a smaller one. And of course, the opposition doesn't cancel both degrading effects. Couldn't we skip both operations for a better result?

The idea would be to take 4 raw pixels (1 red, 1 blue, 2 greens), to have one single pixel with the 3 color levels (one green pixel is thrown away, or bot are averaged). This way, I have a smaller image (9 MP for my D800, which is more than enough for almost everything) with a (theoretically) better image quality.

The questions

1/ Is there some sens in this logic ?

2/ Is there a software that allows me to do so?


Before you say it...

I suppose some people will tell me "Why buying a 36MP camera if you do not intend to use them?"
1/ In some cases, I intend to use them (heavy cropping, v. large prints), but most of the time, I wont
2/ I want a Nikon Full-Frame. I have no use for the D4 burst, neither the money. I can’t rely on the D600 autofocus in low-light situations (much use of it).
3/ I'm really interested to know if this method, if possible, could upgrade image quality, compared to a regular down-sizing

Thank you in advance for your thoughts and answers

[revi]

Afaik, the Foveon sensor has its own set of problems (not least a much lower pixel count), so that the net benefit of
the system is not quite as great as the simple description could make you expect.

Another point is that the better demosaicing algorithms can use the information in all channels to estimate each pixel,
which means that the actual information content is somewhere between the manufacturers value (which counts the
total number of photosites) and the number of (4-pixel RGBG) bayer arrays. Also, most cameras still have an anti-
aliasing filter in front of the sensor, which slightly blurs the image, complicating things even more (in theory, part of the
information is still there and can be made visible with a technique called deconvolution sharpening, which is rather CPU-
intensive, say at least 10-20 times longer compared to unsharp masking).

Then, you are going to do some editing on your image presumably. Any photographic editing can only lose information
(not talking about adding watermarks and such), so after editing you will have less information than before.

If you want to do fairly extensive editing, I would stick with RAW format and the full-sized, demosaiced image. If all you
need in certain cases is the 9MP image, with little additional editing, you could try and extract the jpg embedded
in the RAW file*, afaik that one is a quarter-sized image, generated from combining one bayer array in one pixel. If you do
that, you want to pay extra attention to your camera settings (contrast, white balance, sharpening, saturation, ...)
to get the best possible camera jpeg. I don't know how the smaller sized in-camera jpegs are generated, but that also
might be an option to get what you want.

There's one other point to take into account, and that's the dynamic range of your images. A RAW file has typically 12-14
bits/pixel, a jpeg only 8. This (in combination with the in-camera processing) gives you a lot more dynamic range in RAW
compared to jpeg: you can capture a wider range of tones in your image than will show up in the camera jpeg. And my experience
with low light situations is, that they are often also high contrast, i.e. cover a large range of tones (large dynamic range). Starting
from a raw file can give you a much better result in such cases.

You'll probably want to downsize (and not necessarily to 25%) when using raw, so you might see the aliasing problems mentioned
in your second link. However, you need a repeating texture in your image to generate the moiré, and sample it with a frequency
very close to the texture frequency. This happens not all that often in practice, and you can reduce it with a very slight blurring
prior to down-sizing (as explained in the tutorial article), or a change in your final size. So not really something to worry about too
much.

Hope this helps a bit,

Remco

*: for instance, dcraw can do that, using the '-e' option. And yes, I really prefer capturing in RAW format and resizing afterwards.
It takes more time, but I get closer to what I want/thought I saw.

[Francois]

Hello Remco,

Thanks for taking the time of the reply !

Yes, I heard the Foveon's sensor is far from perfect, I just used it as an illustration of the idea.

Thank you for the information about the 1/4 size jpg embedded, I didn't know about that. In a way, it does what I'm looking for : combining the 4 "Bayer-pixel" into a single one.

But I love processing the raw files : it gives so much flexibility ! In fact, I import even my most basic shots into LR just to have minor adjustments. And for me who loves low-light photography, 12 bit is a minimum.

I know I can look for the optimum workflow working with the full size raw image (what I'm currently doing). But what I don't like in LR is that the down-sizing is done at the export, so after all the fine tuning of the image. I love this software, that easily covers 95% of my needs, and I'd like to stick to it.

What I'd really love to do is creating a file like this embedded jpg you told me about, that doesn't need demosaicing (but is made of the combined 4 Bayer Pix.), but with the bit rate of a raw file (it would be a TIFF I guess ?) to keep all the flexibility in post processing.

Is there any software to do so ?

[dje]

Hi Francois

I'm not aware of any software that can do what you are suggesting, but I'm no expert !

I have heard of a similar technique to what you are describing called "pixel binning" where a group of pixels are somehow combined in camera to produce a smaller number of larger pixels. But the manufacturers are very secretive about their techniques, presumably for commercial reasons. So it's a little difficult to get reliable information on it. I think Fuji may have used it a bit in their high end still cameras and I have seen it referred to being used for video capture in some DSLR's (for HD video, you only need about 2MP). However I think the motivation here is mainly improved noise performance in low light conditions and also reduced video processing load in the case of video capture.

To be honest, I think you are being overly concerned about this issue. I also have a gut feeling that de-mosaicing may be more effective if it is done at the finest pixel density available. From all accounts, the D800 is an excellent performer when it comes to image quality. The downside I suppose is the size of the raw files but storage is cheap these days.

Dave

[ajohnw]

The general impression of the Fovian sensor is good but blues may vary. The reduced pixel count doesn't really matter compare with a bayer set up with 4 times as many pixels. The bayer aspect is interpolated out when the raw file is developed. This can cause artefacts. The 2 green pixels in a bayer matrix are down to colour correction. It has to do with trying to mimic the eyes response to colour. Our eyes are much more sensitive to green than blue or red so dropping one green isn't really an option.

I don't use any of Adobe's products but find that after a shot has been reduced all is ok apart from the need for a certain amount of sharpening. The amount varies and it's easy to apply to much especially on an uncalibrated monitor. It's needed as detail has been lost and must be lost to a certain extent when a shot is reduced. I would hope that you can do further processing on the jpg's that I assume you end up with.

Another option that will probably solve you problem most of the time which tends to cause frowns on here is to use your camera in jpg mode. If you look at the review of your camera on dpreview you will see that your camera has a number of what I call camera curves. These do what is normally done from raw within the camera. Say your camera has a 12bit colour space - 3 12bit channels - a jpg has only 8 bit colour channels. The camera curves map the 12 bits into 8 in a variety of ways. The number and type of camera curves varies from one camera to another but most compress both the dark and light ends of the cameras dynamic range to some extent. Often there will be options to go more into the dark or more into the light end of things and various other options. In fact some cameras now have user settings. As the information is in the jpg it's possible to bring it out in the normal way with little loss of flexibility in real terms. JPG is often stated as being a lossy compression but many packages will save at 100% quality and there is always PNG anyway or tiff for that matter. In terms of saving what is effectively all of the information your camera captured you will probably find that the raw file is much more compact than a tiff.

There seems to be a general failure of people realising that the software in cameras is far more sophisticated than it used to be. Even Canon's can compress high lights now rather than clip them off. Using camera jpg's effectively is a case of taking the time to find out just how the camera behaves under different circumstances and what each of the various facilities in them actually do. I generally shoot jpg + raw with the jpg at the highest quality level. I'm not the greatest of photographers but my failing is mostly post processing. It's improving and I have learnt a lot from several people on here. Most of the time the camera jpg's are good enough for post processing and working from raw would make little or no difference on the vast majority of shots. My biggest problem in post processing is adding vibrance if I feel a shot needs it. Some times I can, some times I can't - raw or jpg. Tonal range can always be boosted where needed from a jpg if the info is there and in real terms when people are working on an 8bit monitor raw has similar problems of only being able to show image is 256 tonal steps on 3 colour channels.

John
-

[Francois]

Hi Dave,

Thanks for the answer.

However I think the motivation here is mainly improved noise performance in low light conditions and also reduced video processing load in the case of video capture.

That's exactly what I'm looking for. Plus I thought there shoud also be some improvement in sharpness.

To be honest, I think you are being overly concerned about this issue.

Not wrong In fact, in can live with the current method. But I just thought it makes sense, and if there is an easy to do it, I’d love to. But I'm no pro, and the hypothetical gain would surely be marginal, so this is not crucial. Well, even if I'm not a pro, I’d like to tend to be (as for the technique), so this is why I'm asking.

[Francois]

Hi John,

Thank you for the answer.

Another option that will probably solve you problem most of the time which tends to cause frowns on here is to use your camera in jpg mode.

Well, for now, this is really not something I would do. To be honest, my photographic skills are not good enough to allow me taking pictures without the security of a raw file. Sometimes I must save a picture with +/- 2 stops exposure correction, which is completely impossible with a jpg file. I also forget about my white balance, leaving it to the camera, knowing that in the 2-3% cases it fails I can easily correct it in post-production.

There seems to be a general failure of people realizing that the software in cameras is far more sophisticated than it used to be

In fact, I may be one of these people, and I don't know much about the in-camera processing. I didn't even customize it, doing all the work myself in PP. Here are 2 ideas that make me want to use a dedicated software and stay away from the in-camera jpg.

1/ I assume I will always do better than the camera to choose how I want the picture to look, being the judge and having all the time to define every single parameter
2/ The raw file allows me to benefit from further improvement of the algorithm: I'll be able to reprocess them. A jpg technically "frozes" the picture in time

In conclusion, if I can't find the software I was talking about, I'll stick to the currente raw processing method. But thank you for the suggestions.

[Francois]

I have heard of a similar technique to what you are describing called "pixel binning" where a group of pixels are somehow combined in camera to produce a smaller number of larger pixels.

V. useful to have the exact term, much easyer to Google, thanks. I foud some interesting articles, but nothing about what I'm looking for.

http://focusonphotography.blogspot.c...g-is-here.html

[revi]

(...)
But I love processing the raw files : it gives so much flexibility ! In fact, I import even my most basic shots into LR just to have minor adjustments. And for me who loves low-light photography, 12 bit is a minimum.

I know I can look for the optimum workflow working with the full size raw image (what I'm currently doing). But what I don't like in LR is that the down-sizing is done at the export, so after all the fine tuning of the image. I love this software, that easily covers 95% of my needs, and I'd like to stick to it.

What I'd really love to do is creating a file like this embedded jpg you told me about, that doesn't need demosaicing (but is made of the combined 4 Bayer Pix.), but with the bit rate of a raw file (it would be a TIFF I guess ?) to keep all the flexibility in post processing.

Is there any software to do so ?

Well, you can do the down-sizing straight after the demosaicing step, and work on the saved copy. But I think that's not optimal, as the modern demosaicing algorithms manage to use information of all colours to fill in the missing colour information for each pixel. Have a look at this article from 2009, where various demosaicing routines are compared with an in-camera jpeg.

If you really want to play with a half-sized image, you could try dcraw with the -h and -6 options (but that's a command-line program, so perhaps a bit more difficult to use than GUI-based programs, in the beginning): -h gives you a half-sized image and is claimed to be twice as fast as the simplest demosaicing for full size, -6 shuld give you 16-bit output.
Note that you never get 12-bit output, as that's a very difficult size to work with on standard CPU's, which are constructed to work with multiples of 8 bit. Any processing on other data sizes is very time-consuming.

[Francois]

Have a look at this article from 2009, where various demosaicing routines are compared with an in-camera jpeg.

I had a look, but I don't really get it. The image is v. bad. Am i missing something ? Is the way the image is shown supposed to help see the differences better ?

Anyway, the hole idea was avoiding demosaicing. If I can't, I'll stick to a simple down-sizing export in LR.

[dje]

Francois I found this article from Fuji on pixel binning which gives an insight in to their approach. It seems that they use a specially designed sensor with a different configuration of the bayer filter. This is intended to avoid some problems with colour accuracy after de-mosaicing that apparently were caused by binning on a conventional sensor. I think this supports my view that de-mosaicing is best done with the pixels as close as possible (ie the full resolution image).

Dave

[Francois]

Intersting article, thanks.

I also read that to be effective, the binning has to be done in-camera, at the sensor level. If it's done in PP, it combines read noise, and it is less effective.

Reading a little bit on the subject also makes me understand why the method I was looking for is wrong : avoiding demosaicing, I would maybe have a slight gain in sharpness, but none in noise. I would even be much worse, as I would have lost so much information.

For a gain in high iso, I could see how the binning is done in-camera with the D800. It looks that maybe John was right, depending on how Nikon produces smaller JPG files. But I'd lose the high bit rate of raw... Well, maybe only field-testing will answer this.

[dje]

Well, maybe only field-testing will answer this.

Hope you enjoy thr D800 !

Dave

[revi]

I had a look, but I don't really get it. The image is v. bad. Am i missing something ? Is the way the image is shown supposed to help see the differences better ?

Anyway, the hole idea was avoiding demosaicing. If I can't, I'll stick to a simple down-sizing export in LR.

In that case, the only program I know that can do that is dcraw (but I'm not that well informed about raw converters)

[pnodrog]

Have you thought of buying a second hand D3s? It has superb focusing and low light performance and is only 12mpx.

[Manfred M]

You might want to look at On-one software's Perfect Resize for a more advanced resizing tool. It is usually used to upsample an image (I believe they use fractals, rather than interpolation), but it does a very nice job.

http://www.ononesoftware.com/products/perfect-resize/

That being said, I've shot the D800 for just over a year now and have never had a problem with downsizing an image and not having it come out looking great.

[revi]

I think there's a misunderstanding somewhere...

I do not see why you would get a better image from using a 'binned' image (of 1/4 the full size),
instead of working with the full sized demosaiced image.

I tried to explain my reasons, but those could very well be wrong or not clearly explained.

[Manfred M]

I think there's a misunderstanding somewhere...

I do not see why you would get a better image from using a 'binned' image (of 1/4 the full size),
instead of working with the full sized demosaiced image.

I tried to explain my reasons, but those could very well be wrong or not clearly explained.

Remco; you are absolutely correct in what you say; but you can still get a result that is "good enough" working from a binned image. If storage space is a priority, it will give adequate results; after all, I have gotten some stunning images that I edited out of 4MP and 5MP point & shoot cameras that output jpegs only.

[Francois]

Have you thought of buying a second hand D3s? It has superb focusing and low light performance and is only 12mpx.

That's a question I asked myself... But even a used D3s is more expensive than a new D800. You also don't get the amazing dynamic range of the new sensors. And when I said 9MP is good for me, it's after PP, so sometimes I like to have more. The 16MP of the D7000 work quite well for me.

That being said, the D3s is an amazing camera I've been dreaming about for a while, and if I could find it under 40k click for the same price as a new D800, I'd go for it.

[Francois]

You might want to look at On-one software's Perfect Resize for a more advanced resizing tool. It is usually used to upsample an image (I believe they use fractals, rather than interpolation), but it does a very nice job.

http://www.ononesoftware.com/products/perfect-resize/

That being said, I've shot the D800 for just over a year now and have never had a problem with downsizing an image and not having it come out looking great.

Thank you for the information.

The idea of avoiding demosaicing was to not have to opposite algorithm operations (one creating more information by "filling the gap" of the Bayer sensor, the other re-combining this extrapolated information for a smaller image). But I understand ther was a wrong thinking, as there would be a huge loss of information.