Hot/Stuck pixels

[Pearlman]

I just bought a canon XSI and i noticed some stuck pixels in every long exposure i take. there's 4 of them. i know stuck pixels is something that we all must deal with but how much is too much? what do you guys think, is my camera a keeper? i posted a 1600 ISO 30 sec dark frame. you can see the three red pixels near the center. the one on the bottom right is harder to see.

[Colin Southern]

Welcome to the forums

Hot pixels are quite common with long exposures - and only 4 is nothing to worry about. If you haven't already turned on long-exposure noise reduction, do it now, and you'll probably find that they disappear altogether (although on your camera it'll probably double the time it take to complete an exposure) (1/2 exposure, 1/2 dark frame subtraction).

Also, if you're shooting RAW - and post-processing with Adobe ACR - then it should automatically remove them for you. (I shoot with a 1Ds3 and I have a lot more than you, but with ACR they just disappear and I don't think twice about them).

Hope this helps,

[shreds]

Its a common phenomenon that is unlikely to be resolved by getting another camera (It may have five or six stuck pixels) Unless you are shooting time exposures all the time, then don't lose too much sleep over such an issue.

[Pearlman]

Thanks for the replies guys. that seems to be the consensus in the other forum i posted too. but once you see them, you just focus on them in every picture.

so what ive found out is that canon's internal software, i think specifically when you have the sensor cleaning mode on, maps those pixels out in short exposures. but the long exposures are another story. you can turn on the long-exposure noise reduction which will remove those pixels along with any other noise but doubling the time between exposures.

i plan on doing some astrophotography with this camera so this is definitely an issue for me. most the time i will have the long exposure noise reduction turn on when i do this, or perform my own dark frame subtraction but there are times when im not going to do this. it just takes too much time when taking hundreds of pictures.

anyway, im going to contact cannon tommorow and see what they think and ill probably get them to map those pixels out. ill post again when i talk to them.

[Colin Southern]

Out of interest, which format are you shooting in, and what package are you using for post-processing?

I see quite a few hot pixels from my 1Ds3 when doing night work - they're visable in bridge until ACR kicks in and refreshes the image with it's processing defaults. After that they just "disappear" (I think it's a fairly trivial task for the software to filter them out).

My understanding is that the ICS (Integrated Cleaning System) is for dust spots - not sure if or how if handles hot pixels.

[Pearlman]

i was shooting in both raw and jpg. the pixels look much worse in raw but i can definitely see them in the JPGs. and i noticed the same thing that bridge and acr both remove the pixels in the raw file very nicely. its a great feature, but it actually kind of pisses me off at the same time because you cant stop it from processing. it would be nice to veiw the raw file before processing. i dont know why but it would be nice to see how the camera sees and also so you can see these pixels.

about the sensor cleaning mode, im not positive but ive heard it maps out stuck pixels in short exposures. quote from someone in another forum:

"Canon executes a dead pixel detection exposure whenever you initiate "Manual Sensor Clean". It then uses this dead pixel map to average out the dead pixels before writing the RAW file. The mapping/removal is only for short exposures, so long exposures will have additional hot pixels. Canon does not admit (or deny) the existence of this feature anywhere."

and ive also heard a story where someone with a stuck pixel in all pictures was told to turn on the sensor cleaning mode and it worked. who knows if its true, we need more proof.

in my case i dont see the stuck pixels in my normal pictures, only in long exposures. so its either that they map out those pixels somehow, or i have stuck pixels that have a slow leak. i cant be sure... i will say that the stuck pixels do get dimmer with shorter exposures. Who knows whats going on....

and i also talked to canon today and they didnt know anything. they just told me to send it in. i asked them what they would do with the camera, and they didnt know that either. their phone technical support is completely separate from the repair lab. im not sure if im going to take my chances and exchange it or send it in just to see whats going on...i think im probably going to send it in. ill post my results.

oh, how many stuck pixels do you have colin? not just the regular noise but the ones that are consistently hot. can you post a dark frame please. thanks for the support too, colin

[Colin Southern]

"it would be nice to veiw the raw file before processing. i dont know why but it would be nice to see how the camera sees and also so you can see these pixels."

I don't know of any way to turn this "hot pixel detection" off in ACR (haven't looked), so possibly the only other way would be to use a utility to look at the actual pixel data in the file (I don't know of any, but I know of a chap on usenet who was able to give someone the exact row and column number of a hot pixel for someone else, so I guess it's possible that one exists).

"Canon does not admit (or deny) the existence of this feature anywhere."

... Hmmm - it MUST be true then!

"in my case i dont see the stuck pixels in my normal pictures, only in long exposures. so its either that they map out those pixels somehow, or i have stuck pixels that have a slow leak. i cant be sure... i will say that the stuck pixels do get dimmer with shorter exposures. Who knows whats going on...."

I reclined in my favourite chair for over 2 hours last night with 1Ds3 in hand doing some testing of such things ... interesting results.

With ISO maxed out, ...

- a 1 minute exposure produces a few hot spots (Red, gree, or blue)

- a 10 minute exposure produces lots and lots and lots more lots, but the vast majority on the right hand side of the screen (and the further to the right, the more dense they were) - almost like dim embers of a fire - reasonably dense, and mostly red)

- a 10 minute exposure with long exposure noise reduction turned on killed each and every one of them - quite impressive actually.

- a 10 minute exposure with high-ISO noice reduction on (but long exposure noise reduction off) reduced the "glow" a little - perhaps by about 20%

- a 10 Minute exposure at F2.8 produced slightly less noise that at F22 - can't figure that one out.

- a 10 minute exposure at ISO 100 produces virtually nothing (may have been a few hot spots, but they weren't obvious)

The most interesting this of all is that the 1Ds3 allows you to take more shots whilst it's still doing what I assume is the dark field subtraction of a previous shot - I need to do more testing here as this is actually quite interesting; I did a 10 minute exposure and then 1 minute after the shutter closed I started another 10 minute exposure. 10 minutes after the shutter closed on my 2nd exposure I was then presented with both images. I assumed initially that it probably just aborted processing on the first and carried it out on the second, but BOTH appeared to have had noise reduction applied. Now I'm left wondering if 1 minutes worth of reduction on the first shot was enough to eliminate hot pixels or if the camera is smart enough to apply the same dark field subtraction to both frames (which would make sense). More testing required!

"oh, how many stuck pixels do you have colin? not just the regular noise but the ones that are consistently hot. can you post a dark frame please. thanks for the support too, colin "

To be honest, I'm not sure about the "terms of reference", - but on a 1 minute exposure where the alleged stuck pixels were red, gree, blue, or even a few "white" ones, I seem to have about 30. Probably not much point in posting a frame as it gets down sampled so much it probably makes the exercise a bit futile. Happy to upload one to sendthisfile.com and eMail you a link to it so you can download the original though if you want - just tell me what exposure length and ISO you'd like though.

Cheers,

Colin - pbase.com/cjsouthern

[Dave Humphries]

This post as much to see the new signature as anything helpful so onto my random musings ..

From what you've discovered Colin; in being able to take more pictures whilst earlier ones are processing, I would guess that the multishot buffer is being used. It still does point to there possibly being one dark field frame held somewhere (probably the buffer), which gets re-used as needed. This either persists for the duration of camera use, or at least until all such images have been processed. I would assume it gets wiped if you switch the camera off and on. The actual noise encountered will depend on many things like ambient temperature, etc., so it would need to be refreshed each shooting session, at least.

But that leaves me wondering why the subsequent images still take so long to process, if they're not re-exposing the sensor for the same time as the exposure.

I can understand 100 ISO giving less noise than say 1600 (because less gain in the signal path where it is analogue), but as you say, the f2.8/f22 effect is weird.

I might hazard a guess that my camera with it's tiny 1/2.5" sensor is limited to a 30 second max exposure (and no B) for reasons such as this, I'd probably have a raging bush fire rather than your dim embers!

Regards,

[Pearlman]

Thats interesting. i would say that if the camera is applying dark frame subtraction to multiple pictures with the camera only taking one dark frame, rustyshutter is on the right track. thats a real cool feature if it is. try testing it again but with 30 second exposures because 10 minutes is a long time to wait

and yes the terminology is confusing. from what ive found out it is as follows:

Stuck pixel = a pixel that always reads high (maximum) on all exposures.
Hot pixel = a pixel that reads high on longer exposures.
Dead pixel = a pixel that reads zero (black) on all exposures.

this is a good website even though its talking about CCD sensors
http://webpages.charter.net/bbiggers...ot_pixels.html

reading it again i think that i have hot pixels. these 4 pixels have a higher than normal charge leakage and come up blazing hot after 10 seconds and somewhat hot after 5 seconds. allthough i cant be sure because the camera might map them out in short exposures or it might not.

anyway, Colin, try a dark frame exposure of 30 seconds at ISO 1600 without the ISO or long exposure noise reduction. for me that seems to show my hot pixels very well without much other noise. a 2 minute exposure for me shows TONS of hot pixels. and actually kind of masks the 4 that i'm talking about, although you can still make them out.

and i just checked your site and really really beautiful pictures colin. New Zealand is such a beautiful place and you capture it very nicely.

[shreds]

Last night I went to an interesting talk by Paul Money:

www.astrospace.co.uk

and ironically hot/stuck pixel issues were briefly discussed. It is a great shame that this discussion had not taken place a few days ago or I would have asked him.

I will ask him to take a look on the forum and see if he has anything to add to the debate, considering extremely long exposures are his speciality and he does now do digital.

[Pearlman]

Thanks shreds, hopefully Paul can shed some light on the matter.

i wanted to mention one more thing about taking the dark frames. take the frames in both jpg and raw. the jpg can sometimes hide them a little bit, the raw does not. the only way i found that you can see these hot pixels in the raw is to view it in bridge and capture a screen shot before it processes them out. you can really see them in the raw file much better. heres my screen shot with one of the pixels zoomed in on. they look like little squares. in the jpgs they can look like American footballs or like a little X.

notice the three red and the green one on the bottom.

[Colin Southern]

"the only way i found that you can see these hot pixels in the raw is to view it in bridge and capture a screen shot before it processes them out."

Even then it makes me wonder if we're seeing them all ... Most cameras probably have close to 10 times the number of pixels that our screens have (or more) - it probably all depends on how the software renders the image in that if the monitor is only displaying 1/10th of the true data - and we're seeing 4 pixels, then possibly there could be another 36 that are being rendered out when displayed (and even that's assuming it's using the entire screen).

Personally, I really don't get too upset by them as they're removed without any user intervention and don't appear to detract from the final image in any significant way.

Cheers,

Colin - pbase.com/cjsouthern

[shreds]

Here's what Paul had to say:


I'm no expert (yet) on deep sky digital astrophotography but I'm
steadily getting there :-)

On long exposures there are hot pixels and what we do is to take a dark
frame by keeping the lens cap on and doing the same length of exposure
as the image you want to take. Ideally you should take several then
average them out using whatever editing prog you use. You then use any
photo editing software to subtract the dark frame from the main image to
take out the hot pixels. There's more to it and I myself am in the
process of learning it all but that's the basic idea.

As a rule the easiest form of deep sky astrophotography is to take a
series of shortish (up to a minute or so) exposures. Use one of the
dedicated bits of software out there (K3CCDTools is a good one or
Registax) to both combine the images into the equivalent of a single
long exposure whilst at the same time the software also subtracts your
dark frame to remove the hot pixels.

I noted that Colin mentions the following:

****
- a 10 minute exposure produces lots and lots and lots more lots, but
the vast majority on the right hand side of the screen (and the further
to the right, the more dense they were) - almost like dim embers of a
fire - reasonably dense, and mostly red)
****

The right hand side of the screen is (if I remember right from a good
friend of mine) thermal noise from the chip - I think it's on the side
where the chip is being 'read out' from - I could be wrong. When we do
exposures that long to bring that problem up then you still use the dark
frame subtraction to remove it but it can degrade the raw data you're
trying to capture. That's part of the reason why we tend to do shorter
exposures - but lots of them.

Feel free to post the above to the forum if you think it will help.
Remember though that I myself am still learning :-)

Incidentally there is a good tutorial on how to do astroimaging by my
friend Pete Lawrence in the Dec issue of BBC Sky at Night Magazine -
covers the basics well. Pete is the imaging expert for the mag and I am
the Reviews Editor.

[Colin Southern]

Thanks for that - very interesting.

I wonder how variable the dark field subtraction shot is - like for a given length exposure at a given ISO, I wonder if I could just take 1 reference dark field exposure shot and use it forever more when doing long exposures at that ISO and exposure length?

At the moment I'm not doing much in the way of long night exposures - probably out to a max of 20 min or so at no more than ISO 400. What compounds the times for me is that I like the star effect you get from lights at small apertures - and end up shooting at F16 & F22.

Cheers,

Colin - pbase.com/cjsouthern

[Dave Humphries]

~ I wonder if I could just take 1 reference dark field exposure shot and use it forever more when doing long exposures at that ISO and exposure length?

Hi Colin,

I suspect not if the ambient temperature is more than say, 5 deg (Celsius) different.
Someone in the UK/Europe might like to try an experiment, do two dark field exposures (as photos to keep), one indoors, assuming that's around 20 deg, and another at outside temps at least 10 or 15 deg different, but do let the camera 'soak' at the outdoor temperature first to make a fair comparison. For northern hemi people, outdoors is going to be colder this time of year, but as for Nelson, NZ, I wouldn't know, it may not be different enough, hence my suggestion as to who tries it.

This should prove, or otherwise, whether your comment above is valid.

Of course another factor for long term re-use is that as CCDs age I believe more pixels may be afflicted, but I'm not sure if they'd go bright or just die (go dark).

Disclaimer
As usual, that's just me bringing warped engineering principles to bear on a subject I have no experience of

Regards to all, Dave

[Colin Southern]

Hi Dave,

I suspect you're right - temperature is one of the variables I'd like to test at some time when I get a spare evening.

Cheers,

Colin

[shreds]

Of course another factor for long term re-use is that as CCDs age I believe more pixels may be afflicted, but I'm not sure if they'd go bright or just die (go dark).

I think that the principle of sensor ageing will lead to a change in the number of pixels affected over time, so as a once and for all solution, it may not be ideal. You would either need to deal with additional affected pixels, or just do another dark field subtraction.

[Colin Southern]

I think you're probably right. First test I want to do though is to see just how bad the problem is at 100 ISO for various exposures.

I was thinking of doing a mega exposure throughout the night - including the moon - but will have to do a bit of testing first.

Cheers,

Colin

[Pearlman]

hey guys, i havent looked at the forum for awhile and i have a few things to add.

QUOTE:

"As a rule the easiest form of deep sky astrophotography is to take a
series of shortish (up to a minute or so) exposures. Use one of the
dedicated bits of software out there (K3CCDTools is a good one or
Registax) to both combine the images into the equivalent of a single
long exposure"

i have heard of stacking photos, and initially i though that stacking would essentailly increase the overall exposure time but thats not entirely true. heres some forums on the subject:

http://www.cloudynights.com/ubbthrea...v=#Post1696629


this one is more in depth but a little confusing:
http://www.cloudynights.com/ubbthrea.../o/all/fpart/1

so basicaly from what ive read from these forums and from my VERY limited experience is that stacking images is all about noise. all stacking images does, no matter if you sum, average or meadian the layers, is decrease the signal to noise ratio and wont brighten the image. but having a lower SNR allows you to stretch the histigram more to reveal fainter details. so stacking many short subexposures does decrease the SNR but one long exposure would give the lowest SNR. but there are some limits to how long you can keep the shutter open though like, how well your polar alignment is, light pollution, and things like that. another way to reduce noise is by subtracting a dark frame from the sub exposures.


Quote:

"I wonder how variable the dark field subtraction shot is - like for a given length exposure at a given ISO, I wonder if I could just take 1 reference dark field exposure shot and use it forever more when doing long exposures at that ISO and exposure length?"


yes you can. many people do this. the key is to match the ISO and the exposure length and the temperature to the picture you want to remove the noise from. some photographers take the dark frames at the same ISO and exposure at the end of the night when they are breaking down their equipment as to match the temp. some take dark frames in there refrigerator before hand to match the temp. the thing with dark frames though is that one is not adequate. heres a link on that subject:

http://www.cloudynights.com/ubbthrea...a456a2f3a2ada4

basically they say that 1 dark will actually add noise. it will get rid of those hot pixels in my camera for example but will add more random noise. i found this hard to believe and i decided to check my pics and the ones where i used the in camera long exposure noise reduction did indeed look like they had more noise. i only checked one picture so i may be mistaken. try it yourself. anyway, most people seem to average about 25 darks for their dark frame subtraction. these people seem to take as many pictures of the back of there lens cap as they do take pictures of the sky. what a pain in the neck.

i hope this helps. and if your interested in astrophotography i suggest that you join the Cloudy Nights forum if you arnt already a member. Everything you need to know is there.

[Colin Southern]

hey guys, i havent looked at the forum for awhile and i have a few things to add.

Thanks Pearlman - very interesting. Next time don't leave it so long between visits!