A question came to mind recently while doing extreme editing on one of my shots. I’m wondering about Sidecar or xmp or whatever the files are called that are created when doing non-destructive editing. I’m talking about ACR or Lightroom, but I’m sure there are others cases where these files are created.
I take it that every time a slider is moved or any little tweak is done information is added to these files. I have a reset button in Lighroom that appears to delete the information in the file and send me back to the original, but I am wondering if the “Reset” is just appended to what is there already.
Do these files just keep growing and growing?
Is there any way to check the size of the files? (In a DNG format in particular but others also)
Can they become so large and convoluted that they could cause the computer to become unstable or overworked trying to read it all?
Could the file sizes and the code that’s in them make editing take longer or slow things down?
Is it something that should be taken into consideration when editing and sliding sliders up and down and back and forth and back again? Every move must add code to the file, but is it possible for these files to become so cumbersome and convoluted that they would be detrimental in any way?
Just curious
Wendy
Question about sidecar/xmp files
Printable View
-
20th March 2010, 06:40 PMScoutRQuestion about sidecar/xmp files
-
20th March 2010, 08:09 PMColin SouthernRe: Question about sidecar/xmp files
Hi Wendy,
Avoiding sidecar files is THE main reason I use DNG format. Interesting question - I don't know the answer - but all I can say is it's never been a problem for me. I don't use LR - but I imagine you could do a few tests yourself to see what it does - it can be your assignment for the week :) -
20th March 2010, 08:13 PMScoutRRe: Question about sidecar/xmp filesLOL, so many assignments, so little time. :( I convert to DNG when I upload from the camera. How would I go about checking this. Is there some way I can actually see the file with the editing info, or would I just go by the changes in filesize as I am editing. I'd love to find out, but I don't know where to start.Quote:
but I imagine you could do a few tests yourself to see what it does - it can be your assignment for the week :)
Edit: sidecar files are seperate (I think) can they actually be opened and read? -
20th March 2010, 08:38 PMDave HumphriesRe: Question about sidecar/xmp filesOh yes; any text editor will do, here's part of the content of one I found, it's not mine, must have been something I PP'd for someone. I think it was Pops' bird feeder one.Quote:
Originally Posted by ScoutR
Boring, isn't it?Quote:
<rdf:Description rdf:about=""
xmlns:tiff="http://ns.adobe.com/tiff/1.0/">
<tiff:Make>NIKON CORPORATION</tiff:Make>
<tiff:Model>NIKON D40</tiff:Model>
<tiff:Orientation>1</tiff:Orientation>
</rdf:Description>
<rdf:Description rdf:about=""
xmlns:exif="http: //ns.adobe.com/exif/1.0/">
<exif:ExifVersion>0221</exif:ExifVersion>
<exif:ExposureTime>1/640</exif:ExposureTime>
<exif:ShutterSpeedValue>9321928/1000000</exif:ShutterSpeedValue>
<exif:FNumber>56/10</exif:FNumber>
<exif:ApertureValue>4970854/1000000</exif:ApertureValue>
<exif:ExposureProgram>2</exif:ExposureProgram>
<exif:ISOSpeedRatings>
<rdf:Seq>
<rdf:li>200</rdf:li>
</rdf:Seq>
</exif:ISOSpeedRatings>
<exif:DateTimeOriginal>2010-01-01T12:00:50.10Z</exif:DateTimeOriginal>
<exif:DateTimeDigitized>2010-01-01T12:00:50.10Z</exif:DateTimeDigitized>
<exif:ExposureBiasValue>-4/6</exif:ExposureBiasValue>
<exif:MaxApertureValue>50/10</exif:MaxApertureValue>
<exif:MeteringMode>5</exif:MeteringMode>
<exif:LightSource>0</exif:LightSource>
<exif:Flash rdf:parseType="Resource">
<exif:Fired>False</exif:Fired>
<exif:Return>0</exif:Return>
<exif:Mode>0</exif:Mode>
<exif:Function>False</exif:Function>
<exif:RedEyeMode>False</exif:RedEyeMode>
</exif:Flash>
<exif:FocalLength>2000/10</exif:FocalLength>
<exif:SensingMethod>2</exif:SensingMethod>
<exif:FileSource>3</exif:FileSource>
<exif:SceneType>1</exif:SceneType>
<exif:FocalLengthIn35mmFilm>300</exif:FocalLengthIn35mmFilm>
<exif:CustomRendered>0</exif:CustomRendered>
<exif:ExposureMode>0</exif:ExposureMode>
<exif:WhiteBalance>0</exif:WhiteBalance>
<exif:SceneCaptureType>0</exif:SceneCaptureType>
<exif:GainControl>0</exif:GainControl>
<exif:Contrast>2</exif:Contrast>
<exif:Saturation>2</exif:Saturation>
<exif:Sharpness>2</exif:Sharpness>
<exif:SubjectDistanceRange>0</exif:SubjectDistanceRange>
<exif:DigitalZoomRatio>1/1</exif:DigitalZoomRatio>
~~~
<exif:UserComment>
<rdf:Alt>
<rdf:li xml:lang="x-default">(c) Pops Carter www.PopsPhotos.us</rdf:li>
</rdf:Alt>
</exif:UserComment>
~~~
<rdf:Description rdf:about=""
xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/">
<crs:RawFileName>DSC_2178.NEF</crs:RawFileName>
<crs:Version>5.4</crs:Version>
<crs:WhiteBalance>Custom</crs:WhiteBalance>
<crs:Temperature>5700</crs:Temperature>
<crs:Tint>-19</crs:Tint>
<crs:Exposure>+1.45</crs:Exposure>
<crs:Shadows>20</crs:Shadows>
<crs:Brightness>+40</crs:Brightness>
<crs:Contrast>+50</crs:Contrast>
<crs:Saturation>+50</crs:Saturation>
<crs:Sharpness>15</crs:Sharpness>
<crs:LuminanceSmoothing>100</crs:LuminanceSmoothing>
<crs:ColorNoiseReduction>100</crs:ColorNoiseReduction>
<crs:ChromaticAberrationR>0</crs:ChromaticAberrationR>
<crs:ChromaticAberrationB>0</crs:ChromaticAberrationB>
<crs:VignetteAmount>0</crs:VignetteAmount>
<crs:ShadowTint>0</crs:ShadowTint>
<crs:RedHue>0</crs:RedHue>
<crs:RedSaturation>0</crs:RedSaturation>
<crs:GreenHue>0</crs:GreenHue>
<crs:GreenSaturation>0</crs:GreenSaturation>
<crs:BlueHue>0</crs:BlueHue>
<crs:BlueSaturation>0</crs:BlueSaturation>
<crs:FillLight>0</crs:FillLight>
<crs:Vibrance>+25</crs:Vibrance>
<crs:HighlightRecovery>0</crs:HighlightRecovery>
<crs:Clarity>+60</crs:Clarity>
<crs:Defringe>0</crs:Defringe>
<crs:HueAdjustmentRed>0</crs:HueAdjustmentRed>
<crs:HueAdjustmentOrange>0</crs:HueAdjustmentOrange>
<crs:HueAdjustmentYellow>0</crs:HueAdjustmentYellow>
<crs:HueAdjustmentGreen>0</crs:HueAdjustmentGreen>
<crs:HueAdjustmentAqua>0</crs:HueAdjustmentAqua>
<crs:HueAdjustmentBlue>0</crs:HueAdjustmentBlue>
<crs:HueAdjustmentPurple>0</crs:HueAdjustmentPurple>
<crs:HueAdjustmentMagenta>0</crs:HueAdjustmentMagenta>
<crs:SaturationAdjustmentRed>0</crs:SaturationAdjustmentRed>
<crs:SaturationAdjustmentOrange>0</crs:SaturationAdjustmentOrange>
<crs:SaturationAdjustmentYellow>0</crs:SaturationAdjustmentYellow>
<crs:SaturationAdjustmentGreen>0</crs:SaturationAdjustmentGreen>
<crs:SaturationAdjustmentAqua>0</crs:SaturationAdjustmentAqua>
<crs:SaturationAdjustmentBlue>0</crs:SaturationAdjustmentBlue>
<crs:SaturationAdjustmentPurple>0</crs:SaturationAdjustmentPurple>
<crs:SaturationAdjustmentMagenta>0</crs:SaturationAdjustmentMagenta>
<crs:LuminanceAdjustmentRed>0</crs:LuminanceAdjustmentRed>
<crs:LuminanceAdjustmentOrange>0</crs:LuminanceAdjustmentOrange>
<crs:LuminanceAdjustmentYellow>0</crs:LuminanceAdjustmentYellow>
<crs:LuminanceAdjustmentGreen>0</crs:LuminanceAdjustmentGreen>
<crs:LuminanceAdjustmentAqua>0</crs:LuminanceAdjustmentAqua>
<crs:LuminanceAdjustmentBlue>0</crs:LuminanceAdjustmentBlue>
<crs:LuminanceAdjustmentPurple>0</crs:LuminanceAdjustmentPurple>
<crs:LuminanceAdjustmentMagenta>0</crs:LuminanceAdjustmentMagenta>
<crs:SplitToningShadowHue>0</crs:SplitToningShadowHue>
<crs:SplitToningShadowSaturation>0</crs:SplitToningShadowSaturation>
<crs:SplitToningHighlightHue>0</crs:SplitToningHighlightHue>
<crs:SplitToningHighlightSaturation>0</crs:SplitToningHighlightSaturation>
<crs:SplitToningBalance>0</crs:SplitToningBalance>
<crs:ParametricShadows>0</crs:ParametricShadows>
<crs:ParametricDarks>0</crs:ParametricDarks>
<crs:ParametricLights>0</crs:ParametricLights>
<crs:ParametricHighlights>0</crs:ParametricHighlights>
<crs:ParametricShadowSplit>25</crs:ParametricShadowSplit>
<crs:ParametricMidtoneSplit>50</crs:ParametricMidtoneSplit>
<crs:ParametricHighlightSplit>75</crs:ParametricHighlightSplit>
<crs:SharpenRadius>+0.5</crs:SharpenRadius>
<crs:SharpenDetail>10</crs:SharpenDetail>
<crs:SharpenEdgeMasking>0</crs:SharpenEdgeMasking>
<crs:PostCropVignetteAmount>0</crs:PostCropVignetteAmount>
<crs:ConvertToGrayscale>False</crs:ConvertToGrayscale>
<crs:ToneCurveName>Medium Contrast</crs:ToneCurveName>
<crs:ToneCurve>
<rdf:Seq>
<rdf:li>0, 0</rdf:li>
<rdf:li>32, 22</rdf:li>
<rdf:li>64, 56</rdf:li>
<rdf:li>128, 128</rdf:li>
<rdf:li>192, 196</rdf:li>
<rdf:li>255, 255</rdf:li>
</rdf:Seq>
</crs:ToneCurve>
<crs:CameraProfile>Adobe Standard</crs:CameraProfile>
<crs:CameraProfileDigest>3F2682AB38BCE545952EB7FBD 1374A1D</crs:CameraProfileDigest>
<crs:HasSettings>True</crs:HasSettings>
<crs:HasCrop>False</crs:HasCrop>
<crs:AlreadyApplied>False</crs:AlreadyApplied>
</rdf:Description>
<rdf:Description rdf:about=""
xmlns:photoshop="http: //ns.adobe.com/photoshop/1.0/">
<photoshop:SidecarForExtension>NEF</photoshop:SidecarForExtension>
</rdf:Description>
-
20th March 2010, 08:51 PMScoutRRe: Question about sidecar/xmp files
Aaaaha! I think that tells me what I wanted to know, for sidecar anyway, but the file imbedded in DNG would probably (maybe) follow a simalar format.
This seems to follow a database like format where there is a field for every possible adjustment, so whenever a change gets made the old setting would be deleted in place of the new one. They must allow so many entires per field though and have them stored somehow for the history part of things, but basically the filesize would change depending on how many controls were used, but not on how many times each control was changed. (taking into account the history storage part of it) I think
I was imagining a file that added lines every time a change was made anywhere, and just kept appending add infinitum (sp)
This type of format looks like it should not matter.
Thanks Dave :) -
20th March 2010, 09:06 PMTerry TedorRe: Question about sidecar/xmp filesHi Wendy. Let me preface my reply by saying I'm by no means an "expert" on xmp files, so take what I say with a grain of salt.Quote:
Originally Posted by ScoutR
The xmp files can be opened with an xml editor. I looked at the structure of them and it appears that the various adjustments to the RAW file are just entries in the xmp file, one entry corresponding to each different adjustment in ACR (I imagine it would be the same for LR). It does not appear that the file is appended, rather the values are reset to the default.Quote:
Originally Posted by ScoutR
Not sure about that, but I don't think so. You can use (assuming you're using Wnindows) Explorer to check the size of the file. Most of the ones I examined on my computer were about 7kb, with a few as small as 3kb. I did not take a very big sample though.Quote:
Originally Posted by ScoutR
Once again, I can't say for certain, but I don't think so. As they are not very big, I don't think you have anything to worry about.Quote:
Originally Posted by ScoutR
Here's a sample of the data contained in an xmp file for an image that I had previously edited in ACR:
http://i40.tinypic.com/2ajo02w.jpg
You can see how the entries in the right hand pane correspond to the different adjustments in ACR shown in the left hand pane.
I opened the file in ACR and reset all the adustments to "Camera Default" and here's the resulting XMP:
http://i39.tinypic.com/w0qk38.jpg
You can see the values were just reset to the ACR default.
Here's what the XMP file looks like when first opened in the XML editor:
http://i40.tinypic.com/35005mh.jpg
Each entry in the left hand pane labeled "rdf : Description" contain information about the RAW file, such as:
1st Entry : Camera Make, Model & Image Orientation
2nd Entry: All the EXIF data is here.
3rd Entry: Modify & Created Dates
4th Entry: Creator & Rights Data
5th Entry: Camera Body Serial #, Len Info, Flash Comp, Owner's Name, Firmware Version
6th Entry: This is where the ACR adjustment slider data is contained
7th Entry: Only contains one entry, "photoshop:SidecarForExtension"
Regards,
Terry -
20th March 2010, 09:34 PMScoutRRe: Question about sidecar/xmp files
Many thanks Terry for putting that all together. It tells me what I need to know. My only problem is that I convert to DNG and don't have any NEF files to play around with right now. I won't convert the next batch to DNG then I can play around and make sure I know how to view the xml file in the Editor.
Is that just Notepad or is it a special XML editor? I have Windows 7, do I have what I need already?
I usually use Lightroom for RAW editing. It stores the steps for me and I can go back to any step and reset to that point. I'm curious how it does that.
With the files that you have shown, it looks like you can only go back to the beginning and reset everything. Is that correct?
I get the main idea now, but am wondering how they store the history in Lightroom, and also if there is any way to get at the embedded file in DNG format.
I'm not that keen on having a seperate xml file and like Colin that is the main reason I convert to DNG, however, it almost looks like it could be advantageous to have the xml seperate. I'm not sure how or why, but there could come a time when it would be good to be able to get at it.
Thanks again for taking the time to put that together.
Wendy -
20th March 2010, 10:38 PMTerry TedorRe: Question about sidecar/xmp files
You're welcome Wendy. They can be opened in Notepad, but they're messy. Very messy. I would recommend an XML editor.
These were opened with Microsoft's free "XML Notepad", available here. Nothing fancy, but it works. I've also used Altova's "XML Spy", lots of bells and whistles and probably overkill. Just be careful - I used XML Notepad to edit the master XMP file that CS4 uses when you edit the file info (File>File Info) and somehow failed to save the file correctly and after that, CS4 wouldn't work correctly when I edited the file info. I ended up deleting the file and CS4 then rebuilt it when I reopened CS4. No big deal, just something to be aware of.
All I tried was resetting to "Camera Defaults" in ACR. I don't think there are any other options in ACR. I'm not familiar with LR; ACR allows you to save presets, which are useful if you want to apply the same corrections to numerous files, but I use it to allow me to revert to different "versions" of adjustments. I would venture a guess that the history is stored in an XMP/XML file somewhere. Poke around in your User Directory (the user name you use to log into your computer) under %systemdrive%\User>(Your User Name)>AppData>Roaming>Adobe. There should be an entry there for LR. If not there, try User>(Your User Name)>Local>Adobe.
I'm not sure I understand why you'd want to get to the embedded XMP data in the DNG file, or edit the XMP data, for that matter. I elected to keep the XMP data in a seperate sidecar file and manage the two as a pair. I don't know exactly why, but I just elected to do it that way when I started out years ago and it seemed easier to stick with it.
You're welcome again. Hollar if I can be of further assistance.
Cheers,
Terry -
20th March 2010, 10:56 PMScoutRRe: Question about sidecar/xmp filesI'm just curious to see how the data is stored in the DNG file, but I imagine it's probably in a similar format as the xmp.Quote:
I'm not sure I understand why you'd want to get to the embedded XMP data in the DNG file, or edit the XMP data, for that matter. I elected to keep the XMP data in a seperate sidecar file and manage the two as a pair. I don't know exactly why, but I just elected to do it that way when I started out years ago and it seemed easier to stick with it.
I don't really care to edit it, I have enough problems already :eek: and cannot at this time think of any reason that I would want to, but it just seems that if the occasion ever came up, that with the seperate sidecar xmp file you could but with DNG you could not edit.
My main question is answered though, and I am no longer worried about the size of the file that I'm creating when I start reworking and reworking and reworking the same image. :o
Wendy -
20th March 2010, 11:11 PMTerry TedorRe: Question about sidecar/xmp filesSounds good!Quote:
Originally Posted by ScoutR
I know what you mean there, as I found out firsthand. Fortunately, no lasting damage was done. True, but only if you choose to embed the XMP data in the DNG. Not being familiar with LR has me at a disadvantage as my knowledge is based solely on my experience with ACR. In ACR you have the option to embed the data or save it as a seperate XMP file. Does LR have the same option?Quote:
Originally Posted by ScoutR
Glad to hear that! Edit away to your heart's content without fear of hurting anything. Isn't the digital darkroom grand!?!Quote:
Originally Posted by ScoutR
-
20th March 2010, 11:30 PMScoutRRe: Question about sidecar/xmp filesSeeing as I always use DNG I'm not really sure, but the help file says I have the choice to save xmp to the Camera Raw Database Or to a seperate xmp file. It also says that I can save xmp data for DNG's separately. I did not know that so I'll be checking it out. One thing leads to another.....Quote:
I know what you mean there, as I found out firsthand. Fortunately, no lasting damage was done. True, but only if you choose to embed the XMP data in the DNG. Not being familiar with LR has me at a disadvantage as my knowledge is based solely on my experience with ACR. In ACR you have the option to embed the data or save it as a seperate XMP file. Does LR have the same option?
Yes, it is, but now I am never happy with anything, and have to keep trying this and that to see if I can make it better. Aaaarrrrggggg will it ever be good enough :eek: Quite often, sometimes after hours of fooling around I end up going back to what I started with :(Quote:
Glad to hear that! Edit away to your heart's content without fear of hurting anything. Isn't the digital darkroom grand!?!
-
20th March 2010, 11:46 PMDave HumphriesRe: Question about sidecar/xmp files
Hi Wendy,
Given that changing a value in the xmp file will only mean that when you next open the image in ACR (via LR, CS4 or Elements), it'll come up with the slider you altered set to the new value - I don't really see the benefit. Well, I say that, but I wonder if there's a checksum stored anywhere that might 'throw a wobbly' if you manually edit it.
I suppose writing a default set, as Terry did, and then Saving As, effectively renaming it, so that it associates to an image you haven't ever opened in ACR yet might have some merit as a batch process, but I'm sure there's an easier way in LR/CS4.
Cheers, -
21st March 2010, 12:07 AMScoutRRe: Question about sidecar/xmp filesNo, I don't really see any benefit either. It's interesting though. Maybe that checksum is what caused Terry problems when he tried to edit an xmp.Quote:
Given that changing a value in the xmp file will only mean that when you next open the image in ACR (via LR, CS4 or Elements), it'll come up with the slider you altered set to the new value - I don't really see the benefit. Well, I say that, but I wonder if there's a checksum stored anywhere that might 'throw a wobbly' if you manually edit it.
Actually, The preset seems like a pretty good idea. I rely on the History in Lightroom, but I think it might be even more helpful to create a preset when I get to a point that I like. Then if I get lost, I can just go back to the original and apply the preset. There are so many options. :confused: :)Quote:
I suppose writing a default set, as Terry did, and then Saving As, effectively renaming it, so that it associates to an image you haven't ever opened in ACR yet might have some merit as a batch process, but I'm sure there's an easier way in LR/CS4.
-
21st March 2010, 12:54 AMTerry TedorRe: Question about sidecar/xmp filesQuote:
Originally Posted by Dave Humphries
Yes, it is possible that is what happened, but I'm inclined to think that I overlooked a setting in the save options of the XML editor.Quote:
Originally Posted by ScoutR
At the recommendation of Colin, I'm reading "Real World Camera Raw with Adobe Photoshop CS4 - Industrial Strength Production Techniques" by Bruce Faser and Jeff Schewe. I was thumbing through the pages ahead and guess what - Chapter 8 - Everything You Wanted To Know About Metadata But Were Afraid To Ask! Seriously, the chapter title is "Mastering Metadata". According to the "book", XMP is an open standard subset of XML that are "simply text files, readable by any text editor or word processor, that conform to a specified syntax and saved with the .xmp extension" (Hope I don't get into any copyright issues here!) If you move or rename the RAW file in Bridge, the sidecar xmp gets moved and renamed too. There is a sidebar that will interest you Wendy: "Tip: Forcing an XMP file for a DNG File. Normally, a DNG file would not have a sidecar file. To write a sidecar .xmp file for a DNG, first select the image in Bridge. In the File Info panel, select the Advanced tab; then click Save. This will allow you to save a sidecar file that you can open or edit in a text reader. Remember this tip when you need to see the embedded metadata in the DNG file. One of DNG's benefits is that you don't have to worry about sidecar files unless you want a peek inside."
Unfortunately, this didn't work when I tried it. I think they left out an intermediate step - before "then click Save..." it should read "click on the drop down window at the bottom next to "Cancel", click on "Export", select the directory where you want the sidecar file to be saved, then click Save". That worked, and all the ACR adjustments were there. Just be careful where you save it; when I tried to put it in the same directory with the DNG file, Bridge immediately deleted it when I closed the File Info window. I had to save it elsewhere to preserve it; I ended up putting it in the root directory of the drive where I have all my photos stored because the default Export path is layered pretty deep in the User's directory, which I think by default is a hidden directory.
Hope this extra info helps. There's more in that chapter about XMP, but gotta run, we're going out for dinner. Will read through it later this evening and report back of there's anything else of interest.
Mods, if you feel I've lifted too much directly from the text and want to delete or edit for fear of copyright issues, please do. Accept my apologies in advance. -
21st March 2010, 12:55 AMTerry TedorRe: Question about sidecar/xmp files
Oh, yes, one other thing... The presets, when those are discussed in "the book", that's one of the things they were designed for Wendy.
-
21st March 2010, 01:06 AMPopsPhotosRe: Question about sidecar/xmp files
Dave, page down once and you will find this:
:DQuote:
<exif:UserComment>
<rdf:Alt>
<rdf:li xml:lang="x-default">(c) Pops Carter www.PopsPhotos.us</rdf:li>
Pops -
21st March 2010, 12:56 PMScoutRRe: Question about sidecar/xmp filesMakes sense. I have this horrible habit of doing everything the hard way. :( I've made a few presets for Develop settings, but usually only use them for the batch that I'm working on when I find myself doing the same thing to every shot.Quote:
Originally Posted by Terry Tedor
I've been reading more on xmp and how Lightroom handles these files. There are a lot of file options in LR as to how to deal with metadata and xmp files. I've pretty much left everything at default because I didn't know what most of the options were, but I've found a few tips that might help, and found an option that will show an icon in the thumbnails if metadata has NOT been updated in the xmp file. I've turned that on now so I will get a warning if for some reason the file has not been updated after editing in another program.
Who knows, this might even end up giving me the solution as to why sometimes EXIF data shows up and sometimes it does not. -
21st March 2010, 03:43 PMTerry TedorRe: Question about sidecar/xmp filesLOL! Me too!Quote:
Originally Posted by ScoutR
Lots of options are good. Sometimes. ;)Quote:
Originally Posted by ScoutR
Didn't think you were still having that problem. Hope you can resolve it.Quote:
Originally Posted by ScoutR
-
21st March 2010, 03:44 PMDave HumphriesRe: Question about sidecar/xmp filesYep, that's why I added; "I think it was Pops' bird feeder one." :)Quote:
Originally Posted by PopsPhotos