I would like to be able to detect the presence of a color profile in JPEG. To be clear: I'm not interested in actually carrying out the color corrections or doing anything else with them.
I did some investigation, but I could not find any definite source of information. Here is what I know so far:
There is a color profile information in the XMP block. The XMP block is in a 0xFFE1 JPEG section (the same as EXIF). The marked of this section is followed by the http://ns.adobe.com/xap/1.0/ string. However, when I stripped this section, Photoshop was still able to detect the color profile.
There is a color space tag in EXIF itself as well (tag = 0xA001). It has only one value defined in the EXIF specification: 1 = sRGB. I noticed that in my image it takes the value 0xFFFF which means uncalibrated. So it’s not this tag.
I did an experiment with one image saving it in two different color profiles, and besides the changes in XMP and EXIF, I noticed that there is probably a whole new JPEG section. It starts by 0xFFE2 and its marker is followed by the string ICC_PROFILE. This seems to be the most likely case.
Does anybody have experience with managing color profiles in JPEG? I would say that the information in EXIF and XMP are just metadata, and that the important thing is the presence of the ICC_PROFILE section.
ICC profiles are stored in a JPEG file's Application Segments. These blocks are for letting applications store whatever data they need, so they are not mandated by the JPEG spec. In other words, the JPEG file format does not define a way to embed color profiles, you are just seeing the de facto ways of doing it.
Since the application segments can be used by anyone for anything, you need to identify and validate the contents, before you can say for certain that it contains an ICC profile.
So there are three ways common ways ICC profiles could be supplied:
By embedding an ICC profile in Application Segment 02, prepended with ICC_PROFILE\0x00\0x01\0x01 for identifying it. I'm currently unsure if the two \0x01 mean anything, or if they are always that. After those 14 characters, the actual ICC profile is stored.
By embedding an ICC profile in an embedded EXIF metadata file in Application Segment 01.
By embedding an ICC profile in an embedded XAM metadata file in Application Segment 01. This is prepended with http://ns.adobe.com/xap/1.0/\0x00.
I have not been able to find any concrete information either. It would be nice if ICC said, "this is how to do it", but I have not yet been able to find any of the sort. Thus as far as I know, any of these could potentially be used as an ICC profile and it would depend on the application on whether it would load the profile or not.
Application Segment 02 seems to be the common way of doing it, though what you want to check might depend on your use case. As the other comment showed, some HELIOS EtherShare software has/had its own way of doing it. (I have not yet confirmed the presence of an ICC profile in the metadata, however I haven't really looked that hard and it wouldn't surprise me if you could find files with multiple conflicting ICC profiles in the wild.)
Related
I have cached preview files from Capture One (a photo processing program, similar to Lightroom) where I have lost the originals. Capture One saves previews in their proprietary .cop format and I'm not sure how to go about identifying what's what in there.
There are the strings ETIFFTagInteropIFD and JPEG Embedded TIFF Tags seen in the HEX view which suggests that they are somehow embedding a TIFF in there.
I do have original JPEG files with their corresponding COP-file, but when comparing them there isn't much that's similar - which makes sense I guess, since the preview COP-file is roughly half the size of the original.
What conclusions can I draw from this and what are some good tools for going further?
Anyone knows why the image, shown below, is partially gray while loading? I assume it has something to do with progressive JPG, but I haven't seen the grayscale effect before. Is there some kind of a JPG export setting that enables that? How is it called? Can I disable it?
Direct image link
Edit:
The problem could either be the problem described in #KonradDzwinel answer OR my answer, both could look identical:
What is the problem
For some reason the file is saved or produced in way that make the file we call it Lossless JPEG.
I am not sure which software, tool or how this problem is was produced, but that won’t change the fact that your file had have some issues or even corrupted.
How to find the problem in the file
Your file was saved in way that it works identical but it had issues. To check whether your file is healthy or not, it requires a tool to scan/analysis the image file. I use a very nice and easy to use tool (software) called JPEGsnoop (link for download). This tool is free and I used it to scan and analysis your image files for problems.
Your file comes up with few places with following note, (the screen-shot btw shows only one note, but there were more):
NOTE: Scan parsing doesn’t support this SOF mode.
After I have re-saved the file correctly and analysis it again, it does not show any problems or notes:
What I did to create a problem FREE image file
For testing I have used Photoshop 5.5 to open the Lossless file and save it with new name with default jpeg (since I don't have the original file). But correctly I assume you have the original file, and I assume you have Photoshop any version does not matter OR any other image tool. Try to optimize your jpeg file best possible, analysis it and remember/note the saving setting for future wise image saving.
Here is an example of my Photoshop save settings, which result in problem FREE file:
How can I learn more about Lossless JPEG
I won’t go in details of this problem but I will leave some link that explain it in details, I hope also you find this is useful to solve your issue:
http://www.impulseadventure.com/photo/fix-corrupt-jpeg-photo.html
http://fileformats.archiveteam.org/wiki/JPEG
http://fileformats.archiveteam.org/wiki/Lossless_JPEG_%28original%29
http://www.howtogeek.com/142174/what-lossless-file-formats-are-why-you-shouldnt-convert-lossy-to-lossless/
Software JPEGsnoop
http://www.impulseadventure.com/photo/jpeg-snoop.html
Progressively encoded JPEGs contain ten scan layers by default. That means ten iterative layers of image information build on each other to deliver the final visual quality of the image. The first visible scan layer of a progressive JPEG is always highly pixelated and often black & white because it saves on color channel information.
Source (emphasis mine)
Another resource that explores this issue in lengh: http://cloudinary.com/blog/progressive_jpegs_and_green_martians
I am developing a game in j2me. How to handle images in .dat format.
I downloaded some games and extracted jar , found some dat format images and not able to open that images and images size also very less.. what tools I need to use?
Ref link
enter link description here
Not able to find solution?
A dat file could be anything. Depends what the developer felt like doing.
Some developers chose to strip PNG files of their header, and added the header back in the code. This was partly done in order to save a few bytes (because they mattered back then), and partly because of the challenge in doing it like that, and partly because it ensured all images used the exact same palette.
So that's one possibility, but it really could be anything.
As stated by mr_lou, there really isn't anything special about a .dat extension.
The steps to re-compile a file usually start with opening the file up in a hex editor and then looking at the first bits of information in the file. You then basically work from there to re-compile the data necessary for a 'normal' program to interpret the file. In particular, the first 8-16 bytes are often very helpful for determining what type of file it is "supposed" to be.
If you are looking at a png file (that's what I usually prefer to use for art assets) then you can reference http://en.wikipedia.org/wiki/Portable_Network_Graphics to see how a 'normal' png might look. When you're tweaking to save bytes you often strip unnecessary fields from png headers (things like the ancillary chunks) and using a common palette.
However, remember that it's not necessarily image data. It could be things like level data, sound, default stats or any particular amount of stuff.
Is there any documentation on how to write software that uses the framebuffer device in Linux? I've seen a couple simple examples that basically say: "open it, mmap it, write pixels to mapped area." But no comprehensive documentation on how to use the different IOCTLS for it anything. I've seen references to "panning" and other capabilities but "googling it" gives way too many hits of useless information.
Edit:
Is the only documentation from a programming standpoint, not a "User's howto configure your system to use the fb," documentation the code?
You could have a look at fbi's source code, an image viewer which uses the linux framebuffer. You can get it here : http://linux.bytesex.org/fbida/
-- It appears there might not be too many options possible to programming with the fb from user space on a desktop beyond what you mentioned. This might be one reason why some of the docs are so old. Look at this howto for device driver writers and which is referenced from some official linux docs: www.linux-fbdev.org [slash] HOWTO [slash] index.html . It does not reference too many interfaces.. although looking at the linux source tree does offer larger code examples.
-- opentom.org [slash] Hardware_Framebuffer is not for a desktop environment. It reinforces the main methodology, but it does seem to avoid explaining all the ingredients necessary to doing the "fast" double buffer switching it mentions. Another one for a different device and which leaves some key buffering details out is wiki.gp2x.org [slash] wiki [slash] Writing_to_the_framebuffer_device , although it does at least suggest you might be able use fb1 and fb0 to engage double buffering (on this device.. though for desktop, fb1 may not be possible or it may access different hardware), that using volatile keyword might be appropriate, and that we should pay attention to the vsync.
-- asm.sourceforge.net [slash] articles [slash] fb.html assembly language routines that also appear (?) to just do the basics of querying, opening, setting a few basics, mmap, drawing pixel values to storage, and copying over to the fb memory (making sure to use a short stosb loop, I suppose, rather than some longer approach).
-- Beware of 16 bpp comments when googling Linux frame buffer: I used fbgrab and fb2png during an X session to no avail. These each rendered an image that suggested a snapshot of my desktop screen as if the picture of the desktop had been taken using a very bad camera, underwater, and then overexposed in a dark room. The image was completely broken in color, size, and missing much detail (dotted all over with pixel colors that didn't belong). It seems that /proc /sys on the computer I used (new kernel with at most minor modifications.. from a PCLOS derivative) claim that fb0 uses 16 bpp, and most things I googled stated something along those lines, but experiments lead me to a very different conclusion. Besides the results of these two failures from standard frame buffer grab utilities (for the versions held by this distro) that may have assumed 16 bits, I had a different successful test result treating frame buffer pixel data as 32 bits. I created a file from data pulled in via cat /dev/fb0. The file's size ended up being 1920000. I then wrote a small C program to try and manipulate that data (under the assumption it was pixel data in some encoding or other). I nailed it eventually, and the pixel format matched exactly what I had gotten from X when queried (TrueColor RGB 8 bits, no alpha but padded to 32 bits). Notice another clue: my screen resolution of 800x600 times 4 bytes gives 1920000 exactly. The 16 bit approaches I tried initially all produced a similar broken image to fbgrap, so it's not like if I may not have been looking at the right data. [Let me know if you want the code I used to test the data. Basically I just read in the entire fb0 dump and then spit it back out to file, after adding a header "P6\n800 600\n255\n" that creates the suitable ppm file, and while looping over all the pixels manipulating their order or expanding them,.. with the end successful result for me being to drop every 4th byte and switch the first and third in every 4 byte unit. In short, I turned the apparent BGRA fb0 dump into a ppm RGB file. ppm can be viewed with many pic viewers on Linux.]
-- You may want to reconsider the reasons for wanting to program using fb0 (this might also account for why few examples exist). You may not achieve any worthwhile performance gains over X (this was my, if limited, experience) while giving up benefits of using X. This reason might also account for why few code examples exist.
-- Note that DirectFB is not fb. DirectFB has of late gotten more love than the older fb, as it is more focused on the sexier 3d hw accel. If you want to render to a desktop screen as fast as possible without leveraging 3d hardware accel (or even 2d hw accel), then fb might be fine but won't give you anything much that X doesn't give you. X apparently uses fb, and the overhead is likely negligible compared to other costs your program will likely have (don't call X in any tight loop, but instead at the end once you have set up all the pixels for the frame). On the other hand, it can be neat to play around with fb as covered in this comment: Paint Pixels to Screen via Linux FrameBuffer
Check for MPlayer sources.
Under the /libvo directory there are a lot of Video Output plugins used by Mplayer to display multimedia. There you can find the fbdev (vo_fbdev* sources) plugin which uses the Linux frame buffer.
There are a lot of ioctl calls, with the following codes:
FBIOGET_VSCREENINFO
FBIOPUT_VSCREENINFO
FBIOGET_FSCREENINFO
FBIOGETCMAP
FBIOPUTCMAP
FBIOPAN_DISPLAY
It's not like a good documentation, but this is surely a good application implementation.
Look at source code of any of: fbxat,fbida, fbterm, fbtv, directFB library, libxineliboutput-fbe, ppmtofb, xserver-fbdev all are debian packages apps. Just apt-get source from debian libraries. there are many others...
hint: search for framebuffer in package description using your favorite package manager.
ok, even if reading the code is sometimes called "Guru documentation" it can be a bit too much to actually do it.
The source to any splash screen (i.e. during booting) should give you a good start.
My custom homebrew photography processing software, running on 64 bit Linux/GNU, writes out PNG and TIFF files. These are to be sent to a quality printing shop to be made into fine art. Working with interior designers - it's important to get the colors just right!
The print shops usually have no trouble with TIFF and PNGs made from commercial software such as Photoshop. Even though i have the TIFF 6.0 specs, PNG specs, and other info in hand, it is not clear how to include color calibration data or implement color management system on linux. My files are often rejected as faulty, without sufficient error reports to make fixes.
This has been a nasty problem for a while for many. Even my contacts at the Hollywood postproduction studios are struggling with this issue. One studio even wanted to hire me to take care of their color calibration, thinking i was the expert - but no, i am just as blind and lost as everyone!
Does anyone know of good code examples, detailed technical information, or have any other enlightenment? Or time to switch to pure Apple?
Take a look at LittleCMS
http://www.littlecms.com/
This page has the code for applying it to TIFF
http://www.littlecms.com/newutils.htm
The basic thing you need to know is that Color profile data is something you need to store in the meta-data of the file itself.
There is a consultant called Charles Poynton who specialises in this area. I work for one of the post production studios you mention (albeit in london not hollywood), and have seen him speak on the subject a couple of times. His website contains a lot of the material he presents and you might find something of use there. He also has a book called Digital Video and HDTV Algorithms and Interfaces which is not as heavy as the title might suggest! While these resources might not answer your question directly, it might provide a spring board to other solutions.
More specifically, which libraries are you using to write the png and tif files - you mention they are homebrew, but how custom are they exactly? Postprocessing the images in an image manipulation program (such as ImageMagick or dcraw) might allow you to inject this information into the header more successfully.
Sorry, I don't have any specific answers, but maybe something that will point you a bit further in the right direction...
As a GNU/Linux user, you’ll want to consider DispcalGUI – http://dispcalgui.hoech.net/ – a GNOME-based GUI that centralizes color management, ICC profile management, and (crucially for your case) device calibration. It can talk to well-known pro- and mid-level hardware, e.g, i1, X-Rite, Spyder, etc.
But before you get into that – you say you are generating your files to spec; are you validating your output using a test suite specific to the format in question? If not, here are three to get you started:
imagetestsuite supports the well-known formats: https://code.google.com/p/imagetestsuite/w/list?can=1&q=
The Luminous* test suite is a JIRA plugin, if that’s your thing: https://marketplace.atlassian.com/plugins/com.luminouslead.plugin.jira.testsuite.LuminousTestSuite
FLOSS Decoder implementations often have one you can use, i.e. OpenJPEG – https://code.google.com/p/openjpeg/wiki/TestSuiteDocumentation
But even barring all of those, it seems like your problem is with embedded ICC data – which is two specs in one. First, there’s the host image-file format, and they all handle embedding differently (meaning the ICC data will likely look totally different when embedded in a TIFF than, say, a JPEG or WebP file). Second, there is the ICC spec itself. It is documented here: http://color.org/v4spec.xalter – and you may also want to look at the source for the aforementioned dispcalGUI, which includes a very legible and hackable ICC profile class in Python: http://sourceforge.net/p/dispcalgui/code/HEAD/tree/trunk/dispcalGUI/ICCProfile.py
Full disclosure: I have contributed to that very ICC profile class, to which I just linked in that last ¶
That’s the basics (many of which you have no doubt covered)... beyond that, if you post more information about what exactly is going wrong, I’d be interested to look it over. Good luck with it either way.
* NB. This project is unrelated to the long-standing photography website, “the Luminous Landscape”