Color Management across Apple Frameworks. Part 1: Understand Colors
This series of articles delve into the concept of Color Management across Apple’s frameworks, particularly on Core Video, Core Image, and Metal. This article is the first part: Understanding Colors.
The math behind Colors
While this series is aimed to explain the color management technique in Apple platforms, the physiology and math behind colors is an interesting and important topic to discover and learn. I recommend you read some explanations about how we humans perceive and interpret colors before reading this article. One recommendation is a YouTube video below.
Colors are beyond numbers
Have you ever asked for someone which color it is and you get the answer like “#FF6302” or (Red = 255, G = 99, B = 2)? That’s both conventionally correct and technically incorrect.
The “conventionally correct” part is, we almost always assume that a given color representation (like the hex form or the RGB integer form) is in the sRGB Color Space, which defines the RGB Color Model, the sRGB gamut, and the gamma to encode the color values. Creating a new document in Photoshop, capturing photos using your mobile phones, or any other things you do on your computer will most likely default to use the sRGB Color Space.
The “technically incorrect” part is, if the person who sends you the color representation as “#FF6302” is working in a Display P3 Color Space, by typing the exact same “#FF6302” in the workspace you are working on (presumably it’s sRGB Color Space) will have the wrong look. Given the fact that different Color Spaces define their own Color Gamuts and encoding methods, the differences in look are obvious.
For another instance, you inspect and send your designer colleague the color of a pixel in the image by some screenshot apps. Normally you would get the color RGB values in the native space (for MacBook Pro’s default profile, it’s Display P3), and your colleague may be working in the sRGB Photoshop/Sketch document. Hence, both of you won’t see the exact same color, even using the exact same display.
Note: due to the difference of various displays and how individuals perceive colors, making two colors in different display perceptually and psychologically exactly the same is technically impossible, but that’s not the topic of this article though. The color management mentioned below is the process of doing the best to make sure colors look consistent on different devices.
The key point is, the numbers representing colors are just meaningless numbers if not defined in the corresponding color spaces.
Color and Color Space
Numbers are meaningless without being in a definition of a color space. If you can imagine what you see as the color of “#FF6302”, you are assuming a lot of things. The first thing is, you are assuming the “#FF6302” is a representation of an RGB color model, where “FF”, “63” and “02” are hex numbers ranging from [0, FF] which represents the relative Red, Green and Blue components respectively. Colors can be represented in various models. Aside from the RGB model we are already familiar with, CMYK model and YCbCr model are also widely used.
Note that this article will mainly focus on the RGB color model, as it’s used commonly in image processing pipelines like the fragment shaders.
In terms of the RGB color model, there are a few color spaces based on the RGB color model: the most commonly used sRGB IEC 61966-2.1, Display P3 which is used when you capture and save photos on your current modern iPhone, and so on.
There are a few specifications regarding to the color model, as you can view it in Wikipedia. I am not going to dive into all specifications in that table, but there are still some specifications we should know.
You may have heard the term “Gamut” or “Color Gamut”, which is the set containing the colors that can be accurately represented in a Color Space. And you probably have seen this image below depicting different color gamuts in the CIE 1931 space.
CIE 1931 space defines the combination of all the color sensations that can possibly be seen in the physical world. You may wonder why this sRGB image below can reproduce all the colors in this CIE 1931 space, while there is literally a white triangle defining what colors can be reproduced in sRGB. That’s because the colors we see there are just for illustration only, and they make us easier to understand. The numbers ranging from 460 to 620 from the outline of the shape represent the wavelength of the visible light. To learn more about how we get this CIE 1931 space, please refer to the video mentioned above.
While the term “Gamut” is a set containing indefinite values, it’s defined by the Color Primaries, which are the (x, y) coordinates of the Red, Green, and Blue in the CIE 1931 space. For example, the Color Primaries of sRGB Color Gamut are Red(x=0.64, y=0.33), Green(x=0.3, y=0.6), Blue(x=0.15, y=0.06), which you can see the white triangle in the image above.
Also, there is a term named “White Point” defining the whitest color in the CIE 1931 space. As for D65 white point, its coordinates are (x=0.31, y=0.32).
Note: there is a way to convert between the RGB Color representation to the CIE xyY representation, and I have made a library doing that for you. Changing the Color Primaries in the CIE 1931 space can get the similar result of the “Color Calibration” feature in Lightroom or ACR.
Now we have mentioned the terms of Color Gamut, Color Primaries and White Point, there is one left concerning how the color will be reproduced: Gamma.
Gamma correction or gamma is a nonlinear operation used to encode and decode values in video or still image systems. The reason why we need to gamma encode the color values is that we as human beings have a nonlinear perception of the brightness: we can tell more differences between darker tones than between lighter tones, while the cameras or sensors perceive brightness in a linear way.
Traditionally, we represent colors in the 8-bit integer form, which ranges from [0, 255]. With such a narrow range of colors, how we distribute (compress) the colors based on their tones affects how they can be more accurate when presented on the display for our human eyes to watch. Hence, using a nonlinear method to encode colors according to their tones can result in a details-preserved image without requiring more bits or bandwidth, which yields to the result of reducing the image file size.
So, how to decode (compress) an encode (expand) a color? Well, you will be able to find some equations to do that, which is shown as below. Don’t worry, I am not going to dive into the equations.
The equations above are about how to gamma encode(compression) and gamma decode(expansion) for sRGB. You may have seen in some cases that people may say gamma γ = 2.4, which is a variable that affects the result of the gamma decoding or gamma encoding(1/γ). Tagging encoded color data with the correct gamma value(or transfer function) is crucial as it affects the decode process, which is a topic to be covered in the following sections.
Gamma, or Gamma Curve, is a part of Transfer Function, which also contains some other functions that you may have heard of, like Logarithmic (or Log), or Perceptual quantizer (or PQ), and Hybrid log–gamma (or HLG). Thus, a Color Space doesn’t always have the constant Gamma value. For example, a Display P3 HLG Color Space will have the same Color Gamut as Display P3, but instead it uses HLG as its transfer function.
The color spaces mentioned like sRGB and Display P3 are referred to as the ones that have a non-linear gamma curve. In some platform APIs, you may find the linear versions of those color spaces like linearSRGB and extendedLinearDisplayP3.
What’s Color Management
There are a lot of things to process. Now fully understanding the content (especially the algorithm to gamma encoding and decoding) is okay, but you still have to remember: Numbers are meaningless without being in a definition of a color space. Digital images apply the same rule as they are sets of color numbers.
As the technology moves forward, there are tons of images and display or intermediate processing systems (like Photoshop, our apps, or the operating systems) that contain or process colors in different color spaces. How to probably process and display colors is what Color Management does.
Color management is the process of ensuring consistent and accurate colors across various devices, such as monitors, printers, and cameras.
Say you have a pixel, which has a color of Red(R=255, G=0, B=0, where three components range from [0, 255]) in the sRGB color space, and it’s about to be displayed in the Display P3 color space system.
What will happen if we just ignored the color space of the Red(R=255, G=0, B=0) color?
Since the Display P3 has a larger color gamut than sRGB, a full red(R=255, G=0, B=0) is apparently different in Display P3 than sRGB. To make sure that whoever created the sRGB Red(R=255, G=0, B=0) can see it the same on both the sRGB Display and Display P3 Display, Color Management must do its job and apply color matching from the sRGB Red(R=255, G=0, B=0) to the value in Display P3 (which is approximately R=233, G=51, B=36 using the relative colorimetric rendering intent).
Likewise, displaying a Display P3 Red(R=255, G=0, B=0) in a sRGB Display requires some techniques called gamut mapping. Gamut mapping, like from a wide gamut to a narrow gamut, has different approaches and may have significantly different visual results. Detailed techniques and algorithms are not discussed here.
Luckily, auto Color Management is performed in some of the systems and leaves us not to worry about it. For regular content consumers, knowing less is for the best. But for content creators or us graphics developers, care must be taken to avoid breaking the Color Management system.
An example of how to break the Color Management system: Creating your visual artwork in a Display P3 Color Profile using Photoshop and not embedding the Display P3 Color Profile when exporting the artwork will result in a different look when you inspect the result, as the software that tries to decode and render the image will usually assume it to be sRGB if no Color Profile is found in the file.
Especially, there are some cases that we don’t want Color Management as the Color Data is not treated as perceptible Colors, like the Depth Data describing the Depth in meters or the Disparity (1/Depth) of the image.
Note: there is a way for you to do color management “manually”, by converting colors between different Models and Color Spaces. You can try that on Mac using the ColorSync Utility app.
Color Profile in Color Management
For reproducing various colors in various devices, a Color Profile, or ICC Profile(International Color Consortium Profile), is used. ICC Profile describes the characteristics of colors like Color Model, PCS(Profile Connection Space), and so on to help mapping colors from one Color Space to another. Image files embed ICC Profiles; your MacBook installs an ICC Profile for your display, and your printer comes with an ICC Profile for you to use when printing.
macOS provides a tool for you to inspect the ICC Profile. If you have an ICC Profile file that ends with the “icc” extension, then by default, it will be opened by the ColorSync Utility app. Below is an inspection of the Display P3 ICC Profile. In the header section, you can find the brief descriptions of this profile and below are the characteristic.
Remember the gamma decoding equation mentioned earlier? If you look at the “rTRC” tag, you’ll find that the function it represents is the same as the equation. I’ve overlayed the equation on top of the “rTRC” tag description page so that you can compare them easily.
For image files, there are a few ways to examine the embedded ICC Profile. One way is using the command-line tool Exiftool:
exiftool -icc_profile -b -w icc your_image.heic.Then you will get the ICC Profile file to examine.
In Photoshop, you can do a few tests to examine the ICC Profile:
Open an image file, and try assigning a different profile to see how the colors go wrong via Edit -> Assign Profile.
Create an empty document with a Color Profile like Display P3, draw something on it and try converting to another profile to see how the pixel color values change via Edit -> Convert Profile.
For an image file that is embedded with this Display P3 Profile, to display a pixel color in a sRGB Display, Color Management needs to do the following conversion:
Decode the image file to Bitmap Data with RGB components.
Linearize the RGB using the gamma-decoding equation.
Convert the Linearized RGB to CIE 1931 xyY Color Space, which is the Profile Connection Space(PCS).
Color match to the destination color gamut (in this case, it’s sRGB) using a specific Rendering Intent.
Convert back the xyY values to Linearized RGB values.
Gamma Encode the Linearized RGB values.
The above steps are just an approximation of how Color Management works. Of course, there are so many trivial things the operating system has done for you.
Not knowing every technical detail about an ICC Profile is still okay, as we are not going to create a custom or calibrated ICC Profile anyway. But what we should learn in this section are:
Color management is automatically handled for most of our users in most cases.
Not breaking the Color Management is what we should do. Don’t rip off the ICC Profile that should be embedded in an image file.
Afterwords
This article concludes with some fundamental concepts to help us comprehend colors in a digital context. The subsequent part of this series will delve into color representations and management across Apple platforms, focusing on Core Video, Core Image, and Metal. Stay tuned for the next article, and thank you for reading.
