Process Apple Gain Map: The ImageIO & the Core Image approaches

Background

Since 2020, iPhone cameras have been capturing images with embedded gain maps to enhance their appearance. The gain map is embedded in HEIF or even JPEG images, and when displayed on compatible devices and apps, the gain maps are extracted and combined with the calculated headroom to display HDR effects.

As noted during this year’s WWDC24, Apple introduced the new ISO Gain Map, also called the Adaptive Gain Map, alongside the existing Apple Gain Map in the “Use HDR for dynamic image experiences in your app” session. The Adaptive Gain Map aims to standardize gain map technology, making it widely adopted across the industry.

Although the standardization process is still ongoing in 2024, it doesn’t mean that we need to process HDR gain maps using the newest API introduced in iOS 18. In fact, a gain map-specific API was introduced in iOS 14.1 as part of the ImageIO framework, which is a lower-level yet easy-to-use framework. This year, Apple also added both the old and new API in Core Image to help developers process Apple Gain Maps. By “old and new,” I mean the API may have existed since iOS 14.1 but was made public in the iOS 18 SDK. Compatibility shouldn’t be a concern since it’s been available since iOS 14.1.

To process Apple Gain Maps, two APIs are available: one in ImageIO and another in Core Image. Since documentation on these is scarce, particularly for the ImageIO approach, this article will demonstrate how to use these APIs to extract, process, and save Apple Gain Maps in your final image.

When capturing and editing photos with my PhotonCam app, you’ll have the option to preserve or discard the HDR Gain Map. I introduced this Gain Map editing feature in early 2024, utilizing the ImageIO APIs to implement it. While there were some challenges when the iOS 18 Beta was released, I have since resolved those issues, and I will discuss the solutions in this article.

The content of table are:

1. How is Gain Map stored in a file?

2. Why & When should you process Gain Map?

3. The ImageIO Approach

4. The Core Image Approach

5. Remove HDR effects

How is Gain Map stored in a file?

The Gain Map is also an image, except that it’s a single channel grayscale image in Apple Gain Map technology. For the newest Adaptive Gain Map, it could be colorful. When it comes to the dimensions, the Gain Map should share the same aspect ratio as the main image and is typically half the size of the main image. Therefore, when calculating which part of the images should apply HDR effect, the Gain Map should be scaled up first to match the dimensions of the main image.

The Gain Map image serves as an auxiliary image storing with the main image in the same file.

For HEIF files, multiple images can be stored in a HEIF file. It can be useful to differentiate between them by assigning them certain roles. Gain Map image is stored as an auxiliary image in HEIF file. Note that for the images captured in Portrait Mode in iPhone, the depth data is also stored as an auxiliary data. For more information about how the auxiliary data is stored in a HEIF file, please refer to this WWDC video.

For JPEG files, the Gain Map image is also stored in the same JPEG file. Though JPEG files are old and not designed for storing multiply images, as far as I know, the depth map data is appended at the bottom of the image data and I suppose the Gain Map image also does this trick.

Why & When should you process Gain Map?

As I observe in practice, a pretty large part of image editing apps won’t preserve Gain Map that is attached in the original image. Sure if the app is simulating the old retro film effect, adding HDR to the final image would look a bit weird. But users may just use the apps to crop the image or just do some basic light adjustments. They may expect that the saved images should still have HDR effect when viewing in the system Photos app or in some HDR supported apps like Instagram, at least they should have the option to do that. Additionally, when editing images in the Photos app, Apple also preserve the Gain Map in the final images.

So when should you process the Gain Map? As I just mentioned, the Gain Map image should share the same aspect ratio with the main image, and when it’s scaled up to the same size of the image, it should have the exact same content as the main image, which means that if there is a 100x100 square in the (10, 10) location of the main image, it should exists in the exact same location in the Gain Map image. Otherwise, when displaying the image with HDR effect, you would notice that there is an overlapping effect on the screen.

With those information in mind, you should process the Gain Map image as what you do to the main image in the following situation:

• The main image is cropped or done with any other transformations by the users.

• The tone of the main image is changed drastically, like being applied a color reverted filter that can change the black color to the white.

• The main image is applied with some advanced effect like applying a bokeh blur in some part given the Depth Map image.

The ImageIO Approach

To process Gain Map image using ImageIO framework, the general steps are:

• Extracting the corresponding auxiliary data from the original image data.

• Parse the auxiliary data, get the metadata information and the bitmap data of the Gain Map image.

• Process the Gain Map image using your favorite APIs.

• Reassemble the auxiliary data with the modified Gain Map image and store into the final image.

The ImageIO approach looks quite complex. As I said, before this year there is not high level API for that, and if you are new to this, it’s recommended that you use the Core Image approach if that’s applicable to you.

Extract the auxiliary data

To extract the auxiliary data, you must first obtain the Data of the original file. Once you have the original data, you can first create a CGImageSource with that data, and then copy the auxiliary data with kCGImageAuxiliaryDataTypeHDRGainMap type and get the CFDictionary result.

func extractAuxiliaryDictionary(data: Data) -> CFDictionary? {
    let options: [String: Any] = [
        kCGImageSourceShouldCacheImmediately as String: false,
    ]
    
    guard let source = CGImageSourceCreateWithData(data as CFData, options as CFDictionary) else {
        return nil
    }
    
    guard let auxiliaryData = CGImageSourceCopyAuxiliaryDataInfoAtIndex(
        source,
        0,
        kCGImageAuxiliaryDataTypeHDRGainMap
    ) else {
        return nil
    }
    
    return auxiliaryData
}

The CFDictionary result, as its type suggests, is a dictionary that contains those information(Note that you can safely cast it to the Dictionary<CFString, Any> type):

Get the actual Gain Map image

The value of kCGImageAuxiliaryDataInfoData is a CFData, which is actually a bitmap data that contains RGB pixel data.

For simplicity, I recommend using Core Image to process the Gain Map image, like cropping and applying filters. To create a CIImage from the bitmap data, you use the init(bitmapData:bytesPerRow:size:format:colorSpace:) initializer of CIImage.

func createFrom(bitmapData: Data, bytesPerRow: Int, size: CGSize) -> CIImage {
    return CIImage(
        bitmapData: bitmapData,
        bytesPerRow: bytesPerRow,
        size: size,
        format: .L8,
        colorSpace: nil
    )
}

Note that:

1. The size and the bytesPerRow properties can be extracted in the kCGImageAuxiliaryDataInfoDataDescription dictionary.

2. The color format is L8, which represents an 8-bit-per-pixel, fixed-point pixel format in which the sole component is luminance.

3. The color space should be nil, as the image doesn’t contain color data and nil should be passed according to the documentation.

4. IMPORTANT: Starting with iOS 18, the kCGImageAuxiliaryDataInfoDataDescription dictionary no longer contains the Orientation key, which indicates the rotation that should be applied to the Gain Map image. The Gain Map image attached to the original image by the system shares the same orientation with the main image. Thus you should also read the Orientation from the EXIF of the main image and apply the orientation before doing some other transformations to it.

Process the Gain Map image

Once you have the CIImage instance of the Gain Map image, you can do whatever you want to it. Note that the transformations applied to the main image being edited should also be applied to the Gain Map image.

IMPORTANT NOTE:

You should scale the Gain Map image to the half size of the main image. If the main image is 4000x3000, the Gain Map image should be the size of 2000x1500. Without doing so, on iOS 18, there will be some artificial defeats when zooming the image in the Photos app. I have addressed this issue since the launch of iOS 18 Beta and only find this workaround to avoid this issue and no other documentations or feedback replies from Apple. Also, be aware of the floating point issue and try rounding the value.

Reassemble the auxiliary data

After applying transformations and effects to the Gain Map image, to save it to the final image, you should reassemble the auxiliary data. The general steps to reassemble the auxiliary data:

1. Getting the bitmap data from the modified CIImage.

2. Replacing the corresponding values of the original CFDictionary created by the CGImageSourceCopyAuxiliaryDataInfoAtIndex method, like updating the size and bytesPerRow properties.

You use the following method to get the bitmap data from a CIImage instance:

func getBitmapData(ciContext: CIContext, bytesPerRow: Int) -> Data? {
    let height = self.extent.height
    
    let dataSize = bytesPerRow * Int(height)
    var gainMapImageData = Data(count: Int(dataSize))
    
    gainMapImageData.withUnsafeMutableBytes {
        if let baseAddress = $0.baseAddress {
            ciContext.render(
                self,
                toBitmap: baseAddress,
                rowBytes: bytesPerRow,
                bounds: self.extent,
                format: .L8,
                colorSpace: nil
            )
        }
    }
    
    return gainMapImageData
}

Note that the bytesPerRow should be the multiple of 4, otherwise you may receive a warning in the console.

func updateData(
    auxiliaryMap: Dictionary<CFString, Any>,
    transformedData: Data,
    description: Dictionary<CFString, Any>
) async -> Dictionary<CFString, Any> {
    var mutable = auxiliaryMap
    mutable[kCGImageAuxiliaryDataInfoData] = transformedData
    mutable[kCGImageAuxiliaryDataInfoDataDescription] = description
    return mutable
}

Saving auxiliary data to the final image

To save Apple Gain Map auxiliary data to the final image. You should first create a CGImageDestination with a file URL, and then add the main CGImage to the destination. Finally, calling CGImageDestinationAddAuxiliaryDataInfo to add the auxiliary data to the destination.

func saveToFile(
    file: URL,
    cgImage: CGImage,
    utType: UTType,
    properties: CFDictionary? = nil,
    auxiliaryData: CFDictionary? = nil
) throws -> URL {
    guard let dest = CGImageDestinationCreateWithURL(
        file as CFURL,
        utType.identifier as CFString,
        1,
        nil
    ) else {
        throw IOError("Failed to create image destination")
    }
    
    CGImageDestinationAddImage(dest, cgImage, properties)
    
    if let auxiliaryData = auxiliaryData {
        CGImageDestinationAddAuxiliaryDataInfo(dest, kCGImageAuxiliaryDataTypeHDRGainMap, auxiliaryData)
    }
    
    if CGImageDestinationFinalize(dest) {
        return file
    }
    
    throw IOError("Failed to finalize")
}

Note that:

1. The CFDictionary type of properties represents the properties(like TIFF and EXIF metadata) of the original image. As it also contains some information in the AppleMaker dictionary that helps calculating the headroom of the image, it should also be added to the final image in the CGImageDestinationAddImage method. To get the properties in the first place, use CGImageSourceCopyPropertiesAtIndex method.

2. If you are processing the main image using Core Image, you should get the CGImage from the output CIImage first before calling this method. You can use createCGImage(_:from:) from CIContext to create the CGImage. This method will create a CGImage that lives in the main memory that can be accessed by CPU rather than GPU, so it’s better to do this at the end of your image processing pipeline to avoid overhead.

The Core Image Approach

While the ImageIO approach is quite complex, as it should handle the metadata and the bitmap data carefully, the Core Image approach is quite easy. To perform the same task, the general steps are:

1. Use Core Image API to get the instance of CIImage that represents the Gain Map.

2. Use Core Image API to write the main CIImage along with t he Gain Map image to the output file URL.

Get and process the Gain Map CIImage

To get the CIImage instance of Gain Map, instead of getting the main image, you should set the CIImageOption of CIImageOption.auxiliaryHDRGainMap to true when creating CIImage from data. To get the orientation applied CIImage, you should also set the CIImageOption.applyOrientationProperty to true.

func getGainMapImage(data: Data) -> CIImage? {
    let options: [CIImageOption: Any] = [.auxiliaryHDRGainMap: true, .applyOrientationProperty: true]
    return CIImage(data: data, options: options)
}

Then you can apply transformations and effects to the CIImage like you would normally do.

Write CIImage to File

You can use writeHEIFRepresentation(of:to:format:colorSpace:options:) or other overloaded methods from CIContext to write both the main image and the Gain Map image to the output file.

func writeToFile(
    outputFile: URL,
    primayImage: CIImage,
    gainMapImage: CIImage,
    context: CIContext
) async throws {
    try context.writeHEIFRepresentation(
        of: primayImage,
        to: outputFile,
        format: .ARGB8,
        colorSpace: primayImage.colorSpace ?? CGColorSpace(name: CGColorSpace.displayP3)!,
        options: [.hdrGainMapImage: gainMapImage]
    )
}

And that’s it! The Core Image approach encapsulates all the details that the ImageIO has and creates a clean API for us to use.

Additionally, as a heads-up, the important notes mentioned above still apply when using the Core Image approach.

You should scale the Gain Map image to the half size of the main image. If the main image is 4000x3000, the Gain Map image should be the size of 2000x1500. Without doing so, on iOS 18, there will be some artificial defeats when zooming the image in the Photos app. I have addressed this issue since the launch of iOS 18 Beta and only find this workaround to avoid this issue and no other documentations or feedback replies from Apple. Also, be aware of the floating point issue and try rounding the value.

Finally, the properties of the final image will be copied from the properties of the main image. So if you are intended to modify some metadata, make sure you override the CIImage’s properties via the settingProperties(_:) method. Similarly, if you want to set the compression rate of the final image, you should set the kCGImageDestinationLossyCompressionQuality to a CFNumber in the properties of the input image.

As I was saying, the Core Image APIs is public until this year, and it’s still compatible back to iOS 14.1 or macOS 11.0. For overall simplicity, it’s recommended that you use the Core Image APIs, especially when the whole image processing pipeline is done using Core Image.

Remove HDR effects

Since you will need to do explicit work to attach HDR Gain Map to the final image, do you think that as long as the final image won’t have the Gain Map auxiliary data, it won’t render HDR effects?

The answer is a no.

For some reasons, the iOS’s Photos apps can still render HDR effects even the Gain Map is missing, and it’s rendered in a weird way. To utterly remove the HDR effects from a photo, you must also remove the metadata (specifically, the headroom metadata) in the properties of the image.

Apple’s documentation has talked about some keys that related to getting the headroom of the image, however, the concrete details of the values are still missing from the documentation. I managed to remove the weird HDR effects from a Gain-Map-missing image by removing some keys mentioned in the documentation:

func removeHDRInfoInAppleMaker(_ properties: inout Dictionary<String, Any>) {
    if var appleMaker = properties[kCGImagePropertyMakerAppleDictionary as String] as? Dictionary<String, Any> {
        // The values are mentioned in this documentation:
        // https://developer.apple.com/documentation/appkit/images_and_pdf/applying_apple_hdr_effect_to_your_photos
        appleMaker.removeValue(forKey: "48")
        appleMaker.removeValue(forKey: "33")
        properties[kCGImagePropertyMakerAppleDictionary as String] = appleMaker
    }
}

Adaptive Gain Map

iOS 18 and macOS 15 introduce the new standardized Adaptive Gain Map, which aims to be the standard in the industry. Since this article is all about the Apple Gain Map, the Adaptive Gain Map part will be talked in the next article.

If you have any questions, feel free to leave them in the comment sections. Thanks for reading this far.