HEIF：Technical information
https://nokiatech.github.io/heif/technical.html
Technical information
Introduction
High Efficiency Image File Format (HEIF, ISO/IEC 23008-12) specifies the storage of individual images, image sequences and their metadata into a container file conforming to the ISO Base Media File Format (ISO/IEC 14496-12). HEIF includes the storage specification of HEVC intra images and HEVC image sequences in which inter prediction is applied in a constrained manner. Use cases supported by HEIF include:
- Storage of still images as well as collection of images in a single container file,
- Storage of burst photos,
- Storage and efficient representation of video animations and cinemagraphs.
- Support for simultaneous capture of video and still images, i.e. storing still images and timed image sequences into the same file.
- Storage of images derived from other images of the file, either as derived images represented by non-destructive image editing operations or as pre-computed derived images.
- Storage of focal and exposure stacks into the same container file.
- Support for storing other media, such as audio and timed text, into the same container file with timed image sequences and synchronizing their playback.
As HEVC provides support for various chroma formats and sample fidelities up to lossless coding, the format can serve the whole spectrum of use cases from today's consumer devices storing images typically at 8 bits per sample to high-end professional devices with sample fidelity and dynamic range requirements going all the way up to 16 bits per sample.
Computational photography forms a new category of use cases that can benefit from the HEIF file format. Now a set of related images can be stored in a single file with associated metadata indicating relationships between different pictures.
HEIF specifies a structural format, from which codec-specific image formats can be derived. HEIF also includes the specification for encapsulating images and image sequences conforming to the High Efficiency Video Coding (HEVC, ISO/IEC 23008-2 | ITU-T Rec. H.265).
In ISOBMFF, a continuous or timed media or metadata stream forms a track, whereas static media or metadata is stored as items. Consequently, HEIF has the following basic design:
- 1. Still images are stored as items. All image items are independently coded and do not depend on any other item in their decoding. Any number of image items can be included in the same file.
- 2. Image sequences are stored as tracks. An image sequence track can be indicated to be displayed either as a timed sequence or in a non-timed manner, such as a gallery of images. An image sequence track may be used instead of image items when there is coding dependency between images.
A file may contain both image items and image sequence tracks along with other media. For example, it is possible to create a file, which includes image items or image sequence tracks conforming to HEIF, and audio and timed text tracks conforming to any derivative format of the ISOBMFF.
Image I. ISOBMFF compliant file high level structure with meta box and items
Files conforming to ISOBMFF consist of a sequence of data structures called boxes, each containing a four-character code (4CC) indicating the type of the box, the size of the box in terms of bytes, and the payload of the box. Boxes may be nested, i.e. a box may contain other boxes. ISOBMFF and HEIF specify constraints on the allowed box order and hierarchy.
HEIF Related Brands and MIME Types
Files conforming to HEIF start with a FileTypeBox as defined in the ISOBMFF standard, which contains a list of brands the file complies with. As the FileTypeBox is located at the start of the file, it provides easily accessible indications of the file contents to media players. Each brand is identified by its unique four-character code. The specification of a brand can include requirements and constraints for files of the brand and for file players supporting the brand. A brand included in the FileTypeBox permits a player that supports the requirements of the brand to play the file.
The brands specified in the HEIF standard are presented in Table I. The HEIF standard specifies the 'mif1' and 'msf1' structural brands. Additionally, HEVC-specific brands are specified as listed in Table I. The dedicated brand names, 'heic' and 'hevc' indicates that HEVC Main Profile is utilized.
Table I. Brands, MIME subtypes, and file extensions for HEIF.
| Brand | Coding format | Image or sequence? | MIME Type | MIME subtype | File extension |
|---|---|---|---|---|---|
| mif1 | Any | image | image | heif | .heif |
| msf1 | Any | sequence | image | heif-sequence | .heif |
| heic | HEVC (Main or Main Still Picture profile) | image | image | heic | .heic |
| heix | HEVC (Main 10 or format range extensions profile) | image | image | heic | .heic |
| hevc | HEVC (Main or Main Still Picture profile) | sequence | image | heic-sequence | .heic |
| hevx | HEVC (Main 10 or format range extensions profile) | sequence | image | heic-sequence | .heic |
Internet media types, also known as MIME (Multipurpose Internet Mail Extensions) types, are used by various applications to identify the type of a resource or a file. MIME types consist of a media type ('image' in the case of HEIF files), a subtype, and zero or more optional parameters. For multi-purpose files, the selection of the subtype can be made on the basis of the primary use of the file.
An optional codecs MIME parameter can be present to indicate the used coding formats of the tracks and items present in the file. The codecs MIME parameter also includes the profile-tier-level value to which an HEVC-coded image item or an image sequence track conforms. More information about the MIME type registration and optional parameter can be found in Annex D and Annex E of the HEIF standard.
Comparison to other Image Formats
Table IX provides a comparison of the features of HEIF to other selected image formats. It can be observed that HEIF is more extensible and comprehensive than the other compared file formats. Particularly the possibility to include other media types, the advanced multi-picture features, and the support for non-destructive editing make HEIF more advanced than the other formats. The rich set of features make HEIF suitable for a broad range of devices and applications, including for example burst photography.
The references used to conclude the information in Table II are included in Table X.
Table II. Comparison of the features of some image file formats.
| .heic | JPEG/Exif | PNG | GIF (89a) | WebP | JPEG-XR / TIFF | JPEG-XR / JPX | BPG | |
|---|---|---|---|---|---|---|---|---|
| Formats and extensibility | ||||||||
| Base container file format | ISOBMFF | TIFF | - | - | RIFF | TIFF | - 4 | - |
| Lossy compression | Yes (HEVC) | Yes (JPEG) | No | No | Yes (VP8) | Yes | Yes | Yes (HEVC10) |
| Lossless compression | Yes (HEVC) | Yes (TIFF Rev 6.0) | Yes (PNG)1 | Yes (GIF)1 | Yes (VP8L) | Yes | Yes | Yes (HEVC10) |
| Extensible to other coding formats | Yes | Yes8 | No | No | No | Yes8 | Yes5 | No |
| Metadata format (on top of internal) | Exif, XMP, MPEG-7 | Exif | - | - | Exif, XMP | Exif, XMP | JPX, (XMP)6 | Exif, XMP |
| Extensible to other metadata formats | Yes | No | No | No | No | No | Yes (XML-based) | Yes |
| Other media types (audio, text, etc.) | Yes | Audio2 | No | No | No | No | Yes7 | No |
| Multi-picture features | ||||||||
| Multiple images in the same file | Yes | No11 | No | Yes3 | Yes3 | No | Yes | Yes9 |
| Image sequences / animations | Yes | No | No | Yes | Yes | No | Yes | Yes |
| Image coding | Yes | No | No | No | No | No | No | Yes |
| Derived images | ||||||||
| Multiple-of-90-degree rotations | Yes | Yes | No | No | No | Yes | Yes | No |
| Cropping | Yes | No | No | No | No | No | Yes | No |
| Tiling/overlaying | Yes | No | No | No | Yes | No | Yes | No |
| Extensible to other editing operations | Yes | No | No | No | No | No | No | No |
| Auxiliary picture information | ||||||||
| Transparency (alpha plane) | Yes | No | Yes | No12 | Yes | Yes | Yes | Yes |
| Thumbnail image | Yes | Yes | No | No | No | Yes | Yes | Yes |
1 In GIF and indexed color PNG encoding, lossy color quantization is applied while the color-quantized image is losslessly compressed.
2 PCM, µ-Law PCM and ADPCM encapsulated in RIFF WAV
3 Only for animations and tiling/overlaying
4 JPX is a box-structured format compatible with ISOBMFF. However, only the File Type box is common in JPX and ISOBMFF.
5 Encapsulation of JPEG-2000 and JPEG-XR have been specified for JPX container. Mappings for other codecs could be similarly specified.
6 JPX (ITU-T T.800 and T.801) specifies an own metadata schema, but is capable of carrying an XML formatted metadata, such as XMP.
7 JPX can contain media complying with ISOBMFF (or derivatives thereof). No accurate synchronization between JPX animations and other media.
8 TIFF as a container format facilitates extensions to other coding formats.
9 Only for animations, thumbnails, and alpha planes. Non-timed image collections not supported.
10 HEVC Main 4:4:4 16 Still Picture profile, Level 8.5, with additional constraints
11 Can be enabled through the MP extension
12 A palette index for fully transparency can be specified
It is acknowledged that a summary such as that in Table II might be somewhat incomplete when it comes to features of different formats. For example, the table does not cover some of the extensions of JPEG. We welcome feedback and corrections to the table.
Table III. References for the compared image file formats
| Image format | Version or date | Reference and/or URL |
|---|---|---|
| HEIF (.heic) | 03/2015 | ISO/IEC 23008-12 |
| JPEG | ISO/IEC 10918-1 | ITU-T Rec. T.81 http://www.w3.org/Graphics/JPEG/itu-t81.pdf |
|
| Exif | http://www.cipa.jp/std/documents/e/DC-008-2012_E.pdf | |
| PNG | http://www.w3.org/TR/PNG/ | |
| GIF | 89a | http://www.w3.org/Graphics/GIF/spec-gif89a.txt |
| WebP |
https://developers.google.com/speed/webp/docs/riff_container VP8L: https://developers.google.com/speed/webp/docs/webp_lossless_bitstream_specification |
|
| JPEG-XR | ISO/IEC 29199-2 | ITU-T Rec. T.832 ISO/IEC 15444-2 | ITU-T Rec. T.801 (for JPX) |
|
| BPG | 0.9.5 | http://bellard.org/bpg/bpg_spec.txt |
The file size of container files are directly affected by the compression performance of the image/video codec being utilized. Such a comparison can be found in Annex B of this document.
Compression Performance
Table IV illustrates coding efficiency of HEVC intra coding with respect to well-known still picture codecs. The results indicate that JPEG would require on average 139 % higher bitrate than HEVC (i.e. 2.39 times the file size) in order to achieve the same objective picture quality. For JPEG-XR and JPEG-2000 the average increase in bitrates are 66 % and 44 %, respectively.
Table IV. HEVC intra coding performance with respect to legacy formats. Bitrate increase required to achieve the objective quality provided by HEVC intra coding is reported for each test category.| Class | Resolution | Charasteristics | JPEG | JPEG XR | JPEG 2000 |
|---|---|---|---|---|---|
| Class A | 2560x1600 | Cropped 4Kx2K sequences for Ultra HDTV services | 87 % | 44 % | 48 % |
| Class B | 1920x1080 | High resolution sequences for streaming and broadcast services | 124 % | 62 % | 15 % |
| Class C | 832x480 | Medium resolution sequences for Internet/mobile video services | 122 % | 53 % | 50 % |
| Class C | 832x480 | Medium resolution sequences for Internet/mobile video services | 122 % | 53 % | 50 % |
| Class D | 416x240 | Low resolution sequences for services to resource constrained devices | 110 % | 47 % | 43 % |
| Class E | 1280x720 | 720p sequences for video conferencing applications | 170 % | 73 % | 23 % |
| Class F | 1024x768, 1280x720 | Computer screen content and computer generated content | 223 % | 118 % | 87 % |
| Average | 139 % | 66 % | 44 % |
Table V indicate that one can expect that for natural content the restricted inter coding can typically provide two to three times better compression than intra picture coding. In special cases like animations where majority of the scene is static the compression efficiency can significantly exceed those levels and be tens of times more efficient than intra coding.
Table V. Coding efficiency improvements provided by low latency predictive coding of the HEVC Image File Format. Bitrate impact and coding gain are reported with respect to HEVC intra coding.| Content | Type | Frames | Bitrate change | Coding gain |
|---|---|---|---|---|
| Class A | Image burst | 8 | -46 % | 1.9 |
| Class B | Image burst | 8 | -51 % | 2.0 |
| Class C | Image burst | 8 | -60 % | 2.5 |
| Class D | Image burst | 8 | -63 % | 2.7 |
| Class E | Image burst | 8 | -79 % | 4.8 |
| Class F | Image burst | 8 | -55 % | 2.2 |
| Memorial | Exposure stack | 16 | -29 % | 1.4 |
| Mersu | Focal stack | 13 | -25 % | 1.3 |
| Car&Tractor | Cinemagraph | 48 | -97 % | 33.3 |
Images in HEIF Files
Multiple images can be stored in a HEIF file. It can be useful to differentiate between them by assigning them certain roles. The roles specified in HEIF are listed and described in Table VI. Note that a single image can be associated with more than one role.
Table VI. Roles of images.
Properties of Images
HEIF allows the storage of image properties which are shared among different image items in a compact way. These properties are stored in ItemPropertyContainerBox. There are mainly two types of properties: descriptive and transformative. Descriptive properties provide information about the image item without applying modifications on the image itself. Transformative properties provide information about the transformative modification that needs to be done on the image item. The order of application of these properties to the image items are defined in the standard. Table VII lists currently defined properties. In addition to descriptive image properties, image items can optionally be characterized with metadata items, the format of which follows Exif, XMP, or MPEG-7 metadata.
Table VII. Image Properties
| Name | Type | Description |
|---|---|---|
| Decoder configuration and initialization | Descriptive Property | The information needed to initialize the decoder. The structure of this information is usually defined in the related image coding format specification. |
| Image spatial Extents (‘ispe’) | Descriptive Property | indicates the width and height of the associated image item |
| Pixel Aspect Ratio (‘pasp’) | Descriptive Property | has the same syntax as the PixelAspectRatioBox as defined in ISO/IEC 14496-12. |
| Color Information (‘colr’) | Descriptive Property | has the same syntax as the ColourInformationBox as defined in ISO/IEC 14496-12. |
| Pixel Information (‘pixi’) | Descriptive Property | indicates the number and bit depth of colour components in the reconstructed image of the associated image item. |
| Relative Location (‘rloc’) | Descriptive Property | indicates the horizontal and vertical offset of the reconstructed image item relative to the associated image item. |
| Image Properties for Auxiliary Images (‘auxC’) | Descriptive Property | Auxiliary images must be associated with an AuxiliaryTypeProperty. |
| Image Rotation (‘irot’) | Transformative | Rotation by 90, 180, or 270 degrees. |
| Clean Aperture (‘clap’) | Transformative | Cropping according to a given cropping rectangle. |
Derived Images
Derived images enable non-destructive image editing, where the original coded images are kept in the file, while new images, called derived images, can be introduced by specifying a transformation operation that is applied to one or more input images. HEIF specifies the generic structures used for storing derived images as items as well as a few specific types of derived images. Derived images can also have descriptive or transformative image properties. Item references of type 'dimg' specify the input image(s) of the derived image. The input images can be coded images or derived images. The derived image types specified in the HEIF standard are listed in Table VIII. Other types may be specified in other documents or later versions of the HEIF standard.
Table VIII. Derived Images
| Name | Description |
|---|---|
| Identity transformation (‘iden’) | Cropping and/or rotation by 90, 180, or 270 degrees, imposed through the respective transformative properties. |
| Image Overlay (‘iovl’) | Overlaying any number of input images in indicated order and locations onto the canvas of the output image. |
| Image Grid (‘grid’) | Reconstructing a grid of input images of the same width and height. |
Here are some examples of Derived Images
Image Metadata
HEIF files allow the storage of the metadata which can be related to images and image sequences. Such metadata can be information related to integrity checks, EXIF or XMP data or MPEG-7 related metadata. For image items, such metadata can be stored as a metadata item and reference the related image items with a ‘cdsc’ item reference. For image sequences, timed metadata tracks can refer to an image sequence track with a ‘cdsc’ track reference.
Indicating Alternative Representations
HEIF files can contain several image items representing the same image content (i.e. alternative representations). A media player may select one of the alternative representations of the same image content for displaying. A mechanism, known as entity grouping (defined in the GroupsListBox), is used for indicating alternate groups, which can contain both image items and media tracks (image sequences). The capability to define alternate groups between tracks and image items provides a unique mechanism for display initialization based on preferences of the media player and the content creator. Examples where alternate groups can be useful are:
- The same original image is coded into two or more image items with different properties, such as spatial resolution, bit-depth, or color gamut.
- The same original image is coded into two or more image items with a different coding format or profile.
- A timed image sequence, such as a cinemagraph, is an alternative to an image item.
Image sequences
Images that have a particular correlation (e.g. image bursts or animations like cinemagraphs) can be efficiently stored in HEIF files, thanks to the inherited media track features of ISOBMFF. Such images are called image sequences and they can reside in the same file with image items. Table IX lists the roles of the image sequence tracks which are currently defined.
Table IX. Roles of image sequence tracks.
| Role | Description |
|---|---|
| thumbnail image sequence | A smaller-resolution representation of a master image sequence. T |
| auxiliary image sequence | An image sequence that complements a master image sequence. For example, a sequence of alpha plane or depth map images. Can assist in displaying the master image sequence but is not typically displayed as such. |
| master image sequence | An image sequence that is not a thumbnail image sequence or an auxiliary image sequence. Typically contains full-resolution displayable images. |
HEIF enables the usage of Inter-picture prediction for compact storage of image sequences. Moreover, image sequence use cases may require faster access to individual images and the ability to edit individual images without affecting any other images. HEIF therefore includes the following two features:
- CodingConstraintsBox to indicate prediction constraints which applies to all the linked samples. For example, it can be indicated that all reference images used for inter-picture prediction are intra-coded images.
- DirectReferenceSamplesList to have sample-wise listing of referenced samples using the sample grouping mechanism of ISOBMFF. This feature enables decoding of only those samples that are needed for accessing a particular sample (e.g. random access) and avoids the need of sequential decoding of each sample starting from the previous intra-coded image.
Processing of image items by file players
Figure 1 illustrates how a file player processes the coded images and the derived images included in a file. The file player decodes a coded image into a reconstructed image. Similarly, the file player applies the operation of the derived image to the indicated one or more input images to obtain the respective reconstructed image. The descriptive image properties generally describe the reconstructed image, with the exception of the decoder configuration and initialization information, which is associated with the coded image. The transformative image properties, if any, are applied to the reconstructed image to obtain an output image. The output image can be displayed, when the coded image or the derived image is not a hidden image. The output image can also act as an input image to derived images.
Figure 1. Operations performed by file players
HEIF File Playback Options
The most important features that enable controlling the playback of an HEIF file are listed in Table X. Some of these features were introduced in the ISOBMFF or ISO/IEC 14496-15 and are explicitly inherited by HEIF, while other features were specifically designed for the HEIF standard.
Table X. Features controlling image sequence playback.
| Feature | First appeared in | Description |
|---|---|---|
| non-displayable sample | ISO/IEC 14496-15 | Is never displayed, but can be used as a reference for predicting other images in the track. |
| timed vs. non-timed playback | HEIF | In timed playback, the image sequence is played as video, whereas in non-timed playback the samples of the track are displayed by other means, such as an image gallery. Non-timed playback may be indicated e.g. when a track is used for achieving a better compression efficiency for an exposure stack. |
| edit list | ISOBMFF | A list of ranges of the image sequence track in their playback order. Enables modifying the playback order and pace of samples. |
| looping | HEIF | HEIF allows indicating edit list repetition e.g. for looping animations. The repetition can be indicated to last for a certain duration or be infinite. |
| cropping and rotation | ISOBMFF | Rectangular cropping and rotation by 90, 180, 270 degrees can be specified. |
Storing HEVC Encoded Pictures in HEIF: HEVC Image Files
The HEIF standard includes the specification for encapsulating HEVC-coded images and image sequences into HEIF-compliant files. The specification includes the following aspects:
- The sample entry format and the sample format of HEVC image sequence tracks are identical to those of the HEVC video tracks specified in ISO/IEC 14496-15. For compatibility with HEVC-capable video players, it is therefore advisable to create both an HEVC image sequence track and an HEVC video track that refer to the same coded images. This is called a Dual-Branded File and has an insignificant impact on the file size.
- The format of the decoder configuration information in the HEVC configuration item property is identical to that in the sample entry format. Likewise, the item format is identical to the sample format of HEVC video and image sequence tracks. This enables having the same intra-coded image as an image item and as a sample in an HEVC image sequence or video track.
- The configuration information of alpha planes and depth maps follows the respective supplemental enhancement information of HEVC.