High Efficiency Image File Format

History

The requirements and main use cases of HEIF were defined in 2013. The technical development of the specification took about 1.5 years and was finalized in summer 2015.

Use Cases

Some main use cases supported by HEIF are as follows:

As HEVC provides support for various chroma formats and sample fidelities up to lossless coding, the format can serve the whole spectrum of use cases from today's consumer devices storing images typically at 8 bits per sample to high-end professional devices with sample fidelity and dynamic range requirements going all the way up to 16 bits per sample. Some usage scenarios that are facilitated by HEIF are described below:

Digital cameras and smartphones

In order to save storage space, HEIF-encapsulated HEVC-coded images can be used for compressing the full-resolution images while keeping a lower-resolution JPEG copy (e.g. at 4K resolution or below) for on-screen displaying purposes.

Digital cameras and smartphones can use HEIF to achieve single-file packaging of burst photos, focal stacks, and exposure stacks. Similarly, simultaneously captured video and still images can be stored in the same HEIF file. HEIF also enables storage of any image collections into a single file, hence they can be shared easily.

Web pages and Internet-connected image applications

The picture element of HTML5.1 provides the capability of indicating multiple alternatives for the same image, out of which the web browser can select the one that best suits its purpose. A motivation for web pages and connected applications to start using HEIF is to reduce the web page and/or image content download times.

Image editing

Changing of the orientation and cropping are basic features of HEIF and require no re-encoding of the images. Additionally, HEIF introduces a framework for non-destructive editing operations, which can be specified by external specifications. This feature can be used by image editing applications so that the editing instructions are kept in the same file as the original image.

Features

In ISOBMFF, a continuous or timed media/metadata stream forms a track, whereas static media or metadata are stored as items. Consequently, HEIF has the following basic design:

Keeping the above-mentioned design principles in mind, HEIF files can store the following types of data:

Examples of above-mentioned entities can be found here.

Feature Comparison

Below table shows a comparison of different image formats:

1 In GIF and indexed color PNG encoding, lossy color quantization is applied while the color-quantized image is losslessly compressed.

2 PCM, µ-Law PCM and ADPCM encapsulated in RIFF WAV

3 Only for animations and tiling/overlaying

4 JPX is a box-structured format compatible with ISOBMFF. However, only the File Type box is common in JPX and ISOBMFF.

5 Encapsulation of JPEG-2000 and JPEG-XR have been specified for JPX container. Mappings for other codecs could be similarly specified.

6 JPX (ITU-T T.800 and T.801) specifies an own metadata schema, but is capable of carrying an XML formatted metadata, such as XMP.

7 JPX can contain media complying with ISOBMFF (or derivatives thereof). No accurate synchronization between JPX animations and other media.

8 TIFF as a container format facilitates extensions to other coding formats.

9 Only for animations, thumbnails, and alpha planes. Non-timed image collections not supported.

10 HEVC Main 4:4:4 16 Still Picture profile, Level 8.5, with additional constraints

11 Can be enabled through the MP extension

12 A palette index for fully transparency can be specified.

The following references are used in the comparison:

Note: It is acknowledged that a summary such as that in the comparison table might be somewhat incomplete when it comes to features of different formats. For example, the table does not cover some of the extensions of JPEG.

Compression performance comparison

The table below illustrates coding efficiency of HEVC intra coding with respect to well-known still picture codecs. The results indicate that JPEG would require 2.39 times the file size in order to achieve the same objective picture quality. For JPEG-XR this number is 1.66 and for JPEG-2000 it is 1.44.

For natural content, the restricted inter coding can typically provide two to three times better compression than intra picture coding. In special cases like animations where majority of the scene is static the compression efficiency can significantly exceed those levels and be tens of times more efficient than intra coding. the following table provides several comparison information:

Syntax and Structure

HEIF conforms to the ISO Base Media File Format (ISO/IEC 14496-12). Image items are stored in root level meta box and image sequences are stored as video media tracks.

A file may contain both image items and image sequence tracks along with other media. For example, it is possible to create a file, which includes image items or image sequence tracks conforming to HEIF, and audio and timed text tracks conforming to any derivative format of the ISOBMFF.

Files conforming to ISOBMFF consist of a sequence of data structures called boxes, each containing a four-character code (4CC) indicating the type of the box, the size of the box in terms of bytes, and the payload of the box. Boxes may be nested, i.e. a box may contain other boxes. ISOBMFF and HEIF specify constraints on the allowed box order and hierarchy.

Image Items

The roles of image items as specified in HEIF are listed and described below. Note that a single image can be associated with more than one role.

HEIF allows the storage of image properties which are shared among different image items. There are two types of properties: descriptive and transformative. Descriptive properties provide information about the image item without applying modifications on the image itself. Transformative properties provide information about the transformative modification that needs to be done on the image item. The HEIF standard specifies transformative properties for rotation by 90, 180, and 270 degrees and cropping. In addition to descriptive image properties, image items can optionally be characterized with metadata items, the format of which follows e.g. Exif or XMP metadata.

Derived Images

Derived images enable non-destructive image editing, where the original coded images are kept in the file, while new images, called derived images, can be introduced by specifying a transformation operation that is applied to one or more input images. HEIF specifies the generic structures used for storing derived images as items as well as a few specific types of derived images. The item type of a derived image item indicates the transformation operation, while the item payload contains the input parameters to the operation. The input images can be coded images or derived images. The derived image types specified in the HEIF standard are listed below. Other types may be specified in other documents or later versions of the HEIF standard.

Image Sequences

An image sequence track inherits the properties and features of a video track of ISOBMFF. A key difference between an image sequence track and a video track lies in the interpretation of the timing information given for the track: While the timing information given for a video track adheres the decoding and output timing of the contained video bitstream, the timing information of an image sequence track can represent the capture times (e.g. for separate images of an exposure stack) or the suggested display timing (e.g. for a slide show). It can be indicated whether an image sequence track should be played as a timed sequence or by some non-timed means, such as an image gallery.

HEIF allows the use of inter-picture prediction (a.k.a. inter coding), which can provide significant coding efficiency improvement to image sequences when images are correlated. Such correlation is especially evident in content like image bursts or animation clips. When inter coding is in use, accessing and decoding a particular picture from an image sequence also requires decoding of its reference pictures for inter prediction. HEIF specifies indications assisting players in random access.

Roles can be indicated for image sequence tracks similarly to image items. Below is the list of roles of the image sequence tracks:

ISOBMFF specifies indications for video and image sequence tracks, such as width and height of the coded images, color information, such as the ICC profile or color primaries, rectangular cropping, transformation matrix, e.g. for rotation. Additionally, timed metadata track(s) can be used to convey Exif, XMP, and MPEG-7 metadata applying to time-parallel samples of the linked image sequence track.

The most important features that enable controlling the playback of an HEIF file are listed below. Some of these features were introduced in the ISOBMFF or ISO/IEC 14496-15 and are explicitly inherited by HEIF, while other features were specifically designed for the HEIF standard.

Supported Media Codecs

HEIF specifies a structural format, from which codec-specific image formats can be derived. HEIF also includes the specification for encapsulating images and image sequences conforming to the High Efficiency Video Coding (HEVC, ISO/IEC 23008-2 | ITU-T Rec. H.265). Future versions of the standard may include other video and still image media codecs.

HEVC Image File Format

The HEIF standard includes the specification of encapsulating HEVC-coded images and image sequences into HEIF-compliant files. The specification includes the following aspects:

Brands and MIME Types

Files conforming to HEIF start with a FileTypeBox as defined in the ISOBMFF standard, which contains a list of brands the file complies with. As the FileTypeBox is located at the start of the file, it provides easily accessible indications of the file contents to media players. Each brand is identified by its unique four-character code. The specification of a brand can include requirements and constraints for files of the brand and for file players supporting the brand. A brand included in the FileTypeBox permits a player that supports the requirements of the brand to play the file.

The brands specified in the HEIF standard are presented in the table below. The HEIF standard specifies the 'mif1' and 'msf1' structural brands. Additionally, HEVC-specific brands are specified as listed in Table I. The dedicated brand names, 'heic' and 'hevc' indicates that HEVC Main Profile is utilized.

Internet media types, also known as MIME (Multipurpose Internet Mail Extensions) types are used by various applications to identify the type of a resource or a file. MIME types consist of a media type ('image' in the case of HEIF files), a subtype, and zero or more optional parameters. For multi-purpose files, the selection of the subtype can be made on the basis of the primary use of the file.

An optional codecs MIME parameter can be present to indicate the used coding formats of the tracks and items present in the file. The codecs MIME parameter also includes the profile-tier-level value to which an HEVC-coded image item or an image sequence track conforms.

As of January 2016 image/heif, image/heif-sequence, and image/heic were not registered as MIME types.

Related Standards

Other Image Formats

Implementations

1. High Efficiency Image File Format web-site and publicly available source code (Nokia Technologies, 2015): The source code is written in C++. The reader implementation and API is also ported to JavaScript and used in the HEIF web-site. The source code repository contains the following components:
2. ISO Base Media File Format (ISOBMFF) parser/writer source code
3. HEIF Reader API and Library
4. HEIF Writer Library and Example Application
5. HEIF Reader API Usage Example Code
6. HEIF Source Code Documentation
7. HEIF Web Site Content
8. HEIF Reader JavaScript Implementation