In this tutorial we will describe what is H264, why to use it and how to fetch interesting information from the H264 stream

# Introduction

## Once upon a time, there was a parser…

In Membrane, we have a concept of an H.264 parser - a processing element that is designed to put the incoming H.264 stream "in order" by splitting the stream into packets containing the given amount of information (i.e. into separate video frames) or fetching some information about the video (like resolution or framerate). Such processing is often required before muxing the stream into a container or putting it into some transport protocol's packets. 
We used to use an Elixir wrapper on ffmpeg SDK that performed the type of processing described above - we have called that wrapper H264.FFMpeg.Parser, and it's still available in our repository: membrane_h264_ffmpeg_plugin, along with the H264 software decoder and encoder. 
Initially, such a solution perfectly fit our needs - we had a concise yet bug-free parser implementation, and we didn't need to get our hands dirty with the h264 stream structure at all. The difficult stuff was already done by the great FFmpeg team, and we were just harvesting the fruits of their labor.

However, it wasn't that long until we faced a need to add some custom functionalities to the parser. Nothing particularly complicated: dropping frames from the stream before the parameters or keyframe appeared for the first time. 
That was the moment that we started struggling with a tremendous number of problems, which mostly originated in the fact that the state of the processing element was held in three different places:
* in the Elixir's GenServer state
* in the C bridge code between Elixir and FFMpeg SDK 
* internally, in the data structures provided by FFMpeg SDK
A stack of options, parameters and behavioral patterns had been growing, some options had become more or less explicitly incompatible with others, and more and more time was being spent on debugging the problems caused by the use of our parser. A decision was made - we need to perform a refactor. Soon we realized that there was in fact nothing to refactor - the whole code would have needed to be thrown out and the element would have needed to be written from scratch.

## The new hope

We realized that we were more experienced with multimedia processing than we had been when the wrapped parser was first created. So we asked ourselves - why not create the parser in plain Elixir and get rid of some native FFMpeg dependency? It shouldn't be that difficult, should it? …
Long story short, we learned a lot ;) And we would like to share some of our findings with you.


Introduction

The first order of business was to put our knowledge in order and supplement it. There is no better knowledge source than the official ITU-T standard defining the H264 codec: "Advanced video coding for generic audiovisual services."
It might be overwhelming - the standard consists of 680 pages of printed text written in an ultra technical manner, however, for our needs, we could skip the parts concerning the video encoding and decoding processes - which make up most of the standard's volume.

## AVC, H.264 or MPEG-4 p. 10?
Firstly, we would like to explain the nomenclature that we will be using, since it might be misleading.
The codec, or more precisely - a coding standard, that we are describing is known under three names:
* **AVC**  - that stands for: Advanced Video Coding
* **H.264** - the name is derived from the name of **ITU-T** (*International Telecommunication Union  Standardization Sector*) (recommendation)[https://www.itu.int/rec/T-REC-H.264] that defines that coding standard 
* **MPEG-4 Part 10** - the name is derived from the name of a (document)[https://www.mpeg.org/standards/MPEG-4/10/] defining the standard, prepared by the **ISO/IEC JTC 1** (*Joint Technical Committee* (JTC) of the *International Organization for Standardization* (ISO) and the *International Electrotechnical Commission*) **Moving Picture Experts Group**

The existence of multiple names originates from the fact that the codec has been standardized as a cooperation of two groups - Video Coding Experts Group of ITU-T and Moving Picture Experts Group of ISO/IEC JTC 1. Within the text of the article we will most frequently use the "H.264" name, as it is most commonly used. 

## Why do we use H.264?
There are multiple reasons! The most important is that it allows us to represent video in a format that:
* supports compression - compression happens on multiple levels (for more information about some "tricks" used to compress the images and video, we invite you to read a tutorial published on [our website](https://membrane.stream/learn/digital_video_introduction)):
    * chroma subsampling - the compression standard, which takes advantage of the imperfections of human perception and removes the information about the  pixels' color that the human's eye can't even process
    * DCT (*discrete cosine transform*) and rounding it's coefficients to the integers
    * intra-frame prediction - each video frame is divided into so called "macroblocks" (square areas consisting of 16x16 pixels) and instead of storing the information about all the pixels of each frame, only information of all of the pixels for some frames is stored (these are so called "keyframes"), and the rest of the frames are stored as vectors describing the translation of given macroblocks
* allows for convenient multimedia transport - it's easy to pack an H.264 stream into multimedia containers (i.e. MP4) or packets (like RTP packets) and send it via the internet.
 
It's good to be aware that in fact the H.264 stream is designed to be wrapped in some other data structure, like a multimedia container (i.e. MP4) or some transport protocol packets (i.e. RTP packets) - we shouldn't store the H.264 stream as a plain binary file, as it might lead to many problems. The reason for that is that the H.264 stream itself does not contain all of the information that is required for a video to be properly displayed. Examples of such information are timestamps - H.264 stream does not contain timestamps at all, so unwrapping the H.264 stream from the MP4 container (that contains timing information additionally) and storing just the H.264 stream leads to a loss of information.

However, that's just theory - in practice, tools like FFmpeg allow you to read the plain H.264 files and do some "best effort" approximations of the missing information - i.e. when the framerate of the video is constant, they can simply calculate the missing timestamps by multiplying the ordering number of the given video frame by the inversion of the framerate.
That defines some kind of implicit "standard" of dealing with H.264 stream dumps, which we have also needed to take into consideration while working on our parser.

One might ask - why use H.264, when its successor, **H.265** standard (known also as **HEVC**, which stands for *high efficiency video compression*) is already available. The answer is - because H.264 is still widely used and still there is a need to cope with H.264 streams that are delivered to our system. At the same time, starting using H.265 comes with a great number of benefits, as it:
* allows for a better compression ratio. For a given video quality, up to 50% of bandwidth can be saved with H.265, compared to H.264, which has non-negligible impact on the maintenance costs of a media processing infrastructure.
* supports up to 8K UHDTV (in contrast to H.264, which supports up to 4K).
* supports up to 300 FPS (in contrast to H.264, which supports up to 59.94 FPS)

## H.264 layers - VCL vs NAL

In H.264 two so-called "layers" can be distinguished:
* **VCL** (*Video Coding Layer*) - this layer deals mostly with the video compression standard. H.264 encoders and decoders need to work with that layer, but in the case of the H.264 parser, we won't need to dig into it. 
* **NAL** (*Network Abstraction Layer*) - the layer that comes as a part of the network protocol's stack. It defines an abstraction over the VCL as it describes how the stream could be split into packets - Network Abstraction Layer units (we will refer to these units as NALu). Such an abstraction layer comes in handy for network devices i.e. it delivers information about how important a given packet is for the decoding process - based on that information, a network device knows which packets are high-priority and shouldn't be dropped (i.e. the ones that hold the keyframes), as dropping them would make the decoding impossible. 

The NAL was the subject of our interest when it comes to the H.264 parser - the parser's goal is to read the NAL units and fetch the information that interested us out of them.


![VCL vs NAL scheme](assets/images/VCL_vs_NAL.png)
<br>

What is H264

As noted above, from the NAL perspective, the H.264 stream is just a sequence of NAL units. 
But how do we know where one NAL unit ends and another one starts?
Well, the answer to that question is not straightforward, since the ITU specification does not define a single byte stream format. At the same time, one of the possible formats is described as an annex to the specification - Annex B. That format itself is usually referred to as Annex B and we will also stick to that name. 
We need to be aware that that format is not the only option - the other commonly used byte stream format is AVCC, known also as length prefix.
We will briefly discuss both of these formats.


## Annex B

Annex B is a format in which all NAL units are preceded with a so-called "start code" - one of the following byte sequences: three-byte `0x000001` or four-bytes `0x00000001`. 
In the case of that format, the H.264 stream might look as follows:
```
([0x000001][first NAL unit]) | ([0x000001][second NAL unit]) | ([0x000001][third NAL unit]) …
```
The existence of two "start codes" is dictated by the fact that the first variant is more applicable in certain situations than the second one. For a four-bytes variant, it's easy to find the start of a NAL unit sent through a serial connection, as it's easy to align the stream to 31 "0" bits followed by a single "1" bit. In other cases, to save space, it's common to use a three-byte variant.

The choice of `0x000001` and `0x00000001` as a NAL units separator wasn't unintentional - such a sequence of bytes does not encode much information (recall Shannon's formula for the amount of information) and that's why it's not frequent for the compression algorithm used in H.264 to produce such a sequence of bytes (the compression mechanism tries to encode as much information about video in the shortest bytes sequence possible). However, such a byte sequence might still appear in the stream (please also note, that some parts of the stream are metadata, which is not compressed) - and the format definition must take that into consideration. That's why an "emulation prevention" mechanism has been introduced. That mechanism is nothing more than a regular "escape character" mechanism - each of the following three-bytes sequences is not allowed to exist in the "escaped" byte stream:
* `0x000000` 
* `0x000001` 
* `0x000002`
* `0x000003`
and that's why it gets replaced correspondingly with the following sequences:
* `0x00000300`
* `0x00000301`
* `0x00000302`
* `0x00000303`
The "escape character" is 0x03 - also known as `emulation_prevention_three_byte`.

The representation with "emulation prevention three bytes" inserted is called **SODB** (*String of Data Bits*), and the representation without the "emulation prevention" bytes is called **RBSP** (*Raw Byte Sequence Payload*). Encoders need to take care of escaping the unallowed byte sequences (converting the byte stream from RBSP to SODB), while decoders are obliged with the reverse operation (converting from SODB to RBSP).


## AVCC

In this byte stream format, each NAL unit is preceded with a `nal_size` field, describing its length. That length might be stored with the use of 1, 2 or 4 bytes, and that is why a special additional header (known as "extradata", "sequence header" or "AVCDecoderConfigurationRecord") is required to be present in the stream to specify how many bytes are used to store the NAL unit's length. The stream then might look as follows: 
```
([extradata]) | ([length] NALu) | ([length] NALu) | …
```

The syntax of the extradata is described in MPEG-4 Part 15 "Advanced Video Coding (AVC) file format" section 5.2.4.1:
```
aligned(8) class AVCDecoderConfigurationRecord { 
    unsigned int(8) configurationVersion = 1; 
    unsigned int(8) AVCProfileIndication; 
    unsigned int(8) profile_compatibility; 
    unsigned int(8) AVCLevelIndication; 
    bit(6) reserved = '111111'b; 
    unsigned int(2) lengthSizeMinusOne; 
    bit(3) reserved = '111'b; 
    unsigned int(5) numOfSequenceParameterSets; 
    for (i=0; i< numOfSequenceParameterSets; i++) { 
        unsigned int(16) sequenceParameterSetLength;          
        bit(8*sequenceParameterSetLength) sequenceParameterSetNALUnit; 
    } 
    unsigned int(8) numOfPictureParameterSets; 
    for (i=0; i< numOfPictureParameterSets; i++) { 
        unsigned int(16) pictureParameterSetLength; 
        bit(8*pictureParameterSetLength) pictureParameterSetNALUnit; 
    }
}
```
The value of `lengthSizeMinusOne` field (which can be: 0, 1 or 3) determines how many bytes are used to store a length of NALu (that is, correspondingly, 1, 2 or 4 bytes). The most commonly used one is a 4-bytes size field that allows for NAL units of length up to `2^32-1` bytes.  

 
## NAL unit syntax

Each of the NAL units that we can find in the stream consists of a number of fields. Each field occupies a particular number of bytes and its value has a given semantics. ITU specifications define a syntax to describe NAL units' structure. We will become familiar with some field types defined within that syntax, as well see some exemplary NAL unit descriptions using that syntax.

### Data types

These are some of the types that can be met in the NAL units syntax (for all of the types available, please refer to chapter 7.2 of the ITU Specification):
* **f(N)** - "fixed pattern" sequence of N bits. I.e. f(8) might be some particular pattern of 8 bits - just like `00001111`. For information on how the pattern looks, you need to look in the semantics description of a given field. 
* **u(N)** - an unsigned integer written with the use of N bits - i.e. u(16) means an unsigned integer written with the use of 16 bits.
* **u(v)** - an unsigned integer written with the use of some number of bits. That number is usually specified by the value of some other, previously read field. In order to find out which field that is, you need to refer to the description of the semantics of that field.
* **ue(v)** - unsigned integer of variable size written with *Exponential-Golomb* coding. For more information about that coding you can refer [here](https://en.wikipedia.org/wiki/Exponential-Golomb_coding).
* **se(v)** - same as above, but this time the integer is signed.


### General NAL unit structure

Below there is a description of each NAL unit's structure. The structure description that you can find in chapter 7.3.1 of the ITU Specification is slightly different (and much more complicated). That is because it also covers the fact that there is a need to get rid of emulation_prevention_bytes. In order to simplify things, we assume that in the structure description we use, the byte stream has already been converted to RBSP. 

| nal_unit( NumBytesInNALunit )                          | Descriptor |
|------------------------------------------------------- |------------|
|{                                                       |            |
|   &nbsp;&nbsp;&nbsp;&nbsp; forbidden_zero_bit          |f(1)        |
|   &nbsp;&nbsp;&nbsp;&nbsp; nal_ref_idc                 |u(2)        |
|   &nbsp;&nbsp;&nbsp;&nbsp; nal_unit_type               |u(5)        |
|   &nbsp;&nbsp;&nbsp;&nbsp; nal_unit_data()             |            |
|}                                                       |            |

As you can see, each NAL unit starts with a single byte being a kind of a NAL unit header. Its first bit (`forbidden_zero_bit`) is always equal to 0, then comes a `nal_ref_idc` field consisting of two bits, and finally `nal_unit_type` - an unsigned integer written with the use of 5 bits, which determines the type of NAL unit. 

## NALu types

Each NAL unit can be of a different type. In this chapter we will describe only some of the NAL units types, while all the possible types are listed in the table available in "Appendix/NALu types".

There are two "classes" of NAL units types defined in ITU-T Specification's Annex A - **VCL** and **non-VCL** NAL units. The first one holds the encoded video, while the other does not contain video data at all. In our case, we found the following NAL unit types especially helpful:
* VCL:
    * *Coded slice of a non-IDR picture* (**non-IDR**) - contains a part or a complete non-keyframe (that is: P-frame or a B-frame)
    * *Coded slice of an IDR picture* (**IDR**) -  contains a part or a complete keyframe (also known as I-frame). The name IDR (that stands for *instantaneous decoding refresh*) originates from the fact that the decoder can "forget" the previous frames when the new keyframe appears, since it contains the complete information about the frame. 
* non-VCL:
    * *Sequence parameter set* (**SPS**) - contains metadata that is applicable to one or more *coded video sequences*. In that NALu you will find information allowing you to calculate the video resolution or H.264 profile.
    * *Picture parameter set* (**PPS**) - contains metadata applicable to one or more *coded pictures* 
    * *Access unit delimiter* (**AUD**) - just a separator between *access units*
    * *Supplemental enhancement information* (**SEI**) - contains some additional metadata that "assist in processes related to decoding, display or other purposes". At the same time, information stored in SEI is not required to restore the picture during the decoding process, so the decoders are not obliged to process SEI. In fact, SEI is defined as Annex D to the ITU specification. 

Some of the terms used in these (especially the ones written with *curved* font) might not be known to you yet - don't worry, keep on reading as they will be explained later. Once that happens, we invite you to go back to that section to reread it ;)


H264 NAL units

A video is a sequence of *frames* - images. Each frame might be (but it's not obliged to be!) written with [interlacing](https://en.wikipedia.org/wiki/Interlaced_video) - which means: it might consist of two *fields*. In such a case, a *field* is a group of alternating rows of a frame (even or odd ones). The two fields that form a single frame are referred to as the *bottom field* and the *top field*. 

In H.264, the term *picture* refers to both a single field of a frame, or to the frame itself. 

As noted previously, a H.264 stream consists of NAL units. Some NAL units hold the compressed video, while the others hold some metadata. The NAL units that hold video contain it in the form of a *slice*. Each slice is a group of *macroblocks* (along with a so-called *slice header*, containing some metadata). Right now it's sufficient for you to think about a macroblock as an area of 16x16 pixels on an image. Each picture is partitioned into macroblocks and such a division allows intra-frame prediction, as the translation vectors are defined between macroblocks of two pictures. We have reached the bottom - a macroblock is an atomic term in the context of H.264. 

Getting back to the slices - one or more slices create a *coded picture* - a compressed representation of a picture or a part of a picture (remember that a picture is a single video frame or a field). A coded picture might be one of two types:
* a *primary coded picture* -  a coded representation of a whole picture - it contains all the macroblocks of a picture (that means, that it's enough for a decoder to be able to display a picture based just on the primary coded picture)
* a *redundant coded picture* - a coded representation of a part of or a whole picture - as the name suggests ("redundant"), it repeats some information contained by a given primary coded picture
With the primary coded picture defined, we can define another crucial term used in H.264 - *access unit*.
An access unit is a sequence of NAL units that contain exactly one primary coded picture. Apart from the primary coded picture, a single access unit might also contain some *redundant coded pictures* (as well as some other video data or metadata, but let's not dig into the details). The crucial thing is that decoding a single access unit allows one to obtain a single *decoded picture*.
Below there is a scheme that shows a possible structure of an access unit:
![AU Scheme](assets/images/AU_scheme.png)
<br>     

As shown, an access unit might or might not start with a special type of NALu - access unit delimiter. Then comes zero or more SEI NAL units. Finally, there is the only obligatory part of the access unit - the sequence of NAL units that form a primary coded picture. Then comes zero or more sequences of NAL units forming redundant coded pictures. Then there is optionally the end of sequence NAL unit (it might help in separating the coded video sequences - you will learn what a coded video sequence is very soon). Finally, the end of stream NAL unit might optionally be present if the stream ends (but in the same manner as previously, it's not obligatory - the stream might end without the End of stream NAL unit appearing). 
The access unit might be of two types:
* IDR access unit - an access unit, in which primary coded picture is an IDR picture
* non-IDR access unit - an access unit, in which primary coded picture is not an IDR picture
Finally, we can define a coded video sequence - a sequence of access units that consists of exactly one IDR access unit at the beginning of the sequence, followed by zero or more non-IDR access units.

## Example - H.264 stream structure

Below there is an exemplary H.264 stream consisting of multiple NAL units. The NAL units are forming Access Units, and the Access Units are gathered together in the form of a Coded Video Sequence.
![H264 Stream Example](assets/images/H264_example_stream.png)
<br>     

Let's thoroughly inspect that stream's structure. 
First, there is a Sequence Parameter Set NAL. It contains some metadata common for zero or more coded video sequences. 
It's followed by two Picture Parameter Set NAL units - each of them contains some metadata relevant for zero or more coded pictures. In the slice's header of each NALu holding the video data (that is - IDR NALu or non-IDR NALu), there is a reference to the particular Picture Parameter Set NALu. Note that all the slices forming a single coded picture must refer to the same Picture Parameter Set NALu. In each Picture Parameter Set NALu, there is a reference to a Sequence Parameter Set - all the Picture Parameter Sets referred to by coded pictures from the same coded video sequence must refer to the same Sequence Parameter Set.
Right after the second Picture Parameter Set NALu a coded video sequence starts. It consists of three access units - the first one is an IDR access unit, while the remaining two are non-IDR access units. 
The first access unit starts with an optional SEI NALu, followed by two IDR NAL units. These two IDR NAL units form a single primary coded picture. The second access unit consists of two non-IDR NAL units, which also form a single primary coded video. That second access unit is probably much smaller in size compared to the first access unit - that is because it doesn't contain a keyframe. The last access unit starts with an optional SEI NALu, followed by a non-IDR NALu (which holds a complete primary coded picture) and an optional redundant coded picture that is contained in a single non-IDR NALu.


H264 stream structure

Fetching information from stream

| nal_unit_type | Content of NAL unit & RBSP syntax structure                                                              | NAL unit type class [Annex A] |
| ------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------ |
| 0             | Unspecified                                                                                              | non-VCL                        |
| 1             | Coded slice of a non-IDR picture *slice_layer_without_partitioning_rbsp()*                               | VCL                            |
| 2             | Coded slice data partition A *slice_data_partition_a_layer_rbsp()*                                       | VCL                            |
| 3             | Coded slice data partition B *slice_data_partition_b_layer_rbsp()*                                       | VCL                            |
| 4             | Coded slice data partition C *slice_data_partition_c_layer_rbsp()*                                       | VCL                            |
| 5             | Coded slice of an IDR picture *slice_layer_without_partitioning_rbsp()*                                  | VCL                            |
| 6             | Supplemental enhancement information (SEI) *sei_rbsp()*                                                  | non-VCL                        |
| 7             | Sequence parameter set *seq_parameter_set_rbsp()*                                                        | non-VCL                        |
| 8             | Picture parameter set *pic_parameter_set_rbsp()*                                                         | non-VCL                        |
| 9             | Access unit delimiter *access_unit_delimiter_rbsp()*                                                     | non-VCL                        |
| 10            | End of sequence *end_of_seq_rbsp()*                                                                      | non-VCL                        |
| 11            | End of stream *end_of_stream_rbsp()*                                                                     | non-VCL                        |
| 12            | Filler data *filler_data_rbsp()*                                                                         | non-VCL                        |
| 13            | Sequence parameter set extension *seq_parameter_set_extension_rbsp()*                                    | non-VCL                        |
| 14            | Prefix NAL unit *prefix_nal_unit_rbsp()*                                                                 | non-VCL                        |
| 15            | Subset sequence parameter set *subset_seq_parameter_set_rbsp()*                                          | non-VCL                        |
| 16 - 18       | Reserved                                                                                                 | non-VCL                        |
| 19            | Coded slice of an auxiliary coded picture without partitioning *slice_layer_without_partitioning_rbsp()* | non-VCL                        |
| 20            | Coded slice extension *slice_layer_extension_rbsp()*                                                     | non-VCL                        |
| 21            | Coded slice extension for depth view components *slice_layer_extension_rbsp()* (specified in Annex I)    | non-VCL                        |
| 22 - 23       | Reserved                                                                                                 | non-VCL                        |
| 24 - 31       | Unspecified                                                                                              | non-VCL                        |