# Architecture

Now let's discuss how the architecture of our solution will look like.
It will be a little different from the RTMP to HLS architecture.
The main component will be the pipeline, which will ingest RTP stream and convert it to HLS. Beyond that we will also need a Connection Manager, which will be responsible for establishing an RTSP connection with the server.

![image](assets/rtsp_architecture.drawio.png)

When initializing, the pipeline will start a Connection Manager which starts an RTSP connection with the server. Once the connection is fully established, the pipeline will be notified.

Let's take a closer look on each of those components:

## Connection Manager
The role of the connection manager is to initialize RTSP session and start playing the stream.
It communicates with the server using the [RTSP requests](https://antmedia.io/rtsp-explained-what-is-rtsp-how-it-works/#RTSP_requests). In fact, we won't need many requests to start playing the stream - take a look at the desired message flow:

![image](assets/connection_manager.drawio.png)

First we want to get the details of the video we will be playing, by sending the `DESCRIBE` method. 
Then we call the `SETUP` method, defining the transport protocol (RTP) and client port used for receiving the stream.
Now we can start the stream using `PLAY` method.

## Pipeline

The pipeline consists of a couple elements, each of them performing a specific media processing task. You can definitely notice some similarities to the pipeline described in the [RTMP architecture](02_RTMP_Introduction.md). However, we will only be processing video so only the video processing elements will be necessary.

![image](assets/rtsp_pipeline.drawio.png)

We have already used the, `H264 Parser`, `MP4 H264 Payloader`, `CMAF Muxer` and `HLS Sink` elements in the RTMP pipeline, take a look at the [RTMP to HLS architecture](03_RTMP_SystemArchitecture.md) chapter for details of the purpose of those elements.

Let us describe briefly what is the purpose of the other components:

### UDP Source
This element is quite simple - it receives UDP packets from the network and sends their payloads to the next element.

### RTP SessionBin
RTP SessionBin is a Membrane's Bin, which is a Membrane's container used for creating reusable groups of elements. In our case the Bin handles the RTP session with the server, which has been set up by the Connection Manager.


RTSP to HLS system architecture

Create your own media broadcasting solution!

We are glad you have decided to continue your journey into the oceans of multimedia with us!
In this tutorial, you will be able to see the Membrane in action - as we will prepare a pipeline responsible for converting
the incoming RTMP stream into an HLS stream, ready to be easily served on the internet.
Later on, we will substitute the front part of our pipeline, so that to make it compatible with another popular streaming protocol - RTSP.
That is how we will try to prove the great dose of flexibility that comes with the use of the Membrane Framework!

## General tutorial's structure
As mentioned earlier, the tutorial consists of two parts:
* "RTMP to HLS" - this part describes how to create a pipeline capable of receiving RTMP stream, muxing it into CMAF files, and serving them with the use of HTTP
* "RTSP to HLS" - this part is based on the previous one and describes how to change the pipeline from the previous part so that to make it capable of handling the RTSP stream instead of the RTMP stream.

As a result of each part, you will have a web application capable of playing media streamed with the use of the appropriate protocols - RTMP and RTSP.

We strongly encourage you to follow the tutorial part by part, as the different chapters are tightly coupled. However, in case you wanted just to see the result solutions or even one of them, we invite you to take a look at the appropriate directories of the `membrane_demo` repository:
* [RTMP to HLS](https://github.com/membraneframework/membrane_demo/tree/master/rtmp_to_hls)
* [RTSP to HLS](https://github.com/membraneframework/membrane_demo/tree/master/rtsp_to_hls)

General introduction

## What do we want to achieve?

Imagine that we need to build a video streaming platform. That means - we want a particular system user, the streamer, to stream its multimedia on a server, where it will be available for other participants - the viewers.
We need to take advantage of the fact, that there will be only one person streaming, and take into consideration, that there might be multiple, possibly many, viewers. What's the best solution for such a use case?
Well, as the tutorial name suggests - we can use RTMP for the streamer to stream its multimedia to the server, and make them accessible for the viewers by broadcasting them via HLS protocol!
Although it might sound quite complicated, with the Membrane Framework this will be easier than you can expect!
As a final product of this tutorial, we want to have a completely working solution for broadcasting a stream. It will consist of two parts:

- The pipeline, responsible for receiving an RTMP stream and preparing an HLS stream out of it
- The web player, capable of playing the HLS stream

The Membrane Framework will be advantageous in the first part where we will create a Membrane's Pipeline for converting the stream.
When it comes to the web player, we will use an existing solution - the [HLS.js](https://github.com/video-dev/hls.js/) player.

## Why one would need such a solution?

You might wonder why one would need to convert an RTMP stream into an HLS stream at all - couldn't we simply make the streamer broadcast its multimedia with the RTMP to all the viewers?
Technically speaking we could...but surprisingly it wouldn't be the easiest solution, since each of the viewers would need to act as an RTMP server. And definitely, it wouldn't be a solution that would scale - RTMP is based on TCP which implies, that there is no broadcast mechanism. It would be the streamer who would need to keep the direct connection to each of the viewers.
In contrast, the solution described above has plenty of advantages - the streamer needs to create a single connection with the RTMP server, and then the multimedia can be shared with the use of a regular HTTP server which is designed to serve multiple clients.

## A brief description of the technology we will use

As stated previously, we will use the preexisting protocols. You might find reading about them beneficial, and that is why we provide you with some links to a brief description of the technology we will be using:

- [How does RTMP work?](https://blog.stackpath.com/rtmp/) - if you are interested in how the connection in RTMP is established, take your time and read this short description!
- [What is RTMP and why should we care about it?](https://www.wowza.com/blog/rtmp-streaming-real-time-messaging-protocol) - here you can find another short description of RTMP, with the focus laid on the history of the protocol and comparison with other available protocols. Whatsmore, there is a comprehensive explanation of why we need to transcode RTMP to some HTTP-based protocol, just like HLS - which is the use case in our tutorial.
- [HLS behind the scenes](https://www.toptal.com/apple/introduction-to-http-live-streaming-hls) - dig into the ideas which stand behind the HLS, one of the most common HTTP-based streaming protocols.
- [Have you heard about CMAF?](https://www.wowza.com/blog/what-is-cmaf) - if you haven't make sure to read what is the purpose of having a `Common Media Application Format`!


RTMP introduction

As noted previously, our system will be divided into two parts:

- the server, which is responsible for receiving the RTMP stream, converting it into HLS, and then publishing the created files as an HTTP server
- the client, responsible for playing the incoming HLS stream.

Considering our point of view, the first part will be more complex, since we will need to prepare the processing pipeline on our own. Let's start with that part.

## The server

Below you can find the desired flow of data in our application server:
![Pipeline scheme](assets/RTMP_to_HLS_pipeline.drawio.png)

As you can see, the server architecture consists of two bigger processing units - in Membrane Framework we refer to such units as 'bins'.
The following bins will be used in our solution:

- RTMP Source Bin - responsible for receiving RTMP stream
- HLS Sink Bin - responsible for "packing" the data into the container format

Each of these bins consists of some subunits (in Membrane we refer to them as _elements_), responsible for completing an atomic multimedia processing task - i.e. parsing or payloading the incoming stream.

Let's take a quick walk through the whole processing line and describe more specifically what its given parts are meant to do.

### RTMP Source

The very first element, just at the beginning of the RTMP Source Bin is the RTMP Source. It acts as an RTMP server, listening for the incoming stream. It will be exposing a TCP port, and the streamer will be able to connect and start sending RTMP packets through that channel.
The incoming packets will be demuxed - meaning, that the packets will be unpacked and split, based on the track (video or audio) which data they are transporting.

## Parsers

Buffers containing the given track data will be sent to the appropriate parser - H264 parser for video data and AAC parser for audio data.

### H264 Parser

H264 parser is quite a complex element, designed to read the incoming H264 stream. We have prepared a [separate, supplemental chapter of this tutorial](13_H264_codec.md) to describe the H264 codec and our parser's implementation - we invite you to read it, however, that knowledge is not necessary to successfully run the application.

### AAC Parser

At the same time, parsing happens to the buffers containing audio data. Since the audio track has been encoded with the use of AAC, we need [AAC parser](https://github.com/membraneframework/membrane_aac_plugin) to decode it.
That part of the tutorial does not describe the AAC codec itself, and in case you are interested in digging into that codec, we highly recommend visiting [that page](https://wiki.multimedia.cx/index.php/Understanding_AAC).

### HLS converter

Once data reaches the HLS bin, it needs to be put into the appropriate container files. Since we will be using Common Media Application Format (CMAF) to distribute our media, we need to put all the tracks into the `fragmented MP4` container. The first step is to payload the track's data. Payloading transforms the media encoded with a given codec into a form that is suitable to be put into the container.
The audio track is payloaded within the `Membrane.MP4.Payloader.AAC` module and the video track is payloaded with an H264 payloader, implemented by the `Membrane.MP4.Payloader.H264` module.
The payloaded streams (both the audio stream and the video stream) are then put in the result CMAF file - and the `Membrane.MP4.Muxer.CMAF` is the element responsible for that process. The name 'muxer', relates to the process
of 'muxing' - putting multiple tracks in one file, which is done by that element. Furthermore, the `CMAF muxer`, splits the streams into so-called segments - stream parts of the desired duration. Along that procedure, the manifest files, which contain metadata about the media and names of the appropriate segment file names, are generated.
Finally, the output files: '.m4s' files for each of the segments, as well as manifest files, are written on the disk.
Let's take a look at how do the generated files look like:
![Pipeline scheme](assets/output_files_structure.png)
As we can see, there are:

- index.m3u8 - the manifest file which contains metadata about the media, as well as the URI of the manifest files for both the video and audio track.

```
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:TYPE=AUDIO,NAME="audio_default_name",GROUP-ID="audio_default_id",AUTOSELECT=YES,DEFAULT=YES,URI="audio.m3u8"
#EXT-X-STREAM-INF:BANDWIDTH=4507734,CODECS="avc1.42e00a",AUDIO="audio_default_id"
video_g2QABXZpZGVv.m3u8
```

- audio.m3u8 - the manifest file for the audio track.

```
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:8
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-DISCONTINUITY-SEQUENCE:0
#EXT-X-MAP:URI="audio_header_g2QABWF1ZGlv_part0_.mp4"
#EXTINF:8.0,
audio_segment_9_g2QABWF1ZGlv.m4s
#EXTINF:8.0,
audio_segment_10_g2QABWF1ZGlv.m4s
#EXTINF:8.0,
audio_segment_11_g2QABWF1ZGlv.m4s
```

That file contains some metadata (like the desired duration of the segment), along with the URI pointing to the '.mp4' header file of the fMP4 container, and the list of the segment files, in '.m4s' format.
Each segment is described with `EXTINF` directive, indicating the duration of the segment, in seconds.

- video\_<identifier>.m3u8 - the manifest file for the video track. Its structure is similar to the structure of the audio.m3u8 file. However, it is worth noting, that the desired duration of each segment is equal to 10 seconds - and the '.m4s' files are holding a video stream of that length.

```
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-DISCONTINUITY-SEQUENCE:0
#EXT-X-MAP:URI="video_header_g2QABXZpZGVv_part0_.mp4"
#EXTINF:10.0,
video_segment_8_g2QABXZpZGVv.m4s
#EXTINF:10.0,
video_segment_9_g2QABXZpZGVv.m4s
```

- audio_header\_<identifier>_part\<part_ordering_number>_.mp4 - a binary file, the header meant to describe the audio track in the format required by the fragmented MP4 container.
- video_header\_<identifier>\_part\<part_ordering_number>.mp4 - a binary file, the header meant to describe the video track in the format required by the fragmented MP4 container.
- audio_segment\_\<segment_ordering_number>\_\<>.m4s - particular segments containing fragments of the audio stream.
- video_segment\_\<segment_ordering_number>\_\<>.m4s - particular segments containing fragments of the video stream.

### HTTP server

The HTTP server sends the requested files to the clients. Its implementation is based on the API provided by the Phoenix Framework.

## The client

When it comes to our application client, we will be using the `Hls.js` library, available in the javascript ecosystem.
We won't dig into the details of how the client application media player is implemented, however, we would like to point out some steps which take place while playing the media.
First, the client needs to request the master manifest file. Based on that file, the client asks for the appropriate audio and video track manifest files.
With these files, the client knows the MP4 header files for both the tracks, as well as knows the filenames of the particular segments, along with their duration. The client downloads the MP4 header file and starts downloading the appropriate segment files, based on which part of the media should be played in the nearest future.


RTMP to HLS system architecture

This time we won't create the application from the scratch nor will we be based on some kind of template.
Since most of the application's code is media-agnostic, we don't see a point in focusing on it. Instead, we will indicate the most crucial parts of the
[RTMP to HLS demo](https://github.com/membraneframework/membrane_demo/tree/master/rtmp_to_hls), one of many publicly available [Membrane demos](https://github.com/membraneframework/membrane_demo/).
The Membrane demos are supplementing the Membrane Tutorials on providing a source of knowledge in the field of multimedia processing - we warmly invite you to look at them as you may find some interesting
use cases of Membrane in real-life scenarios.

## Running the application

In order to run the demo, you need to clone the Membrane demos repository first:

```console
git clone https://github.com/membraneframework/membrane_demo
cd membrane_demo/rtmp_to_hls
```

Once in the project directory, you need to get the dependencies of the project:

```console
mix deps.get
```

Finally, you can run the application with the following command:

```console
mix phx.server
```

The server will be waiting for an RTMP stream on localhost:9009, and the client of the application will be available on localhost:4000.

## Exemplary stream generation with OBS

You can send RTMP stream onto `localhost:9006` with your favorite streaming tool. As an example, we will show you how to generate an RTMP stream with
[OBS](https://obsproject.com).
Once you have OBS installed, you can perform the following steps:

1. Open the OBS application
1. Open the `Settings` windows
1. Go to the `Stream` tab and set the value in the `Server` field to: `rtmp://localhost:9006` (the address where the server is waiting for the stream)
1. Finally, you can go back to the main window and start streaming with the `Start Streaming` button.

Below you can see how to set the appropriate settings (steps 2) and 3) from the list of steps above):
![OBS settings](assets/OBS_settings.webp)

Once the streaming has started, you can visit `localhost:4000`, where the client application should be available. After a few seconds, you should be able to play
the video you are streaming. Most likely, a kind of latency will occur and the stream played by the client application will be delayed. That latency can reach values as great as 10 seconds, which is the result
of a relatively complex processing taking place in the pipeline. Note, that the latency here is not a result of the packets traveling far via the network, as the packets do not leave our computer in that setup - in the case of a real streaming server, we would need to take into consideration the time it takes a packet to reach the destination.


Running the RTMP to HLS demo

In this chapter, we will discuss the multimedia-specific part of the application.

## The pipeline

Let's start with `lib/rtmp_to_hls/pipeline.ex` file. All the logic is put inside the [`Membrane.Pipeline.handle_init/1`](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#c:handle_init/1) callback,
which is invoked once the pipeline is initialized.

**_`lib/rtmp_to_hls/pipeline.ex`_**

```elixir
 @impl true
 def handle_init(_opts) do
 ...
 children: %{
     src: %Membrane.RTMP.SourceBin{port: 9009},
     sink: %Membrane.HTTPAdaptiveStream.SinkBin{
         manifest_module: Membrane.HTTPAdaptiveStream.HLS,
         target_window_duration: 20 |> Membrane.Time.seconds(),
         muxer_segment_duration: 8 |> Membrane.Time.seconds(),
         storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{directory: "output"}
     }
 },
 ...
 end
```

First, we define the list of children. The following children are defined:

- `:src` - a `Membrane.RTMP.SourceBin`, an RTMP server, which, according to its `:port` configuration, will be listening on port `9009`. This bin will be acting as a source for our pipeline. For more information on RTMP Source Bin please visit [the documentation](https://hexdocs.pm/membrane_rtmp_plugin/Membrane.RTMP.SourceBin.html).
- `:sink` - a `Membrane.HTTPAdaptiveStream.SinkBin`, acting as a sink of the pipeline. The full documentation of that bin is available [here](https://hexdocs.pm/membrane_http_adaptive_stream_plugin/Membrane.HTTPAdaptiveStream.SinkBin.html). We need to specify some of its options:
- `:manifest_module` - a module which implements [`Membrane.HTTPAdaptiveStream.Manifest`](https://hexdocs.pm/membrane_http_adaptive_stream_plugin/Membrane.HTTPAdaptiveStream.Manifest.html#c:serialize/1) behavior. A manifest allows aggregate tracks (of a different type, i.e. an audio track and a video track as well as many tracks of the same type, i.e. a few video tracks with different resolutions). For each track, the manifest holds a reference to a list of segments, which form that track. Furthermore, the manifest module is equipped with the `serialize/1` method, which allows transforming that manifest to a string (which later on can be written to a file). In that case, we use a built-in implementation of a manifest module - the `Membrane.HTTPAdaptiveStream.HLS`, designed to serialize a manifest into a form required by HLS.
- `:target_window_duriation` - that value determines the minimal manifest's duration. The oldest segments of the tracks will be removed whenever possible if persisting them would result in exceeding the manifest duration.
- `:muxer_segment_duration` - the maximal duration of a segment. Each segment of each track shouldn't exceed that value. In our case, we have decided to limit the length of each segment to 8 seconds.
- `:storage` - the sink element, the module responsible for writing down the HLS playlist and manifest files. In our case, we use a pre-implemented `Membrane.HTTPAdaptiveStream.FileStorage` module, designed to write the files to the local filesystem. We configure it so that the directory where the files will be put in the `output/` directory (make sure that that directory exists as the storage module won't create it itself).

The fact that the configuration of a pipeline, which performs relatively complex processing, consists of just two elements, proves the power of [bins](/basic_pipeline_extension/02_Bin.md). Feel free to stop for a moment and read about them if you haven't done it yet.

After providing the children's specifications, we are ready to connect the pads between these children. Take a look at that part of the code:
**_`lib/rtmp_to_hls/pipeline.ex`_**

```elixir
 @impl true
 def handle_init(_opts) do
 ...
 links: [
     link(:src)
     |> via_out(:audio)
     |> via_in(Pad.ref(:input, :audio), options: [encoding: :AAC])
     |> to(:sink),
     link(:src)
     |> via_out(:video)
     |> via_in(Pad.ref(:input, :video), options: [encoding: :H264])
     |> to(:sink)
 ]
 ...
 end
```

The structure of links reflects the desired architecture of the application, described in the [chapter about system architecture](../videoroom/3_SystemArchitecture.md).
`:src` has two output pads: the `:audio` pad and the `:video` pad, transferring the appropriate media tracks.
The source's `:audio` pad is linked to the input `:audio` pad of the sink - along with the `:encoding` option. That option is an atom, describing the codec which is used to encode the media data - when it comes to audio data,
we will be using AAC coded.
At the time of the writing, only `:H264` and `:AAC` codecs are available to be passed as an `:encoding` option - the first one is used with video data, and the second one is used with audio data.
By analogy, the source's `:video` pad is linked with the sink's `:video` pad - and the `:encoding` to be used is H264.

The final thing that is done in the `handle_init/1` callback's implementation is returning the desired actions:
**_`lib/rtmp_to_hls/pipeline.ex`_**

```elixir
 @impl true
 def handle_init(_opts) do
 ...
 { {:ok, spec: spec, playback: :playing}, %{} }
 end
```

The first action is the `:spec` action, which spawns the children. The second action changes the playback state of the pipeline into the `:playing` - meaning, that data can start flowing through the pipeline.

## Starting the pipeline

The pipeline is started with `Supervisor.start_link`, as a child of the application, inside the `lib/rtmp_to_hls/application.ex` file:

**_`lib/rtmp_to_hls/application.ex`_**

```elixir
 @impl true
 def start(_type, _args) do
    children = [
        # Start the Pipeline
        Membrane.Demo.RtmpToHls,
        ...
    ]
    opts = [strategy: :one_for_one, name: RtmpToHls.Supervisor]
    Supervisor.start_link(children, opts)
 end
```

## HLS controller

The files produced with the pipeline are written down to the `output/` directory. We need to make them accessible via HTTP.
The Phoenix Framework provides tools to achieve that - take a look at the `RtmpToHlsWeb.Router`:
**_`lib/rtmp_to_hls_web/router.ex`_**

```elixir
scope "/", RtmpToHlsWeb do
    pipe_through :browser

    get "/", PageController, :index
    get "/video/:filename", HlsController, :index
 end
 ```

We are directing HTTP requests on `/video/:filename` to the HlsController, whose implementation is shown below:
**_`lib/rtmp_to_hls_web/controllers/hls_controller.ex`_**

```elixir
defmodule RtmpToHlsWeb.HlsController do
    use RtmpToHlsWeb, :controller

    alias Plug

    def index(conn, %{"filename" => filename}) do
        path = "output/#{filename}"

        if File.exists?(path) do
            conn |> Plug.Conn.send_file(200, path)
        else
            conn |> Plug.Conn.send_resp(404, "File not found")
        end
    end
end
```

This part of the code is responsible for sending the file `output/<filename>` to the HTTP request sender.


RTMP to HLS - pipeline

In this really short chapter, we will take a look at how one can play the video delivered with HLS on the website.

In our demo, we are taking advantage of the HLS.js player.
With such a tool onboard, the creation of the player is simple as that:
**_`lib/rtmp_to_hls_web/templates/page/index.html.heex`_**

```js
<script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>

<div class="container">
  <video id="player" controls autoplay class="Player"></video>
</div>
<script>
  var video = document.getElementById('player');
  var videoSrc = window.location.origin + `/video/index.m3u8`;
  if (Hls.isSupported()) {
    var hls = new Hls();
    hls.loadSource(videoSrc);
    hls.attachMedia(video);
  } else if (video.canPlayType('application/vnd.apple.mpegurl')) {
    video.src = videoSrc;
  }
</script>
```

First, we load a `hls.js` script.
Then, in the DOM we add a div holding a video element of a class `Player`.
Finally, we add our custom script, which decorates the video player element with the functionalities
provided by the `hls.js` library.
The `videoSrc` is a URL of the manifest file of the HLS playlist. As described in the previous chapter, the files from the playlist are available
at `/video/<filename>`.
If the HLS is supported by the client's browser, we create an object of type `Hls` (that class is a part of the `hls.js` library), then we set its source manifest file and a DOM element that acts as a video player. For more options that can be specified for the player, see the [documentation](https://github.com/video-dev/hls.js/blob/master/docs/API.md).


Web player

# Introduction

Other than RTMP, another protocol which is sometimes used for streaming media to the server is RTSP. It isn't so popular today anymore, however, there are still areas where it is still in use, primarily by surveillance cameras.
In the next couple chapters, we would like to show you a modification of the current RTMP to HLS converter, that will allow us to convert an RTSP stream instead.

## Use case

Suppose that we have a surveillance camera, which is providing us with an RTSP stream. We would like to stream this video to multiple viewers, for them to be easily accessible in the web browser.

Note, that we want this solution to be scalable as the number of users can be quite big.  

## Solution

The reasons for converting RTSP to HLS are very similar to the ones we explained with RTMP, as RTSP comes with similar shortcomings as RTMP. RTSP is very rarely supported on playback devices, has problems with traversing firewalls and proxies and doesn't support adaptive bitrate.

## What technology we will use

Before going further, make sure you have some basic understanding of the protocols used:
  
[Real Time Streaming Protocol (RTSP)](https://antmedia.io/rtsp-explained-what-is-rtsp-how-it-works) - what RTSP is and how does it work?

[Real-time Transport Protocol (RTP) and RTP Control Protocol (RTCP)](https://www.techtarget.com/searchnetworking/definition/Real-Time-Transport-Protocol) are protocols used for delivering audio and video over the internet. While the RTP transports the data, the RTCP is responsible for QoS and synchronization of multiple streams.

RTSP to HLS introduction

In the tutorial we won't explain how to implement the solution from the ground up - instead, we will run the existing code from [Membrane demos](https://github.com/membraneframework/membrane_demo).

To run the RTSP to HLS converter first clone the demos repo:
```console
git clone https://github.com/membraneframework/membrane_demo.git
```

```console
cd membrane_demo/rtsp_to_hls
```

Install the dependencies
```console
mix deps.get
```

Make sure you have those libraries installed as well:
- gcc
- libc-dev
- ffmpeg

On ubuntu:
```console
apt-get install gcc libc-dev ffmpeg
```

Take a look inside the `lib/application.ex` file. It's responsible for starting the pipeline.
We need to give a few arguments to the pipeline:
```elixir
@rtsp_stream_url "rtsp://rtsp.membrane.work:554/testsrc.264"
@output_path "hls_output"
@rtp_port 20000
```

The `@output_path` attribute defines the storage directory for hls files and the `@rtp_port` defines on which port we will be expecting the rtp stream, once the RTSP connection is established.

The `@rtsp_stream_url` attribute contains the address of the stream, which we will be converting. It is a sample stream prepared for the purpose of the demo. 

Now we can start the application:
```console
mix run --no-halt
```

The pipeline will start playing, after a couple of seconds the HLS files should appear in the `@output_path` directory. In order to play the stream we need to first serve them. We can do it using simple python server.

```console
python3 -m http.server 8000
```

Then we can play the stream using [ffmpeg](https://ffmpeg.org/), by pointing to the location of the manifest file:
```console
ffplay http://YOUR_MACHINE_IP:8000/rtsp_to_hls/hls_output/index.m3u8
```


Running the RTSP to HLS demo

Now let's focus on how the Connection Manager works. As mentioned previously its role is to establish the RTSP connection with the RTSP server.

The `ConnectionManager` module will use the [`Connection`](https://hexdocs.pm/connection/Connection.html) behaviour, which provides additional callbacks to [`GenServer`](https://hexdocs.pm/elixir/GenServer.html) behaviour, aiding with building a connection process.

First of all we are defining the `ConnectionStatus` struct, which we will use to keep the state of the ConnectionManager:

##### lib/connection_manager.ex
```elixir
defmodule ConnectionStatus do
  @moduledoc false
  @type t :: %__MODULE__{
          stream_url: binary(),
          rtsp_session: pid(),
          pipeline: pid(),
          keep_alive: pid(),
          pipeline_options: keyword()
          }

  @enforce_keys [
      :stream_url,
      :pipeline,
      :pipeline_options
  ]

  defstruct @enforce_keys ++
              [
                  :rtsp_session,
                  :keep_alive
              ]
end
```

It holds the `rtsp_session`, which is the pid of a process started with `Membrane.RTSP.start/1`. The `Membrane.RTSP` allows us to execute RTSP client commands. You can read more about it [here](https://hexdocs.pm/membrane_rtsp/readme.html).
The `pipeline` field is the pid of the pipeline, we will need it to notify the pipeline, that the RTSP connection is ready. In such notification we send `pipeline_options`, which contain necessary information about the stream.
The `keep_alive` is a process which repeatedly pings the RTSP server, in order to keep the connection alive and prevent a timeout.

Let's take a look at the `connect/2` callback, which is called immediately after the `init/1`:

##### lib/connection_manager.ex
```elixir
def connect(_info, %ConnectionStatus{} = connection_status) do
  rtsp_session = start_rtsp_session(connection_status)
  connection_status = %{connection_status | rtsp_session: rtsp_session}

  if is_nil(rtsp_session) do
    {:backoff, @delay, connection_status}
  else
    with {:ok, connection_status} <- get_rtsp_description(connection_status),
          :ok <- setup_rtsp_connection(connection_status),
          {:ok, connection_status} <- start_keep_alive(connection_status),
          :ok <- play(connection_status) do    

      send(
        connection_status.pipeline,
        {:rtsp_setup_complete, connection_status.pipeline_options}
      )

      {:ok, connection_status}
    else
      {:error, error_message} ->
        {:backoff, @delay, connection_status}
    end
  end
end
```

In the callback we go through the whole process of establishing RTSP connection - first starting the RTSP session, then getting the video parameters with, setting up the session, starting the keep alive process and finally playing the stream.
If all those steps succeed we can notify the pipeline, otherwise we back off and try to set up the connection after a `@delay` amount of time.

What might seem unclear to you is the `get_sps_pps` function.
It is responsible for getting the [sps and pps](https://www.cardinalpeak.com/blog/the-h-264-sequence-parameter-set) parameters from the RTSP DESCRIBE method. In short, sps and pps are parameters used by the H.264 codec and are required to decode the stream. Once the RTSP connection is complete we are sending them to the pipeline.

Connection manager

As explained in the [Architecture chapter](07_RTSP_Architecture.md), the pipeline will consist of a couple of elements, that will be processing the RTP stream.

The flow of the pipeline will consist of three steps. First, when the pipeline is initialized we will start the Connection Manager, which will set up the RTP stream via the RTSP.
Once that is finished, we will set up two initial elements in the pipeline - the `UDP Source` and `RTP SessionBin`, which will allow us to receive RTP packets and process them.
When the SessionBin detects that the RTP stream has been started, it will notify the pipeline with the `:new_rtp_stream` notification. Later on, we will add the remaining elements to the pipeline, allowing for the whole conversion process to take place.

Those steps take place, respectively, in the: `handle_init/1`, `handle_other/3` and `handle_notification/4` callbacks. While the `handle_init/1` is rather intuitive, we will describe in detail what's happening in the other callbacks.

Let us explain what's going on in the `handle_other` callback:

##### lib/pipeline.ex
```elixir
@impl true
def handle_other({:rtsp_setup_complete, options}, _ctx, state) do
  children = %{
    app_source: %Membrane.UDP.Source{
      local_port_no: state[:port],
      recv_buffer_size: 500_000
    },
    rtp: %Membrane.RTP.SessionBin{
      fmt_mapping: %{96 => {:H264, 90_000}}
    },
    hls: %Membrane.HTTPAdaptiveStream.Sink{
      manifest_module: Membrane.HTTPAdaptiveStream.HLS,
      target_window_duration: 120 |> Membrane.Time.seconds(),
      target_segment_duration: 4 |> Membrane.Time.seconds(),
      storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{
        directory: state[:output_path]
      }
    }
  }

  links = [
    link(:app_source)
    |> via_in(Pad.ref(:rtp_input, make_ref()))
    |> to(:rtp)
  ]

  spec = %ParentSpec{children: children, links: links}
  { {:ok, spec: spec}, %{state | video: %{sps: options[:sps], pps: options[:pps]}} }
end
```

When we receive the `rtsp_setup_complete` message, we first define the new children for the pipeline, and links between them - the UDP Source and the RTP SessionBin. We also create the HLS Sink, however we won't be linking it just yet. With the message we receive the sps and pps inside the options, and we add them to the pipeline's state.

Only after we receive the `:new_rtp_stream` notification we add the rest of the elements and link them with each other:

##### lib/pipeline.ex
```elixir
@impl true
def handle_notification({:new_rtp_stream, ssrc, 96, _extensions}, :rtp, _ctx, state) do
  actions =
    if Map.has_key?(state, :rtp_started) do
      []
    else
      children = %{
        video_nal_parser: %Membrane.H264.FFmpeg.Parser{
          sps: state.video.sps,
          pps: state.video.pps,
          skip_until_keyframe?: true,
          framerate: {30, 1},
          alignment: :au,
          attach_nalus?: true
        },
        video_payloader: Membrane.MP4.Payloader.H264,
        video_cmaf_muxer: Membrane.MP4.Muxer.CMAF
      }

      links = [
        link(:rtp)
        |> via_out(Pad.ref(:output, ssrc),
          options: [depayloader: Membrane.RTP.H264.Depayloader]
        )
        |> to(:video_nal_parser)
        |> to(:video_payloader)
        |> to(:video_cmaf_muxer)
        |> via_in(:input)
        |> to(:hls)
      ]

      [spec: %ParentSpec{children: children, links: links}]
    end

  { {:ok, actions}, Map.put(state, :rtp_started, true) }
end
```

First we check, if the stream hasn't started yet. That's because if we are restarting the pipeline there might be a previous RTP stream still being sent, so we might receive the `:new_rtp_stream` notification twice - once for the old and then for the new stream. We want to ignore any notification after the first one, as we want only a single copy of each media processing element.
Notice the sps and pps being passed to the H264 parser - they are necessary for decoding the stream.


RTSP to HLS - pipeline

As you can see converting an RTMP or RTSP stream to HLS might not be the easiest task. However, some understanding of the protocols and the usage of a couple of prefabricated elements in the Membrane Framework make up a rather compact solution. Importantly, such solution is open for modification and easily extensible.

HLS is one of the most popular media streaming protocols. Its advantages are using HTTP, allowing it for traversing over firewalls. It also allows for adaptive bitrate, i.e. it changes the resolution of the video based on the network bandwidth. Unlike RTMP or RTSP it is highly scalable. You will often come across HLS in the media streaming industry, so it's useful to have some understanding of it.

We hope that you grasped the basic ideas of the RTMP and RTSP protocols as well as that you know how to create your 'to HLS' converter with the use of the Membrane Framework.

If you liked this tutorial and want to extend your knowledge of the Membrane Framework, make sure to check out [other tutorials](https://membrane.stream/learn).


Summary

In this chapter you will learn about the H264 codec and how its processing is done with the Membrane Framework plugin - the [H264 plugin](https://github.com/membraneframework/membrane_h264_ffmpeg_plugin).

In H264 we can distinguish two layers of abstraction - the video coding layer (VCL), focused on the visual representation of the video, and the network abstraction layer (NAL), focused on how to structurize the stream so that it can be sent via the network.

When it comes to VCL, a video encoded with the H264 codec is represented as a sequence of pictures. Each picture consists of many **macroblocks**.
The macroblock is simply a part of the picture (in the context of some spatial dependence) - i.e. the top left corner of the picture.
The macroblocks are grouped together into so-called **slices**. Each slice consists of some macroblocks, which share the same **slice header**, containing some metadata common for all these macroblocks.

Each slice (which in fact is a part of a video picture), can be packed into a single
**NALu** (*Network Abstraction Layer units*). As the name suggests, here is where we start to deal with NAL. NALu is just an atomic piece of information sent through the network - it might contain some visual data (i.e. a list of macroblocks forming a part of a video frame), but it is not limited to the visual data - within NALus metadata used to properly decode the stream is also sent. We can distinguish two types of NAL units:

- VCL NALus - which stands for "Video coding layer" NALus
- Non-VCL NALus - which stands for "Non-video coding layer" NALus

There are different types of both VCL and Non-VCL units - for more information on them you can refer [here](https://yumichan.net/video-processing/video-compression/introduction-to-h264-nal-unit/)
> **Parse H264 stream on your own!**
> 
> With the use of Membrane Framework, you can inspect the types of NALus in your h264 file. To do so, you need to clone the H264 parser repository with:
>
> ```console
>   git clone https://github.com/membraneframework/membrane_h264_ffmepg_plugin
> ```
>
> Later on, inside the repository's directory, you can launch the Elixir's interactive shell, with compiled modules from the repository's mix project, by typing:
>
> ```console
>   iex -S mix
> ```
>
> Once in iex, you can do the following thing:
>
> ```elixir
>   alias Membrane.H264.FFmpeg.Parser.NALu
>
>   # Of course you can read your own file, here is just an example file from the membrane_rtmp_plugin's test directory,
>   # available at: https://github.com/membraneframework/membrane_rtmp_plugin/tree/master/test/fixtures/testvideo.h264
>   binaries = File.read!("test/fixtures/testvideo.h264") 
>
>   NALu.parse(binaries)
> ```
>
> You should see the following response:
>
> ```elixir
>   {[
>       %{
>           metadata: %{h264: %{new_access_unit: %{key_frame?: true}, type: :sps}},
>           prefixed_poslen: {0, 29},
>            unprefixed_poslen: {4, 25}
>        },
>        %{
>            metadata: %{h264: %{type: :pps}},
>            prefixed_poslen: {29, 9},
>            unprefixed_poslen: {33, 5}
>        },
>        %{
>            metadata: %{h264: %{type: :sei}},
>            prefixed_poslen: {38, 690},
>            unprefixed_poslen: {41, 687}
>         },
>        %{
>            metadata: %{h264: %{type: :idr}},
>            prefixed_poslen: {728, 8284},
>            unprefixed_poslen: {731, 8281}
>        },
>        %{
>            metadata: %{h264: %{type: :non_idr}},
>            prefixed_poslen: {9012, 1536},
>            unprefixed_poslen: {9016, 1532}
>        },
>        ....
>   ]}
> ```
>
> As you can see, NALus of different types have appeared, just like:
>
> - SPS - *Sequence Parameter Set*, a Non-VCL NALu, with a set of parameters that rarely change, applicable to the series of consecutive coded video pictures, called the **coded video sequence**. Each coded video sequence can be decoded independently of any other coded video sequence.
> - PPS - *Picture parameter Set* - a set of parameters that are applicable to some pictures from the coded video sequence. Furthermore, inside a PPS NALu there is a reference to SPS.
> - SEI - *Supplemental Enhancement Information* - some additional information that enhances the usability of the decoded video, i.e. timing information.
> - IDR - *Instantaneous Decoding Refresh*, a VCL unit containing the I-frame (known also as `intra frame`) - a picture that can be decoded without knowledge of any other frame, in contrast to P-frames and B-frames, which might need previously presented frames or frames that need to be presented in the future. As you might guess, the I-frames size is much greater than P-frame or B-frame size - that is because the whole information about the content of the picture needs to be encoded in such a frame.
> - NON_IDR - *Non-Instantaneous Decoding Refresh* - a VCL unit containing a P-frame or B-frame or parts of such a non-key frame. Note the size of a Non-IDR NALu (1536 B), compared to the size of IDR NALu (8284 B).

The sequence of NALus of a special form creates an **Access Unit**.
Each access unit holds a single picture of the video.
When such a picture is the keyframe, we refer to it as a **IDR Access Unit**. Otherwise, we call it **Non-IDR Access Unit**. Of course, the IDR Access Units are much bigger than Non-IDR Access Units when it comes to their binary size.
Sometimes, for the convenience of the decoding, the access units are separated with the use of another Non-VCL NALu, called *AUD* (**Access Unit Delimeter**).
Below you can find a diagram showing the structure of an access unit:
![Access Unit structure](assets/au_structure.png)
The access unit consists of the **primary coded picture** - a set of macroblocks representing the picture, and optionally of the **redundant coded picture** - the set of macroblocks that hold the redundant information about the given areas on the picture.
A parser needs to be aware that some of the NALus are optional are might not appear in the stream.

The existence of a coded video sequence is determined by the presence of an IDR NALu in the first access unit. Each coded video sequence can be decoded independently of the other coded video sequences.

As a summary we would like to present a diagram showing an exemplary NALus stream, along with the structures we can distinguish in that stream:
![H264 NALus stream](assets/h264_structure.png)
