# Native code integration

In Membrane we try to make use of Elixir goodness as often as possible. However, sometimes it is necessary to write some native code, for example to integrate with existing C libraries. To achieve that, we use _natives_, which are either [NIFs](http://erlang.org/doc/man/erl_nif.html) or [C nodes](http://erlang.org/doc/man/ei_connect.html). Both of them have their tradeoffs:
- NIFs don't introduce significant overhead, but in case of failure they can even crash entire VM,
- CNodes are isolated and can be supervised like Elixir processes, but they are slower due to the inter-process communication.

That's why we use them only when necessary and created some tools to make dealing with them easier. For interoperability with C and C++, we use [Bundlex](https://github.com/membraneframework/bundlex) and [Unifex](https://github.com/membraneframework/unifex). For calling Rust, we use [Rustler](https://github.com/rusterlium/rustler).

## [Bundlex](https://github.com/membraneframework/bundlex)

To simplify and unify the process of writing and compiling natives, we use our own build tool - [Bundlex](https://github.com/membraneframework/bundlex). It is a multi-platform tool that provides a convenient way of compiling and accessing natives from Elixir code.
For more information, see [Bundlex's GitHub page](https://github.com/membraneframework/bundlex).

## [Unifex](https://github.com/membraneframework/unifex)

Process of creating natives is not only difficult but also quite arduous, because it requires using cumbersome Erlang APIs, and thus a lot of boilerplate code. To make it more pleasant, we created [Unifex](https://github.com/membraneframework/unifex), a tool that is responsible for generating interfaces between C or C++ libraries and Elixir on the base of short `.exs` configuration files. An important feature of Unifex is that you can write the C/C++ code once and use it either as a NIF or as a C node.

A quick introduction to Unifex is available [here](https://hexdocs.pm/unifex/creating_unifex_natives.html).



Native code integration

This tutorial explains how membrane works. It covers environment preparation, pipelines, and the basic concepts behind the framework.

Hello there, and a warm welcome to the Membrane tutorials. We're glad you chose to learn Membrane; and we'd like to invite you on a journey around multimedia with us, where we explore how to utilize the Membrane Framework to build applications that process audio, video, and other multimedia content in interesting ways.

## What is Membrane?

Membrane is a multimedia processing framework that focuses on reliability, concurrency, and scalability. It is primarily written in Elixir, while some platform-specific or time-constrained parts are written in Rust C. With a range of existing packages and an easy-to-use interface for writing your own, Membrane can be used to process almost any type of multimedia, for example:
- stream via WebRTC, RTSP, RTMP, HLS, HTTP and other protocols,
- transcode, mix and apply custom processing of video & audio,
- accept and generate / record to MP4, MKV, FLV and other containers,
- handle dynamically connecting and disconnecting streams,
- seamlessly scale and recover from errors,
- do whatever you imagine if you implement it yourself :D Membrane makes it easy to plug in your code at almost any point of processing.

If the abbreviations above don't ring any bells, don't worry - this tutorial doesn't require any multimedia-specific knowledge. Then, [other tutorials](https://membrane.stream/learn) and [demos](https://github.com/membraneframework/membrane_demo) will introduce you to the multimedia world!

## Structure of the framework

The most basic media processing entities of Membrane are `Element`s. An element might be able, for example, to mux incoming audio and video streams into MP4, or play raw audio using your sound card. You can create elements yourself, or choose from the ones provided by the framework.

Elements can be organized into a pipeline - a sequence of linked elements that perform a specific task. For example, a pipeline might receive an incoming RTSP stream from a webcam and convert it to an HLS stream, or act as a selective forwarding unit (SFU) to implement your own videoconferencing room. You'll see how to create a pipeline in the subsequent chapter.

### Membrane packages

To embrace modularity, Membrane is delivered to you in multiple packages, including plugins, formats, core and standalone libraries. The list of all Membrane packages is available [here](https://github.com/membraneframework/membrane_core/#All-packages). It contains all the packages maintained by the Membrane team and some third-party packages.

**Plugins**

Plugins provide elements that you can use in your pipeline. Each plugin lives in a `membrane_X_plugin` repository, where `X` can be a protocol, codec, container or functionality, for example [mebrane_opus_plugin](https://github.com/membraneframework/membrane_opus_plugin). Plugins wrapping a tool or library are named `membrane_X_LIBRARYNAME_plugin` or just `membrane_LIBRARYNAME_plugin`, like [membrane_mp3_mad_plugin](https://github.com/membraneframework/membrane_mp3_mad_plugin). Plugins are published on [hex.pm](https://hex.pm), for example [hex.pm/packages/membrane_opus_plugin](https://hex.pm/pakcages/membrane_opus_plugin) and docs are at [hexdocs](https://hexdocs.pm), like [hexdocs.pm/membrane_opus_plugin](https://hexdocs.pm/membrane_opus_plugin). Some plugins require native libraries installed in your OS. Those requirements, along with usage examples are outlined in each plugin's readme.

**Formats**

Apart from plugins, Membrane has stream formats, which live in `membrane_X_format` repositories, where `X` is usually a codec or container, for example, [mebrane_opus_format](https://github.com/membraneframework/mebrane_opus_format). Stream formats are published the same way as packages and are used by elements to define what kind of stream can be sent or received. They also provide utility functions to deal with a given codec/container.

**Core**

[Membrane Core](https://github.com/membraneframework/membrane_core) is the heart and soul of the Membrane Framework. It is written entirely in Elixir and provides the internal mechanisms and API that allow you to prepare processing elements and link them together in a convenient yet reliable way. Note that Membrane Core does not contain any multimedia-specific logic. 
The documentation for the developer's API is available at [hexdocs](https://hexdocs.pm/membrane_core/readme.html).

**Standalone libraries**

Last but not least, Membrane provides tools and libraries that can be used standalone and don't depend on the `membrane_core`, for example, [video_compositor](https://github.com/membraneframework/video_compositor), [ex_sdp](https://github.com/membraneframework/ex_sdp) or [unifex](https://github.com/membraneframework/unifex).

## Where can I learn Membrane?
There are a number of resources available for learning about Membrane:

### This guide
The following sections in that guide will introduce the main concepts of creating Membrane elements and pipelines, without focusing on the specific details of multimedia processing.

### Demos
The [membrane_demo](https://github.com/membraneframework/membrane_demo) repository contains many projects, scripts and livebooks that cover different use cases of the framework. It's a good place to learn by example.

### Tutorials
For a step-by-step guide to implementing a specific system using Membrane, check out our [tutorials](https://membrane.stream/learn).

### Documentation
For more detailed information, you can refer to the Membrane Core documentation and the documentation for the Membrane packages maintained by the Membrane team, both of which can be accessed [here](https://hex.pm/orgs/membraneframework).

If you see something requiring improvement in this guide, feel free to create an issue or open a PR in the [membrane_tutorials](https://github.com/membraneframework/membrane_tutorials) repository.


Get Started with Membrane

# Pipelines

Building `Pipeline`s is the way to create streaming applications with Membrane. Pipeline allows you to spawn `Element`s and establish data flow between them. Pipelines can also communicate with elements or terminate them. Elements within a pipeline are often referred to as its `children` and the pipeline is their `parent`.

Connecting elements together is called `linking`. 

To create a pipeline, you need to implement the [Membrane.Pipeline](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html) behavior. It boils down to implementing callbacks and returning actions from them. For a simple pipeline, it's sufficient to implement the [handle_init](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#c:handle_init/2) callback, which is called upon the pipeline startup, and return the [spec](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html#t:spec/0) action, which spawns and links elements. Let's see it in an example:

## Sample pipeline

```elixir
Mix.install([
  :membrane_hackney_plugin,
  :membrane_mp3_mad_plugin,
  :membrane_portaudio_plugin,
])

defmodule MyPipeline do
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, mp3_url) do
    spec =
      child(%Membrane.Hackney.Source{
        location: mp3_url, hackney_opts: [follow_redirect: true]
      })
      |> child(Membrane.MP3.MAD.Decoder)
      |> child(Membrane.PortAudio.Sink)

    {[spec: spec], %{}}
  end
end

mp3_url = "https://raw.githubusercontent.com/membraneframework/membrane_demo/master/simple_pipeline/sample.mp3"

Membrane.Pipeline.start_link(MyPipeline, mp3_url)
```

The code above is one of the simplest examples of Membrane usage. It plays an MP3 file through your computer's default audio playback device with the help of the [PortAudio](http://www.portaudio.com/) audio I/O library. Let's digest this code and put it to work playing some sound.
> **Elixir**
>
> Membrane is written in Elixir. It's an awesome programming language of the functional paradigm with great fault tolerance and process management, which made it the best choice for Membrane.
> If you're not familiar with it, you can use [this cheatsheet](https://devhints.io/elixir) for a quick look-up.
> We encourage you to also take a [deep look into Elixir](https://elixir-lang.org/getting-started/introduction.html) and learn how to use it to take full advantage of all its awesomeness. We believe you'll fall in love with Elixir too!


You have two options to run the snippet:

- Option 1: Click [here](https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2Fmembraneframework%2Fmembrane_core%2Fblob%2Fmaster%2Fexample.livemd). You'll be directed to install [Livebook](https://livebook.dev), an interactive notebook similar to Jupyter, and it'll open the snippet in there for you. Then just click the 'run' button in there.

- Option 2: If you don't want to use Livebook, you can [install Elixir](https://elixir-lang.org/install.html), type `iex` to run the interactive shell and paste the snippet there.


## Sample pipeline explained

Let's figure out step-by-step what happens in the sample pipeline.

Firstly, we install the needed dependencies. They are plugins, that contain elements that we will use in the pipeline.

```elixir
Mix.install([
  :membrane_hackney_plugin,
  :membrane_mp3_mad_plugin,
  :membrane_portaudio_plugin,
])
```

Instead of creating a script and using `Mix.install`, you can also [create a Mix project](https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html) and add these dependencies to `deps` in `mix.exs` file.

After installing the dependencies, we can create a module for our pipeline:

```elixir
defmodule MyPipeline do
  use Membrane.Pipeline

end
```

Using the `Membrane.Pipeline` behaviour means we are treating our module as a Membrane Pipeline, so we will have access to functions defined in the `Membrane.Pipeline` module, and we can implement some of its callbacks. Let's implement the `handle_init/2` callback, which is a function that is invoked to initialize a pipeline during start-up:

```elixir
defmodule MyPipeline do
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, path_to_mp3) do

  end
end
```

> If the concept of callbacks and behaviours is new to you, you can read more about it [here](https://elixir-lang.org/getting-started/typespecs-and-behaviours.html#behaviours), or see examples of behaviours in the Elixir standard library, for example the [GenServer](https://elixir-lang.org/getting-started/mix-otp/genserver.html) and [Supervisor](https://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html) behaviours.

We'll use `handle_init` to specify all its [elements](../glossary/glossary.md#element) as children and set up links between them to define the order in which data will flow through the pipeline:

```elixir
@impl true
def handle_init(_ctx, path_to_mp3) do
  spec =
    child(%Membrane.Hackney.Source{
      location: mp3_url, hackney_opts: [follow_redirect: true]
    })
    |> child(Membrane.MP3.MAD.Decoder)
    |> child(Membrane.PortAudio.Sink)

  {[spec: spec], %{}}
end
```

The [child](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#child/2) function allows us to spawn particular elements. By piping one child to another with the `|>` operator, we can specify the order of the elements in the pipeline. Thus, the code above links `Membrane.Hackney.Source` to `Membrane.MP3.MAD.Decoder` and `Membrane.MP3.MAD.Decoder` to `Membrane.PortAudio.Sink`. Here's what they are:
- Hackney source - an element based on the [Hackney HTTP library](https://github.com/benoitc/hackney), that downloads a file via HTTP chunk by chunk, and sends these chunks through its `output` pad. We pass two options to it: a URL where the MP3 is stored and a flag to make it follow HTTP redirects.
- MP3 decoder - an element based on [libmad](https://github.com/markjeee/libmad), that accepts MP3 audio on the `input` pad and sends the [decoded](../glossary/glossary.md#decoding) audio through the `output` pad.
- PortAudio sink - an element that accepts decoded audio on its `input` pad and uses the [PortAudio](https://github.com/PortAudio/portaudio) library to play in on the speaker.

In our spec, we don't mention the names of the pads, because `input` and `output` are the defaults. However, we could explicitly specify them:

```elixir
spec =
  child(%Membrane.Hackney.Source{
    location: mp3_url, hackney_opts: [follow_redirect: true]
  })
  |> via_out(:output)
  |> via_in(:input)
  |> child(Membrane.MP3.MAD.Decoder)
  |> via_out(:output)
  |> via_in(:input)
  |> child(Membrane.PortAudio.Sink)
```

Even though not necessary here, [via_in](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_in/3) and [via_out](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_out/3) are useful in more complex scenarios that we'll cover later.

The value returned from `handle_init`:

```elixir
{[spec: spec], %{}}
```

is a tuple containing the list of actions and the state.
- Actions are the way to interact with Membrane. Apart from `spec`, you can for example return `terminate: reason` that will stop the elements and terminate the pipeline. Most actions, including `spec`, can be returned from multiple callbacks, allowing, for example, to spawn elements on demand. Check the [Membrane.Pipeline](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html) behavior for the available callbacks and [Membrane.Pipeline.Action](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html) for the available actions.
- State is an arbitrary data that will be passed to subsequent callbacks as the last argument. It's usually a map. As we have no use for the state in this case, we just set it to an empty map.

When we have created our pipeline module, we can call [Membrane.Pipeline.start_link](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#start_link/3) to run it:

```elixir
Membrane.Pipeline.start_link(MyPipeline, mp3_url)
```

We pass to it the pipeline module and options, which in our case is the `mp3_url`. The options are passed directly to the `handle_init` callback.

Congratulations! You've just built and run your first Membrane application. Let's now have a deeper look at the elements.


Pipelines

# Elements

Elements in Membrane are the most basic entities responsible for processing multimedia.
Each instance of an element is an Elixir process, that has an internal state and communicates by message passing. You've already seen some examples of elements in the previous chapter.

Elements are spawned and controlled by their parent, which can be a pipeline or a bin (we'll cover bins in the subsequent chapter).

## Element types

The basic types of elements are the following:

* `Source` - fetches the stream from outside of the pipeline and delivers it to other elements
* `Sink` - consumes the stream from other elements
* `Filter` - receives the stream from other elements, processes it and sends it further to the subsequent elements
* `Endpoint` - a `Source` and a `Sink` combined - can both deliver and consume the data from other elements

To create an element, you need to `use` the appropriate module - [Membrane.Source](https://hexdocs.pm/membrane_core/Membrane.Source.html), [Membrane.Sink](https://hexdocs.pm/membrane_core/Membrane.Sink.html), [Membrane.Filter](https://hexdocs.pm/membrane_core/Membrane.Filter.html) or [Membrane.Endpoint](https://hexdocs.pm/membrane_core/Membrane.Endpoint.html), for example:

```elixir
defmodule MyElement do
  use Membrane.Filter

  # Element implementation
end
```

The `Element implementation` consists of defining pads, options and callbacks. Let's find out how to do that.

## Pads

As you already learned, pads allow the creation of the flow of data between elements. Pads, much like contact pads on a printed circuit board, are inputs and outputs of an element and are used to connect the elements with one another.
Because of that, there are two types of pads: `input` and `output`. It is worth mentioning that `Source` elements may only contain `output` pads, `Sink` elements contain only `input` pads, and `Filter` and `Endpoint` elements can have both of them.

Every pad should define the format of data that it is expecting. This format can be, for example,
raw audio with a specific sample rate or encoded audio in a given format.

To send data between elements, their pads need to be linked. There are a couple of rules that apply to pad linking:

* One pad of an element can only be linked with one pad from another element.
  (Dynamic pads can help with that limitation; you'll learn about them in the `pads_and_linking` chapter)
* Only links between `output` and `input` pads are allowed.
* Accepted stream formats of pads have to be compatible.

### Defining pads

Pads can be defined using [def_input_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#def_input_pad/2) and [def_output_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithOutputPads.html#def_output_pad/2) macros. They both accept the pad name and the list of properties. The name allows for the identification of the pad. If an element has a single input or output pad, the convention is to name it `input` or `output`, respectively. The pad properties are listed below:

* `accepted_format` - A pattern for a stream format expected on the pad, for example `Membrane.RawAudio` or `%Membrane.RawAudio{channels: 2}`. It serves documentation purposes and is validated in runtime.
* `flow_control` - Configures how back pressure should be handled on the pad. You can choose from the following options:
  * `:auto` - Membrane automatically manages the flow control. It works under the assumption that the element does not need to block or slow down the processing rate, it just processes or consumes the stream as it flows. This option is not available for output pads of `Source` end `Endpoint` elements, while for all other pads it's the default.
  * `:manual` - You need to manually control the flow control by using the `demand` action on `input` pads and implementing the `handle_demand` callback for `output` pads.
  * `:push` - It's a simple mode where an element producing data pushes it right away through the `output` pad. An `input` pad in this mode should be always ready to process that data.
* `demand_unit` - Either `:bytes` or `:buffers`, specifies what unit will be used to request or receive demands. Must be specified for inputs that have `flow_control` is set to `:manual`.
* `availability` - Either `:always` (default) - meaning the pad is static and available from the moment an element is spawned, or `:on_request` meaning it is dynamic. We'll learn more about it in the `Pads and linking` chapter.
* `options` - Optional; specification of options accepted by the pad. We'll learn more about it in the `Pads and linking` chapter.

A pad definition may look like this:

```elixir
def_input_pad :input, flow_control: :auto, accepted_format: %Membrane.RawAudio{channels: 2}
```

It means that the element has a static input pad called `input`, with automatic flow control, that accepts [raw audio](http://hexdocs.pm/membrane_raw_audio_format) with two channels.

## Options

Element options make it possible to pass configuration data to an element. Elements aren't required to accept any options, but it's useful in many cases. Available options can be specified using the [def_options](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#def_options/1) macro, for example:

```elixir
def_options some_option: [
              spec: integer() | string(),
              default: 0,
              description: """
              This option is intended for...
              """
            ],
            other_option: [
              # ...
            ]
```

Each option can have the following fields:
- `spec` - The [typespec](https://hexdocs.pm/elixir/1.12/typespecs.html) of the values that option can have. Defaults to `any`.
- `default` - The value that the option will have if it's not specified. If the default is not provided, the option must be always explicitly specified.
- `description` - Write here what the option does. It will be included in the module documentation.

We'll see a practical example of defining options in the [sample element](#sample-element). 

## Callbacks

Apart from specifying pads and options, creating an element involves implementing callbacks. They have different responsibilities and are called in a specific order. As in the case of pipelines, callbacks interact with the framework by returning [actions](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html). Here are some most useful callbacks:

[handle_init](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_init/2) is invoked once, upon the element creation.
It receives options specified by the user, which should be parsed and on their base,
the element should create and initialize its internal state. It is called synchronously (the parent waits until it returns), thus you shouldn't perform any long tasks there.

[handle_setup](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_setup/2) is invoked right after `handle_init`. It's intended for resource allocation or some potentially time-consuming initialization. If you need to make sure that resources are properly released upon element termination, use [Membrane.ResourceGuard](https://hexdocs.pm/membrane_core/Membrane.ResourceGuard.html) or [Membrane.UtilitySupervisor](https://hexdocs.pm/membrane_core/Membrane.UtilitySupervisor.html)

After `handle_setup`, the following callbacks can be called at any point:
- [handle_pad_added](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_added/3) and [handle_pad_removed](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_removed/3) are called when a dynamic pad is added and removed, respectively
- [handle_parent_notification](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_parent_notification/3) is called whenever the parent sends a notification to the element; elements can send notifications the other way with the **notify** action

[handle_playing](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_playing/2) is called when the stream processing is ready to start. From that point, you can return the following actions:
- [stream_format](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:stream_format/0) tells the subsequent element what kind of stream it should expect on the given pad
- [buffer](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:buffer/0) sends media data to the subsequent element; stream_format has to be sent before the first buffer
- [event](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:event/0) sends a custom struct to the subsequent or preceding element; downstream events are sent in order with buffers
- [demand](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:demand/0) requests data from the previous element; only works for pads in `flow_control: manual` mode
- [end_of_stream](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:end_of_stream/0) tells the subsequent element that the stream has finished, nothing can be sent through that pad afterward

After `handle_playing`, you should expect the following callbacks to be called:

- [handle_stream_format](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_stream_format/4) tells you what kind of stream you should expect on the given pad; called at least once, before `handle_start_of_stream`, may be called later when the stream format changes

- [handle_start_of_stream](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_start_of_stream/3) is called just before the first buffer arrives from the preceding element

- [handle_buffer](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_buffer/4) is called every time a buffer arrives from the preceding element

- [handle_event](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_event/4) is called once an event arrives from the preceding or subsequent element

- [handle_demand](https://hexdocs.pm/membrane_core/Membrane.Element.WithOutputPads.html#c:handle_demand/5) is called when the subsequent element requests data on the given pad; only works for pads in `flow_control: :manual` mode

- [handle_end_of_stream](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_end_of_stream/3) is called when the stream has finished; it may be because the preceding element explicitly returned `end_of_stream` action, the pad is about to be unlinked or the current element is about to terminate

Finally, [handle_terminate_request](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_terminate_request/2) is called when the parent decides to remove the element. By default, it returns the `terminate: :normal` action and the element terminates gracefully. Note that this callback is only called when the element is gracefully asked to terminate.


## Sample element

That's enough for the theory, let's write some code! We'll create a sample element and plug it into the pipeline from the previous chapter. Here's the element:

```elixir
defmodule VolumeKnob do
  @moduledoc """
  Membrane filter that changes the audio volume
  by the gain passed via options.
  """
  use Membrane.Filter

  alias Membrane.RawAudio

  def_input_pad :input, accepted_format: RawAudio, flow_control: :auto
  def_output_pad :output, accepted_format: RawAudio, flow_control: :auto

  def_options gain: [
    spec: float(),
    description: """
    The factor by which the volume will be changed.

    A gain smaller than 1 reduces the volume and gain
    greater than 1 increases it.
    """
  ]

  @impl true
  def handle_init(_ctx, options) do
    {[], %{gain: options.gain}}
  end
  
  @impl true
  def handle_buffer(:input, buffer, ctx, state) do
    stream_format = ctx.pads.input.stream_format
    sample_size = RawAudio.sample_size(stream_format)
    payload =
      for <<sample::binary-size(sample_size) <- buffer.payload>>, into: <<>> do
        value = RawAudio.sample_to_value(sample, stream_format)
        scaled_value = round(value * state.gain)
        RawAudio.value_to_sample(scaled_value, stream_format)
      end

    buffer = %Membrane.Buffer{buffer | payload: payload}
    {[buffer: {:output, buffer}], state}
  end
end
```

As the `moduledoc` says, the element can be used to adjust the audio volume. As we create a filter, we start with `use Membrane.Filter` clause. Then we define pads, one input and one output:

```elixir
alias Membrane.RawAudio

def_input_pad :input, accepted_format: RawAudio, flow_control: :auto
def_output_pad :output, accepted_format: RawAudio, flow_control: :auto
```

The element is going to receive raw audio and send the raw audio too. The raw audio (sometimes referred to as PCM - Pulse Code Modulation) is a simple digital representation of an audio wave, that we can operate on - for example, change the volume. The `Membrane.RawAudio` format is defined in the `membrane_raw_audio_format` package.

Since the element only transforms the stream as it flows, we can safely set `flow_control` to `auto` on both pads.

After defining the pads, we can define options. In this case, it's a single option - `gain` by which the volume will be changed.

```elixir
def_options gain: [
  spec: number(),
  description: """
  The factor by which the volume will be changed.

  A gain smaller than 1 reduces the volume and gain
  greater than 1 increases it.
  """
]
```

It's important to provide the type spec and description for each option so that everyone knows how to use it.

Next, we implement the first callback - `handle_init`:

```elixir
@impl true
def handle_init(_ctx, options) do
  {[], %{gain: options.gain}}
end
```

The callback does not return any actions (thus the empty list), but it saves the gain passed through options in the state.

Then goes the main part of the element - the `handle_buffer` callback:

```elixir
@impl true
def handle_buffer(:input, buffer, ctx, state) do
```

The callback is called whenever a buffer arrives on a pad, and receives four arguments:
- the pad where the buffer arrived,
- the [Membrane.Buffer](https://hexdocs.pm/membrane_core/Membrane.Buffer.html) structure carrying the stream data,
- [Membrane.Element.CallbackContext.t](https://hexdocs.pm/membrane_core/Membrane.Element.CallbackContext.html#t:t/0), providing some useful information about the element,
- the element's state that we created in `handle_init`.

Firstly, we use the callback context to get the stream format present on the pad and use a utility from [Membrane.RawAudio](https://hexdocs.pm/membrane_raw_audio_format/Membrane.RawAudio.html) to calculate the sample size:
```elixir
stream_format = ctx.pads.input.stream_format
sample_size = RawAudio.sample_size(stream_format)
```

We could have implemented the `handle_stream_format` callback and stored the `sample_size` in the element's state too. When there's more work to be done once the stream format arrives, it's the preferred approach, though in a simple case like this we're good using the callback context.

The sample size is the amount of bytes that each audio sample takes. We'll use it to extract each sample from the payload:

```elixir
payload =
  for <<sample::binary-size(sample_size) <- buffer.payload>>, into: <<>> do
```

Now we can convert each sample to an integer with another utility from `Membrane.RawAudio`: [sample_to_value](https://hexdocs.pm/membrane_raw_audio_format/Membrane.RawAudio.html#sample_to_value/2). Having the integer, we can multiply it by the gain and convert it back to the binary representation.

```elixir
    value = RawAudio.sample_to_value(sample, stream_format)
    scaled_value = round(value * state.gain)
    RawAudio.value_to_sample(scaled_value, stream_format)
  end
```

Finally, we can update the payload and forward the buffer to the output pad using the [buffer](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:buffer/0) action.

```elixir
  buffer = %Membrane.Buffer{buffer | payload: payload}
  {[buffer: {:output, buffer}], state}
end
```

Let's test our element by plugging it into the pipeline from the previous chapter. Since it accepts `Membrane.RawAudio`, we should plug it after the decoder, which accepts encoded audio and outputs raw audio. Let's update the spec in the `handle_init` callback of our pipeline:

```elixir
spec =
  child(%Membrane.Hackney.Source{
    location: mp3_url, hackney_opts: [follow_redirect: true]
  })
  |> child(Membrane.MP3.MAD.Decoder)
  |> child(%VolumeKnob{gain: 0.2})
  |> child(Membrane.PortAudio.Sink)
```

Let's run the pipeline again:

```elixir
Membrane.Pipeline.start_link(MyPipeline, mp3_url)
```

Since we set the gain to `0.2`, the audio should play quieter than before.

In this chapter, you learned how elements work and how to create one. Now let's figure out what are `Bin`s.


Elements

# Bins

Bins, similarly to pipelines, are containers for elements. However, at the same time, they can be placed and linked within pipelines. Although a bin is a separate Membrane entity, it can be perceived as a pipeline within an element. Bins can also be nested within one another, though we don't recommend too much nesting, as it may end up hard to maintain. For an example of a bin, have a look at the [RTMP source bin](https://github.com/membraneframework/membrane_rtmp_plugin/blob/master/lib/membrane_rtmp_plugin/rtmp/source/bin.ex) or [HTTP adaptive stream sink bin](https://github.com/membraneframework/membrane_http_adaptive_stream_plugin/blob/master/lib/membrane_http_adaptive_stream/sink_bin.ex).

The main use cases for a bin are:
- creating reusable element groups,
- encapsulating children's management logic, for instance, dynamically spawning or replacing elements as the stream changes.

## Bin's pads

Bin's pads are defined and linked similarly to element's pads. However, their role is limited to proxy the stream to other elements and bins inside (inputs) or outside (outputs). To achieve that, each input pad of a bin needs to be linked to both an output pad from the outside of a bin and an input pad of its child inside. Accordingly, each bin's output should be linked to output inside and input outside of the bin.

## Bin and the stream

Although the bin passes the stream through its pads, it does not access it directly, so callbacks such as `handle_process` or `handle_event` are not found there. This is because the responsibility of the bin is to manage its children, not to process the stream. Whatever the bin needs to know about the stream, it should get through notifications from the children.

## Bin as a black box

Bins are designed to take as much responsibility for their children as possible so that pipelines (or parent bins) don't have to depend on bins' internals. That's why notifications from the children are sent to their direct parents only. Also, messages received by a bin or pipeline can be forwarded only to its direct children.


Bins

# Pads and linking

You learned how to link elements in the `Pipeline` chapter and how to define pads in the `Element` chapter. Now, let's have a deeper dive into pads and their capabilities.

## Dynamic Pads

A dynamic pad is a type of pad that acts as a template - each time some other pad is linked to a dynamic pad, a new instance of it is created.

Dynamic pads don't have to be linked when the element is started. This needs to be handled by the element, but in return, it gives new possibilities when the number of pads can change on the fly.

Another use case for dynamic pads is when the number of pads is not known at the compile time.
For example, an audio mixer may have any number of inputs.


### Creating an element with dynamic pads

To make a pad dynamic, you need to set its `availability` to `:on_request` in [def_input_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#def_input_pad/2) or [def_output_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithOutputPads.html#def_output_pad/2).

Now, each time some element is linked to this pad, a new instance of the pad is created and [handle_pad_added](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_added/3) callback is invoked. Instances of a dynamic pad can be referenced as `Pad.ref(pad_name, pad_id)`. When a dynamic pad is unlinked, the [handle_pad_removed](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_removed/3) callback is called.

### Gotchas

As usual, with great power comes great responsibility. When implementing an element with dynamic pads, you need to consider what should happen when they are added or removed. Usually, you need to implement `handle_pad_added` and `handle_pad_removed` callbacks, and the logic of an element may generally become more complicated.

### Linking dynamic pads

Let's see how to link dynamic pads. We'll use [membrane_file_plugin](https://github.com/membraneframework/membrane_file_plugin), a plugin that allows reading and writing to files, and [membrane_tee_plugin](https://github.com/membraneframework/membrane_tee_plugin) which allows forwarding the stream from a single input to multiple outputs. Running the pipeline below with `Membrane.Pipeline.start_link(MyPipeline)` should copy the "source" file to "target1", "target2" and "target3" files. Don't forget to create the "source" file before.

```elixir
Mix.install([
  :membrane_file_plugin,
  :membrane_tee_plugin
])

defmodule MyPipeline do
  use Membrane.Pipeline

  alias Membrane.{File, Tee}

  @impl true
  def handle_init(_ctx, _options)
    spec = [
      child(%File.Source{location: "source"})
      |> child(:tee, Tee.Parallel),
      get_child(:tee) |> child(%File.Sink{location: "target1"}),
      get_child(:tee) |> child(%File.Sink{location: "target2"}),
      get_child(:tee) |> child(%File.Sink{location: "target3"})
    ]
  end
end
```

The [Membrane.Tee.Parallel](https://hexdocs.pm/membrane_tee_plugin/Membrane.Tee.Parallel.html) element has a single static input and a single dynamic output pad. Because the output is dynamic, each time we link it, a new pad instance is created with a unique reference. In the example above, pad references were generated automatically. It's possible to specify them directly with [via_in](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_in/3) or [via_out](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_out/3) and [Membrane.Pad.ref](https://hexdocs.pm/membrane_core/Membrane.Pad.html#ref/1):

```elixir
spec = [
  child(%File.Source{location: "source"})
  |> child(:tee, Tee.Parallel),
  get_child(:tee) |> via_out(Pad.ref(:output, 1)) |> child(%File.Sink{location: "target1"}),
  get_child(:tee) |> via_out(Pad.ref(:output, 2)) |> child(%File.Sink{location: "target2"}),
  get_child(:tee) |> via_out(Pad.ref(:output, 3)) |> child(%File.Sink{location: "target3"})
]
```

In this case, it won't make a difference, but elements can rely on pad references and use them to identify the stream that should be sent or received through the given pad.

## Pad options

Just like elements, pads can have options. They have to be defined with the `options` key in `def_input_pad` and `def_output_pad`, using the same syntax as the element options. For example, [Membrane.AudioMixer](https://hexdocs.pm/membrane_audio_mix_plugin/Membrane.AudioMixer.html) does it to make its `input` pad accept the `offset` option:

```elixir
def_input_pad :input,
    availability: :on_request,
    options: [
      offset: [
        spec: Time.non_neg(),
        default: 0,
        description: "Offset of the input audio at the pad."
      ]
    ],
    # ...
```

Pad options can be set via a keyword list within the second argument of `via_in` or `via_out`. Here's how to provide the `offset` option to the mixer:

```elixir
spec = [
  # ...
  child(Membrane.MP3.MAD.Decoder)
  |> via_in(:input, options: [offset: Membrane.Time.seconds(2)])
  |> child(Membrane.AudioMixer)
  # ...
]
```

Pad options' values can be accessed in the context of the `handle_pad_added` callback:

```elixir
@impl true
def handle_pad_added(pad, context, state) do
  %{offset: offset} = context.pad_options
  # ...
end
```


Pads &amp; linking

# Flow Control

Once we have linked the elements together, there comes a moment when they can start sending buffers through the pads. However, how to control the flow of buffers between elements?

Membrane solves this problem using its own backpressure mechanism and demands.

## Types of `flow_control`

When defining a pad od an element, you can pass one of 3 values in the `flow_control` field: `auto`, `push`, or `manual`. They determine how the backpressure mechanism works on the pad they relate to.

- Input pad with `flow_control: :push` - an element with such a pad must always be ready to receive a new buffer on that pad. The output pad linked with such an input pad must also have `flow control` set to `:push`. It is not advised to use `flow_control: :push` in input pads, as it makes them prone to overflow.

- Output pad with `flow_control: :push` - the element decides for itself when to send how many buffers via a given pad.

- Input pad with `flow_control: :manual` - the element directly, using the `demand` action, requests how much data it wants to receive from this pad. Membrane guarantees that more data will never come to the input pad than was demanded.

- Output pad with `flow_control: :manual` - the element gets the information, through the `c:Membrane.Element.WithOutputPads.handle_demand/5` callback, how much data a specific output pad needs. An element having an output pad with `flow_control: :manual` is obligated to satisfy the demand on it.

- Input pad with `flow_control: :auto` - unlike in manual demands, the element doesn't have to directly specify the demand value on the input pad to receive buffers on it. In `:auto` `flow control`, the framework will manage the demand value itself. In the case of Sources and Endpoints, it will make sure that the demand on the input pad is positive as long as the number of buffers waiting for processing in the mailbox is not too large. In the case of filters, Membrane will take care of a positive demand under one additional condition - the demand value on all output pads with `flow_control: :auto` of this element must also be positive.

- Output pad with `flow_control: :auto` - an auto output pad does not require from the element to implement the `handle_demand` callback. The demand value on this pad only matters when the framework itself calculates the value of the auto demand on the input pads. The only type of Membrane Element that can have output pads with `flow_control: :auto` is a `Membrane.Filter`. 

`flow_control: :manual` is useful in cases, when element wants to get a specific amount of data on each pad (e.g. to satisfy demand on 1 buffer on pad `:output`, element need 1 buffer from pad `:input_a`, 10 buffers from pad `:input_b` and 100 bytes from pad `:input_c`). Comparing to `flow_control: :auto`, `flow_control: :manual` can solve a broader class of problems, but, on the other hand, it requires more code in your element and is less performant.

`flow_control: :auto` is the best solution, when you want to provide a backpressure mechanism, but your element has only one input pad or it doesn't care about the proportion of the amount of data incoming on each input pad. A good example might be a pipeline made of filters linked one after the other, with a sink or a source on each end. In such a case, using `flow_control: :auto` in filters would ensure, that the pipeline would adapt its processing pace to the slowest element, without the necessity to implement `c:Membrane.Element.WithOutputPads.handle_demand/5` in every element. Additionally, the whole pipeline will be faster, because of the lack of performance overhead caused by `flow_control: :manual`.

## Linking pads with different `flow_control`

Two pads with the same type of `flow_control` can always be linked together. `flow_control: auto` can also always be linked with `flow_control: manual`. However, the situation becomes complicated when we try to link two pads with different types of `flow_control`, where one of them has `flow_control: push`.

When the input pad has `flow_control: push`, the output pad linked to it must also have `flow_control: push`. An attempt to link different type of an output pad will cause an error to be thrown.

Output pads with `flow_control: push` can be linked with input pads having `flow control` `:auto` or `:manual`. The demand value on such an input pad does not affect the behavior of the element with the push output pad.

When an element with a push output pad sends amount of data, that exceeds demands on the input pad, when input pad `flow_control` is: 
- `manual`, these buffers will wait in line until the element raises the demand on its input by returning the `demand` action.
- `auto`, these buffers will be processed, and the demand value on the input pad will drop below zero.

```elixir
get_child(:mixer)
|> via_in(:input, toilet_capacity: 500)
|> get_child(:sink)
```

Flow control

# Observability & logging

To observe and debug Membrane, you can use a variety of tools provided by the BEAM, like [observer](https://www.erlang.org/doc/man/observer.html) or [Process.info](https://hexdocs.pm/elixir/1.15.5/Process.html#info/1). Apart from them, Membrane provides dedicated utilities for observability and logging.

## Introspecting pipelines with kino_membrane

[kino_membrane](https://github.com/membraneframework/kino_membrane) allows for introspecting Membrane pipelines in [Livebook](https://livebook.dev). You can either run the pipeline inside livebook ([see example](https://hexdocs.pm/kino_membrane/pipeline_in_livebook.html)) or connect to a running VM ([see example](https://hexdocs.pm/kino_membrane/connect_to_node.html)). Then, it's enough to install kino_membrane:

```elixir
Mix.install([:kino_membrane])
```

and run

```elixir
KinoMembrane.pipeline_dashboard(pipeline_pid)
```

in Livebook, and you'll get a panel with the pipeline graph, metrics and other utilities:

![Kino Membrane](assets/images/kino_membrane.png)

Check [kino_membrane](https://github.com/membraneframework/kino_membrane) for details.

## Logging

Apart from the usual `Logger` configuration, logging in Membrane can be additionally configured, also via Elixir's `Config`. It allows to enable verbose mode and customize metadata, for example:

```elixir
import Config
config :membrane_core, :logger, verbose: true
```

For this change to take effect, it may be necessary to recompile the whole project with `mix deps.compile --force && mix compile --force`. See [`Membrane.Logger`](https://hexdocs.pm/membrane_core/Membrane.Logger.html) for details.

Moreover, in a pipeline or bin, you can add logger metadata to the [children spec](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html):

```elixir
@impl true
def handle_init(_ctx, opts) do
  spec = {
    child(Some.Element) |> child(Some.Bin),
    log_metadata: [pipeline_id: opts.id]
  }

  {[spec: spec], state}
end
```

and it will add the `pipeline_id` metadata key to all the logs coming from `Some.Element` and `Some.Bin` and all its descendants.

To have the metadata displayed in the logs, you need to configure it as well:
```elixir
import Config
config :logger, :console, metadata: [:pipeline_id]
```
