Do you mean `Membrane.LiveAudioMixer`? I'm struggling to find any plugin with module `Membrane.LiveMixer`. Also Karol Konkol

Here's some more context / clarification: 

- The chatbot's "voice" may come from several different locations (e.g. HTTP requests, static audio files, etc.)
- The user can trigger a "speak" event that makes the chatbot say something. I want to ensure that the chatbot always speaks clearly, without distortion or dropped audio. 
My understanding is that any solution likely involves the Chatbot endpoint mixing a `Voice` stream with samples of generated silence to fill the gaps between "speak" events.
My current approach is this:
  - A `Chatbot` Endpoint that mixes audio from two sources (`Voice` and `Silence`) using `Membrane.LiveAudioMixer`:
    - `Voice` is a static source that listens for "speak" events triggered by the user and makes an HTTP request to fetch audio from the appropriate location, and emits the audio from the response. The `Voice` source is linked through a decoder then `Membrane.RawAudioParser` then linked to the mixer. 
  - `Silence` is a `Membrane.SilenceGenerator` source linked to `Membrane.Realtimer`, then linked to the mixer.

In order for `Membrane.LiveAudioMixer` to mix the two sources correctly,  `Membrane.RawAudioParser` needs to set a `pts_offset` for the `Voice` source, and this `pts_offset` should be the duration between the beginning of the `Silence` stream and the beginning of the `Voice` stream. 


Without updating the `pts_offset` for each new response, `Membrane.LiveAudioMixer` ends up dropping most buffers from the `Voice` stream because they are "late" relative to the `Silence` stream.
I also suspect that a single static `Voice` source is the wrong approach, and that I might need to add a new `Voice` source for each request in order to set the `pts_offset` correctly.

Any ideas as to how I can synchronize timestamps between `Silence` and `Voice` streams so that `Membrane.LiveAudioMixer` can combine them without dropping most of the `Voice` buffers?

Let me know if I can clarify anything -- any help is appreciated.

Hi, sorry for the late response.

We came up with several possible solutions to your problem, but to choose the proper one, you need to consider what exactly happens to the PTSs (presentation timestamps) of your audio files.

First thighs first, with `LiveAudioMixer` you don't need to add `SilenceGenerator`. If `LiveAudioMixer` don't have any audio with correct PTSs for generated buffer it automatically fills it with silence.

What you need to do for the rest of the pipeline to work properly, depends on whether the PTSs of buffers that `LiveAudioMixer` receives from `Voice` input are correct.

For `LiveAudioMixer` to work properly, it should receive buffers with audio PTSs equal to or larger than the PTS of the next mixed buffer. If `LiveAudioMixer` hasn't received any audio with PTSs between [last mixed buffer pts, next mixed buffer PTS], it will produce silence.

Keep in mind that some decoders/parsers may ignore input audio PTSs and just produce buffers with stable rate PTSs (e.g. decoder and parser receiving audios: [0s, 10s] and [20s, 30s] might output raw audio with timestamps [0s, 20s]).

Therefore it's crucial to determine what `LiveAudioMixer` input PTSs look like. 
If you want to have audio files like this:
[0s, 2s] - silence,
[2s, 10s] - first audio,
[10s, 20s] - silence,
[20s, 30s] - second audio

`LiveAudioMixer` have to receive:
first audio with PTSs [2s, 10s] before `LiveAudioMixer` produce buffer with PTS 2s (i.e. in the first 2s of `LiveAudioMixer` work)
second audio with PTSs [20s, 30s]  before `LiveAudioMixer` produce buffer with PTS 20s (i.e. in the first 2s of `LiveAudioMixer` work)

However, in your case, most likely the simplest solution is to modify the logic of  `LiveAudioMixer` a little bit.
If you want to just play audio whenever you receive it, you can just add it to the `LiveAudioMixer` queue without checking buffer pts.
You just need to modify a little bit `add_buffer` function in: https://github.com/membraneframework/membrane_audio_mix_plugin/blob/master/lib/membrane_live_audio_mixer/live_queue.ex

Currently, it looks like this:
```elixir
  @doc """
  Adds to a specific queue.

  When a buffer is too old it will be dropped
  When a part of a buffer is too old, only the part that is "fresh" will be added.
  When a whole buffer is "fresh", the whole buffer will be added.
  All the holes between audio packets will be filled with silence.

  The state of the buffer, whether it's too old or not, is based on LiveQueue's `current_time`.
  """
  @spec add_buffer(t(), any(), Membrane.Buffer.t()) :: t()
  def add_buffer(
        %{
          stream_format: stream_format,
          current_time: current_time,
          queues: queues
        } = lq,
        id,
        %{pts: pts, payload: payload}
      ) do
    queue = Map.fetch!(queues, id)
    pts = pts + queue.offset
    payload_duration = RawAudio.bytes_to_time(byte_size(payload), stream_format)
    end_pts = pts + payload_duration
    queue_ts = current_time + queue.buffer_duration

    case {pts > queue_ts, end_pts > queue_ts} do
      {false, true} ->
        drop_duration = queue_ts - pts
        drop_bytes = RawAudio.time_to_bytes(drop_duration, stream_format)
        <<_rest::binary-size(drop_bytes), to_add::binary>> = payload

        to_add_duration = payload_duration - drop_duration

        update_in(lq, [:queues, id], fn queue ->
          queue
          |> Map.update!(:buffer, &(&1 <> to_add))
          |> Map.update!(:buffer_duration, &(&1 + to_add_duration))
        end)

      {true, true} ->
        silence_duration = pts - queue_ts
        silence = RawAudio.silence(stream_format, silence_duration)

        new_lq =
          update_in(lq, [:queues, id], fn queue ->
            queue
            |> Map.update!(:buffer, &(&1 <> silence <> payload))
            |> Map.update!(:buffer_duration, &(&1 + silence_duration + payload_duration))
          end)

        new_lq

      _else ->
        lq
    end
  end
```
and you probably don't want to check buffer pts and add the appropriate silence part.

Thanks so much, this was very helpful.

I modified the `add_buffer` function like so and it seems to have solved the issue:
```elixir
@spec add_buffer(t(), any(), Membrane.Buffer.t()) :: t()
  def add_buffer(
        %{
          stream_format: stream_format,
          current_time: current_time,
          queues: queues
        } = lq,
        id,
        %{pts: pts, payload: payload}
      ) do
    queue = Map.fetch!(queues, id)
    pts = pts + queue.offset
    payload_duration = RawAudio.bytes_to_time(byte_size(payload), stream_format)
    queue_ts = current_time + queue.buffer_duration

    if pts > queue_ts do
      silence_duration = pts - queue_ts
      silence = RawAudio.silence(stream_format, silence_duration)

      new_lq =
        update_in(lq, [:queues, id], fn queue ->
          queue
          |> Map.update!(:buffer, &(&1 <> silence <> payload))
          |> Map.update!(:buffer_duration, &(&1 + silence_duration + payload_duration))
        end)

      new_lq
    else
      update_in(lq, [:queues, id], fn queue ->
        queue
        |> Map.update!(:buffer, &(&1 <> payload))
        |> Map.update!(:buffer_duration, &(&1 + payload_duration))
      end)
    end
  end
```

RTC Engine, HTTP Sources, and pts offsets

naillik7710

8 responses

Join out Discord Community