Here is my endpoint implementation:
```elixir
defmodule MediaServer.Endpoint.Speech.Chatbot do
  use Membrane.Bin

  def_output_pad(:output, availability: :on_request, demand_unit: :buffers, accepted_format: _any)

  @impl true
  def handle_init(_ctx, opts) do
    state = %{
      room_pid: opts.room_pid,
      track: create_track(opts.peer_id),
      peer_id: opts.peer_id
    }
    
    {[
      notify_parent: {:ready, nil},
      notify_parent: {:publish, {:new_tracks, [state.track]}}
    ], state}
  end

  @impl true
  def handle_playing(_ctx, state) do
    {[
      notify_parent: {:track_ready, state.track.id, :high, state.track.encoding}
    ], state}
  end

  @impl true
  def handle_pad_added(Pad.ref(:output, _) = pad, _ctx, state) do
    structure =
      audio_source_spec(:audio_mixer) ++
        silencer_spec(:audio_mixer) ++
        generate_track_spec(:audio_mixer, pad, state)

    {[spec: structure], state}
  end

  defp speech_spec(child, url \\ "https://example.com/download_mp3_audio") do
    [
      child(:audio_source, %Hackney.Source{location: url})
      |> via_out(Pad.ref(:output))
      |> child(:decoder, Membrane.MP3.MAD.Decoder)
      |> child(:converter, get_converter())
      |> child(:raw_parser, get_input_parser())
      |> get_child(child)
    ]
  end

  defp silencer_spec(child) do
    [
      child(:silence_generator, get_silence_generator())
      |> child(:realtimer, Membrane.Realtimer)
      |> get_child(child)
    ]
  end

  defp generate_track_spec(name, pad, state) do
    [
      child(name, Membrane.LiveAudioMixer)
      |> child(:encoder, get_encoder())
      |> child(:parser, get_output_parser())
      |> child(:payloader, get_payloader(state.track))
      |> via_in(Pad.ref(:input, {state.track.id, :high}))
      |> child(:track_sender, get_track_sender(state.track))
      |> via_out(pad)
      |> bin_output(pad)
    ]
  end
  # Helper functions omitted for brevity
end
```

I think I need to implement this `handle_parent_notification/3` function, but keep running into issues with either adding duplicate elements or linking pads multiple times and feel like there must be something simple that I'm not seeing for how this Bin could be structured for this use case
```elixir
def handle_parent_notification({:user_event, {:play_audio, url}}, _ctx, state) do
    # 1. interrupt whatever audio is currently playing 
    #   - I think I need to remove every element from :audio_source to before :track_sender with a :remove_child action?
    # 2. Replace it with audio from url
    #   - I think I need to create a new spec for the elements from :audio_source to before :track sender with the new URL with the :spec action?
end
```

From my understanding of the rtc engine, it should work as follows:
When you receive this notification, you need to publish the following events to the rtc engine:
1. Removed tracks event to delete the old track
2. New tracks event with the new track
3. Track ready event

Removing the track will trigger handle_pad_removed (which you should add) callback on which you should delete the elements of the old track

Publishing and making ready the new track will trigger handle_pad_added and you should create and link your elements.

When creating the elements, don't use a simple name like :payloader, use {:payloader, track_id} for example to have unique names. Also consider using group names, it'll be easier to delete all the elements.

Thanks billal , this worked to get the behaviour I was looking for.

However, I'm curious if there's a way to do this without creating a new track for each event received?

My application is built with the assumption that there are two parties to any WebRTC call -- the user, and the chatbot.

I have some other endpoints that persist each side of the conversation to AWS S3 (i.e. `user.wav` + `chatbot.wav`), and these assume that there is a constant `track_id` for both the WebRTC peer and the Chatbot peer. Publishing a new track triggers `handle_pad_added` in these endpoints as well, which results in either 1 audio file being saved for each new track instead of just 1 for each side of the conversation (if saved under different names) or the chatbots audio file being overwritten with the contents of each new track (if saved under the same name).

Hi naillik7710 , this should be possible to achieve with one track only. The drawback of removing and adding track once again (besides your business logic problems) is that it triggers ICE restart in WebRTC Endpoint. The easiest way, as you said, would be to remove everything up to the track sender and spawn this path once again. You can do this in the same membrane callback i.e.

```elixir
def handle_parent_notification(:user_event) do
  [remove_child: old_path, spec: new_path]
end
```

Thanks mickel8 , I've been struggling to get this approach working at all.

When I return both `remove_child: [:children, :to, :remove]` and `spec: new_path` in the same callback (where `new_path` is a `Builder` containing the children being removed), I get a `(Membrane.ParentError) Duplicated names in children specification: [:children, :to, :remove]`. 

I'm not sure why this is happening, since I am explicitly removing all children that precede the track sender with `remove_child: old_path`, and my `spec` action spawns an identical path each time. 

I suppose that there's no guarantee that actions returned from callbacks will be executed in sequence? Is there a workaround for this?

That's a good question. Maybe mat_hek can help. For now maybe let's try to give all the elements some random name that you will save in process state? You can generate id with :crypto.strong_rand_bytes()

So whenever someone passes user_event you generate random id that you associate with every element and you save this id in your state. When user_event is received next time you remove old path using id from the state, you generate new id, add a new path and save a new id in the state

Hi naillik7710 , actions are guaranteed to be executed in sequence, but removing children is not atomic, so there's no guarantee that the child will actually be removed before the new one is started. As mickel8 said, you can add unique identifiers to a child name (we usually use `make_ref()` for this purpose). Since v0.12 you can use children groups and anonymous children too: `[remove_child: :my_group, spec: {child(Anonymous.Child) |> child({:name, make_ref()}, Named.Child), group: :my_group}]`

Dynamically add & remove source elements for custom RTC Engine endpoint

naillik7710

10 responses

Join out Discord Community