Membrane.Realtimer and OPUS stream: dropping packets

06/12/2023, 17:56:19

Hi. We're trying to stream PCM audio we're getting from Amazon Polly (text-to-speech) over WebRTC in OPUS format and are so very close 😁 , but the problem now is that the packets are being sent too fast and the browser is just discarding the earliest ones 😭 . So if the message is short ("hello"), we hear the whole thing. Above a certain length ("hello, how are you. Is the weather nice?"), it drops the beginning and we hear only "...ther nice?", for example. The payload from Polly has the full audio and as the message length increases, we see in chrome://webrtc-internals/ that packets received increases. When some of the message is dropped, we see that too (see screenshot).

We believe Realtimer is supposed to limit playback speed to realtime but it doesn't seem to have any effect. If anyone could take a look at these code snippets and suggest what we might be doing wrong or missing, I'd appreciate it. The value of state.track.encoding is :OPUS and state.track.clock_rate is 4800.

payloader = Membrane.RTP.PayloadFormat.get(state.track.encoding).payloader
payloader_bin = %Membrane.RTP.PayloaderBin{
  payloader: payloader,
  ssrc: 22,
  payload_type: 96,
  clock_rate: state.track.clock_rate
spec = [
  child(:input, %Polly.Source{})
  |> child(:encoder, %Membrane.Opus.Encoder{
    application: :audio,
    input_stream_format: %RawAudio{
      channels: 1,
      sample_format: :s16le,
      sample_rate: 16_000
  |> child(:parser, %Membrane.Opus.Parser{delimitation: :undelimit})
  |> child(:payloader, payloader_bin)
  |> via_in(:input, toilet_capacity: @toilet_capacity)
  |> child(:realtimer, Membrane.Realtimer)
  |> via_in(Pad.ref(:input, {, :high}))
  |> child(:track_sender, %Membrane.RTC.Engine.Endpoint.WebRTC.TrackSender{
    track: state.track,
    variant_bitrates: %{high: 5},
  |> via_out(Pad.ref(:output, {, :high}))
  |> child(:inspector, InterpretoWeb.InspectElement)
  |> bin_output(pad)
9 responses