Membrane help

Hello everyone, we have a music streaming app (newnew.media) currently written in React/nodejs having a simple player with mp3 url provided as audio source.

We'd like to have something more robust and get some load off the front end app so want to give membrane a try. Open for ideas from experienced ones ~

mickel8 do u have any recommendation what's best way to integrate membrane to a react app?

I am actually trying to store segment files in a dynamically created directory based on stream_key in rtmp_to_hls pipeline. I am getting stream_key via custom validator from %SourceBin{socket: socket, validator: %Message.Handler{}}. But not sure how to change output value in %Membrane.HTTPAdaptiveStream.Storages.FileStorage{directory: "output"}.

My pipleine

 structure = [
      child(:src, %SourceBin{socket: socket, validator: %RTMPServer.StreamHandler{}})
      |> via_out(:audio)
      |> via_in(Pad.ref(:input, :audio),
        options: [encoding: :AAC, segment_duration: Membrane.Time.seconds(4)]
      )
      |> child(:sink, %Membrane.HTTPAdaptiveStream.SinkBin{
        manifest_module: Membrane.HTTPAdaptiveStream.HLS,
        target_window_duration: Membrane.Time.seconds(30),
        persist?: false,
        mode: :live,
        hls_mode: :muxed_av,
        storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{directory: "output"}
      }),
      get_child(:src)
      |> via_out(:video)
      |> via_in(Pad.ref(:input, :video),
        options: [encoding: :H264, segment_duration: Membrane.Time.seconds(4)]
      )
      |> get_child(:sink)

I'm struggling to get a react-native client to connect to my membrane server. I'm just running locally right now. I start my membrane server with EXTERNAL_IP={my ip} mix phx.server. I'm using the @jellyfish-dev/react-native-membrane-webrtc client in my react native code

Then I have the following connection code in my react-native view. I see the console log statements for init connect and attempting connect. But it never connects. I don't see a connection message in my phoenix server, nor the successful connection message.

I tried increasing the log verbosity, but didn't get anything out of the logs from react-native. Is there something obviously wrong with my connection string? Is it expecting something different for the server URL?

const startServerConnection = async () => {
    const deviceID = await getUniqueId();
    
    try {
      console.log('attempting connnect')
      //Should make this environment aware at some point
      connect('https://192.168.121.87:4001/socket/', "room:" + deviceID, {
        endpointMetadata: {
          displayName: deviceID,
        },
        socketChannelParams: {
          childrenNames: params.childrenNames,
          talkAbout: params.talkAbout          
        }
      }).then(() => {
        console.log('connected. starting mic')
        startMicrophone({ audioTrackMetadata: { active: true, type: 'audio' } });
      }).catch((e) => {
        console.log('connection error: ' + e);  
      });
      
    } catch (e) {
      console.log('connection error: ' + e);
    }
  };

  useEffect(()=>{
    let isMounted = true;
    const logLevel: LoggingSeverity = LoggingSeverity.Verbose
    changeWebRTCLoggingSeverity(logLevel)
    const initConnection = async ()=>{
      try {
        console.log('init connnect')
        await startServerConnection();
        if (isMounted) {
          setIsConnected(true)
        }
      } catch (error) {
        console.log('init connection error: ' + error)
      }
    }

Hello! I'm new to Membrane, but seasoned Elixir dev. I'm currently working on a project (similar to 3D printer) where a RPi Nerves device is controlling an ongoing process. As a part of this, there are two USB cameras which plug into the pi and I would like to livestream to the user. One good resource that I found was this excellent tutorial by pressy4pie, one of the Nerves maintainers (huge thank you).

One major difference between what is documented here though is that I don't have a main server running on the internet somewhere. My Nerves project also bundles and starts a Phoenix LiveView UI on device boot. What this means: user connects directly to https://nerves.local:4000 which is running on the device. The way it currently works, the Phoenix project starts a GenServer which constantly reads the cameras using the OpenCV bindings (Evision project). Upon receiving the frame, it does some basic image processing, encodes the image as base64 JPG and then pushes it over the websocket to the LiveView client img tag. It works, but under poor network conditions we are starting to see that the LiveView process mailbox is getting clogged, resulting in significant lag in the live stream as well as for user input events.

I'd like to see if Membrane is suitable to tackle this problem. Preferably, we could use the hardware h264 accelerator in the pi for better performance. My requirements are to live stream to browser (via HLS? WebRTC? idk), and allow the software running on device grab the most recent frame from the camera stream for async image processing.

Some recommendations to get started in the right direction would be appreciated 🙂

Thanks!

In our web and mobile app(react native) we need to provide 1:1 voice call between two user. I am not sure that either we use membrane webrtc engine or not.We need Stun and Turn server and prefer Stun if not then Turn mechanism as fallback.Membrane SFU is suitable for this as it have builtin turn server. Or you prefer to use other turn server like coturn, xturn.

Hello everyone in the Elixir Membrane community! I've recently taken an interest in Membrane and have started learning about it. I've been working with Elixir for about three years, so I am fairly comfortable with Elixir. However, I'm not very familiar with media-related technologies, so I'm facing some challenges. While going through the tutorials, I've come across a question. Below is a demo provided by Membrane that reads a music file and outputs it to a speaker. How can I make the music play automatically again after it ends? To put it in terms of a music player, I would like to implement a loop feature. If you have any good ideas, or if there are documents or code that I should look into further, your guidance would be greatly appreciated.

def handle_init(_ctx, path_to_mp3) do
    # Setup the flow of the data
    # Stream from file
    spec =
      child(:file, %Membrane.File.Source{location: path_to_mp3})
      # Decode frames
      |> child(:decoder, Membrane.MP3.MAD.Decoder)
      # Convert Raw :s24le to Raw :s16le
      |> child(:converter, %Membrane.FFmpeg.SWResample.Converter{
        output_stream_format: %Membrane.RawAudio{
          sample_format: :s16le,
          sample_rate: 48000,
          channels: 2
        }
      })
      # Stream data into PortAudio to play it on speakers.
      |> child(:portaudio, Membrane.PortAudio.Sink)

    {[spec: spec], %{}}
  end

Hey! Trying to set up a basic pipeline that will take an MP3 file and stream it via HLS.

I'm able to execute the pipeline and generate an index.m3u8 file (mainly adapting from the camera to HLS example), but I cannot play the file via ffplay when hosted with the example Python server.

[warning] <0.861.0>/:encoder Processing queue is not empty, but EndOfStream event was received

I'm assuming that error warning is related to why my file is unplayable, but I'm unsure how to proceed.

My pipeline is defined as:

defmodule MyApp.Audio.Pipeline do
  use Membrane.Pipeline

  @impl true
  def handle_init(_ctx, path) do
    spec =
      child(:source, %Membrane.File.Source{location: path})
      |> child(:decoder, Membrane.MP3.MAD.Decoder)
      |> child(:converter, %Membrane.FFmpeg.SWResample.Converter{
        output_stream_format: %Membrane.RawAudio{
          sample_format: :s16le,
          sample_rate: 48000,
          channels: 2
        }
      })
      |> child(:encoder, Membrane.AAC.FDK.Encoder)
      |> child(:parser, %Membrane.AAC.Parser{out_encapsulation: :none})
      |> via_in(:input,
        options: [
          encoding: :AAC,
          segment_duration: Membrane.Time.seconds(5)
        ]
      )
      |> child(:sink, %Membrane.HTTPAdaptiveStream.SinkBin{
        manifest_module: Membrane.HTTPAdaptiveStream.HLS,
        storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{directory: "output"}
      })

    {[spec: spec], %{}}
  end
end

Thanks in advance!

HI folks, we have a pipeline with many crash groups. It looks like when a crash group goes down, an end_of_stream message is delivered immediately to the input pad of the element attached to it (which is in a crash group by itself). Is this expected? We're on membrane_core v0.12.5!

Hi, I'm looking for some help with dynamically adding and removing audio sources from a Membrane Bin making sure that all audio from previously added sources is interrupted when a new source is added, in the context of an application built with membrane_rtc_engine,

The users of my application can join a WebRTC room along with a chatbot "peer", represented by the Chatbot module below:

While a user is in the room with the chatbot peer, they can submit an event {:user_event, url} that trigger the Chatbot endpoint to fetch audio from url using membrane_hackney_plugin.

When a user submits such an event, I would like to interrupt whatever audio the chatbot is currently playing and replace it with audio from the new url.

Ultimately I'm trying to determine a way to structure this Bin so that I can implement handle_parent_notification/3

defmodule MediaServer.Endpoint.Speech.Chatbot do
  def handle_parent_notification({:user_event, {:play_audio, url}}, _ctx, state) do
    # 1. interrupt whatever audio is currently playing 
    #   - I think I need to remove every element from :audio_source to before :track_sender with a :remove_child action?
    # 2. Replace it with audio from url
    #   - I think I need to create a new spec for the elements from :audio_source to before :track sender with the new URL with the :spec action?
  end
end

I've tried a few variations of adding / removing children, but keep running into issues -- either with adding duplicate children, or linking pads multiple times. Any advice or examples of how to do this would be appreciated!

Hello friends, I'm trying to get microphone input (via Membrane.PortAudio.Source) packaged into an RTP stream and sent to a server and can't quite seem to get it right.

Excerpt below based on the demo in membrane-demo/rtp but with microphone input substituted and newer syntax.

alias Membrane.{RTP, UDP, PortAudio, RawAudio}
...
links = [
      child(:mic_input, %PortAudio.Source{
        channels: 2,
        sample_rate: 24_000
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
        application: :audio,
        input_stream_format: %RawAudio{
          channels: 2,
          sample_format: :s16le,
          sample_rate: 24_000
        }
      })
      |> via_in(Pad.ref(:input, audio_ssrc), options: [payloader: RTP.Opus.Payloader])
      |> child(:rtp, %RTP.SessionBin{
        secure?: secure?,
        srtp_policies: [
          %ExLibSRTP.Policy{
            ssrc: :any_inbound,
            key: srtp_key
          }
        ]
      })
      |> via_out(Pad.ref(:rtp_output, audio_ssrc), options: [encoding: :OPUS])
      |> child(:audio_realtimer, Membrane.Realtimer)
      |> child(:audio_sink, %UDP.Sink{
        destination_port_no: destination_port,
        destination_address: destination_address
      })
    ]

This seems to throw an error when generating the headers because buffer.pts is nil:

[error] <0.815.0>/:rtp/{:stream_send_bin, 1236}/:payloader/:header_generator Error handling action {:split, {:handle_buffer, [[:input, %Membrane.Buffer{payload: <<220, 255, 254>>, pts: nil, dts: nil, metadata: %{}}]]}} returned by callback Membrane.RTP.HeaderGenerator.handle_buffers_batch

[error] GenServer #PID<0.849.0> terminating
** (FunctionClauseError) no function clause matching in Ratio.mult/2
    (ratio 2.4.2) lib/ratio.ex:418: Ratio.mult(nil, 48000)
    (membrane_rtp_plugin 0.23.0) lib/membrane/rtp/header_generator.ex:71: Membrane.RTP.HeaderGenerator.handle_process/4

Does anyone have any tips on what I might be doing wrong?

Hey people, we get a RTC Engine (#PID<0.6475.0>) crashed with reason: {%Membrane.CallbackError{message: \"Callback handle_child_pad_removed/4 is not implemented in Membrane.RTC.Engine\\n\"} error when removing an endpoint from the RTC engine. We're on the release/0.15 branch and membrane_core v0.12.5. We use the RTC.Engine.remove_endpoint(state.rtc_engine, peer_id) function to get rid of the endpoint. Are we missing something? I see that the rtc engine uses v0.12.3 internally but I suspect the two versions are supposed to be compatible given the version code difference... What do we do? 😄

I want to create video chat application in vanilla JavaScript can someone guide me step by step

Hello, I apologize if this question is trivial as I am new to the Erlang/Elixir ecosystem. I recently cloned https://github.com/membraneframework/membrane_demo to try out the webrtc_videoroom demo. Everything went fine up until running mix phx.server:

== Compilation error in file lib/membrane/testing/endpoint.ex ==
** (FunctionClauseError) no function clause matching in String.trim/1

    The following arguments were given to String.trim/1:

        # 1
        nil

    Attempted function clauses (showing 1 out of 1):

        def trim(string) when is_binary(string)

    (elixir 1.15.0) lib/string.ex:1270: String.trim/1
    lib/membrane/core/child.ex:28: Membrane.Core.Child.generate_moduledoc/2
    expanding macro: Membrane.Element.Base.__before_compile__/1
    lib/membrane/testing/endpoint.ex:1: Membrane.Testing.Endpoint (module)
could not compile dependency :membrane_core, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile membrane_core --force", update it with "mix deps.update membrane_core" or clean it with "mix deps.clean membrane_core"

I am using Elixir v1.15.0, Erlang v26. This compile error was thrown both with the default mix.exs and after everything was updated to the latest versions listed on Hex. Thank you for any assistance, and I apologize if this is a beginner Elixir error -- it definitely feels like a user-error what with String being the 'problem'.

A change was made to the RTC engine, implementing a to_type_string function for all existing endpoints. This function seems to be necessary for an endpoint to be added.

This has the side-effect of removing the ability to create new endpoints - only the predefined ones are allowed: https://github.com/jellyfish-dev/membrane_rtc_engine/blame/master/lib/membrane_rtc_engine/endpoints/webrtc/media_event.ex#L366-L368

Can this be worked around? Was this an intended outcome?

[error] GenServer #PID<0.2030.0> terminating ** (MatchError) no match of right hand side value: <<141, 8, 0, 1, 83, 0, 0, 89, 0, 0, 0, 0, 175, 1, 33, 123, 78, 85, 255, 238, 0, 0, 42, 34, 44, 205, 18, 15, 66, 193, 65, 105, 221, 249, 188, 235, 121, 174, 250, 231, 137, 18, 243, 65, 172, 67, 185, 198, 224, 174, ...>> (membrane_h264_ffmpeg_plugin 0.25.4) lib/membrane_h264_ffmpeg/parser/nalu.ex:80: Membrane.H264.FFmpeg.Parser.NALu.parse_nalu/3

Hi folks, I'm building an application with Membrane and MembraneRTCEngine and am having some trouble with garbled & dropped audio resulting from some issues with timestamping audio from an HTTP source.

The application lets a user join a WebRTC audio call with a chatbot "peer".

The user can click a button in their browser which triggers the chatbot to "speak" -- under the hood, this requests audio data via HTTP which is then piped through a chatbot "peer" endpoint.

Currently, I am able to hear the audio received from the HTTP source, but it is garbled, truncated, or otherwise incoherent. My Chatbot endpoint uses Membrane.LiveMixer to mix audio from two sources:

• An HTTP Source, parsed with Membrane.RawAudioParser to apply timestamps
• A Silence generator which is Membrane.SilenceGenerator passed through Membrane.RealTimer

Membrane.RawAudioParser allows you to set pts_offset value, which Membrane.LiveMixer uses to select the correct audio to mix from each stream.

The examples for Membrane.RawAudioParser and Membrane.LiveMixer use constant offsets (e.g. "If Source B starts 5 seconds after Source A..."). However, I do not know the offset to set in advance -- it should be the difference between the start of the Chatbot endpoint / silence generator, and the start of the stream from the HTTP source.

Do you have any suggestions as to how to correctly set the pts_offset for Membrane.RawAudioParser when I do not know the offset in advance? Is there a way to set it dynamically (e.g. using start_of_stream events?)

Hey team I have an RTMP pipeline (which I simplified for the purpose of the question):

  def handle_init(ctx, socket: socket) do
    Logger.info("Starting RTMP pipeline")

    structure = [
      child(:source, %Membrane.RTMP.Source{
        socket: socket,
        validator: Membrane.RTMP.DefaultMessageValidator
      })
      |> child(:demuxer, Membrane.FLV.Demuxer)
      |> via_out(Pad.ref(:audio, 0))
      # we use a fake sink to terminate the pipeline
      |> child(:fake_sink, Membrane.Fake.Sink.Buffers)
    ]

    {[spec: structure, playback: :playing], %{}}
  end

Which is served by an TCP server:

 {Membrane.RTMP.Source.TcpServer, rtmp_server_options()},

defp rtmp_server_options do
    %Membrane.RTMP.Source.TcpServer{
      port: rtmp_port(),
      listen_options: [
        :binary,
        packet: :raw,
        active: false,
        ip: rtmp_ip()
      ],
      socket_handler: fn socket ->
        # On new connection a pipeline is started
        {:ok, _supervisor, pipeline} = MyPipeline.Pipeline.start_link(socket: socket)
        {:ok, pipeline}
      end
    }
  end

This works great, I can initiate an rtmp session using this kind of url for example: rtmp://localhost:5000

Now I'm trying to use url like this: rtmp://localhost:5000?rtmp_id=xxxx or even something like this: rtmp://localhost:5000/xxxx/ So within my pipeline I can get this xxxx and use if for different things (maybe you can think of a filename which could be anice usecase) So far I don't manage to find how I can get this xxxx within my pipeline, happy to know if there is anotherway to do that 😄 ! Thanks

Hey fellas, we want to make something similar to what .sloy brought up recently with react native integration. I couldn't really grasp how the architecture will be (especially the player).

So how does an integration with membrane would really work? Currently we have a music player where it plays from 2 kinds of sources; one is a remote postgres db, other is IPFS for NFT music.

Currently we have a react front-end and our player just receives a list of src urls to play on the client-side. It sometimes receives mp3 and sometimes mp4 (especially on NFTs it is mostly in audiovisual format).

In case we get membrane up, will it work as a rest service or something? Would like to make use of membrane for the player, what'd be the best architecture? Would be really cool to get our new architecture's multimedia service developed together.

This is the project I'm talking about: https://newnew.media

Thank you for the support in advance ~

Hi there, in a video conferencing scenario. Fully peer to peer connectivity would you say 5-8 peers is a reasonable limit?

After that moving to a server based relay system makes sense? Like Jellyfish? Do you have any numbers of clients / server size that makes sense in a relay scenario?

Do you have any recommendations on how to obtain SSL certs for use in the handshake_opts of the WebRTC Endpoint?

I posting this as sometimes I'm not getting it. Consider a pipeline test such this one

 @tag :tmp_dir
 test "pipeline starts", %{tmp_dir: tmp_dir} do
   master_playlist_uri = URI.new!("test/data/stream.m3u8")
   reader = HLS.FS.OS.new()
   rtmp_output_url = "rtmp://localhost:4000/live"
   output_path = Path.join([tmp_dir, "output.mp4"])

   links = [
     child({:endpoint, "st"}, %Studio.StreamingEndpoint{
       owner_pid: self(),
       rtc_engine: nil,
       rtmp_output_url: rtmp_output_url,
       master_playlist_uri: master_playlist_uri,
       reader: reader
     })
   ]

   pid = Membrane.Testing.Pipeline.start_link_supervised!(spec: links)

   ffmpeg =
     Task.async(fn ->
       # We do not want ffmpeg to block us.
       start_ffmpeg(rtmp_output_url, output_path)
     end)

   assert_end_of_stream(pid, :sink, :audio, @timeout)
   assert_end_of_stream(pid, :sink, :video, @timeout)
   assert :ok == Task.await(ffmpeg)
 end

As soon as the bin is indeed receiving the EOS on both audio and video pads, I would expect this assert to pass. Instead, no such messages are received. Notice that the StreamingEndpoint is a Bin and the Sink we're talking about here is a Bin as well, but the video and audio pads are exposed in the StreamingEndpoint's links, so it's not a nested thing in theory. What is supposed to happen? 😄

I had cloned membrane video room repo few months ago.

I am using it on my staging server everything is working fine I am using it for multiple applications through android, ios, web

After sometime it get disconnected without any reason but after restarting everything works fine for some time

**I want to make it live So, I can not restart it everytime **

How do you handle graceful shutdown? We would like to leave the pipeline running until the sink receives an end_of_stream and finishes his work. Is there any recommended way to handle this?

When stopping the application I can see the following logs:

iex(4)> :init.stop
:ok
Pepe.Pipeline/ Pipeline supervisor got exit from a linked process, stopping pipeline
Pepe.Pipeline/ Pipeline supervisor got exit from pipeline with reason :shutdown, stopping subprocess supervisor
....and for all the components

can anyone help me with this ?

Hi everyone 🙂 (message can be redundant with the one I posted on slack, sorry about that) I'm currently building a POC for live transcription using whisper model. I checked Lawik example already which uses either the mic as or a file as a source to achieve it. In my case I do have RTMP source with AAC codec for audio and flv format. I've built a pipeline which seems to be "ok":

  def handle_init(_ctx, socket: socket) do
    structure = [
      child(:source, %Membrane.RTMP.SourceBin{
        socket: socket,
        validator: Membrane.RTMP.DefaultMessageValidator
      })
      |> via_out(:audio)
      |> child(:decoder, Membrane.AAC.FDK.Decoder)
      |> child(:converter, %Converter{
        output_stream_format: %RawAudio{
          sample_format: :f32le,
          sample_rate: 16000,
          channels: 1
        }
      })
      |> child(:fake_sink, Membrane.Fake.Sink.Buffers)
    ]

    {[spec: structure, playback: :playing], %{}}}
  end

When I start it it initialise without issues however it doesn't do anything (from what I can see) (I see debug logs showing elements received the play request)

I would be happy to get advises on that one :D, thanks !

We are running into some intermittent failures with the RTMP sink. What we notice is that sometimes a given pipeline will stream all the way through and sometimes the RTMP sink will raise when writing frames and crash the pipeline.

22:01:58.807 [error] <0.5365.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 139, 207, 65, 154, 128, 54, 188, 23, 73, 255, 152, 130, 98, 10, 43, 88, 28, 64, 176, 10, 39, 247, 233, 179, 54, 27, 17, 168, 97, 24, 82, 152, 175, 21, 138, 252, 216, 108, 205, 134, ...>>, pts: 27240000000, dts: 27240000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:01:58.808 [error] GenServer #PID<0.5369.0> terminating
** (RuntimeError) writing audio frame failed with reason: "End of file"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:269: Membrane.RTMP.Sink.write_frame/3

this one crashed ~20 seconds into a 1 hour long stream, so I don't understand how it could have been the end of file. Additionally, in the error log I see an error handling a video buffer, however, the crash is in the write audio frame path.

22:57:02.461 [error] <0.3134.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 0, 7, 65, 154, 0, 122, 0, 63, 204>>, pts: 3626320000000, dts: 3626320000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:57:02.461 [error] GenServer #PID<0.3138.0> terminating
** (RuntimeError) writing video frame failed with reason: "Invalid argument"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:285: Membrane.RTMP.Sink.write_frame/3

Looking at the demo I can't see a "normal" pipeline. I want to process video frames in a pipeline. Do ML to them and stuff. I want the video from the browser. WebRTC should be a reasonable way to stream that. I want to apply a bunch of stuff between input and final HLS output.

Is there a straightforward way for that or is the engine mostly repackaging the video and audio streams?

Hi! I'm trying to implement a solution in which I can stream audio from an API to the client via the Membrane RTC Engine.

I'm basing my Membrane.Source implementation on Membrane.Hackney.Source, but now I'm a bit stuck on getting the audio to send from that to the WebRTC Endpoint.

I've created an endpoint that fowards the output of my pad to Membrane.RTC.Engine.Endpoint.WebRTC.TrackSender, but I'm wondering:

1. Is this the right approach?
2. If so, where do I get the correct track with which to configure the TrackSender?

If there are any examples out there implementing any sort of Source-based Membrane RTC Endpoint, I'd love to study it.

Thanks!

Hi,

I'm trying to get my IP camera to work with the rtsp_to_hls demo. I get this:

[debug] Application is starting
[debug] Pipeline start link: module: Membrane.Demo.RtspToHls.Pipeline,
pipeline options: %{output_path: "hls_output", port: 20000, stream_url: "rtsp://tapoadmin:[email protected]:554/stream1"},
process options: []

[debug] Source handle_init options: %{output_path: "hls_output", port: 20000, stream_url: "rtsp://tapoadmin:[email protected]:554/stream1"}
[debug] ConnectionManager: start_link, args: [stream_url: "rtsp://tapoadmin:[email protected]:554/stream1", port: 20000, pipeline: #PID<0.345.0>]
[debug] ConnectionManager: Initializing
[debug] ConnectionManager: Connecting
[debug] [pipeline@<0.345.0>] Changing playback state from stopped to prepared
[debug] [pipeline@<0.345.0>] Playback state changed from stopped to prepared
[debug] [pipeline@<0.345.0>] Changing playback state from prepared to playing
[debug] [pipeline@<0.345.0>] Playback state changed from prepared to playing
[debug] ConnectionManager: Setting up RTSP description
[warning] ConnectionManager: Connection failed: :getting_rtsp_description_failed

Nothing ends up in the hls_outputdirectory. The camera works, both via the Tapo App on my mobile, as well as answering ping from me. The use of the User:Password , I got from: https://www.tp-link.com/se/support/faq/2680/

Any hints on how to debug this would be very nice 🙂

Hi, is there documentation on using Membrane with external STUN? Option was in older version, seems moved

I am trying to get the membrane_hackney_plugin to send files to a CDN. Trying to get it to work with one file and then set it up for HLS if all goes well. I'm getting an error whenever I run it.

The Code:

defmodule Membrane.Demo.SimplePipeline do
  use Membrane.Pipeline

  alias Membrane.{File, Hackney}

  @impl true
  def handle_init(_context, _) do

    structure = [
      child(:source, %Membrane.File.Source{location: "./sample.mp3"})
      |> child(:sink, %Membrane.Hackney.Sink{method: :put, location: build_uri("my-bucket", "sample.mp3"), headers: [access_key(),
{"content-type", "audio/flac"}]}),
    ]

    {[spec: {structure, []}, playback: :playing], %{}}
  end

  @impl true
  def handle_notification(%Hackney.Sink.Response{} = response, from, _ctx, state) do
    IO.inspect({from, response})
    {[], state}
  end

  def handle_notification(_notification, _from, _ctx, state) do
    {[], state}
  end

  defp access_key do
    {"AccessKey", "foo-bar"}
  end

  defp build_uri(bucket, name) do
    "https://storage.bunnycdn.com/ny.storage.bunnycdn.com/#{bucket}/" <>
      URI.encode_query(upload_type: "media", name: name)
  end
end

and the error

** (MatchError) no match of right hand side value: {:error, :req_not_found}
    (membrane_hackney_plugin 0.9.0) lib/membrane_hackney/sink.ex:93: Membrane.Hackney.Sink.handle_end_of_stream/3
    (membrane_core 0.11.3) lib/membrane/core/callback_handler.ex:138: Membrane.Core.CallbackHandler.exec_callback/4
    (membrane_core 0.11.3) lib/membrane/core/callback_handler.ex:62: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 0.11.3) lib/membrane/core/element/event_controller.ex:79: 
...

Can someone help me figure out what I'm doing wrong here? I tried to follow the update to v0.11.0 guides with the simple_pipeline demo (modified to use the Hackney.Sink) and I'm getting errors.

structure = [ child(:source, %Membrane.File.Source{location: "sample.mp3"}), child(:sink, %Membrane.Hackney.Sink{method: :put, location: build_uri("my_store", "sample.mp3"), headers: [access_key(), {"content-type", "audio/flac"}]}), get_child(:source) |> child(:sink) ]

 {[spec: structure, playback: :playing], %{}}

I am getting the error

** (MatchError) no match of right hand side value: {:error, {%Membrane.ParentError{message: "Invalid children config: {:sink, [], %{get_if_exists: false}}"}, [{Membrane.Core.Parent.ChildEntryParser, :parse_child, 1, [file: 'lib/membrane/core/parent/child_entry_parser.ex', line: 40]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1658]}, {Membrane.Core.Parent.ChildLifeController, :setup_children, 3, [file: 'lib/membrane/core/parent/child_life_controller.ex', line: 189]}, {Enum, :"-flat_map_reduce/3-fun-1-", 3, [file: 'lib/enum.ex', line: 1288]}, {Enumerable.List, :reduce, 3, [file: 'lib/enum.ex', line: 4751]}, {Enum, :flat_map_reduce, 3, [file: 'lib/enum.ex', line: 1287]}, {Membrane.Core.Parent.ChildLifeController, :handle_spec, 2, [file: 'lib/membrane/core/parent/child_life_controller.ex', line: 124]}, {Membrane.Core.CallbackHandler, :"-handle_callback_result/5-fun-0-", 5, [file: 'lib/membrane/core/callback_handler.ex', line: 187]}]}} lib/membrane_demo/simple_pipeline.ex:80: (file) (elixir 1.14.2) lib/kernel/parallel_compiler.ex:346: anonymous fn/5 in Kernel.ParallelCompiler.spawn_workers/7

No content

When attempting to connect to Membrane VideoRoom from a client on a different origin, the WebRTC connection fails to establish. The following error message is displayed in the console:

"ICE failed, your TURN server appears to be broken"

Steps to Reproduce:

• Open the Membrane VideoRoom application on a server with a TURN server configured
• Attempt to connect to the application from a client on a different origin
• Observe that the WebRTC connection fails to establish and the error message "ICE failed, your TURN server appears to be broken" is displayed in the console.

Expected Result: The WebRTC connection should successfully establish, allowing the client to participate in the video conference.

Actual Result: The WebRTC connection fails to establish and the error message "ICE failed, your TURN server appears to be broken" is displayed in the console.

Workarounds: None found.

Additional Information:

• The TURN server has been configured correctly and is operational.
• The server's firewall has been configured to allow incoming traffic on the required ports.
• The WebRTC connection is successfully established when connecting from a client on the same origin.
• The issue only occurs when attempting to connect to the application from a client on a different origin.
• I'm developing a frontend-only application to connect to the server using the React application from membrane_videoroom hosted on a different origin. I've used the Corsica plug to allow all origins.

Environment: Membrane VideoRoom version: 2.0.1 Server OS: Ubuntu 20.04 Client OS: Windows 10 Browser: Google Chrome 97.0.4692.99

Attachments: Log files from the server and client are attached for further investigation.

Request: Could you please provide a beginner-friendly explanation of the root cause of the issue and suggest any possible solutions that could help resolve it? Thank you in advance for your help!

Hello there,

I've made it so far as to have a pipeline that has an RTMP SourceBin with a validator to ensure the I have a correctly formatted stream key, but I would like to be able to retrieve/inspect the key affiliated with a given pipeline after calling start_link.

The concrete use case is that I would like to be able to match a pipeline & stream key, to be able to use these in tandem in other areas of my application.

Would you have any pointers on how to go by accomplishing this?

I have a pipeline which I see fail intermittently on startup due to a toilet overflow of a realtimer element. The downstream element of the realtimer is a rtmp sink.

In my pipeline I have specified the toilet capacity with via_in to the realtimer as shown in the docs:

      # shortened for brevity
      |> child(:h264_encoder, %Membrane.H264.FFmpeg.Encoder{preset: :ultrafast})
      |> child(:h264_parser, %Membrane.H264.FFmpeg.Parser{
        framerate: {30, 1},
        alignment: :au,
        attach_nalus?: true,
        skip_until_keyframe?: false
      })
      |> child(:video_payloader, Membrane.MP4.Payloader.H264)
      |> via_in(:input, toilet_capacity: 500)
      |> child(:video_realtimer, Membrane.Realtimer)
      |> via_in(:video)
      |> get_child(:rtmp_sink),
      ################################################################
      # shortened for brevity
      |> child(:aac_encoder, Membrane.AAC.FDK.Encoder)
      |> child(:aac_parser, %Membrane.AAC.Parser{
        in_encapsulation: :ADTS,
        out_encapsulation: :ADTS
      })
      |> via_in(:input, toilet_capacity: 500)
      |> child(:audio_realtimer, Membrane.Realtimer)
      |> via_in(:audio)
      |> get_child(:rtmp_sink)

However, I see the reported toilet size as 200 instead of the specified value when the pipeline. crashes:

10:58:09.755 [error] <0.591.0>/:audio_realtimer/ Toilet overflow.

Reached the size of 201, which is above toilet capacity (200)
when storing data from output working in push mode. It means that some element in the pipeline
processes the stream too slow or doesn't process it at all.
To have control over amount of buffers being produced, consider using output in pull mode
(see `Membrane.Pad.mode_t`).
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.

Any thoughts on why the config isn't bubbling down to the realtimer?

Hello there,

I'm a first time user of Membrane and very much a fan. I have setup a Pipeline where I have a RTMP SourceBin that forwards the stream to a given RTMP sink. I'm attempting to use the Tee plugin to duplicate the source input to then conditionally be able to forward it to a secondary RTMP sink.

Optimally, I'd be able to only start pushing the output to the secondary sink based on an interaction with a separate web interface- but for now, I'm just trying to successfully duplicate the source stream.

I have had no success accomplishing this using the Tee plugin- is it viable to use this plugin to duplicate a RTMP source? or should I have two tee's, one for each of the video & audio outputs?

Any pointers in the right direction would be very welcome!

Hi, I'm trying to write a simple pipeline which take in an mp4 source and streams it out via the RTMP sink. I have some general confusion on how to properly wire up the MP4.Demuxer.ISOM.

1. How do I decode/parse the output pads stream format downstream from the source? I haven't seen any examples or demos of transforming the MP4.Payload.{AAC.AVC1}

2. How to properly handle dynamically attaching the output pads for each track to downstream elements? If I want to handle the :new_track message to attach to some sink with pads in the :always availability mode (such as the RTMP sink) I can't attach that track to some grouping of elements which end at the sink temporarily. For example, if I get the :new_track notification for an AAC track I can't attach just the :audio pad of the RTMP sink, because when handling that callback there is no video pad to attach.

3. Is it better to handle the track determination statically? IE, should I ffprobe the mp4 file and parse the available tracks beforehand?

4. Does the demuxer only handle mp4 with embedded h264/aac? A container with h264/mp3 won't be able to be demuxed?

Thanks!

Hello everyone.

I'm pretty new to the media streaming world and still learning all the protocols, how things work, etc. So bear with me if my question seems a little bit basic.

I'm developing a LiveView system where one of the requisites is to create a continuous audio stream of the user's microphone via the browser he is using and send that data to my server. In the server, I want to receive that stream and pass it to a speech-to-text model so I have a transcription of what the user is doing in (soft) real-time.

I believe Membrane can help me here with the part of creating an audio stream from the user's browser microphone input and sending it to the server to consume it, but I'm a bit lost on what parts of membrane would help me with that, I believe I would not use webRTC for that since it is, AFAIK, a peer-to-peer protocol, and in this case, I just want a stream from client to server.

Can you guys have suggestions on how I should do that?

Hi i'am trying to get the rtp_to_hls working but I got a issue. The HLS player seems to looping old movie segments Is there a config I need to set here. I want to send a RTP stream to the web client with HLS

Not sure what the issue stems from. Opening a microphone source gives me:

** (Membrane.CallbackError) Error returned from Membrane.PortAudio.Source.handle_prepared_to_playing:
:pa_open_stream
Internal state: %{
  endpoint_id: :default,
  latency: :high,
  native: nil,
  portaudio_buffer_size: 256
}

    (membrane_core 0.10.2) lib/membrane/core/callback_handler.ex:209: Membrane.Core.CallbackHandler.parse_callback_result/3
    (membrane_core 0.10.2) lib/membrane/core/callback_handler.ex:65: Membrane.Core.CallbackHandler.exec_and_handle_callback/5
    (membrane_core 0.10.2) lib/membrane/core/element/lifecycle_controller.ex:143: Membrane.Core.Element.LifecycleController.handle_playback_state/3
    (membrane_core 0.10.2) lib/membrane/core/playback_handler.ex:115: Membrane.Core.PlaybackHandler.do_change_playback_state/2
    (membrane_core 0.10.2) lib/membrane/core/element.ex:170: Membrane.Core.Element.handle_info/2
    (stdlib 4.1.1) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.1.1) gen_server.erl:1200: :gen_server.handle_msg/6
    (stdlib 4.1.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: {Membrane.Core.Message, :change_playback_state, :playing, []}

Is there anyway to add UnfexPid into UnifexState? The state is defined in the header file, and trying to put UnifexPid into the state struct results in error: unknown type name 'UnifexPid'

Am I missing something here?

Hi I'm getting following error when try to join from iOS membrane webrtc sdk (4.0.0). Can anybody points me what I'm missing here. Kinda stuck on the issue for days. It'll be really helpful if someone can share any insight regarding the issue.

Hi, I'm trying to update to the latest release but I keep getting many dependency requires errors when I do mix deps.get My current dependencies are:

{:membrane_core, "~> 1.0.0-rc0", override: true},
{:membrane_http_adaptive_stream_plugin, "~> 0.8.1"},
{:membrane_mp4_plugin, "~> 0.16.0"},
{:membrane_h264_ffmpeg_plugin, "~> 0.21.6"},
{:membrane_aac_plugin, "~> 0.12.1"}

But when I run the command trying to update everything I receive this error:

Resolving Hex dependencies...

Failed to use "bunch" (version 1.6.0) because
  membrane_core (version 1.0.0-rc0) requires ~> 1.6
  membrane_h264_ffmpeg_plugin (version 0.21.6) requires ~> 1.3.0

** (Mix) Hex dependency resolution failed, change the version requirements of your dependencies or unlock them (by using mix deps.update or mix deps.unlock). If you are unable to resolve the conflicts you can try overriding with {:dependency, "~> 1.0", override: true}

And when I try to update the other dependencies below the membrane_core one, every other dep complains about a required version of a different dependency. I want to try the new features but not sure how to solve this issue.

I've been trying to get events of tracks added/removed with no luck. I copied the video conference example and been trying on that code to get notified of tracks changes for the reason of updating a state of peers with which peer has audio/video.

Thanks in advance.

Hi everyone, first of all I would like to appreciate membrane team who have put up so much effort to make the framework usable. I'm deploying membrane_videoroom on a private server and having streams transmission/receiving issues. I've deployed the master branch with the following commands

docker build -t videoroom . docker run -p 4443:4443/tcp -p 50000-50050:50000-50050/udp --env-file .env -e INTEGRATED_TURN_CERT_PKEY=/usr/local/cert.pem -v /priv/key/integrated_turn_cert_pkey.pem:/usr/local/cert.pem videoroom

Server seems to be up and serving requests, peers are also connecting without any issues. Only problem I'm facing is that after peers join room, no video/audio is transmitted/received

following is contents of my .env file:

EXTERNAL_IP=135.181.49.66 VIRTUAL_HOST=videoroom.invo.zone APP_PORT=4443

DATABASE=videoroom DB_USERNAME=postgres DB_PASSWORD=postgres DB_HOSTNAME=localhost DB_PORT=5432

INTEGRATED_TURN_CERT_PKEY=priv/key/integrated_turn_cert_pkey.pem

P.S. Same issue occurs on https://videoroom.membrane.stream/ but https://videoroom.membrane.work/ works fine

If I have an MKV container, what can I do to either use that as is in a pipeline for some operation, or how can I pull out the audio and video from the container in order to use it within a pipeline?

If it matters, I am just trying to test taking an MKV file and put it to an RTMP sink.

Hi 🙂 I'm working on a POC app and evaluating membrane for a project at work and I have some questions about how the timestamp offset for the video compositor plugin is supposed to work.

First the goal of the POC app is to take an arbitrary list of videos and time offsets and dynamically stitch them together into one continuous video (and then stream the result somewhere but I haven't gotten that far yet).

Some other members of my team chatted with some of the membrane contributors and they suggested we take a look at the video compositor plugin and provided us some skeleton code, however, I'm struggling to understand how the timestamp offset option that the compositor takes can be used to seek through the input.

No matter what input I give for the timestamp offset (except a negative number which raises as expected) the video starts playing from the first frame. Does the timestamp offset not do what I am expecting it to do?

My approach for the POC is to dynamically spawn (and remove previous children) at each cue point and attempt to provide the compositor timestamp offset to seek the new video to the correct offset. Is this the right approach?

Thanks! Sorry for the long post 🙂

Hi again! I have a bug for users in Safari 16, in the onTrack handler the event object passed in to the handler has no tranceiver member, so event.transceiver.mid returns undefined which is passed to midToTrackId which then throws an error in checkIfTrackBelongToPeer (trackId is undefined and startsWith explodes). In debugging a bit I can see that the event object DOES have a Receiver object but no tranceiver. Is this some difference in Safari WebRTC implementation? Do you have any suggestions on a fix?

When I run a local test with both sides running on chrome browser on ubuntu I get a successful connection, but when one end is a mobile browser I don't see any JS errors but I also dont see any incoming OPUS stream on the server for the person joining on mobile. I also now see this behavior when someone joins from a windows browser. Is there some way I can debug the membrane.js to see why theres no OPUS track sent? I profiled the js and I see addTranceiver being called

Hi, i was wondering whether its possible to use Membrane to record a live screencasting from browser and save it to a S3 bucket We can send data in WebRTC but im not sure whether membrane can accept the stream and store it.

There can be more than 1 connected users who stream to our server

I've been having trouble getting the membrane_videoroom app working locally. But only tested the demo instance now (https://videoroom.membrane.stream/) and realize it's not working either. There's no video/audio from other peers. Is this the most up to date repo to experiment with? Any idea why there's no video/audio from other peers? Thanks!

I’m getting this error

(RuntimeError DTLS handshake reached max retransmission number)

Hello! I am starting a project with membrane and so far I have succeeded in failing to play an mp3 in my Ubuntu Linux following the first tutorial of the webpage. The code compiles without fail (newbie victory), but it fails at playing (no problems there, my Ubuntu is not configured to play anything).

I am not too bothered about the mp3 not playing. I really want to play it through a website, not at my local computer. Now that I have a functioning mix phoenix project with the Membrane dependencies installed, which would my next step be to make an mp3 play on the site?

Thank you 🙂