Support for streams with fragmented MP4 segments

[samrat]

Hello,

Thanks for this plugin!

It looks like the plugin currently only supports MPEG-TS segments.

Would it be possible to add support for streams with fMP4 segments?

[dmorn]

Hi @samrat! I don't think there is anything that could stop you from doing it. This plugin does not decode nor parse the renditions you select, it just forwards the contents of the files.

Using the test as an example, when you receive the master playlist from the source notification just select the fMP4 rendition.

The Source then delivers the stream format message, which is indeed going to be incorrect (it will report MPEGTS with the fMP4 codecs), but you can easily put a filter after that and take care of correcting the format before handing the bytestream over to a decoder.

If you don't find a Membrane decoder for your format (check membrane's dash plugin I think that one uses this format), you can "easily" craft one using Exile and ffmpeg, something like this

defmodule Decoder do
  use Membrane.Filter

  defmodule FFmpegError do
    defexception [:message]

    @impl true
    def exception(value) do
      %FFmpegError{message: "FFmpeg error: #{inspect(value)}"}
    end
  end

  def_input_pad(:input,
    accepted_format: Membrane.RemoteStream,
    availability: :always,
    flow_control: :auto
  )

  def_output_pad(:output,
    accepted_format: Membrane.RawAudio,
    availability: :always,
    flow_control: :auto
  )

  @impl true
  def handle_init(_ctx, _opts) do
    {:ok, p} =
      Exile.Process.start_link(
        ~w(ffmpeg -hide_banner -loglevel error -i - -ac 1 -ar 48000 -f s16le -)
      )

    parent = self()

    read_loop_task =
      Task.Supervisor.async(Speech.Core.Task.Supervisor, fn ->
        :ok = Exile.Process.change_pipe_owner(p, :stdout, self())
        read_loop(p, parent)
      end)

    {[], %{ffmpeg: p, read_loop_task: read_loop_task}}
  end

  @impl true
  def handle_stream_format(:input, _input_format, _ctx, state) do
    {[
       stream_format:
         {:output,
          %Membrane.RawAudio{
            channels: 1,
            sample_format: :s16le,
            sample_rate: 48000
          }}
     ], state}
  end

  @impl true
  def handle_buffer(:input, buffer, _ctx, state) do
    :ok = Exile.Process.write(state.ffmpeg, buffer.payload)
    {[], state}
  end

  @impl true
  def handle_end_of_stream(:input, _ctx, state) do
    Exile.Process.close_stdin(state.ffmpeg)
    {[], state}
  end

  @impl true
  def handle_info({:exile, {:data, payload}}, _ctx, state) do
    {[buffer: {:output, %Membrane.Buffer{payload: payload}}], state}
  end

  def handle_info({ref, :eof}, _ctx, state = %{read_loop_task: %Task{ref: ref}}) do
    # Avoid receiving the DOWN message.
    Process.demonitor(ref, [:flush])
    {:ok, 0} = Exile.Process.await_exit(state.ffmpeg)
    {[end_of_stream: :output], state}
  end

  def handle_info({ref, {:error, any}}, _ctx, %{read_loop_task: %Task{ref: ref}}) do
    raise FFmpegError, any
  end

  defp read_loop(p, parent) do
    case Exile.Process.read(p) do
      {:ok, data} ->
        send(parent, {:exile, {:data, data}})
        read_loop(p, parent)

      :eof ->
        :eof

      {:error, any} ->
        {:error, any}
    end
  end
end

This one will turn an audio stream into raw audio, depending on your use case change the ffmpeg command. Afterwards, you'll for sure find a Membrane parser that can produce well-behaving buffers for your pipeline.

Let me know if you manage to do it!

[samrat]

Thanks @dmorn .

Any ideas on how I could distinguish between when fmp4 segments are sent vs. when MPEG segments are being sent?

Looks like the only way is to actually look at whether segments have a EXT-X-MAP, but the format is determined here when we haven't seen any segments yet.

[dmorn]

Hi @samrat! I think you should be able to distinguish the two using some of the fields of the variant/alternative renditions? Like the codecs?

[samrat]

I think the codecs won't necessarily differ with fMP4 streams-- it can be the same codecs but stored in different container formats. For example, this is from an fMP4 playlist:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:BANDWIDTH=31491438,AVERAGE-BANDWIDTH=21880235,RESOLUTION=3840x2160,CODECS="avc1.640033,mp4a.40.2"
video_track.m3u8

And this is from a MPEG-TS playlist:

❯ cat master.m3u8
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=5711200,RESOLUTION=3840x2160,CODECS="avc1.640033,mp4a.40.2"
stream_0/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=3511200,RESOLUTION=1280x720,CODECS="avc1.64001f,mp4a.40.2"
stream_1/playlist.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=1311200,RESOLUTION=640x360,CODECS="avc1.64001e,mp4a.40.2"
stream_2/playlist.m3u8

[samrat]

I was able to pass along fmp4 segments in a very hacky way by converting the segments to mpeg-ts in membrane_hls_plugin: samrat@4e8c8c6

The only check I was able to use to distinguish fmp4 streams from MPEG-TS was the presence of the EXT-X-MAP tag, but I'm not sure if that's the right thing to do.

[dmorn]

There must be something in the RFCs about it 🤔