Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Browser vs Pion and Ffmpeg stream #69

Open
silverark opened this issue Nov 13, 2020 · 7 comments
Open

Browser vs Pion and Ffmpeg stream #69

silverark opened this issue Nov 13, 2020 · 7 comments

Comments

@silverark
Copy link

Not usre where exactly yo ask this, but I'm looking for some advise on processing a stream. Firstly, thank you for Pion, it's awesome!

We have it in a project that connects in to a call and monitors the users by doing some facial recognition and adult content detection. Everything is working really well, apart from when a user has a bad wifi connection.

We started building from the janus-gateway example and to get the image so we can process it in gocv we use the example in gocv-receive.

This uses ffmpeg with an input and output pipe. An ivfWriter sends it to the input, and we read it from the 'output' as per the example. We then open it as an opencv mat.

The problem:
When viewing the stream in a browser, when it get's choppy due to a bad wifi signal the browser drops the frames rather than showing garbled data. With pion -> ffmpeg -> gocv we get garbled frames as shown in the image below.

I've tried various ffmpeg commands to see if we can detect these bad fames/packets: "ffmpeg", "-err_detect", "buffer", "-fflags", "discardcorrupt" but whatever I try, when we simulate a bad connection it destroys the image.

Is there a way of detecting when the image is bad? Is there a way of stopping or dropping these frames? I'm not 100% sure where the issue is so even pointing me in the right direction would be greatly appreciated!

What did you expect?

When the signal is bad, for bad frames to be dropped.

What happened?

I'm seeing garbled frames when the connection is bad, or just before it disconnects.

image

@iammeizu
Copy link

met the same problem, any suggestion?

@iammeizu
Copy link

iammeizu commented Nov 17, 2020

hi, @silverark I solved this problem. ivfwriter is not reliable, I recommand samplebuilder instead of ivfwriter because samplebuilder will sort rtp packet by sequence number. Check the expamle named "Save-to-webm", and use ffmpeg pipe as videowriter. Later, I will commit a new example, Good Luck!

@silverark
Copy link
Author

Thank you @iammeizu

I will give it a go tomorrow! I was also contacted by Sean DuBois toio who said he was workign on Interceptors for V3 too: pion/webrtc-v3-design#34

@silverark
Copy link
Author

@iammeizu - I've got this working with sampleBuilder but strangely found the image to be worse using this method! What value did you have for the maxLate? When testing with a simulted bad network, values of 10 - 20 almost didn't work at all and produced the image below. A value of 60 worked quite well but turned to snow sometimes. Values over 200 introduce quite a delay.

I'm wondering if it's my ffmpeg command?
"ffmpeg", "-i", "pipe:0", "-s", strconv.Itoa(width)+"x"+strconv.Itoa(height), "-pix_fmt", colourFormat, "-f", "rawvideo", "pipe:1"

I tried using the example from the twilio example, but outputting to pipe1 instead, but I found that the stream never started!

image

@iammeizu
Copy link

hi @silverark , my Maxlate is 50, and you can use send PLI by rtcp, which is used to control keyframe internal. I set 1 PLI per second, which means server get a complete frame per second. so the snow phenomenon can be relief by this way.

@Sean-Der
Copy link
Member

Sean-Der commented Dec 1, 2020

Sorry for the slow response on this! @silverark emailed me directly, copying my respond here

So the issue is that Pion doesn't provide error concealment by default. We are fixing that in V3 so you get it by default.

For now what I would suggest is sending a PLI every couple seconds. You can see an example of that here https://github.com/pion/webrtc/blob/master/examples/save-to-disk/main.go#L95-L104

This requests that the encoder sends a new frame, so even if you have packet loss it will recover much faster. Thanks for sharing all your comments/feedback on the GitHub issue so far.

I am also only Slack https://pion.ly/slack and happy to answer questions there as well. I hope you keep using Pion, and will try my best to make this all work :) 

@marlocarlo
Copy link

marlocarlo commented Aug 27, 2021

I am having the exact same issue with rtp-forwarder. But its happening always, not only when the signal is bad. To get this ready for production, i need to somehow solve this.

I would appreciate any guidance

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

4 participants