Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: add destructor to FFmpegWriter #429

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from

Conversation

chad3814
Copy link
Contributor

@chad3814 chad3814 commented Feb 6, 2020

This fixes up some memory leaks, specifically in the video and audio codec contexts

@chad3814
Copy link
Contributor Author

chad3814 commented Feb 7, 2020

well, not surprising that FFmpeg4 on Ubuntu 18.04 passed, as that's what I'm running...

I'm setting up some docker containers to try the others

@@ -102,6 +102,34 @@ FFmpegWriter::FFmpegWriter(std::string path) :
auto_detect_format();
}

FFmpegWriter::~FFmpegWriter() {
if (is_open) {
Close();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've only just taken a cursory look at this, but one thing I will say: If at all possible, it would be better to avoid calling Close() in the destructor. There's too much of that already in libopenshot, and we need to avoid it because our Close() methods are waaaay too heavy. They do a lot of housekeeping "stuff" in preparation for being reopened again, which of course makes no sense in a destructor.

See the discussion at #378, and in particular #378 (comment), for more on that.

It may mean having to reimplement some parts of Close() in the destructor, or separate them out into a new method (Finalize()?) that Close() also calls, but it would be good to avoid all of the unnecessary "stuff" in Close().

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. FFmpegWriter's Close() is actually pretty good, as libopenshot Close() methods go. It doesn't do too much rearranging of deck chairs on the Titanic, as the saying goes, and most of what it does is properly conditional.

It does unconditionally do this, though:

// Free the context which frees the streams too
avformat_free_context(oc);
oc = NULL;

...which may be a problem in a destructor, if there is no valid oc to free. May be as simple as slapping an if (oc) around it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also calls ZeroMQ as its very last action:

ZmqLogger::Instance()->AppendDebugMethod("FFmpegWriter::Close");

which DEFINITELY should not be done from a destructor. But since that's just a debug printf() with no useful information, really it can just be removed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I take it back. Close() itself is fairly lightweight, but the methods it calls are WAY too heavy. WriteTrailer(), close_video(), close_audio(), they all do a lot of processing that will bite us in a destructor.

If the video file hasn't been successfully written by the time the destructor is called, I don't think it's realistic for the destructor to try and complete the encoding process before it goes away. That's what an explicit Close() is for, and if the caller hasn't called it by the time the FFmpegWriter object is being destroyed, IMHO it should just close the output stream and drop everything on the floor, not try and write it all out first.

@SuslikV
Copy link
Contributor

SuslikV commented Feb 7, 2020

It's band-aid. There is nothing to fix. It should be rewritten completely. If program uses resources that it can't free in time, then something is wrong here.

@chad3814
Copy link
Contributor Author

chad3814 commented Feb 7, 2020

the ordering of freeing things changed in ffmpeg 2-4; last night I got the tests to pass on all of them, but I'll go through @ferdnyc's comments and move/copy some of the freeing to the destructor

@ferdnyc
Copy link
Contributor

ferdnyc commented Feb 7, 2020

I built ffmpeg-3.4.7 on my system in ilibopenshot/thirdparty with the following configure line:

./configure --prefix=/ --disable-libvmaf --enable-cuvid --disable-libfdk-aac --enable-nonfree --enable-libfreetype --enable-libdrm --enable-gnutls --enable-gcrypt --enable-fontconfig --enable-libmp3lame --enable-libgsm --enable-libopenjpeg --enable-librsvg --enable-libssh --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libzimg --enable-avfilter --enable-avresample --enable-postproc --enable-pthreads --disable-static --enable-shared --disable-debug --disable-stripping --arch=x86_64

Then I ran a make install DESTDIR=install-x64 after it built. Set up like that, libopenshot can be built with -DFFmpeg_ROOT=../thirdparty/ffmpeg-3.4.7/install-x64 added to the cmake command line to use those libs.

Editing FFmpegWriter_Tests.cpp to set the audio codec to aac, the video codec to mpeg4, and the output filename to output1.mp4, I'm able to reproduce the crash:

$ make test
[  1%] Automatic MOC for target openshot
[  1%] Built target openshot_autogen
[ 78%] Built target openshot
Scanning dependencies of target openshot-test
[ 79%] Building CXX object tests/CMakeFiles/openshot-test.dir/FFmpegWriter_Tests.cpp.o
[ 81%] Linking CXX executable openshot-test
/usr/bin/ld: warning: libavutil.so.56, needed by /usr/lib64/libpostproc.so, may conflict with libavutil.so.55
[100%] Built target openshot-test
----------------------------
     RUNNING ALL TESTS
----------------------------
[mp4 @ 0x1de01a0] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
[aac @ 0x1b98f40] Qavg: 192.111
[aac @ 0x1b98f40] 1 frames left in the queue on closing
double free or corruption (!prev)
make[3]: *** [tests/CMakeFiles/test.dir/build.make:57: tests/CMakeFiles/test] Aborted (core dumped)
make[2]: *** [CMakeFiles/Makefile2:687: tests/CMakeFiles/test.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:694: tests/CMakeFiles/test.dir/rule] Error 2
make: *** [Makefile:420: test] Error 2

Some of that output (especially the "double free..." line) may also be helpful. I'll try and look into it as well, see if I can find anything.

@ferdnyc
Copy link
Contributor

ferdnyc commented Feb 7, 2020

Here's the output of a backtrace in gdb, after running openshot-test through to the crash:

(gdb) bt
#0  0x00007ffff3ef0625 in raise () at /lib64/libc.so.6
#1  0x00007ffff3ed98d9 in abort () at /lib64/libc.so.6
#2  0x00007ffff3f344af in __libc_message () at /lib64/libc.so.6
#3  0x00007ffff3f3ba9c in annobin_top_check.start () at /lib64/libc.so.6
#4  0x00007ffff3f3d92c in _int_free () at /lib64/libc.so.6
#5  0x00007ffff7f045bc in openshot::FFmpegWriter::~FFmpegWriter() ()
    at /home/ferd/rpmbuild/REPOS/libopenshot/build_pr429/src/libopenshot.so.18
#6  0x00007ffff7f046d8 in openshot::FFmpegWriter::~FFmpegWriter() ()
    at /home/ferd/rpmbuild/REPOS/libopenshot/build_pr429/src/libopenshot.so.18
#7  0x0000000000436c3b in TestFFmpegWriter_Test_Webm::RunImpl() const ()
#8  0x00007ffff7da6b56 in UnitTest::ExecuteTest<UnitTest::Test>(UnitTest::Test&, UnitTest::TestDetails const&, bool) (testObject=warning: RTTI symbol not found for class 'TestFFmpegWriter_Test_Webm'

..., details=..., isMockTest=<optimized out>) at UnitTest++/ExecuteTest.h:24
#9  0x00007ffff7da6a91 in UnitTest::Test::Run()
    (this=this@entry=0x495f80 <testFFmpegWriter_Test_WebmInstance>)
    at UnitTest++/Test.cpp:32
#10 0x00007ffff7da7397 in UnitTest::TestRunner::RunTest(UnitTest::TestResults*, UnitTest::Test*, int) const
    (this=<optimized out>, result=0x4be070, curTest=0x495f80 <testFFmpegWriter_Test_WebmInstance>, maxTestTimeInMs=0) at UnitTest++/TestRunner.cpp:67
#11 0x00007ffff7da778c in UnitTest::TestRunner::RunTestsIf<UnitTest::True>(UnitTest::TestList const&, char const*, UnitTest::True const&, int) const
    (suiteName=0x0, maxTestTimeInMs=0, predicate=..., list=..., this=0x7fffffffd--Type <RET> for more, q to quit, c to continue without paging--
440) at UnitTest++/TestRunner.h:36
#12 UnitTest::RunAllTests() () at UnitTest++/TestRunner.cpp:17
#13 0x0000000000412e2b in main ()

@ferdnyc
Copy link
Contributor

ferdnyc commented Feb 7, 2020

Actually, more informative might be this, from the output of the run instruction in gdb:

[Thread 0x7fffe77fe700 (LWP 3369901) exited]
[Thread 0x7fffe7fff700 (LWP 3369902) exited]
[Thread 0x7fffe6ffd700 (LWP 3369900) exited]
[Thread 0x7fffecf04700 (LWP 3369903) exited]
[New Thread 0x7fffecf04700 (LWP 3369904)]
[New Thread 0x7fffe7fff700 (LWP 3369905)]
[New Thread 0x7fffe77fe700 (LWP 3369906)]
[New Thread 0x7fffe6ffd700 (LWP 3369907)]
[New Thread 0x7fffe4ff9700 (LWP 3369908)]
[New Thread 0x7fffc7fff700 (LWP 3369909)]
[New Thread 0x7fffc77fe700 (LWP 3369910)]
[mp4 @ 0x7fa740] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
[Thread 0x7fffecf04700 (LWP 3369904) exited]
[Thread 0x7fffe7fff700 (LWP 3369905) exited]
[Thread 0x7fffe77fe700 (LWP 3369906) exited]
[Thread 0x7fffe6ffd700 (LWP 3369907) exited]
[New Thread 0x7fffe6ffd700 (LWP 3369911)]
[New Thread 0x7fffe77fe700 (LWP 3369912)]
[New Thread 0x7fffe7fff700 (LWP 3369913)]
[New Thread 0x7fffecf04700 (LWP 3369914)]
[aac @ 0x5a32a0] Qavg: 192.111
[aac @ 0x5a32a0] 1 frames left in the queue on closing
[Thread 0x7fffc77fe700 (LWP 3369910) exited]
[Thread 0x7fffc7fff700 (LWP 3369909) exited]
[Thread 0x7fffe4ff9700 (LWP 3369908) exited]
[Thread 0x7fffe6ffd700 (LWP 3369911) exited]
[Thread 0x7fffe77fe700 (LWP 3369912) exited]
[Thread 0x7fffe7fff700 (LWP 3369913) exited]
[Thread 0x7fffecf04700 (LWP 3369914) exited]
double free or corruption (!prev)

Thread 1 "openshot-test" received signal SIGABRT, Aborted.
0x00007ffff3ef0625 in raise () from /lib64/libc.so.6

Presumably, if multiple threads are executing at once, then multiple threads are executing the destructor. Which is an excellent reason to avoid calling Close() from within it (not that I think it's the issue here), and it may also be necessary to protect some of the rest of the code with #pragma omp critical or CriticalSection mutexes, if shared memory is being freed (which I do think is the issue here).

@SuslikV
Copy link
Contributor

SuslikV commented Feb 7, 2020

Mark this PR as work-in-progress, because it has unresolved issues (crash in tests, for example).

@ferdnyc ferdnyc changed the title add destructor to FFmpegWriter WIP: add destructor to FFmpegWriter Feb 7, 2020
@ferdnyc
Copy link
Contributor

ferdnyc commented Feb 7, 2020

Fair point, though a failing PR can't be merged anyway. Still... done.

@codecov-io
Copy link

codecov-io commented Feb 7, 2020

Codecov Report

Merging #429 into develop will decrease coverage by 7.02%.
The diff coverage is 91.66%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #429      +/-   ##
===========================================
- Coverage    49.45%   42.43%   -7.03%     
===========================================
  Files          129      129              
  Lines        10261    13292    +3031     
===========================================
+ Hits          5075     5640     +565     
- Misses        5186     7652    +2466     
Impacted Files Coverage Δ
include/FFmpegWriter.h 0.00% <ø> (ø)
src/FFmpegWriter.cpp 61.76% <91.66%> (-0.95%) ⬇️
tests/ReaderBase_Tests.cpp 52.38% <0.00%> (-47.62%) ⬇️
src/DummyReader.cpp 49.18% <0.00%> (-27.75%) ⬇️
src/QtImageReader.cpp 52.63% <0.00%> (-13.11%) ⬇️
src/ImageReader.cpp 52.11% <0.00%> (-11.83%) ⬇️
tests/Clip_Tests.cpp 89.56% <0.00%> (-10.44%) ⬇️
src/effects/Negate.cpp 25.00% <0.00%> (-6.25%) ⬇️
src/Color.cpp 38.98% <0.00%> (-6.12%) ⬇️
src/Point.cpp 42.42% <0.00%> (-5.86%) ⬇️
... and 150 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07a447c...d26aafb. Read the comment docs.

@chad3814
Copy link
Contributor Author

chad3814 commented Feb 7, 2020

Should still be a WIP, I didn't address the leaks in ffmpeg3

@ferdnyc
Copy link
Contributor

ferdnyc commented Feb 8, 2020

It's coming along nicely, though! As a sort of coarse metric, here are the results of valgrind runs on this code vs. the develop branch, both built against my system FFmpeg 4.2.2:

develop branch

$ valgrind --track-origins=yes --leak-check=full tests/openshot-test
[...]
==3479570== LEAK SUMMARY:
==3479570==    definitely lost: 80,790 bytes in 149 blocks
==3479570==    indirectly lost: 30,008 bytes in 13 blocks
==3479570==      possibly lost: 42,606,904 bytes in 1,077 blocks
==3479570==    still reachable: 609,908 bytes in 2,922 blocks
==3479570==                       of which reachable via heuristic:
==3479570==                         newarray           : 1,536 bytes in 16 blocks
==3479570==         suppressed: 0 bytes in 0 blocks
==3479570== Reachable blocks (those to which a pointer was found) are not shown.
==3479570== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3479570== 
==3479570== For lists of detected and suppressed errors, rerun with: -s
==3479570== ERROR SUMMARY: 180 errors from 176 contexts (suppressed: 0 from 0)

$ # I edited Example.cpp and set `s->HARDWARE_DECODER = 0` before building
$ valgrind --track-origins=yes --leak-check=full src/openshot-example
[...]
==3476791== LEAK SUMMARY:
==3476791==    definitely lost: 3,750 bytes in 8 blocks
==3476791==    indirectly lost: 424,020 bytes in 27 blocks
==3476791==      possibly lost: 499,522 bytes in 73 blocks
==3476791==    still reachable: 78,934,137 bytes in 422 blocks
==3476791==                       of which reachable via heuristic:
==3476791==                         newarray           : 1,536 bytes in 16 blocks
==3476791==         suppressed: 0 bytes in 0 blocks
==3476791== Reachable blocks (those to which a pointer was found) are not shown.
==3476791== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3476791== 
==3476791== For lists of detected and suppressed errors, rerun with: -s
==3476791== ERROR SUMMARY: 29 errors from 29 contexts (suppressed: 0 from 0)

FFmpegWriter-memory branch

$ valgrind --track-origins=yes --leak-check=full tests/openshot-test
[...]
==3475086== LEAK SUMMARY:
==3475086==    definitely lost: 79,734 bytes in 148 blocks
==3475086==    indirectly lost: 0 bytes in 0 blocks
==3475086==      possibly lost: 4,392 bytes in 23 blocks
==3475086==    still reachable: 609,905 bytes in 2,922 blocks
==3475086==                       of which reachable via heuristic:
==3475086==                         newarray           : 1,536 bytes in 16 blocks
==3475086==         suppressed: 0 bytes in 0 blocks
==3475086== Reachable blocks (those to which a pointer was found) are not shown.
==3475086== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3475086== 
==3475086== For lists of detected and suppressed errors, rerun with: -s
==3475086== ERROR SUMMARY: 60 errors from 56 contexts (suppressed: 0 from 0)

$ # I edited Example.cpp and set `s->HARDWARE_DECODER = 0` before building
$ valgrind --track-origins=yes --leak-check=full src/openshot-example
[...]
==3478225== LEAK SUMMARY:
==3478225==    definitely lost: 2,694 bytes in 7 blocks
==3478225==    indirectly lost: 0 bytes in 0 blocks
==3478225==      possibly lost: 4,392 bytes in 23 blocks
==3478225==    still reachable: 77,693 bytes in 274 blocks
==3478225==                       of which reachable via heuristic:
==3478225==                         newarray           : 1,536 bytes in 16 blocks
==3478225==         suppressed: 0 bytes in 0 blocks
==3478225== Reachable blocks (those to which a pointer was found) are not shown.
==3478225== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3478225== 
==3478225== For lists of detected and suppressed errors, rerun with: -s
==3478225== ERROR SUMMARY: 24 errors from 24 contexts (suppressed: 0 from 0)

So, major improvements on all fronts, with FFmpeg 4. I'll post full output once I've gotten all of the missing debug symbols installed, so the tracebacks are readable. (And I'll try to get FFmpeg 3 builds together, as well. The valgrind runs take forever, tho, so it may take a sec.)

@ferdnyc
Copy link
Contributor

ferdnyc commented Feb 8, 2020

(I can say that the remaining "definitely lost" in tests/openshot-test are mostly in FFmpegReader, plus a few in ZmqLogger... though FFmpegWriter::write_audio_packets() and FFmpegWriter::auto_detect_format() do seem to still be leaking a bit.)

@ferdnyc
Copy link
Contributor

ferdnyc commented Feb 8, 2020

BTW, not to contradict your latest commit message, but it seems Close() doesn't call free_resources(), currently. (And in fact, since the only thing the destructor does is call free_resources(), as things stand free_resources() could just be the destructor.)

Though, before going down that road, we should probably add a Close()-and-reopen unit test, if that's something that's meant to work correctly.

Especially since the fix for OpenShot/openshot-qt#2881 will likely introduce a Writer Close()-and-reopen cycle into OpenShot's Encode process.

@ferdnyc
Copy link
Contributor

ferdnyc commented Feb 8, 2020

OK, attached are the complete output of the following two commands, run in my worktree for this PR and built against FFmpeg 4.2.2:

valgrind -s --track-origins=yes --leak-check=full --show-leak-kinds=all tests/openshot-test \
 |& tee valgrind_openshot-test.txt
valgrind -s --track-origins=yes --leak-check=full --show-leak-kinds=all src/openshot-example \
 |& tee valgrind_openshot-example.txt

( |& is a zshism equivalent to 2>&1 |, IOW piping both stdout and stderr together.)

valgrind_openshot-test.txt
valgrind_openshot-example.txt

FFmpegWriter barely appears in either file, though you'll have to search the output files to see which of these are the top of their respective stacks and which are farther down:

$ grep FFmpegWriter valgrind_openshot-test.txt
==71597==    by 0x494A68A: openshot::FFmpegWriter::write_audio_packets(bool) [clone ._omp_fn.0] (FFmpegWriter.cpp:1534)
==71597==    by 0x4948BC8: openshot::FFmpegWriter::write_audio_packets(bool) (FFmpegWriter.cpp:1522)
==71597==    by 0x49427A3: openshot::FFmpegWriter::WriteTrailer() (FFmpegWriter.cpp:768)
==71597==    by 0x4944838: openshot::FFmpegWriter::Close() (FFmpegWriter.cpp:1065)
==71597==    by 0x436C63: TestFFmpegWriter_Test_Webm::RunImpl() const (FFmpegWriter_Tests.cpp:61)
==71597==    by 0x493D846: openshot::FFmpegWriter::auto_detect_format() (FFmpegWriter.cpp:139)
==71597==    by 0x493D3FA: openshot::FFmpegWriter::FFmpegWriter(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (FFmpegWriter.cpp:102)
==71597==    by 0x436AC7: TestFFmpegWriter_Test_Webm::RunImpl() const (FFmpegWriter_Tests.cpp:48)
==71597==    by 0x49425C2: openshot::FFmpegWriter::WriteFrame(openshot::ReaderBase*, long, long) (FFmpegWriter.cpp:754)
==71597==    by 0x436C54: TestFFmpegWriter_Test_Webm::RunImpl() const (FFmpegWriter_Tests.cpp:58)
==71597==    by 0x49425C2: openshot::FFmpegWriter::WriteFrame(openshot::ReaderBase*, long, long) (FFmpegWriter.cpp:754)
==71597==    by 0x436C54: TestFFmpegWriter_Test_Webm::RunImpl() const (FFmpegWriter_Tests.cpp:58)

$ grep FFmpegWriter valgrind_openshot-example.txt
==94857==    by 0x494A68A: openshot::FFmpegWriter::write_audio_packets(bool) [clone ._omp_fn.0] (FFmpegWriter.cpp:1534)
==94857==    by 0x4948BC8: openshot::FFmpegWriter::write_audio_packets(bool) (FFmpegWriter.cpp:1522)
==94857==    by 0x49427A3: openshot::FFmpegWriter::WriteTrailer() (FFmpegWriter.cpp:768)
==94857==    by 0x4944838: openshot::FFmpegWriter::Close() (FFmpegWriter.cpp:1065)
==94857==    by 0x493D846: openshot::FFmpegWriter::auto_detect_format() (FFmpegWriter.cpp:139)
==94857==    by 0x493D3FA: openshot::FFmpegWriter::FFmpegWriter(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (FFmpegWriter.cpp:102)

@chad3814
Copy link
Contributor Author

I've been using address sanitizer (libasan) for leak detection, but it misses some libav stuff. Maybe I should get valgrind working.

This came up because at athenascope, we use libopenshot to make some clips, and if we make more than X clips in a single process it gets OOM killed ;) So this is my current top priority, I think @ckirmse and I should be able to get it down.

@jonoomph
Copy link
Member

Is this PR still WIP or are we getting close to a final version? Looking good!

@ferdnyc
Copy link
Contributor

ferdnyc commented Mar 3, 2020

@jonoomph @chad3814

Mmm, if nothing else I should probably write that Close()-and-reopen unit test I keep threatening, make sure things still work as expected there.

@ferdnyc
Copy link
Contributor

ferdnyc commented Mar 5, 2020

^ For the record, I got most of the way through writing said test, then got derailed coming up with CheckPixel tests for the resulting video file. But it doesn't really need CheckPixel tests, it's the metadata that's important. I'll just wrap that up now, so I can be done with it.

@ferdnyc
Copy link
Contributor

ferdnyc commented Mar 6, 2020

My Close_and_reopen unit test is crashing if I run it against the code in this PR, but it crashes if I run it against the code in the develop branch, too. So it appears FFmpegWriter::Close() on an opened writer, followed by a subsequent FFmpegWriter::Open(), is simply not currently supported.

I wanted to also write a unit test for the new destructor, but unit-testing destructors is a deceptively tricky problem that I'm not feeling up to solving tonight.

Copy link
Contributor

@ferdnyc ferdnyc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted in the PR conversation, I'm down with this. Any issues encountered/raised during testing of the PR code have turned out to be pre-existing, and therefore tangential to this change.

@jonoomph
Copy link
Member

@ferdnyc This one has been here a while. What are your thoughts on this one? I don't want to introduce a crash into the C++ side at this point, and this looks a bit scary without testing it myself.

@github-actions github-actions bot added the conflicts A PR with unresolved merge conflicts label Apr 9, 2021
@github-actions
Copy link

github-actions bot commented Apr 9, 2021

Merge conflicts have been detected on this PR, please resolve.

@ferdnyc ferdnyc added the ffmpeg Issues or PRs involving the a/v processing code label Jul 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
conflicts A PR with unresolved merge conflicts ffmpeg Issues or PRs involving the a/v processing code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants