Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Haoquan/rtp video stream control #308

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

hq286
Copy link

@hq286 hq286 commented Nov 23, 2022

Built a new remote python controller with low latency RTP video stream for mobile network connection. New python controller supports autopilot, recording, without crashing remote video stream.

@izivkov
Copy link
Contributor

izivkov commented Nov 28, 2022

I am trying to run the new branch. One thing I noticed running the main app:

  • after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM
E/AndroidRuntime: FATAL EXCEPTION: main
Process: org.openbot, PID: 26454
java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener
at org.openbot.env.RtpServer.init(RtpServer.java:49)
at org.openbot.env.PhoneController.init(PhoneController.java:64)
at org.openbot.env.PhoneController.getInstance(PhoneController.java:33)
at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76)
at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44)
at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987)
at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546)
at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282)
at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189)
at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106)
at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002)
at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524)
at android.os.Handler.handleCallback(Handler.java:938)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:250)
at android.app.ActivityThread.main(ActivityThread.java:7755)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958)
V/FA: Recording user engagement, ms: 5450

@izivkov
Copy link
Contributor

izivkov commented Nov 28, 2022

Is the new python script called "joystick_controller.py"? I cannot test it since I do not have a joystick. Is it a particular joystick model that works with it? I am wondering if the RTP stream would still be useful with keyboard-only controller, since most people do not have joystick.

@thias15
Copy link
Collaborator

thias15 commented Nov 29, 2022

In my opinion, having also a keyboard controller would be useful.

}

public String[] getVideoServerAddress() {
String ip = get("ip", "172.217.22.14");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should have some neutral IP as default or just something like N/A if no server is setup.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to "127.0.0.1"

@@ -80,6 +80,9 @@ dependencies {
implementation 'com.github.pedroSG94:RTSP-Server:1.0.8'
implementation 'com.github.pedroSG94.rtmp-rtsp-stream-client-java:rtplibrary:2.0.2'

//RTP Client
implementation 'com.github.ar-android:libstreaming:1.0.0'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use the original library.
https://github.com/fyhertz/libstreaming

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried the original library in the beginning. However, the official libstreaming published on jitpack is the very old version, and they didn't publish their latest version. I'm not sure if there is another way around. Do you have any idea?


Controller mapping
```
▲: autodrive
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the BT joystick, autopilot is on R1, data logging on X. Let's be consistent. Triangle, square and circle are for the indicators.

@thias15
Copy link
Collaborator

thias15 commented Dec 3, 2022

@haoquan on what OS did you test this? On Mac I'm unable to install approxeng.input. Is this essential or could it be done differently?

error: legacy-install-failure

@thias15
Copy link
Collaborator

thias15 commented Dec 3, 2022

Also, what kind of latency are you getting with RTP for the video stream?

@thias15
Copy link
Collaborator

thias15 commented Dec 17, 2022

@haoquan, after further testing I also encountered frequent app crashes.

java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener

Did you not face these app crashes?

@hq286
Copy link
Author

hq286 commented Dec 20, 2022

I am trying to run the new branch. One thing I noticed running the main app:

  • after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM E/AndroidRuntime: FATAL EXCEPTION: main Process: org.openbot, PID: 26454 java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener at org.openbot.env.RtpServer.init(RtpServer.java:49) at org.openbot.env.PhoneController.init(PhoneController.java:64) at org.openbot.env.PhoneController.getInstance(PhoneController.java:33) at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76) at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44) at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987) at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546) at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282) at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189) at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106) at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002) at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524) at android.os.Handler.handleCallback(Handler.java:938) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:250) at android.app.ActivityThread.main(ActivityThread.java:7755) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958) V/FA: Recording user engagement, ms: 5450

I have fixed this issue. Check out the latest commit

@hq286
Copy link
Author

hq286 commented Dec 20, 2022

@haoquan, after further testing I also encountered frequent app crashes.

java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener

Did you not face these app crashes?

I fixed the issue. Check out the latest commit

@hq286
Copy link
Author

hq286 commented Dec 20, 2022

Is the new python script called "joystick_controller.py"? I cannot test it since I do not have a joystick. Is it a particular joystick model that works with it? I am wondering if the RTP stream would still be useful with keyboard-only controller, since most people do not have joystick.

Yes. It should work with major bluetooth joystick controllers like PS4, and xbox and etc.
I agree that a keyboard controller would be very convenient. The problem now is that I am using cv2 as the video displayer, which will cause conflict between keyboard control and cv2 video display (waitkey). It turns out that PyGame is introducing a lot of latency when displaying the video. I will think about how to resolve this issue.

@hq286
Copy link
Author

hq286 commented Dec 20, 2022

@haoquan on what OS did you test this? On Mac I'm unable to install approxeng.input. Is this essential or could it be done differently?

error: legacy-install-failure

I developed and tested everything on a Linux computer. Unfortunately approxeng.input only supports Linux system. I'm stilly working on finding a solution for Mac.

@hq286
Copy link
Author

hq286 commented Dec 20, 2022

Also, what kind of latency are you getting with RTP for the video stream?

I tested it on a Linux computer. The latency is about 150ms on average. It could vary depending on your network condition.

@izivkov
Copy link
Contributor

izivkov commented Dec 20, 2022

That is pretty good response time. I will check it out. Can I run the app and see the video without a Joystick?

@izivkov
Copy link
Contributor

izivkov commented Dec 20, 2022

I am trying to run the new branch. One thing I noticed running the main app:

  • after setting the Stream Mode to "RTP" in settings, and then selecting "Free Roam" fragment, the program crashes:

D/AndroidRuntime: Shutting down VM E/AndroidRuntime: FATAL EXCEPTION: main Process: org.openbot, PID: 26454 java.lang.ClassCastException: org.openbot.main.MainActivity cannot be cast to android.media.ImageReader$OnImageAvailableListener at org.openbot.env.RtpServer.init(RtpServer.java:49) at org.openbot.env.PhoneController.init(PhoneController.java:64) at org.openbot.env.PhoneController.getInstance(PhoneController.java:33) at org.openbot.common.ControlsFragment.onViewCreated(ControlsFragment.java:76) at org.openbot.robot.FreeRoamFragment.onViewCreated(FreeRoamFragment.java:44) at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:2987) at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:546) at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:282) at androidx.fragment.app.FragmentManager.executeOpsTogether(FragmentManager.java:2189) at androidx.fragment.app.FragmentManager.removeRedundantOperationsAndExecute(FragmentManager.java:2106) at androidx.fragment.app.FragmentManager.execPendingActions(FragmentManager.java:2002) at androidx.fragment.app.FragmentManager$5.run(FragmentManager.java:524) at android.os.Handler.handleCallback(Handler.java:938) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:250) at android.app.ActivityThread.main(ActivityThread.java:7755) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:958) V/FA: Recording user engagement, ms: 5450

I have fixed this issue. Check out the latest commit

OK, the code works with RTP selected, but it crashes if we select WebRTC, and then select Free Roam fragment. RTSP does not crash.

@hq286
Copy link
Author

hq286 commented Dec 20, 2022

Fixed it. There was a type with webrtc.

@hq286
Copy link
Author

hq286 commented Dec 20, 2022

That is pretty good response time. I will check it out. Can I run the app and see the video without a Joystick?

Yes. You can run keyboard-pygame.py without the video flag. Then run the camera.py. Make sure you are in the RTP mode, and match the ports for control and stream

@hq286
Copy link
Author

hq286 commented Dec 20, 2022

This works at least in Linux. I'm not sure about Mac. Getting GStreamer, opencv and python to work together on Mac needs some work

@thias15
Copy link
Collaborator

thias15 commented Dec 28, 2022

I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error.

Failed to open camera
Src opened, 0x0 @ 0 fps

@izivkov did you manage to get this running under Linux?

@izivkov
Copy link
Contributor

izivkov commented Dec 28, 2022 via email

@thias15
Copy link
Collaborator

thias15 commented Dec 31, 2022

@hq286 any idea what we are doing wrong?

@hq286
Copy link
Author

hq286 commented Jan 1, 2023

No, I got the same error. I never got video to work with the new Python script. I thought it was something wrong with my configuration.

On Wed., Dec. 28, 2022, 5:47 a.m. thias15, @.> wrote: I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error. Failed to open camera Src opened, 0x0 @ 0 fps @izivkov https://github.com/izivkov did you manage to get this running under Linux? — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37RZMKFEDBNHXTSFSO3WPQLDNANCNFSM6AAAAAASJLNUYQ . You are receiving this because you were mentioned.Message ID: @.>

It's the problem of GStreamer pipeline configuration. This configuration works in Ubuntu 20.04. I need to investigate the configuration on MacBook

@thias15
Copy link
Collaborator

thias15 commented Jan 3, 2023

No, I got the same error. I never got video to work with the new Python script. I thought it was something wrong with my configuration.

On Wed., Dec. 28, 2022, 5:47 a.m. thias15, @.> wrote: I compiled OpenCV with GStreamer support on Mac, but when I try to run the camera, I get the following error. Failed to open camera Src opened, 0x0 @ 0 fps @izivkov https://github.com/izivkov did you manage to get this running under Linux? — Reply to this email directly, view it on GitHub <#308 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37RZMKFEDBNHXTSFSO3WPQLDNANCNFSM6AAAAAASJLNUYQ . You are receiving this because you were mentioned.Message ID: @.>

How did you install OpenCV? Can you try the script I added to the repo? Make sure to install GStreamer before.

@izivkov
Copy link
Contributor

izivkov commented Jan 3, 2023

@thias15 What is the name of your "install OpenCV" script and what branch is it in?

@thias15
Copy link
Collaborator

thias15 commented Jan 4, 2023

I pushed it to this PR, it's called install_opencv.sh

@thias15
Copy link
Collaborator

thias15 commented Jan 11, 2023

@izivkov any update on your end?

@thias15
Copy link
Collaborator

thias15 commented Jan 11, 2023

By the way, I think we should also unify the port setting, at least for the controller across the different scripts.

@izivkov
Copy link
Contributor

izivkov commented Jan 11, 2023

@izivkov any update on your end?

I installed GStreamer and OpenCV using your installation script on my Linux PC but did not see video. Basically, the Video Window never appears. I used the following command:

python3 joystick_controller.py -video

Maybe it is my PC - only a Celaron CPU. I'm not sure if it needs a more powerful GPU.

Also, I'm not that familiar with GStreamer and OpenCV, so I may be doing some incorrectly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants