I am trying to mix multiple audio udp rtp packets which created with following command on some other computers, but after a lot of searching I could not find some proper command to mix received audios.
I use this command to stream audio on other computers to my computer:
gst-launch-1.0 autoaudiosrc ! audioconvert ! rtpL24pay ! udpsink host=<MY_COMPUTER_IP> port=<some_port_number>
and I can receive the stream on my computer with this command:
gst-launch-1.0 -v udpsrc port=<port_number> caps="application/x-rtp,channels=(int)2,format=(string)S16LE,media=(string)audio,payload=(int)96,clock-rate=(int)44100,encoding-name=(string)L24" ! rtpL24depay ! audioconvert ! autoaudiosink sync=false
but I want to mix received streams together and play them as one audio in only one pipeline, how can I do that?
To mix two audio streams you can use GStreamer's audiomixer plugin. Very basic example would be:
Generator of 2 parallel RTP (over UDP) streams with test audios in different frequencies
gst-launch-1.0 audiotestsrc freq=523 ! audioconvert ! rtpL24pay ! udpsink host=127.0.0.1 port=5000 \
audiotestsrc freq=659 ! audioconvert ! rtpL24pay ! udpsink host=127.0.0.1 port=5001
Receiver of 2 different RTP (over UDP) streams that mixes 2 audios carried by streams
gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp,channels=(int)2,format=(string)S16LE,media=(string)audio,payload=(int)96,clock-rate=(int)44100,encoding-name=(string)L24" \
! queue ! rtpL24depay ! audioconvert ! audiomixer name=mixer ! autoaudiosink \
udpsrc port=5001 caps="application/x-rtp,channels=(int)2,format=(string)S16LE,media=(string)audio,payload=(int)96,clock-rate=(int)44100,encoding-name=(string)L24" \
! queue ! rtpL24depay ! audioconvert ! mixer.
Related
I am trying to record RTP stream with video and audio to HLS. Using GStreamer and nodejs. I get WARNING: erroneous pipeline: no element "fdkaacenc". I am using macOS with M1 CPU and all plugins base, good, bad, and ugly are installed.
The whole command I am trying to execute is this
GST_DEBUG=3 gst-launch-1.0 -v -e
rtpbin name=rtpbin latency=50 buffer-mode=0 sdes="application/x-rtp-source-sdes, cname=(string)3xtbiiMIcbTDI1Td" !
udpsrc port=40228 caps="application/x-rtp,media=(string)video,clock-rate=(int)90000,payload=(int)105,encoding-name=(string)H264,ssrc=(uint)982701080"
! rtpbin.recv_rtp_sink_0 rtpbin. ! queue ! rtpvp8depay ! vp8dec ! x264enc ! mux.
udpsrc port=40401 caps="application/x-rtp,media=(string)audio,clock-rate=(int)48000,payload=(int)100,encoding-name=(string)OPUS,ssrc=(uint)906623489" !
# the part I am trying to convert opus to aac
rtpbin.recv_rtp_sink_1 rtpbin. ! queue ! rtpopusdepay ! opusdec ! fdkaacenc ! mux.
mpegtsmux name=mux ! hlssink location=storage/recordings/take-3540_question-1996/webcam-029ad85b-7a43-45a5-888f-f5ce873b108f%06d.ts playlist-location=storage/recordings/take-3540_question-1996/webcam-029ad85b-7a43-45a5-888f-f5ce873b108f.m3u8 target-duration=4
udpsrc address=127.0.0.1 port=40463 ! rtpbin.recv_rtcp_sink_0 rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=23257 bind-address=127.0.0.1 bind-port=40463 sync=false async=false udpsrc address=127.0.0.1 port=40499 ! rtpbin.recv_rtcp_sink_1 rtpbin.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=42134 bind-address=127.0.0.1 bind-port=40499 sync=false async=false
any ideas?
I fixed that by avenc_aac instead of fdkaacenc. Not to mention that you should use audioconvert before avenc_aac.
The part of command I was trying to convert to aac:
rtpopusdepay ! opusdec ! audioconvert ! avenc_aac.
More: avenc_aac is part of gstreamer ffmpeg plugins which exists in linux, mac and windows. fdkaacenc is part of gstreamer bad plugins that may not be the same in all systems. That's is why using avenc_aac is more approperiate. So it can be executed in both mac and linux.
I am trying to generate MP4s from HLS streams with discontinuity tags. Since the videos are from the same source the FPS and the WXH are the same.
I tested with the following pipeline to demux and play it and it works fine
gst-launch-1.0 -v souphttpsrc location=<HLS_URL> ! hlsdemux ! decodebin name=decoder \
! queue ! autovideosink decoder. ! queue ! autoaudiosink
To this I added the x264 enc and avenc_aac encoder to save it to a file and it keeps failing on
"gstadaptivedemux.c(2651): _src_chain (): /GstPipeline:pipeline0/GstHLSDemux:hlsdemux0"
Failing Pipeline
gst-launch-1.0 -v mp4mux name=mux faststart=true presentation-time=true ! filesink location=dipoza.mp4 \
souphttpsrc location=<HLS_URL> ! hlsdemux ! decodebin name=decoder ! queue name=q1 ! \
videoconvert ! queue name=q2 ! x264enc name=encoder ! mux. decoder. \
! queue name=q3 ! audioconvert ! queue name=q4 ! avenc_aac ! mux.
I really appreciate any help in this.
After a lot of debugging, I found the issue with my pipeline. Thanks a lot to #FlorianZwoch for asking me to move to voaacenc encoder.
voaacenc is not installed by default in gst-plugins-bad for mac. I so I had to use
brew reinstall gst-plugins-bad --with-libvo-aacenc
The following pipeline worked well with my application.
gst-launch-1.0 --gst-debug=3 mp4mux name=mux ! \
filesink location=xxxx.mp4 souphttpsrc location=<hls url> ! decodebin name=decode ! \
videoconvert ! videorate ! video/x-raw, framerate=50/1 ! queue ! x264enc ! mux. decode. ! \
audioconvert ! voaacenc ! mux.
Also in my HLS stream video segments some had 50FPS and some had 59.97FPS. So I used a videorate to default to 50. This might need to change depending on your segments
For those folks who want a C++ code of the same, please checkout my github page
I have a SAA7134 TV card. I want to record a video with sound using Gstreamer. This command I use to make sure I can hear the audio and it works
gst-launch-1.0 alsasrc device="hw:1,0" ! queue ! audioconvert ! alsasink
This command proves that I can watch the video (also works fine)
gst-launch-1.0 v4l2src device=/dev/video0 ! xvimagesink
This command works fine and allows me to write the sound to a file
gst-launch-1.0 alsasrc device="hw:1,0" ! queue ! audioconvert ! wavenc ! filesink location=/home/out/testout.wav
But this command only writes the video without any sound
gst-launch-1.0 v4l2src device=/dev/video0 ! queue ! videoconvert ! jpegenc ! mux. alsasrc device="hw:1,0" ! queue ! audioconvert ! lamemp3enc bitrate=192 ! mux. avimux name=mux ! filesink location=/home/out/testout.avi
the same for
gst-launch-1.0 v4l2src device=/dev/video0 ! queue ! videoconvert ! theoraenc ! mux. alsasrc device="hw:1,0" ! queue ! audioconvert ! vorbisenc ! mux. oggmux name=mux ! filesink location=/home/out/testout.ogg
How to solve the problem? Thank you.
P.S. I use Ubuntu 16.04.3 LTS.
It looks like I missed one important detail about using gst-launch syntax. I took a better look and found this:
The -e option forces EOS on sources before shutting the pipeline down. This is useful when we write to files and want to shut down by killing gst-launch using CTRL+C or with the kill command
When I tested this option I finally got both the video and audio.
I am successfully streaming a file (audio/video) through UDP on Windows and watching it on another machine with VLC (this was covered on Stackoverflow before):
gst-launch-1.0 -v filesrc location=video.mkv ! decodebin name=dec ! videoconvert ! x264enc ! video/x-h264 ! mpegtsmux name=mux ! queue ! udpsink host=127.0.0.1 port=5000 sync=true dec. ! queue ! audioconvert ! voaacenc ! audio/mpeg ! queue ! mux.
You can test this on VLC: Media > Open Network Stream > Network URL > udp://#:5000
However, while the video is being streamed I would like to also display it on a window, so I could watch the stream myself (no audio needed).
To accomplish this, I started with a series of small experiments so I could change the original pipeline without any surprises. If you are reading this question you know my plan didn't work so well.
My first experiment was to display just the video on single window:
gst-launch-1.0 -v filesrc location=video.mkv ! decodebin ! autovideosink
Then, I changed it to display the same video on 2 windows, to make sure I understood how to work with multithreading:
gst-launch-1.0 -v filesrc location=video.mkv ! decodebin name=dec ! queue ! tee name=t t. ! queue ! videoconvert ! autovideosink t. ! autovideosink
Finally, it came the moment to blend those 2 parts together and stream the video through the network while it is displayed locally. The result is not what I expected, of course: only the first frame appears to be streamed and then everything freezes:
gst-launch-1.0 -v filesrc location=video.mkv ! decodebin name=dec ! tee name=t ! queue ! autovideosink t. ! queue ! videoconvert ! x264enc ! video/x-h264 ! mpegtsmux name=mux ! queue ! udpsink host=127.0.0.1 port=5000 sync=true dec. ! queue ! audioconvert ! voaacenc ! audio/mpeg ! queue ! mux.
It seems that the data is not flowing through the pipeline anymore (for some reason unknown to me) and my attempt to add autovideosink broke everything.
Any tips on how to do this correctly?
The right moment to split the data is right after filesrc:
gst-launch-1.0 -v filesrc location=video.mkv ! tee name=t ! queue ! decodebin ! autovideosink t. ! queue ! decodebin name=dec ! videoconvert ! x264enc ! video/x-h264 ! mpegtsmux name=mux ! queue ! udpsink host=127.0.0.1 port=5000 sync=true dec. ! queue ! audioconvert ! voaacenc ! audio/mpeg ! queue ! mux.
So the data flows directly to autovideosink before anything else happens while the other thread also joins at this very same moment, carrying the data flow to queue and the 2nd decodebin.
I'm initiating RTP stream from my Raspberry camera using:
raspivid -n -vf -fl -t 0 -w 640 -h 480 -b 1200000 -fps 20 -pf baseline -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay pt=96 config-interval=10 ! udpsink host=192.168.2.3 port=5000
on the client site, I'm converting it to HLS and upload it on a web server:
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,payload=96 ! rtph264depay ! mpegtsmux ! hlssink max-files=5 target-duration=5 location=C:/xampp/htdocs/live/segment%%05d.ts playlist-location=C:/xampp/htdocs/live/playlist.m3u8
the above works with me. on the other hand some players does not play the HLS since it has no audio track. I'm trying to figure out how i can add a dummy audio track. I tried many things but no luck e.g.
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,payload=96 ! rtph264depay ! h264parse ! mux. audiotestsrc wave=4 freq=200 ! audioconvert ! queue ! mux. mpegtsmux name=mux ! hlssink max-files=5 target-duration=5 location=C:/xampp/htdocs/live/segment%%05d.ts playlist-location=C:/xampp/htdocs/live/playlist.m3u8
or
gst-launch-1.0 -e -v udpsrc port=5000 name=src ! application/x-rtp,payload=96 ! rtph264depay ! h264parse ! mpegtsmux name=mux ! audiotestsrc wave=silence src. ! audioconvert ! wavenc ! rtpmp4gdepay ! aacparse ! mux. ! hlssink max-files=5 target-duration=5 location=C:/xampp/htdocs/live/segment%%05d.ts playlist-location=C:/xampp/htdocs/live/playlist.m3u8
Any help is appreciated
What was your idea for these pipelines? These look like you are trying to mux uncompressed audio data. I don't think this is what you want. I expected something like this for the audio path:
audiotestsrc wave=silence ! voaacenc ! aacparse ! mux.
Note that there may be more specific requirements - like number of audio channels or specific sample rates that are supported by your HLS players.