i dont konw how build webrtc aec in pjsip，I don't want to download all the code for webrtc, just the aec section，embed cpu is imx6ul，Please give me some guidance, thank you very much
They are both written in C so migration of the WebRTC module and make it compiable should be easy.
You should take advantage of pjsip's Audio Manipulation Algorithms module. It allows you to apply algos on the processing audio sample frames.
https://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__FRAME__OP.htm
For the AEC, make sure you have access to the loopback channel on the codec. (Samples from playing speaker).
Steps:
Create a New Audio Manipulation Algorithms Module
Gain access to audio playback/capture signals with pjmedia
Migrate WebRTC AEC Module
Reroute the signal processing pipeline and enabling the AEC Algorithms module
Related
I am trying to build a Security Camera.
However, I do not want to do something like Port Forwarding to enable streaming. Nor do I want to implement a 3rd party Cloud based Streaming service like Wowza.
Under these conditions, the only way I could find was to implement WebRTC MediaStream on NodeJS
The WebRTC implementations on NodeJS are missing the MediaStream package.
Also the MediaStream package relies heavily on built in browser code to set up and stream audio & video.
How can I do seamless audio video streaming using NodeJS?
Is it even possible? Can NodeJS (a single threaded model) do something as computationally intensive as video transcoding?
This is certainly doable. I read a while back about a webRTC connected drone (https://tech.ipcortex.co.uk/blog/keevioeye) and there's the webRTC connected RC car (http://www.instructables.com/id/WebRTC-Creeper-Drone-Browser-Controlled-RC-Car/). To build it from scratch you would probably need a pico motherboard with Linux Chromium installed to handle the webRTC negotiation to the server receiving the stream.
For server side MediaStream recording, transcoding and even motion detection and facial recognition I'd use the open source Kurento project which has both Java and nodeJS libraries.
Hej
I would like to build an audio effect into a RPi. This effect should be programmed in C. I am not familiar with the software audio interfaces in Linux. The ALSA interface looks very complicated. Port Audio seems to be an alternative.
Any ideas(maybe with a tutorial)?
With some work you can also get OpenAL to stream and render audio using c language - then you could perform your processing in that context ...
Node.js is available on RPi which offers audio modules
PortAudio seems the best approach. A good tutorial can be found here:
http://portaudio.com/docs/v19-doxydocs/tutorial_start.html
Sometimes the Interface configuration needs to be done manually.
I am looking to build an app that needs to process incoming audio on a phone call in real time.
WebRTC allows for this but i think this works only in their browser based P2P audio communications functionality but not for phone calls/ VOIP.
Twilio and Plivo allow you record the audio for batch/later processing.
Is there a library that will give me access to the audio streams in real time? If not, what would I need to build such a service from scratch?
Thanks
If you are open to using a media server (so that the call is not longe P2P but it's mediated by the media server using a B2B model), then perhaps the Kurento Media Server may solve your problem. Kurento Media Server makes possible to create processing capabilities which are applyied in real time onto the media streams. There are many examples in the documentation of computer vision and augmented reality algorithms applied in real time over the video streams. I've never seen an only-audio processing module, but it should be simple to implement just by creating an additional module, which is not too complex if you have some knowledge about C/C++ and media processing concepts.
Disclaimer: I'm part of the Kurento development team.
Suppose that I want to code an audio filter in C++ that is applied on every audio or to a specific microphone/source, where should I start with this on ubuntu ?
edit, to be clear I don't get how to do this and what is the role of Pulseaudio, ALSA and Gstreamer.
Alsa provides an API for accessing and controlling audio and MIDI hardware. One portion of ALSA is a series of kernel-mode device drivers, whilst the other is a user-space library that applications link against. Alsa is single-client.
PulseAudio is framework that facilitates multiple client applications accessing a single audio interface (alsa is single-client). It provides a daemon process which 'owns' the audio interface and provides a IPC transport for audio between the daemon and applications using it. This is used heavily in open source desktop environments. Use of Pulse is largely transparent to applications - they continue to access the audio input and output using the alsa API with audio transport and mixing. There is also Jack which is targeted more towards 'professional' audio applications - perhaps a bit of a misnomer, although what is meant here is low latency music production tools.
gStreamer is a general purpose multi-media framework based on the signal-graph pattern, in which components have a number of inputs and output pins and provide a transformation function. A Graph of these components is build to implement operations such as media decoding, with special nodes for audio and video input or output. It is similar in concept to CoreAudio and DirectShow. VLC and libAV are both open source alternatives that operate along similar lines. Your choice between these is a matter of API style, and implementation language. gStreamer, in particular, is an OO API implemented in C. VLC is C++.
The obvious way of implementing the problem you describe is to implement a gStreamer/libAV/VLC component. If you want to process the audio and then route it to another application, this can be achieved by looping it back through Pulse or Jack.
Alsa provides a plug-in mechanism, but I suspect that implementing this from the ALSA documentation will be tough going.
The de-facto architecture for building effects plug-ins of the type you describe is Steinberg's VST. There are plenty of open source hosts and examples of plug-ins that can be used on Linux, and crucially, there is decent documentation. As with a gStreamer/libAV/VLC, you be able to route audio in an out of this.
Out of these, VST is probably the easiest to pick up.
I want to make (for fun, challenge) a videoconference application, I have some ideas about this:
1) taking the audio/video streams (I don't know what an audio/video stream is)
2) pass this to a server that lets communicate the clients. I can figure out how to write a server(there are a lot of books and documentation about this) but I really don't know how to interact with the webcam and with the audio/video in general.
I want some links, book, suggestions about the basics of digital audio/video expecially on programming. Please help me!!!
I want to make it run on a Linux platform.
Linux makes video grabbing really nice. As long as you have a driver that outputs the video stream to the /dev/video/v* channels. All you have to do is open up a control connection to the device [an exercise for the OP] and then read in the channel like a file [given the parameters set by the control connection. Audio should be the same way, but don't quote me on it.
BTW: Video streaming from a server is a very complex issue. You have to develop or use an existing protocol. You have to be very aware of networking delays, and adjust the information sent (resize or recompress) to the client based on the link size between the client and the server.