I've been trying to find sources online for real-time system audio processing, but cannot seem to find anything I'm looking for. I've checked out some Python libraries, like PyAudio and librosa, but I believe their use-cases are different from what I'm looking for. I made a quick test with PyAudio, and it seems that I can only process my audio input (my system's microphone).
My goal is to play Spotify, a YouTube video, or whatever from my computer and to process the audio-- I'd like to process system sounds in real-time.
Related
I am trying to build an open-source in-ear monitoring system. I have created the UI and was wondering how I would get the channels that are on an audio mixing console so that I can edit the channels and stream them to each musician. Is there a certain protocol that all the mixers use? You can find the project at https://gitlab.com/openstagemix. We would love to have contributors.
I can't really test whether this is the correct answer as I am trapped in my house during the coronavirus time. But, all mixers use something called OSC which is a protocol between mixers, synthesizers, etc. to computers. You can find more information here http://opensoundcontrol.org/introduction-osc.
Update:
It's neither! I am going to use the AES67 standard to receive information from my mixer and with that process the audio. This is because my mixer is ethernet capable.
I am from signal processing background. When I listen audio from youtube, sometime I feel the chances of improvement in audio quality at run time. These basic improvements will be auto adjusted as per the audio content. I mean, there is a possibility of write a generalized or feedback based algorithm.
I am aware to test my algorithm on offline audio(recorded audio file). But I do not know, how can I interface my algorithm with youtube audio? Is it possible to write some windows API which activates whenever any audio plays over the machine.
Suppose that I want to code an audio filter in C++ that is applied on every audio or to a specific microphone/source, where should I start with this on ubuntu ?
edit, to be clear I don't get how to do this and what is the role of Pulseaudio, ALSA and Gstreamer.
Alsa provides an API for accessing and controlling audio and MIDI hardware. One portion of ALSA is a series of kernel-mode device drivers, whilst the other is a user-space library that applications link against. Alsa is single-client.
PulseAudio is framework that facilitates multiple client applications accessing a single audio interface (alsa is single-client). It provides a daemon process which 'owns' the audio interface and provides a IPC transport for audio between the daemon and applications using it. This is used heavily in open source desktop environments. Use of Pulse is largely transparent to applications - they continue to access the audio input and output using the alsa API with audio transport and mixing. There is also Jack which is targeted more towards 'professional' audio applications - perhaps a bit of a misnomer, although what is meant here is low latency music production tools.
gStreamer is a general purpose multi-media framework based on the signal-graph pattern, in which components have a number of inputs and output pins and provide a transformation function. A Graph of these components is build to implement operations such as media decoding, with special nodes for audio and video input or output. It is similar in concept to CoreAudio and DirectShow. VLC and libAV are both open source alternatives that operate along similar lines. Your choice between these is a matter of API style, and implementation language. gStreamer, in particular, is an OO API implemented in C. VLC is C++.
The obvious way of implementing the problem you describe is to implement a gStreamer/libAV/VLC component. If you want to process the audio and then route it to another application, this can be achieved by looping it back through Pulse or Jack.
Alsa provides a plug-in mechanism, but I suspect that implementing this from the ALSA documentation will be tough going.
The de-facto architecture for building effects plug-ins of the type you describe is Steinberg's VST. There are plenty of open source hosts and examples of plug-ins that can be used on Linux, and crucially, there is decent documentation. As with a gStreamer/libAV/VLC, you be able to route audio in an out of this.
Out of these, VST is probably the easiest to pick up.
I'm writing a cross-platform program that involves scrolling a waveform along with uncompressed wav/aiff audio playback. Low latency and accuracy are pretty important. What is the best cross-platform audio library for audio playback when synchronizing to an external clock? By that I mean that I would like to be able to write the playback code so it sends events to a listener many times per second that includes the "hearing frame" at the moment of the notification.
That's all I need to do. No recording, no mixing, no 3d audio, nothing. Just playback with the best possible hearing frame notifications available.
Right now I am considering RTAudio and PortAudio, mostly the former since it uses ALSA.
The target platforms, in order of importance, are Mac OSX 10.5/6, Ubuntu 10.11, Windows XP/7.
C/C++ are both fine.
Thanks for your help!
The best performing cross platform library for this is jack. Properly configured, jack on Linux can outperform Windows asio easily (in terms of low latency processing without dropouts). But you cannot expect normal users to use jack (the demon should be started by the user before the app is started, and it can be a bit tricky to set up). If you are making an app specifically for pro-audio I would highly recommend looking in to jack.
Edit:
Portaudio is not as high-performance, but is much simpler for the user (no special configuration should be needed on their end, unlike jack). Most open source cross platform audio programs that I have used use portaudio (much moreso than openal), but unlike jack I have not used it personally. It is callback based, and looks pretty straightforward though.
OpenAL maybe an option for you.
I have a large amount of audio stored on my web server in a very custom format that can't be replayed by anything other than my own application. That application is a Win32 app that can connect to my web server and stream and replay that audio.
I'd really like to be able to do the streaming and replaying from within a browser, but don't know where to start. Ideally I'd like the technology to be cross-platform (unlike my current Win32 app) and cross-browser (IE 6 and above and Firefox).
My current thoughts are to look at things like:
Flash, but doesn't that only replay mp3 audio?
Java, are VMs freely available still?
Converting the audio to a WAV file on the web server and then using someone else's plugin to replay that file. I'd rather keep the conversion off the web server for performance reasons, but is still an option.
Writing my own custom plugin to do the complete stream and replay operation.
Any guidance would be most useful.
Please note that the audio is not music and that simply converting to another audio format is not trivial. The audio that is stored also changes frequently (every minute) would need constant conversion.
Why are you using a proprietary music format? I'd probably not even bother downloading a program to listen to it.
I would suggest you convert it to mp3 and then use flash.
Building your own plugin would probably be hard, there are so many different platforms you'd have to cater for, something like flash is written for them already.
Apart from converting server-side: Implement a decoder for your format in ActionScript or Java. Then you can write a Flash movie or Java applet that plays it. Both languages/runtimes should be fast enough to decode in realtime unless your format is very complex. Flash would be the more accessible of the two, since nearly everyone has the plugin installed. (It's possible that playing a raw sound buffer isn't supported by older Flash versions than 10, I'm no expert on that.) The Java plugin is definitely free, but you'd require the users to install it.
I'd go with converting the audio to WAV (or MP3) on the server. Writing your own cross-platform browser component would be a lot of work, thanks to the different ways the major OSes handle their audio APIs.
Try taking a look at shoutcast.
Basically its a server app that will stream music to any client that connects to it through a browser (effectively your own radio station). I've never used it myself but should be straight forward.
Another idea is winamp remote. Again you install the app on the server but this time you can browse your music collection on their website and play individual songs.