Screen Recording with both headphone & system audio - audio

I am trying to build a web-application with the functionality of screen-recording with system audio + headphone-mic audio being captured in the saved video.
I have been thoroughly googling on a solution for this, however my findings show multiple browser solutions where the above works so long as headphones are NOT connected, meaning the microphone input is coming from the system rather than headset.
In the case that you connect headphones, all of these solutions capture the screen without video-audio, and the microphone audio from my headset. So to re-clarify on this, it should have recorded video-audio from the video being played whilst recording, and the headset-mic audio also.
This is thoroughly available in native applications, however I am searching for a way to do this on a browser.
If there are no solutions for this currently that anybody knows of, some insight on the limitations around developing this would also really help, thank you.

Your browser manages the media input being received in the selected tab/window
To receive media input, you need to ensure you have the checkbox Share Audio in the image below checked. However this will only record media-audio being played in your headphones, when it comes to receiving microphone audio, the opposite must be done i.e the checkbox should be unchecked, or merge the microphone audio separately on saving the recorded video
https://slack-files.com/T1JA07M6W-F0297CM7F32-89e7407216

create two const, one retrieving on-screen video, other retrieving audio media:
const DISPLAY_STREAM = await navigator.mediaDevices.getDisplayMedia({video: {cursor: "motion"}, audio: {'echoCancellation': true}}); // retrieving screen-media
const VOICE_STREAM = await navigator.mediaDevices.getUserMedia({ audio: {'echoCancellation': true}, video: false }); // retrieving microphone-media
Use AudioContext to retrieve audio sources from getUserMedia() and getDisplayMedia() separately:
const AUDIO_CONTEXT = new AudioContext();
const MEDIA_AUDIO = AUDIO_CONTEXT.createMediaStreamSource(DISPLAY_STREAM); // passing source of on-screen audio
const MIC_AUDIO = AUDIO_CONTEXT.createMediaStreamSource(VOICE_STREAM); // passing source of microphone audio
Use the method below to create a new audio source which will be used as as the merger or merged version of audio, then passing audios into the merger:
const AUDIO_MERGER = AUDIO_CONTEXT.createMediaStreamDestination(); // audio merger
MEDIA_AUDIO.connect(AUDIO_MERGER); // passing media-audio to merger
MIC_AUDIO.connect(AUDIO_MERGER); // passing microphone-audio to merger
Finally, connect the merged-audio and video together into one array to form a track, and pass it to the MediaStreamer:
const TRACKS = [...DISPLAY_STREAM.getVideoTracks(), ...AUDIO_MERGER.stream.getTracks()] // connecting on-screen video with merged-audio
stream = new MediaStream(TRACKS);

Related

DiscordJS audio playback with multiple streams

I'm trying to create a discord bot using discord.js that is able to both play music and sound effects.
The bot correctly plays music, but when trying to play a sound effect during music playback, the sound effect never starts/ends.
I have the following code for playing sound effects.
function play(guildID, serverQueue, resource){
const newAudioPlayer = createAudioPlayer();
const oldAudioPlayer = serverQueue.audioPlayer;
const connection = getVoiceConnection(guildID)
oldAudioPlayer.pause();
connection.subscribe(newAudioPlayer);
newAudioPlayer.on(AudioPlayerStatus.Idle, () => {
console.log("Done playing");
connection.subscribe(oldAudioPlayer);
oldAudioPlayer.unpause();
newAudioPlayer.stop();
}).on('error', err => {
console.log("Something went wrong when trying to play sound effect");
console.log(err);
connection.subscribe(oldAudioPlayer);
oldAudioPlayer.unpause();
newAudioPlayer.stop();
})
newAudioPlayer.play(resource);
console.log("Trying to play");
}
I realize that a voice connection can only subscribe to one audioplayer at a time and that an audioplayer can only play one resource at a time. This is why, I create a new temporary audio player called "newAudioPlayer" that will play the short sound effect. I then pause the old audio player and subscribe the connection to the new audio player.
The sound effect is correctly played as long as the oldAudioPlayer has not been used to play a resource before. As soon as the oldAudioPlayer has been used to play a resource, the newAudioPlayer never starts playing the resource. I have checked all the different AudioPlayerStates for the newAudioPlayer, but none of them get triggered.
serverQueue.audioPlayer is initialized when a voice channel is joined, and is always set.
The program does print "Trying to play" but no audio can be heard.
Apparently this is a known issue with discord.js:
https://github.com/discordjs/discord.js/issues/7232
The workaround is to play the local file as a stream if the file for the other audioPlayer is also reading a stream.

How can I change the recording audio device of SoX?

In the title: How can I change the recording audio device of SoX?
I am using MacOS (installed with homebrew).
I am interacting with SoX through a Node.js library called node-audiorecorder that records sound; let me know if there's a better solution that I should be using for recording audio to a .wav file from a specific input device.
EDIT: Just to be clear, we are NOT talking about recording input from the default input device here.
There is an device option in the constructor.
const AudioRecorder = require('node-audiorecorder');
const options = {
program: `sox`,
device: null, // Recording device you want to use.
};
let audioRecorder = new AudioRecorder(options);

Web Audio API merging audio in single channel doesn't fully work

I have a video and a WebRTC audio stream and want to use Web Audio API to send the audio from the video to the Left channel, while the WebRTC to the right channel. So basically I'm doing:
video = document.getElementsByTagName("video")[0]
video.src = "http://link/to/my/video"
video.load()
audioContext = new AudioContext()
videoSourceL = audioContext.createMediaElementSource(video)
#create merger with 2 inputs, left (0) and right (1)
merger = audioContext.createChannelMerger(2)
merger.connect(audioContext.destination)
#now strange work around for WebRTC
audio = new Audio();
audio.muted = true
audio.srcObject = remoteStream
audioStreamR = audioContext.createMediaStreamSource(remoteStream)
# connect remote audio stream channel 0 to input 1 (right)
audioStreamR.connect(merger, 0, 1)
#connect video source channel 0 to input 0 (left)
videoSourceL.connect(merger, 0, 0)
The problem I have is that although the remote audio does go to the right channel (And is not audible in the Left), the audio from the video is also still slightly present in the right channel. So basically I have audio bleeding. The weird thing is that if I redirect both the remote stream and the video to the same channel, then the other channel has absolute silence.
Whereas if I had used an oscillator in place of the video audio, I would have a perfect separation. Any idea what I'm doing wrong?
EDIT: I also tried from the OS audio settings to turn off the left channel, and the audio bleeding to the right channel stopped (also tried this on a colleagues machine), so is this maybe a hardware/configuration
issue?
Was a hardware issue after all, effect is not there with good headphones.

Does v3 Google Cast receiver parse alternative audio tracks from an hls master playlist automatically or do I have to define them in the sender?

I'm trying to get a multi-audio HLS stream working on a v3 Google Cast custom receiver app. The master playlist of the stream refers to several video renditions of different resolution and two alternative audio tracks:
#EXTM3U
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",LANGUAGE="de",NAME="TV Ton",DEFAULT=YES, AUTOSELECT=YES,URI="index_1_a.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",LANGUAGE="de",NAME="Audiodeskription",DEFAULT=NO, AUTOSELECT=NO,URI="index_2_a.m3u8"
#EXT-X-STREAM-INF:AUDIO="aac",BANDWIDTH=383000,RESOLUTION=320x176,CODECS="avc1.4d001f, mp4a.40.2",CLOSED-CAPTIONS=NONE
index_0_av.m3u8
...more renditions
#EXT-X-STREAM-INF:AUDIO="aac",BANDWIDTH=3697000,RESOLUTION=1280x720,CODECS="avc1.4d001f, mp4a.40.2",CLOSED-CAPTIONS=NONE
index_6_av.m3u8
The video plays fine in both the sender and receiver app, I can see both audio tracks in the sender app, but when casting to the receiver there are no controls for changing the audio tracks.
When accessing the AudioTracksManager's getTracks() method while intercepting the LOAD message like so...
playerManager.setMessageInterceptor(
cast.framework.messages.MessageType.LOAD, loadRequestData => {
loadRequestData.media.hlsSegmentFormat = cast.framework.messages.HlsSegmentFormat.TS
const audioTracksManager = playerManager.getAudioTracksManager();
console.log(audioTracksManager.getTracks())
console.log('Load request: ', loadRequestData);
return loadRequestData;
});
I get an error saying:
Uncaught Error: Tracks info is not available.
Maybe unrelated, but super weird: I can console.log the request's media prop and see its tracks prop (an array with the expected 1 video and 2 audio tracks), however, if I try to access the tracks property in the LOAD message interceptor I get undefined.
I currently cannot look into the iOS sender code yet, so I tried to eliminate error sources on the receiver end. The thing is:
I always assumed that the receiver identifies alternative audio tracks on its own when loading HLS playlists. Is this assumption correct or can the AudioTracksManager only access tracks that have been previously defined in a sender app?
I couldn't find a clear statement on that in the Google Cast reference...
Ok, feeling stupid for the time I spent on this, but I'm finally able to answer my own question. I didn't realize that I was accessing the AudioTracksManager in the wrong place - namely in the LOAD message interceptor instead of in a PLAYER_LOAD_COMPLETE event listener (as it is properly documented here)
After placing my logic into this event listener I was able to access and programmatically set my audio tracks.
So to answer my original question: Yes, the receiver app automatically identifies alternative audio tracks from an HLS playlist.

vlc parameters to set audio stream from microphone

I have C# project where stream from ip-camera recorded to the file, I use libvlc.
This is part of code with vlc parameters:
string VlcArguments = #":sout=#transcode{acodec=mpga,deinterlace}:standard{access=file,mux=mp4,dst="C:\Users\I\Desktop\Output.mp4"}";
var media = factory.CreateMedia<IMedia>(rtsp://184.72.239.149/vod/mp4:BigBuckBunny_175k.mov, VlcArguments);
var player = factory.CreatePlayer<IPlayer>();
player.Open(media);
filename is the path of the result file.
It works fine, but I need to record sound from a microphone Microphone (High Definition Audio Device).
What I need to change to achieve that?
UPD
It should look something like this
var media = factory.CreateMedia<IMedia>("dshow:// dshow-vdev=rtsp://184.72.239.149/vod/mp4:BigBuckBunny_175k.mov dshow-adev=Microphone (High Definition Audio Device)", VlcArguments)
But it doesn't work (
UPD2
So, I think I found the answer
https://forum.videolan.org/viewtopic.php?f=14&t=124229&p=425550&hilit=camera+microphone+dshow#p425550
Unfortunately this will not work

Resources