I'm interested in getting a screenshot from the user and I'm using the getDisplayMedia API to capture the user's screen:
const constraints = { video: true, audio: false };
if (navigator.mediaDevices["getDisplayMedia"]) {
navigator.mediaDevices["getDisplayMedia"](constraints).then(startStream).catch(error);
} else {
navigator.getDisplayMedia(constraints).then(startStream).catch(error);
}
When executed, the browser prompts the user if they want to share their display. After the user accepts the prompt, the provided callback receives a MediaStream. For visualization, I'm binding it directly to a element:
const startStream = (stream: MediaStream) => {
this.video.nativeElement.srcObject = stream;
};
This is simple and very effective so far. Nevertheless, I'm only interested in a single frame and I'd therefore like to manually stop the stream as soon as I've processed it.
What I tried is to remove the video element from the DOM, but Chrome keeps displaying a message that the screen is currently captured. So this only affected the video element but not the stream itself:
I've looked at the Screen Capture API article on MDN but couldn't find any hints on how to stop the stream.
How do I end the stream properly so that the prompt stops as well?
Rather than stopping the stream itself, you can stop its tracks.
Iterate over the tracks using the getTracks method and call stop() on each of them:
stream.getTracks()
.forEach(track => track.stop())
As soon as all tracks are stopped, Chrome's capturing prompt disappears as well.
start screen capture sample code:
async function startCapture() {
logElem.innerHTML = "";
try {
videoElem.srcObject = await navigator.mediaDevices.getDisplayMedia(displayMediaOptions);
dumpOptionsInfo();
} catch(err) {
console.error("Error: " + err);
}
}
stop screen capture sample code:
function stopCapture(evt) {
let tracks = videoElem.srcObject.getTracks();
tracks.forEach(track => track.stop());
videoElem.srcObject = null;
}
more info: MDN - Stopping display capture
Related
I'm making a discord bot with Discord.js v14 that records users' audio as individual files and one collective file. As Discord.js streams do not interpolate silence, my question is how to interpolate silence into streams.
My code is based off the Discord.js recording example.
In essence, a privileged user enters a voice channel (or stage), runs /record and all the users in that channel are recorded up until the point that they run /leave.
I've tried using Node packages like combined-stream, audio-mixer, multistream and multipipe, but I'm not familiar enough with Node streams to use the pros of each to fill in the gaps the cons add to the problem. I'm not entirely sure how to go about interpolating silence, either, whether it be through a Transform (likely requires the stream to be continuous, or for the receiver stream to be applied onto silence) or through a sort of "multi-stream" that swaps between piping the stream and a silence buffer. I also have yet to overlay the audio files (e.g, with ffmpeg).
Would it even be possible for a Readable to await an audio chunk and, if none is given within a certain timeframe, push a chunk of silence instead? My attempt at doing so is below (again, based off the Discord.js recorder example):
// CREDIT TO: https://stackoverflow.com/a/69328242/8387760
const SILENCE = Buffer.from([0xf8, 0xff, 0xfe]);
async function createListeningStream(connection, userId) {
// Creating manually terminated stream
let receiverStream = connection.receiver.subscribe(userId, {
end: {
behavior: EndBehaviorType.Manual
},
});
// Interpolating silence
// TODO Increases file length over tenfold by stretching audio?
let userStream = new Readable({
read() {
receiverStream.on('data', chunk => {
if (chunk) {
this.push(chunk);
}
else {
// Never occurs
this.push(SILENCE);
}
});
}
});
/* Piping userStream to file at 48kHz sample rate */
}
As an unnecessary bonus, it would help if it were possible to check whether a user ever spoke or not to eliminate creating empty recordings.
Thanks in advance.
Related:
Record all users in a voice channel in discord js v12
Adding silent frames to a node js stream when no data is received
After a lot of reading about Node streams, the solution I procured was unexpectedly simple.
Create a boolean variable recording that is true when the recording should continue and false when it should stop
Create a buffer to handle backpressuring (i.e, when data is input at a higher rate than its output)
let buffer = [];
Create a readable stream for which the receiving user audio stream is piped into
// New audio stream (with silence)
let userStream = new Readable({
// ...
});
// User audio stream (without silence)
let receiverStream = connection.receiver.subscribe(userId, {
end: {
behavior: EndBehaviorType.Manual,
},
});
receiverStream.on('data', chunk => buffer.push(chunk));
In that stream's read method, handle stream recording with a 48kHz timer to match the sample rate of the user audio stream
read() {
if (recording) {
let delay = new NanoTimer();
delay.setTimeout(() => {
if (buffer.length > 0) {
this.push(buffer.shift());
}
else {
this.push(SILENCE);
}
}, '', '20m');
}
// ...
}
In the same method, also handle ending the stream
// ...
else if (buffer.length > 0) {
// Stream is ending: sending buffered audio ASAP
this.push(buffer.shift());
}
else {
// Ending stream
this.push(null);
}
If we put it all together:
const NanoTimer = require('nanotimer'); // node
/* import NanoTimer from 'nanotimer'; */ // es6
const SILENCE = Buffer.from([0xf8, 0xff, 0xfe]);
async function createListeningStream(connection, userId) {
// Accumulates very, very slowly, but only when user is speaking: reduces buffer size otherwise
let buffer = [];
// Interpolating silence into user audio stream
let userStream = new Readable({
read() {
if (recording) {
// Pushing audio at the same rate of the receiver
// (Could probably be replaced with standard, less precise timer)
let delay = new NanoTimer();
delay.setTimeout(() => {
if (buffer.length > 0) {
this.push(buffer.shift());
}
else {
this.push(SILENCE);
}
// delay.clearTimeout();
}, '', '20m'); // A 20.833ms period makes for a 48kHz frequency
}
else if (buffer.length > 0) {
// Sending buffered audio ASAP
this.push(buffer.shift());
}
else {
// Ending stream
this.push(null);
}
}
});
// Redirecting user audio to userStream to have silence interpolated
let receiverStream = connection.receiver.subscribe(userId, {
end: {
behavior: EndBehaviorType.Manual, // Manually closed elsewhere
},
// mode: 'pcm',
});
receiverStream.on('data', chunk => buffer.push(chunk));
// pipeline(userStream, ...), etc.
}
From here, you can pipe that stream into a fileWriteStream, etc. for individual purposes. Note that it's a good idea to also close the receiverStream whenever recording = false with something like:
connection.receiver.subscriptions.delete(userId);
As well, the userStream should, too be closed if it's not, e.g, the first argument of the pipeline method.
As a side note, although outside the scope of my original question, there are many other modifications you can make to this. For instance, you can prepend silence to the audio before piping the receiverStream's data to the userStream, e.g, to make multiple audio streams of the same length:
// let startTime = ...
let creationTime;
for (let i = startTime; i < (creationTime = Date.now()); i++) {
buffer.push(SILENCE);
}
Happy coding!
I'm making a program that will output text to speech, the user will need to be able to change the output device (for example to virtual audio cable). At the moment I'm using https://www.npmjs.com/package/say to produce the speech
e.g.
const say = require("say");
say.speak("Hello World");
but I've no clue on how I could go about choosing the audio output.
What I've found so far has been pretty useless, I'm guessing largely because I don't know the proper terminology to search for answers, regardless this is it:
I first found out about navigator.MediaDevices, then I found how I could make an audio element and change the audio device of that element via setSinkId, then I realized these things are probably(?) irrelevant since the say module seems to play sounds using powershell commands. I've even gone as far as trying to change powershell's output device in app volume device preferences(img), but that seems to do nothing.
I'm pretty much stumped right now, so I'd appreciate any help please. I'm not set on using Say as a module, it just seemed easy to use at first.
Edit:
A workaround I've settled with is making my own TTS class and using SpVoice.
I have something similar to this:
const childProcess = require('child_process');
class TTS {
constructor(channel, speed) {
this.speed = speed;
this.channel = channel;
this.baseCommand = "$speak = New-Object -ComObject SAPI.SPVoice;"
}
speak(text){
var command = this.baseCommand +
`$speak.AudioOutput = foreach ($o in $speak.GetAudioOutputs()) {if ($o.getDescription() -eq '${this.channel}') {$o; break;}}; `
+ "$speak.Speak([Console]::In.ReadToEnd());"
this.child = childProcess.spawn('powershell', [command], {shell: true})
this.child.stdin.setEncoding('ascii')
this.child.stdin.end(text);
this.child.addListener('exit', (code, signal) => {
if (code === null || signal !== null) {
console.log(new Error(`error [code: ${code}] [signal: ${signal}]`))
}
this.child = null
})
}
}
Then I can pass in an audio channel like
tts = new TTS("CABLE Input (VB-Audio Virtual Cable)", 0);
tts.speak("Hello there");
and it will output TTS in my desired channel
Some of the browsers support the built in speechSynthesis API.
Save the following code in 'test.html' file and open the file in chrome web browser for testing the speech api.
<script>
//---------------- SpeechAPI Start ------------------------
function speak(message) {
try {
var utterance= new SpeechSynthesisUtterance("");
utterance.lang='en-GB';
utterance.text=message;
window.speechSynthesis.speak(utterance);
}catch(e){
console.log('Exception in speak : ' + e)
}
}
//---------------- SpeechAPI End ------------------------
</script>
<button onclick="speak('Hello, how are you doing?');">Press to speak</button>
Is this what you are looking for ?
I am starting with the sample LibVLCSharp.WPF.Sample and while it plays my VOB, I cannot change the audio track.
I call SetAudioTrack and it always returns false (and doesn't change). Full VLC lets me change the audio track on the same file just fine.
var media = new Media(_libVLC, clip.GetEvoPath(playlist), FromType.FromPath);
var status = await media.Parse();
_mediaPlayer.Playing += (sender, e) =>
{
bool ok = _mediaPlayer.SetAudioTrack(3);
};
_mediaPlayer.Play(media);
When should I call this, to get it to work? Ideally I would like to set the stream before I start playback, but that doesn't work either.
How is it possible to carry out any background tasks while Alexa is playing something with audioPlayer.Play?
The below plays an audio stream, but I need to perform other tasks in the background without intervention while the stream is playing. I know it is possible because other Skills can do it.
var handlers = {
'LaunchRequest': function() {
this.emit('Play');
},
'Play': function() {
this.response.speak('Sure.').
audioPlayerPlay(
'REPLACE_ALL',
stream.url,
stream.url,
null,
0);
this.emit(':responseReady');
}
}
Does anyone know or have any suggestions? From what I can see, once it starts playing the stream, I cannot get it to do anything unless I interrupt the stream to command another intent?
Alexa has a few built in requests that she sends to your skill throughout the lifecycle of an Audio stream for just this purpose! They are as follows:
AudioPlayer.PlaybackStarted - Sent when a new audio item begins playing.
AudioPlayer.PlaybackNearlyFinished - Sent when an audio item is almost over (most commonly used by a skill service to handle queuing the next item.)
AudioPlayer.PlaybackFinished - Sent when an audio item ends.
There are a couple of other ones too that you can read about here, but my guess is these will do just fine for what you need.
To use them, just set up handlers in your code for any of these requests and perform any tasks you need to there!
I don't know node.js at all, but my guess is the end result will look relatively close to this:
var handlers = {
'LaunchRequest': function() { /* your launch request code from above */ }
'Play': function() { /* your play code from above */ }
'AudioPlayer.PlaybackNearlyFinished': function() {
// Perform any background tasks here
}
}
I'm using the videojs-playlist plugin along with Google's videojs-ima plugin. Everything works swimmingly except I am only getting a preload ad before the first video. I want one before each video in the playlist.
Basic setup is boilerplate, but for reference:
this.player = videojs('currentvideo', { autoplay : true, fluid : true });
this.player.playlist(this.playlist);
this.player.playlist.autoadvance(5);
const skippable_linear = {google's test ad};
const options = {
id: 'currentvideo',
adTagUrl: skippable_linear,
debug : true
};
this.player.ima(
options
);
this.player.ima.requestAds();
I have tried various ways of manually calling ads from inside an 'ended' event handler, such as calling requestAds again:
const _this = this;
this.player.on( 'ended', function(){
/* some other stuff */
_this.player.ima.requestAds();
});
This does play an ad where I want it, but
this breaks playlist's 'autoadvance' setting (next video doesn't start playing when the ad is finished), and
this puts the player into "ad display" mode (scrubber is unavailable, etc).
Is there a simple way to just say, "play an ad now" programmatically? I've tried, without joy, to use all of the seemingly applicable methods exposed by both the ima plugin and the contrib-ads plugin it relies on. I'll admit here that this is the first time I've ever had to deal with videos that run ads, so I'm kind of a noob.
I am trying to do the same thing. Just like you I failed when calling player.ima.requestAds() on events. I dug deeper and the best I could come up with is what I share bellow.
According to the videojs-ima API you have to use the setContentWithAdTag method instead of whatever you are using to switch the player content. In our case it is the player.playlist.next method.
I combined the code found in the videojs-ima examples with the original playlist.next to write my own next.
Then quite brutally I overrode the original plugin method.
Here's the code:
player.playlist(myPlayilst);
player.playlist.autoadvance(2);
player.playlistUi(); //videojs-playlist-ui
player.ima({
id: 'video5',
adTagUrl: 'thy adserver request'
});
//override playlist.next
player.playlist.next = function(){
var nextIndex = 0,
playlist = this.player_.playlist,
list = this.player_.playlist();
//everything below is copied directly from the original `next` (except for the "//load with ad")
// Repeat
if (playlist.repeat_) {
nextIndex = playlist.currentIndex_ + 1;
if (nextIndex > list.length - 1) {
nextIndex = 0;
}
} else {
// Don't go past the end of the playlist.
nextIndex = Math.min(playlist.currentIndex_ + 1, list.length - 1);
}
// Make the change
if (nextIndex !== playlist.currentIndex_) {
//load with ad
this.player_.playlist.currentItem(nextIndex);
this.player_.ima.setContentWithAdTag(
this.player_.playlist.currentItem(),
null,
true);
this.player_.ima.requestAds();
/////
return list[playlist.currentItem()];
}
}
You will probably need to override other methods that change the current playback, like playlist.previous.
I use videojs-playlist-ui so in my case it was neccessary to change the onclick handler called switchPlaylistItem_. I used some good old brute force to do that like this:
videojs.getComponent('PlaylistMenuItem').prototype.switchPlaylistItem_ = function(e){
this.player_.playlist.currentItem(this.player_.playlist().indexOf(this.item));
this.player_.ima.setContentWithAdTag(
this.player_.playlist.currentItem(),
null,
true);
this.player_.ima.requestAds();
};
PlaylistMenuItem's prototype should be changed before initializing the player.
This solution works, but it feels hacky, so if anyone can come up with something cleaner, please share!
I ended up forking videojs-playlist, and adding the option to override the player.src method. Feel free to use it:
fw-videojs-playlist
Details on how to use it are all in the github readme (including an example with ima.setContentWithAdTag)