I'm trying out the new spatial audio functions in the AudioGraph 1.1 API and I have sound output from a file working without an emitter, but when I add an emitter to my node creation call it suddenly returns FormatNotSupported. I can't find any information via searching that has been any sort of helpful, probably because it's such a new API. Can anyone see if I'm doing something wrong or missing something? Following is my code:
private async void MainPage_Loaded(object sender, RoutedEventArgs args)
{
AudioGraphSettings settings = new AudioGraphSettings(AudioRenderCategory.Media);
var devices = await DeviceInformation.FindAllAsync();
CreateAudioGraphResult result = await AudioGraph.CreateAsync(settings);
if (result.Status != AudioGraphCreationStatus.Success)
{
return;
}
graph = result.Graph;
FileOpenPicker saveFilePicker = new FileOpenPicker();
saveFilePicker.FileTypeFilter.Add(".wav");
saveFilePicker.FileTypeFilter.Add(".wma");
saveFilePicker.FileTypeFilter.Add(".mp3");
StorageFile file = await saveFilePicker.PickSingleFileAsync();
if (file == null)
{
return;
}
AudioNodeEmitter emitter = new AudioNodeEmitter(AudioNodeEmitterShape.CreateOmnidirectional(),
AudioNodeEmitterDecayModel.CreateNatural(.1,1,10,100),
AudioNodeEmitterSettings.None);
emitter.Position = new Vector3(10, 0, 5);
CreateAudioDeviceOutputNodeResult deviceOutputNodeResult = await graph.CreateDeviceOutputNodeAsync();
var outputNode = deviceOutputNodeResult.DeviceOutputNode;
CreateAudioFileInputNodeResult fileInputNodeResult = await graph.CreateFileInputNodeAsync(file, emitter);
inputNode = fileInputNodeResult.FileInputNode;
inputNode.AddOutgoingConnection(outputNode);
graph.Start();
}
There is no problem with your code, problem is:
Audio node emitters can only process audio that is formatted in mono with a sample rate of 48kHz. Attempting to use stereo audio or audio with a different sample rate will result in an exception.
You can refer to the note part of Spatial audio.
To test this API, you may download this audio here.
Related
Here is the problem: when i'm running this code I get a error saying: song_queue.connection.play is not a function. The bot joins the voicechat correctly but the error comes when it tries to play a song. Sorry for the large amount of code but I really want to fix this so my bot can work. I got the code from a YouTube tutorial recorded in discord.js 12.4.1 (my version is the latest 13.1.0) and I think the error has to do with #discordjs/voice. I would really appreciate any help with getting this to work.
const ytdl = require('ytdl-core');
const ytSearch = require('yt-search');
const { joinVoiceChannel, createAudioPlayer, createAudioResource, } = require('#discordjs/voice');
const queue = new Map();
// queue (message.guild.id, queue_constructor object { voice channel, text channel, connection, song[]});
module.exports = {
name: 'play',
aliases: ['skip', 'stop'],
description: 'Advanced music bot',
async execute(message, args, cmd, client, discord){
const voice_channel = message.member.voice.channel;
if (!voice_channel) return message.channel.send('You need to be in a channel to execute this command');
const permissions = voice_channel.permissionsFor(message.client.user);
if (!permissions.has('CONNECT')) return message.channel.send('You dont have permission to do that');
if (!permissions.has('SPEAK')) return message.channel.send('You dont have permission to do that');
const server_queue = queue.get(message.guild.id);
if (cmd === 'play') {
if (!args.length) return message.channel.send('You need to send the second argument');
let song = {};
if (ytdl.validateURL(args[0])){
const song_info = await ytdl.getInfo(args[0]);
song = { title: song_title.videoDetails.title, url: song_info.videoDetails.video_url }
} else {
//If the video is not a URL then use keywords to find that video.
const video_finder = async (query) =>{
const videoResult = await ytSearch(query);
return (videoResult.videos.length > 1) ? videoResult.videos[0] : null;
}
const video = await video_finder(args.join(' '));
if (video){
song = { title: video.title, url: video.url }
} else {
message.channel.send('Error finding your video');
}
}
if (!server_queue){
const queue_constructor = {
voice_channel: voice_channel,
text_channel: message.channel,
connection: null,
songs: []
}
queue.set(message.guild.id, queue_constructor);
queue_constructor.songs.push(song);
try {
const connection = await joinVoiceChannel({
channelId: message.member.voice.channel.id,
guildId: message.guild.id,
adapterCreator: message.guild.voiceAdapterCreator
})
queue_constructor.connection = connection;
video_player(message.guild, queue_constructor.songs[0]);
} catch (err) {
queue.delete(message.guild.id);
message.channel.send('There was an error connecting');
throw err;
}
} else{
server_queue.songs.push(song);
return message.channel.send(`<:seelio:811951350660595772> **${song.title}** added to queue`);
}
}
}
}
const video_player = async (guild, song) => {
const song_queue = queue.get(guild.id);
if(!song) {
song_queue.voice_channel.leave();
queue.delete(guild.id);
return;
}
const stream = ytdl(song.url, { filter: 'audioonly' });
song_queue.connection.play(stream, { seek: 0, volume: 0.5 }).on('finish', () => {
song_queue.songs.shift();
video_player(guild, song_queue.songs[0]);
});
await song_queue.text_channel.send('(`<:seelio:811951350660595772> Now Playing **${song.title}**`)')
}
Discord.js V13 and #discordjs/voice
Since a relatively recent update to the Discord.js library a lot of things have changed in the way you play audio files or streams over your client in a Discord voice channel. There is a really useful guide by Discord to explain a lot of things to you on a base level right here, but I'm going to compress it down a bit and explain to you what is going wrong and how you can get it to work.
Some prerequisites
It is important to note that for anything to do with voice channels for your bot it is necessary to have the GUILD_VOICE_STATES intent in your client. Without it your bot will not actually be able to connect to a voice channel even though it seems like it is. If you don't know what intents are yet, here is the relevant page from the same guide.
Additionally you will need some extra libraries that will help you with processing and streaming audio files. These things will do a lot of stuff in the background that you do not need to worry about them, but without them playing any audio will not work. To find out what you need you can use the generateDependecyReport() function from #discordjs/voice. Here is the page explaining how to use it and what dependencies you will need. To use the function you will have to import it from the #discordjs/voice library.
Playing audio over a client
So once everything is set up you can get to how to play audio and music. You're already a great few steps on the way by using ytdl-core and getting a stream object from it, but audio is not played by using a .play() command on the connection. Instead you will need to utilize AudioPlayer and AudioResource objects.
AudioPlayer
The AudioPlayer is essentially your jukebox. You can make one by simply calling its function and storing that in a const like so:
const player = createAudioPlayer()
This is a function from the #discordjs/voice library and will have to be imported just like generateDependencyReport().
There are a few parameters you can give it to modify its behavior, but right now that is not important. You can read more about that on its page from the Discord guide right here.
AudioResource
To get your AudioPlayer to actually play anything you will have to create an AudioResource. This is basically a version of your file or stream modified to work with the player. This is very simply done with another function from the #discord.js/voice library called createAudioResource(...). This must once again be imported. As a parameter you can parse the location of an mp3 or webm file, but you can also use a stream object like you have already acquired. Just input stream as the parameter of that function.
To now play the resource there are two more steps. First you must subscribe your connection to the player. This basically tells the connection to broadcast whatever your AudioPlayer is playing. To do this simply call the .subscribe() function on your connection object with the player as a parameter like so:
connection.subscribe(player)
player.play(resource)
The second line of code you see above is how you get your player to play your AudioResource. Just parse the resource as a parameter and it will start playing. You can find more on the AudioResource side of things on its page in the Discord guide right here.
This way takes a few more steps than it did in V12, but once you get the hang of this system it really isn't that bad or difficult.
Leaving a voice channel
There is another thing that is going wrong in your code when you try to leave a voice channel. I can see that you did figure out how to join in V13 already, but .leave() unfortunately is no longer a valid function. Now, to leave a voice channel you must retrieve the connection object that you get from calling joinVoiceChannel(...) and call either .disconnect() or .destroy() on it. They are almost the same, but the latter also makes it so that you cannot use the connection again.
I'm using the following code to decode audio chunks from nodejs's socket
window.AudioContext = window.AudioContext || window.webkitAudioContext;
var context = new AudioContext();
var delayTime = 0;
var init = 0;
var audioStack = [];
var nextTime = 0;
client.on('stream', function(stream, meta){
stream.on('data', function(data) {
context.decodeAudioData(data, function(buffer) {
audioStack.push(buffer);
if ((init!=0) || (audioStack.length > 10)) { // make sure we put at least 10 chunks in the buffer before starting
init++;
scheduleBuffers();
}
}, function(err) {
console.log("err(decodeAudioData): "+err);
});
});
});
function scheduleBuffers() {
while ( audioStack.length) {
var buffer = audioStack.shift();
var source = context.createBufferSource();
source.buffer = buffer;
source.connect(context.destination);
if (nextTime == 0)
nextTime = context.currentTime + 0.05; /// add 50ms latency to work well across systems - tune this if you like
source.start(nextTime);
nextTime+=source.buffer.duration; // Make the next buffer wait the length of the last buffer before being played
};
}
But it has some gaps/glitches between audio chunks that I'm unable to figure out.
I've also read that with MediaSource it's possible to do the same and having the timing handled by the player instead of doing it manually. Can someone provide an example of handling mp3 data?
Moreover, which is the proper way to handle live streaming with web audio API? I've already read almost all questions os SO about this subject and none of them seem to work without glitches. Any ideas?
You can take this code as an example: https://github.com/kmoskwiak/node-tcp-streaming-server
It basically uses media source extensions. All you need to do is to change from video to audio
buffer = mediaSource.addSourceBuffer('audio/mpeg');
yes #Keyne is right,
const mediaSource = new MediaSource()
const sourceBuffer = mediaSource.addSourceBuffer('audio/mpeg')
player.src = URL.createObjectURL(mediaSource)
sourceBuffer.appendBuffer(chunk) // Repeat this for each chunk as ArrayBuffer
player.play()
But do this only if you don't care about IOS support 🤔 (https://developer.mozilla.org/en-US/docs/Web/API/MediaSource#Browser_compatibility)
Otherwise please let me know how you do it !
Good day! I'm into video chat streaming this morning and I've bumped into a problem with the incoming ArrayBuffer which contains binary data of an audio.
Here is the code I found for playing binary audio data (Uint8Array):
function playByteArray(byteArray) {
var arrayBuffer = new ArrayBuffer(byteArray.length);
var bufferView = new Uint8Array(arrayBuffer);
for (i = 0; i < byteArray.length; i++) {
bufferView[i] = byteArray[i];
}
context.decodeAudioData(arrayBuffer, function(buffer) {
buf = buffer;
play();
});
}
// Play the loaded file
function play() {
// Create a source node from the buffer
var source = context.createBufferSource();
source.buffer = buf;
// Connect to the final output node (the speakers)
source.connect(context.destination);
// Play immediately
source.start(0);
}
Now below, I've used MediaStreamRecorder from https://github.com/streamproc/MediaStreamRecorder to record the stream from getUserMedia. This code will continuously send the recorded binary data to the server.
if (navigator.getUserMedia) {
navigator.getUserMedia({audio: true, video: true}, function(stream) {
video.src = (window.URL || window.webkitURL).createObjectURL(stream); //get this for video strewam url
video.muted = true;
multiStreamRecorder = new MultiStreamRecorder(stream);
multiStreamRecorder.canvas = {
width: video.width,
height: video.height
};
multiStreamRecorder.video = video;
multiStreamRecorder.ondataavailable = function(blobs) {
var audioReader = new FileReader();
audioReader.addEventListener("loadend", function() {
var arrBuf = audioReader.result;
var binary = new Uint8Array(arrBuf);
streamToServ.write(binary);
// streamToServ is the binaryjs client
});
audioReader.readAsArrayBuffer(blobs.audio);
};
multiStreamRecorder.start(1);
}, onVideoFail);
} else {
alert ('failed');
}
Convert the blobs produced (audio and video) to binary and send it to binaryjs which will be played on another client with this:
client.on('stream', function (stream, meta) {
stream.on('data', function(data) {
playByteArray(new Uint8Array(data));
});
});
I had no problems with transferring the binary data but the problem is there is a hiccup sound in the playback significantly on every binary data that was played. Is there something wrong on how I play the incoming ArrayBuffers? I'm also thinking of asking streamproc about this.
Thanks in advance!
Cheers.
I found a solution to this problem by making an audio buffer queueing. Most of the code is from here:
Choppy/inaudible playback with chunked audio through Web Audio API
Thanks.
Not sure if this is the problem, but perhaps instead of source.start(0), you should use source.start(time), where time is where you want to start the source. source.start(0) will start playing immediately. If your byte array comes in faster than real-time, the sources might overlap because you start them as soon as possible.
I am trying to build something, where a user can send audio instantly to many people using, socket.io, audioContext, js for the front-end and Node.js,socket.io for the server.
I can record the audio, send it to the server and send it back to other users, but I cannot play the data. I guess it must be a problem of how I send them or how I process the buffer that receives them.
I get the following error: Update!
The buffer passed to decodeAudioData contains an unknown content type.
Audio is passed fine, the buffer is created with no errors but there is no sound feedback.
The User presses record and it started recording/streaming with he following functions:
This is how it all starts:
navigator.getUserMedia({audio: true,video: false}, initializeRecorder, errorCallback);
function initializeRecorder(stream) {
var bufferSize = 2048;
audioCtx = new (window.AudioContext || window.webkitAudioContext)();
var source = audioCtx.createMediaStreamSource(stream);
var recorder = audioCtx.createScriptProcessor(bufferSize, 1, 1);
recorder.onaudioprocess = recorderProcess;
source.connect(recorder);
recorder.connect(audioCtx.destination);
recording = true;
initialized = true;
play = false;
stop = true;
}
function recorderProcess(e) {
var left = e.inputBuffer.getChannelData(0);
socket.emit('audio-blod-send', convertFloat32ToInt16(left));
}
function convertFloat32ToInt16(buffer) {
l = buffer.length;
buf = new Int16Array(l);
while (l--) {
buf[l] = Math.min(1, buffer[l])*0x7FFF;
}
return buf.buffer;
}
Then the server uses the socket to broadcast what the original sender send:
socket.on('audio-blod-send',function(data){
socket.broadcast.to(roomName).emit('audio-blod-receive', data);
});
And then the data are played: Update!
I was using audioContext.decodeData which I found out that it is only used to read/decode audio from MP3 or WAV files not streaming. With the new code no errors appear however there is no Audio feedback.
socket.on('audio-blod-receive',function(data) {
playAudio(data);
});
function playAudio(buffer)
{
var audioCtx;
var started = false;
if(!audioCtx) {
audioCtx = new (window.AudioContext || window.webkitAudioContext)();
}
source = audioCtx.createBufferSource();
audioBuffer = audioCtx.createBuffer( 1, 2048, audioCtx.sampleRate );
audioBuffer.getChannelData( 0 ).set( buffer );
source.buffer = audioBuffer;
source.connect( audioCtx.destination );
source.start(0);
console.log(buffer);
}
P.S If anyone is interested further in what I am trying to do, feel free to contact me.
I am trying to stream audio using node.js + ffmpeg to browsers connected in LAN only using web audio api.
Not using element because it's adding it's own buffer of 8 to 10 secs and I want to get maximum high latency possible (around 1 to 2 sec max).
Audio plays successfully but audio is very choppy and noisy.
Here is my node.js (server side) file:
var ws = require('websocket.io'),
server = ws.listen(3000);
var child_process = require("child_process");
var i = 0;
server.on('connection', function (socket)
{
console.log('New client connected');
var ffmpeg = child_process.spawn("ffmpeg",[
"-re","-i",
"A.mp3","-f",
"f32le",
"pipe:1" // Output to STDOUT
]);
ffmpeg.stdout.on('data', function(data)
{
var buff = new Buffer(data);
socket.send(buff.toString('base64'));
});
});
And here is my HTML:
var audioBuffer = null;
var context = null;
window.addEventListener('load', init, false);
function init() {
try {
context = new webkitAudioContext();
} catch(e) {
alert('Web Audio API is not supported in this browser');
}
}
var ws = new WebSocket("ws://localhost:3000/");
ws.onmessage = function(message)
{
var d1 = base64DecToArr(message.data).buffer;
var d2 = new DataView(d1);
var data = new Float32Array(d2.byteLength / Float32Array.BYTES_PER_ELEMENT);
for (var jj = 0; jj < data.length; ++jj)
{
data[jj] = d2.getFloat32(jj * Float32Array.BYTES_PER_ELEMENT, true);
}
var audioBuffer = context.createBuffer(2, data.length, 44100);
audioBuffer.getChannelData(0).set(data);
var source = context.createBufferSource(); // creates a sound source
source.buffer = audioBuffer;
source.connect(context.destination); // connect the source to the context's destination (the speakers)
source.start(0);
};
Can any one advise what is wrong?
Regards,
Nayan
I got it working !!
All I had to do is adjust the number of channel.
I've set FFMPEG to output mono audio and it worked like a charm. Here is my new FFMOEG command:
var ffmpeg = child_process.spawn("ffmpeg",[
"-re","-i",
"A.mp3",
"-ac","1","-f",
"f32le",
"pipe:1" // Output to STDOUT
]);
You are taking chunks of data, creating separate nodes from them, and starting them based on network timing. For audio to sound correct, the playback of buffers must be without break, and sample-accurate timing. You need to fundamentally change your method.
The way I do this is by creating a ScriptProcessorNode which manages its own buffer of PCM samples. On process, it reads the samples into to the output buffer. This guarantees smooth playback of audio.