We've got a really annoying bug when trying to send mp3 data. We've got the following set up.
Web cam producing aac -> ffmpeg convert to adts -> send to nodejs server -> ffmpeg on server converts adts to mp3 -> mp3 then streamed to browser.
This works *perfectly" on Linux ( chrome with HTML5 and flash, firefox flash only )
However on windows the sound just "stalls", no matter what combination we use ( browser/html5/flash ). If however we shutdown the server the sound then immediately starts to play as we expect.
For some reason on windows based machines it's as if the sound is being buffered "waiting" for something but we don't know what that is.
Any help would be greatly appreciated.
Relevant code in node
res.setHeader('Connection', 'Transfer-Encoding');
res.setHeader('Content-Type', 'audio/mpeg');
res.setHeader('Transfer-Encoding', 'chunked');
res.writeHeader('206');
that.eventEmitter.on('soundData', function (data) {
debug("Got sound data" + data.cameraId + " " + req.params.camera_id);
if (req.params.camera_id == data.cameraId) {
debug("Sending data direct to browser");
res.write(data.sound);
}
});
Code on browser
soundManager.setup({
url: 'http://dashboard.agricamera.co.uk/themes/agricamv2/swf/soundmanager2.swf',
useHTML5Audio: false,
onready: function () {
that.log("Sound manager is now ready")
var mySound = soundManager.createSound({
url: src,
autoLoad: true,
autoPlay: true,
stream: true,
});
}
});
If however we shutdown the server the sound then immediately starts to play as we expect.
For some reason on windows based machines it's as if the sound is being buffered "waiting" for something but we don't know what that is.
That's exactly what's happening.
First off, chrome can play ADTS streams so if possible, just use that directly and save yourself some audio quality by not having to use a second lossy codec in the chain.
Next, don't use soundManager, or at least let it use HTML5 audio. You don't need the Flash fallback these days in most cases, and Chrome is perfectly capable of playing your streams. I suspect this is where your problem lies.
Next, try disabling chunked transfer. Many clients don't like transfer encoding on streams.
Finally, I have seen cases where Chrome's built-in media handling (which I believe varies from OS to OS) cannot sync to the stream. There are a few bug tickets out there for Chromium. If your playback timer isn't incrementing, this is likely your problem and you can simply try to reload the stream programmatically to work around it.
Related
I am trying to build a web-application with the functionality of screen-recording with system audio + headphone-mic audio being captured in the saved video.
I have been thoroughly googling on a solution for this, however my findings show multiple browser solutions where the above works so long as headphones are NOT connected, meaning the microphone input is coming from the system rather than headset.
In the case that you connect headphones, all of these solutions capture the screen without video-audio, and the microphone audio from my headset. So to re-clarify on this, it should have recorded video-audio from the video being played whilst recording, and the headset-mic audio also.
This is thoroughly available in native applications, however I am searching for a way to do this on a browser.
If there are no solutions for this currently that anybody knows of, some insight on the limitations around developing this would also really help, thank you.
Your browser manages the media input being received in the selected tab/window
To receive media input, you need to ensure you have the checkbox Share Audio in the image below checked. However this will only record media-audio being played in your headphones, when it comes to receiving microphone audio, the opposite must be done i.e the checkbox should be unchecked, or merge the microphone audio separately on saving the recorded video
https://slack-files.com/T1JA07M6W-F0297CM7F32-89e7407216
create two const, one retrieving on-screen video, other retrieving audio media:
const DISPLAY_STREAM = await navigator.mediaDevices.getDisplayMedia({video: {cursor: "motion"}, audio: {'echoCancellation': true}}); // retrieving screen-media
const VOICE_STREAM = await navigator.mediaDevices.getUserMedia({ audio: {'echoCancellation': true}, video: false }); // retrieving microphone-media
Use AudioContext to retrieve audio sources from getUserMedia() and getDisplayMedia() separately:
const AUDIO_CONTEXT = new AudioContext();
const MEDIA_AUDIO = AUDIO_CONTEXT.createMediaStreamSource(DISPLAY_STREAM); // passing source of on-screen audio
const MIC_AUDIO = AUDIO_CONTEXT.createMediaStreamSource(VOICE_STREAM); // passing source of microphone audio
Use the method below to create a new audio source which will be used as as the merger or merged version of audio, then passing audios into the merger:
const AUDIO_MERGER = AUDIO_CONTEXT.createMediaStreamDestination(); // audio merger
MEDIA_AUDIO.connect(AUDIO_MERGER); // passing media-audio to merger
MIC_AUDIO.connect(AUDIO_MERGER); // passing microphone-audio to merger
Finally, connect the merged-audio and video together into one array to form a track, and pass it to the MediaStreamer:
const TRACKS = [...DISPLAY_STREAM.getVideoTracks(), ...AUDIO_MERGER.stream.getTracks()] // connecting on-screen video with merged-audio
stream = new MediaStream(TRACKS);
I have a set of animations which I can make in Canvas (fabric.js) or WebGL (three.js). I need to record them automatically, server-side, through a script and output a video file.
The animations include:
Pictures
Videos (with audio)
Other animations/effects
I have researched a lot during last few months on this.
Results
1. Use PhantomJS + FFMPEG
Run HTML Canvas animations on headless browser(PhantomJS) and record with FFMPEG. Here the issue is PhantomJS supports neither WebGL nor Video element. http://phantomjs.org/supported-web-standards.html
2. Use Websockets to send data back to server using DataURL
Here again, we will need to run the animations on browser (which we can't because we have to do everything on server).
3. Use node-canvas
This is a library by TJ Holowaychuk which allows rendering HTML Canvas on Node.js. But it has its own limitations plus I haven't really explored this field much.
(If someone could shed more light on this library)
If anyone has done it before or can guide me somewhere useful.
All we need to do is use some data to create animations and record it into a video, everything on server side.
You can use electron to render WebGL pages with BrowserWindow option "show" set to false and/or use xvfb-run to run headless.
I don't think node-canvas supports the webgl context, so you'll
have to use a library built around 2d drawing, and it certainly
won't have support for any video codecs.
If you can get your animation to work using node-canvas, you can
grab the animated frames at a rate appropriate for your content,
something like this:
Disclosure: I've successfully used FFmpeg to encode a sequence
of externally generated images, but haven't tried the setInterval()
method below. In addition to the animation overhead itself, I don't
know how exporting a canvas to PNG files at 30 FPS would perform.
// assuming "canvas" is asynchronously drawn on some interval
function saveCanvas(canvas, destFile) {
return new Promise((resolve, reject) => {
const ext = path.extname(destFile),
encoder = '.png' === ext ? 'pngStream'
: 'jpegStream';
let writable = fs.createWriteStream(destFile),
readable = canvas[encoder]();
writable
.on('finish', resolve)
.on('error', err => {
let msg = `cannot write "${destFile}": ${err.message}`;
reject(new Error(msg));
});
readable
.on('end', () => writable.end())
.on('error', err => {
let msg = `cannot encode "${destFile}": ${err.message}`;
reject(new Error(msg));
});
readable.pipe(writable);
});
}
const FPS = 30;
let frame = 0,
tasks = [],
interval = setInterval(() => tasks.push(
saveCanvas(canvas, `frame_${frame++}.png`)), 1000 / FPS);
// when animation is done, stop timer
// and wait for images to be written
clearInterval(interval);
Promise.all(tasks).then(encodeVideo);
function encodeVideo() {
// too much code to show here, but basically run FFmpeg
// externally with "-i" option containing "frame_%d.png"
// and "-r" = FPS. If you want to encode to VP9 + WEBM,
// definitely see: http://wiki.webmproject.org/ffmpeg/vp9-encoding-guide
}
And then use FFmpeg to encode a sequence of images into a video.
For the code behind encodeVideo(), you can look at this example.
Edit: There may be an issue with canvas.pngStream() writing
incorrect frames while the animation loop continuously draws on
that one canvas--maybe a copy of the canvas needs to be created
per frame? That would surely create significant memory pressure.
I think that the chromium headless mode might support WebGL already and is another possibility. The video rendering part is yet to come though:
https://bugs.chromium.org/p/chromium/issues/detail?id=781117
CCapture.js makes this pretty easy.
I receive over network PCM audio data stream and this part works fine so I am ending up with
DataReader incomming = args.GetDataReader();
byte[] RcvBuffer = new byte[incomming.UnconsumedBufferLength];
incomming.ReadBytes(RcvBuffer);
I have all audio data in buffer.
How I can play this through telephone Speaker ? Can you point me in some direction ?
Thanks
There're many ways to do that.
You can prepend the WAVE header to your data, and use MediaElement for playback, see the documentation for SetSource method.
If however by “telephone speaker” you mean the earphone, then it is only possible if you are creating a VoIP app.
It took a while but I sorted it, maybe someone else will need help in the future.
First Problem - since I just started app development for Windows Phone I have chosen Blank App (Windows Phone) instead Blank App (Windows Phone Silverlight) and I did not have access to many features that are available in Silverlight projects, so my suggestions for beginners: understand what each project is for.
Like Soonts said there are many ways to do this, this is one that I used.
I simplified this code and retyped this so there can be some typos.
using Microsoft.Xna.Framework.Audio;
using System.IO;
1) Create Stream to load your incoming data:
MemoryStream stream = new MemoryStream();
2) Load data from buffer to stream:
stream.Write(RcvBuffer, 0, RcvBuffer.Length);
3) I am using SoundEfect to play this through Loud-Speaker. Sample rate that I use is 8 kHz
SoundEffect sound;
sound = new SoundEffect(stream.toArray(), 8000, AudioChannels.Mono)
sound.Play();
Currently my audio won't play on safari and on mobile devices.
It works fine on a normal pc on FireFox, Chrome and IE
var manifest = [
{ id: "correct", src: 'assets/correct.mp3|assets/correct.ogg' },
{ id: "wrong", src: 'assets/wrong.mp3|assets/wrong.ogg' }
];
var queue = new createjs.LoadQueue();
queue.installPlugin(createjs.Sound);
queue.loadManifest(manifest, true);
And I'm calling the play function like this;
createjs.Sound.play("correct");
This function is written inside a function that's called when a user presses a div.
That code looks like it should work. Web Audio is initially muted on iOS devices, but when play is called inside of a user event it unmutes.
There are a couple of possibilities (without seeing the rest of the code):
You are working on iPad 1, which does not support web audio and has html audio disabled by default due to severe limitations.
You are not waiting for the audio to finish loading before calling play:
queue.addEventListener("complete", loadComplete);
The audio file path is incorrect and therefore the load is failing, which you can detect by listening for an error event.
You are using a non-default encoding for the mp3 files that is not supported by Safari. Generally that would break in other browsers as well though.
Safari requires quicktime for html audio to play, so that could be a problem.
Using createjs.Sound.registerPlugins, SoundJS is being set to use an unsupported on mobile plugin, such as FlashPlugin. You can check your current plugin with:
createjs.Sound.activePlugin.toString();
You may find the Mobile Safe Tutorial useful. Hope that helps.
There is a way to hack it, play an empty mp3 then play the audio.
It must load the empty mp3 within mainfest array firstly:
var manifest = [
...
{ id: "empty", src: 'assets/empty.mp3|assets/empty.ogg' }
];
...
Before playing the sound, play the empty mp3:
createjs.Sound.play("empty");
createjs.Sound.play("correct");
Background: I'm coding a metro-styled app for Win8. I need to be able to play music-file. Because of quality and space requirements we're using encoded audio (mp3/ogg).
I'm using XAudio2 to play sound effects (.wav files), but since I couldn't figure out a way to play encoded audio with it, I decided to play the music files with Media Foundation (IMFMediaPlayer interface).
I downloaded metro apps sample, and found out that the Media Engine Native C++ video playback sample was closest to what I needed.
Now that my app has MediaPlayer playing musics, I ran into a problem. If the device running the app is slow enough, MediaPlayer hangs. When I'm running the release-version of the app on my device, it's fine and I can hear the music just fine. But when I attach the debugger or run it on a slower device, it hangs when I'm setting bytestream for the MediaPlayer to play.
Here's some code, you'll find it pretty similiar to the sample:
StorageFolder^ installedLocation = Windows::ApplicationModel::Package::Current->InstalledLocation;
m_pickFileTask = Concurrency::task<StorageFile^>(installedLocation->GetFileAsync(filename)), m_tcs.get_token());
auto player = this;
m_pickFileTask.then([player](StorageFile^ fileHandle)
{
player->SetURL(fileHandle->Path);
Concurrency::task<IRandomAccessStream^> fOpenStreamTask = Concurrency::task<IRandomAccessStream^> (fileHandle->OpenAsync(Windows::Storage::FileAccessMode::Read));
fOpenStreamTask.then([player](IRandomAccessStream^ streamHandle)
{
MEDIA::ThrowIfFailed(
player->m_spMediaEngine->Pause()
);
MEDIA::GetMediaError(player->m_spMediaEngine);
player->SetBytestream(streamHandle);
if (player->m_spMediaEngine)
{
MEDIA::ThrowIfFailed(
player->m_spEngineEx->Play()
);
MEDIA::GetMediaError(player->m_spMediaEngine);
}
}
);
}
);
And here's the SetBytestream method:
SetBytestream(IRandomAccessStream^ streamHandle)
{
if(m_spMFByteStream != nullptr)
{
m_spMFByteStream->Close();
m_spMFByteStream = nullptr;
}
MEDIA::ThrowIfFailed(
MFCreateMFByteStreamOnStreamEx((IUnknown*)streamHandle, &m_spMFByteStream)
);
MEDIA::ThrowIfFailed(
m_spEngineEx->SetSourceFromByteStream(m_spMFByteStream.Get(), m_bstrURL)
);
MEDIA::GetMediaError(m_spEngineEx);
return;
}
The line where it hangs is:
m_spEngineEx->SetSourceFromByteStream(m_spMFByteStream.Get(), m_bstrURL)
When I'm debugging the app, I can press pause and see the stack. Well, not much of it, but atleast I can see it that it's indefinitely at
ntdll.dll!77b7f4dc()
Any ideas why my app would hang in such a way?
(OPTIONAL: If you know a better way to play mp3/ogg in a c++ metro-styled app, let me know)
Could not figure out why this is happening, but I managed to code a work-a-round:
IMFSourceReader can be used to decode MP3s and feed bytes into XAudio2SourceVoice.
XAudio2 audio stream effect sample contains good example how to do this.