Using of media foundation Sink Writer IMFSinkWriter for recording H.264 & AAC encoded file was cause huge memory leak at high resolution like a 1920x1080.
Why?
The initial cause of memory leak was MF_SINK_WRITER_DISABLE_THROTTLING attribute at TRUE state setting for IMFAttributes interface.
Related
I am working on RTSP live Streaming. I am getting live stream on my android App using exoplayer RTSP stream player. But latency of that streaming is about 3 seconds. As latency on vlc media player is 1 second. so how to reduce latency in exoplayer. Is there any way please tell me
What you're facing is buffering latency. VLC uses its own engine and buffering algorithms. However, if you wanna reduce buffering latency on ExoPlayer, you gotta get yourself familiarized with LoadControl. ExoPlayer uses a DefaultLoadControl in default instantiation. This LoadControl is a class that belongs to the ExoPlayer library, and it determines the values of time durations the player should spend in order to buffer the stream. If you wanna reduce the delay, you gotta reduce the LoadControl buffer values.
Here's a quick snippet of creating a SimpleExoPlayer instance with a custom load control :
val customLoadControl = DefaultLoadControl.Builder()
.setBufferDurationsMs(minBuffer, maxBuffer, playbackBuffer, playbackRebuffer)
.build()
Parameters in a nutshell : minBuffer is the minimum buffered video duration, maxBuffer is the maximum buffered video duration, playbackBuffer is the required buffered video duration in order to start playing , playbackRebuffer is the required buffered video duration in case it failed and it retries.
In your case, you should set your values really low, especially the playbackBuffer and minBuffer parameters. You should mess around with small values (they are in milliseconds). Going with a value of 1000 for both minBuffer and playbackBuffer
How to use the custom load control : After building the custom load control instance and storing it in a variable, you should pass it when you're building your SimpleExoPlayer :
myMediaPlayer = SimpleExoPlayer.Builder(this#MainActivity)
.setLoadControl(customLoadControl)
.build()
What to expect:
Using the default values is always recommended. If you mess with the values, the app may crash, the stream may be stuck, or the player may get very glitchy. So manipulate the values intelligently.
Here is the javadocs DefaultLoadControl Javadocs
If you don't know what buffering is, exoplayer (or any other player) may need to buffer (load the upcoming portion of the video/audio and store it in memory, rendered, way faster to access and reduces playback problems. Streamed media however needs buffering because it comes in form of chunks. So each chunk that arrives, will eventually be buffered. If you set the required buffered duration to 1000, you are telling ExoPlayer that the first chunk of stream that arrives whose length is 1000 millisecond should be buffered and played right away. I believe there is no simpler way to explain this. Best of luck.
I have a 5MP 2592×1944 pixel camera that writes to GPU via DMA or to a YUV4:2:2 data file.
As this is quite non standard nearly all camera viewing applications can not handle this data and I would only be able to analyze the data after recording. I had the idea of setting up a ring buffer in /dev/ that is 2 frames large (2592x1944x1.5x2) 14763kb, and dump the video into that buffer. I could then access it like a standard video stream with video4linux programs and save/modify/resize it all I want to a new format.
The problem is how to setup a ring buffer and get the video to stream to it instead of yuv_file.data
Any sites that would show me how to set something like this up, or is this not even possible?
I want to capture audio on Linux with low latency in a program I'm writing.
I've run some experiments using the ALSA API, using snd_pcm_readi() to
capture sound, then immediately using snd_pcm_writei() to play it back.
I've tried playing with the number of frames captured, and the buffer size,
but I don't seem to be able to get the latency down to less than a second
or so.
Am I better off using PulseAudio or JACK? Can those be used to play the
captured audio?
To reduce capture latency, reduce the period size of the capture device.
To reduce playback latency, reduce the buffer size of the playback device.
Jack can play the captured audio (just connect the input ports to the output ports), but you still have to configure its periods/buffers.
Also see Relation between period size of speaker and mic and Recording from ALSA - understanding memory mapping.
I've doing some work on low latency audio programming,
My experience is, first, your capture buffer should be small, like 10ms period buffer. (let's assuming you're using 512 frame buffer, and 48000 sample rate).
Then, you should config your Output device start_threshold to at least 2 * frame size ( 1 * frame size if your don't have much process of recorded data).
For record device, like CL. said, use a relative small period size is better, but not too small to avoid too much irq.
Also, you can change your process schedule to FIFO schedule.
Then, hopefully, you will get about 20ms total latency.
I believe you should at first ensure that you are running a Linux kernel which actually allows you to achieve low typical latency.
There are several kernel compile-time configuration options which you might look into:
CONFIG_HZ_1000
CONFIG_IRQ_FORCED_THREADING
CONFIG_PREEMPT
CONFIG_PREEMPT_RT_FULL (available only with RT patch)
Apart from that, there are more things you can do in order to optimize your audio latency in Linux. Some starting reference points can be found there:
http://wiki.linuxaudio.org/wiki/real_time_info
I have an application concept that required real-time audio signal processing that can be broadly described as: a) sampling incoming audio (from microphone), b) performing signal processing functions (such as filtering, fourier transform, filtering and manipulation, inverse fourier transform) c) play-out (via speaker jack)
I believe that the "end to end" round trip timing (a) to (c) would need be in the order of 2 to 5 ms for the application to work in the real-world.
So, my question is this possible on today's generation of iphones and android phones?
On iOS, it is possible, but not guaranteed. I have managed to get ~6ms (22050 sampling rate, 128 samples buffer size) in my iOS app which does real-time processing of speech input. Take a look at Novocaine (https://github.com/alexbw/novocaine) - which provides a nice encapsulation of Audio Units and makes programming easier.
However, keep in mind that even if you request a particular buffer size, at run time iOS may decide to send larger buffers at longer intervals (=> higher latency) based on resource constraints. For example, if you have requested a buffer size of 128 (~6ms), you MAY end up getting 256 size buffers at 12ms instead. Your app has to take this into account and handle the buffer accordingly.
On Android, unfortunately, low-latency round-trip audio is a much bigger problem. This is because latency is driven by a host of device/manufacturer driven factors like hardware/driver level buffers and these vary from device to device. You can find a discussion of this long-standing Android handicap here: https://code.google.com/p/android/issues/detail?id=3434
My suggestion would be to ignore Android for now, and implement/validate your signal processing algorithms on an iOS device. Later, you can consider porting them to Android.
Has anyone tried to change the audio frame size in WebRTC? It uses 10ms frames. I need a different size, and the code doesn't look promising...
Fundamentally there's no reason you can't use non-10ms frames, but much of the code is written with that assumption. It would indeed likely be a serious undertaking to change it.
On the device you can use other audio frame (e.g. 20，40ms). For the codec，you can use other audio frame because codec has a audio buffer. I used silk codec and ios device.