I have been trying to use OMXCodec through Stagefright. I have implemented the code for ICS version of Android.I have two classes CustomDataSource which derives MediaSource and another is CustomOmxCodec which calls OMXCodec::Create method and execute read operation to decode h264 frames. I have tested this implementation on a device with omx.google.video.avc software decoder and it works fine. Now, when I try to run the same implementation on an android phone with hardware h264 decode, it returns error on read call. The error is as below:
[OMX.MTK.VIDEO.DECODER.AVC] ERROR (0x80001005, 0)
0x80001005 is for OMX_ErrorBadParameter.
and I get the error code -1103 on read operation.
I tried various parameters but no success.
The complete log is as below:
[OMX.MTK.VIDEO.DECODER.AVC] mVideoInputErrorRate (0.000000)
!##!>>create tid (21087) O<XCodec mOMXLivesLocally=0, mIsVideoDecoder (1), mIsVideoEncoder (0), mime(video/avc)
[OMX.MTK.VIDEO.DECODER.AVC] video dimensions are 640X480
mSupportesPartialFrames 1 err 0
[OMX.MTK.VIDEO.DECODER.AVC] allocating 10 buffers of size 65536 on input port.
[OMX.MTK.VIDEO.DECODER.AVC] mMemHeapBase = 0x00E8C288, mOutputBufferPoolMemBase=0x51F8E000, size = 9578848
[OMX.MTK.VIDEO.DECODER.AVC] ERROR (0x80001005, 0)
OMXCodec::onEvent--OMX Bad Parameter!!
Read Error : -1103
I'd grateful for any direction on this.
From the question, the hardware codec i.e. OMX.MTK.VIDEO.DECODER.AVC is not supporting one of the parameters being passed as part of the configuration steps.
From OMXCodec::create, configureCodec will be invoked which internally invokes a lot of other functions. Since the error is coming as part of OMXCodec::onEvent, one of the possible scenarios could be that the component encountered an error while decoding the first few bytes of the first frame.
Specifically, when the component encounters SPS and PPS (part of codec specific data), the component would typically trigger a portSettingsChanged. From your response, I feel that during this process, there is some error and hence, onEvent has been triggered.
Please share more logs to analyze further.
The MTK H264 decoder need the parameter csd-0 and csd-1 to init the decoder(You can get some information at http://developer.android.com/reference/android/media/MediaCodec.html). csd-0 and csd-1 stands for SPS and PPS of H264.I have asked a MTK engineer and he said that we can use the code below to set these two parameters.
byte[] sps = {0,0,0,1,103,100,0,40,-84,52,-59,1,-32,17,31,120,11,80,16,16,31
,0,0,3,3,-23,0,0,-22,96,-108};
byte[] pps = {0,0,0,1,104,-18,60,-128};
MediaFormat mFormat = MediaFormat.createVideoFormat("video/avc", width, height);
mFormat.setByteBuffer("csd-0", ByteBuffer.wrap(sps));
mFormat.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
Maybe that's whay we got the OMX Bad Parameter error message.
From the logs and mapping the same to the implemented code, I feel that the following is happening
[OMX.MTK.VIDEO.DECODER.AVC] allocating 10 buffers of size 65536 on input port.
This step allocates the buffers on the input port of the decoder
From the flow of code, after input port buffers are allocated, the buffers on output port are allocated from nativeWindow through allocateOutputBuffersFromNativeWindow.
One of the steps as part of this method implementation is to increase the number of buffers on the output port by 2 and set the same to the OMX component as shown here.
I feel your error might be stemming from this specific point as nBufferSize is a read-only parameter of OMX_IndexParamPortDefinition index. Please refer to the OMX Standard, section 3.1.3.12.1, page 83, which clearly shows that nBufferSize is a read-only parameter.
It appears that your OMX component may be strictly OMX compliant component, whereas in Android from ICS onwards, certain deviations are expected. This could be one of the potential causes of your error.
P.S: If you could share more information, we could help further.
Related
I'm trying to optimise a piece of software for playing video, which internally uses the FFmpeg libraries for decoding. We've found that on some large (4K, 60fps) video, it sometimes takes longer to decode a frame than that frame should be displayed for; sadly, because of the problem domain, simply buffering/skipping frames is not an option.
However, it appears that the FFmpeg executable is able to decode the video in question fine, at about 2x speed, so I've been trying to work out what we're doing wrong.
I've written a very stripped-back decoder program for testing; the source is here (it's about 200 lines). From profiling it, it appears that the one major bottleneck during decoding is the avcodec_send_packet() function, which can take up to 50ms per call. However, measuring the same call in FFmpeg shows strange behaviour:
(these are the times taken for each call to avcodec_send_packet() in milliseconds, when decoding a 4K 25fps VP9-encoded video.)
Basically, it seems that when FFmpeg uses this function, it only really takes any amount of time to complete every N calls, where N is the number of threads being used for decoding. However, both my test decoder and the actual product use 4 threads for decoding, and this doesn't happen; when using frame-based threading, the test decoder behaves like FFmpeg using only 1 thread. This would seem to indicate that we're not using multithreading at all, but we've still seen performance improvements by using more threads.
FFmpeg's results average out to being about twice as fast overall as our decoders, so clearly we're doing something wrong. I've been reading through FFmpeg's source to try to find any clues, but it's so far eluded me.
My question is: what's FFmpeg doing here that we're not? Alternatively, how can we increase the performance of our decoder?
Any help is greatly appreciated.
I was facing the same problem. It took me quite a while to figure out a solution which I want to share here for future references:
Enable multithreading for the decoder. Per default the decoder only uses one thread, depending on the decoder, multithreading can speed up decoding drastically.
Assuming you have AVFormatContext *format_ctx, a matching codec AVCodec* codec and AVCodecContext* codec_ctx (allocated using avcodec_alloc_context3).
Before opening the codec context (using avcodec_open2) you can configure multithreading. Check the capabilites of the codec in order to decide which kind of multithreading you can use:
// set codec to automatically determine how many threads suits best for the decoding job
codec_ctx->thread_count = 0;
if (codec->capabilities | AV_CODEC_CAP_FRAME_THREADS)
codec_ctx->thread_type = FF_THREAD_FRAME;
else if (codec->capabilities | AV_CODEC_CAP_SLICE_THREADS)
codec_ctx->thread_type = FF_THREAD_SLICE;
else
codec_ctx->thread_count = 1; //don't use multithreading
Another speed-up I found out is the following: keep sending packets to the decoder (thats what avcodec_send_packet() is doing) until you get AVERROR(EAGAIN) as return value. This means the internal decoder buffers are full and you first need to collect the decoded frames (but remember to send this last packet again after the decoder is empty again). Now you can collect the decoded frames using avcodec_receive_frame until you get AVERROR(EAGAIN) again.
Some decoders work way faster when they have mutiple frames queued for decoding (thats what the decoder does when codec_ctx->thread_type = FF_THREAD_FRAME is set).
avcodec_send_packet() and avcodec_receive_frame() are wrapper functions most important thing those do is calling selected codec's decode function and returns decoded frame (or error).
Try tweaking the codec options, for example, low latency may not give you what you want. And sometimes old api (I believe it still around) avcodec_decode_video2() outperforms newer one, you may try that too.
I have a very interesting problem.
I am running custom movie player based on NDK/C++/CMake toolchain that opens streaming URL (mp4, H.264 & stereo audio). In order to restart from given position, player opens stream, buffers frames to some length and then seeks to new position and start decoding and playing. This works fine all the times except if we power-cycle the device and follow the same steps.
This was reproduced on few version of the software (plugin build against android-22..26) and hardware (LG G6, G5 and LeEco). This issue does not happen if you keep app open for 10 mins.
I am looking for possible areas of concern. I have played with decode logic (it is based on the approach described as synchronous processing using buffers).
Edit - More Information (4/23)
I modified player to pick a stream and then played only video instead of video+audio. This resulted in constant starvation resulting in buffering. This appears to have changed across android version (no fix data here). I do believe that I am running into decoder starvation. Previously, I had set timeouts of 0 for both AMediaCodec_dequeueInputBuffer and AMediaCodec_dequeueOutputBuffer, which I changed on input side to 1000 and 10000 but does not make much difference.
My player is based on NDK/C++ interface to MediaCodec, CMake build passes -DANDROID_ABI="armeabi-v7a with NEON" and -DANDROID_NATIVE_API_LEVEL="android-22" \ and C++_static.
Anyone can share what timeouts they have used and found success with it or anything that would help avoid starvation or resulting buffering?
This is solved for now. Starvation was not caused from decoding perspective but images were consumed in faster pace as clock value returned were not in sync. I was using clock_gettime method with CLOCK_MONOTONIC clock id, which is recommended way but it was always faster for first 5-10 mins of restarting device. This device only had Wi-Fi connection. Changing clock id to CLOCK_REALTIME ensures correct presentation of images and no starvation.
I am trying to figure out how to adjust the volume level of a PCM audio stream in node.
I have looked all over npmjs.org at all of the modules that I could find for working with audio, but haven't found anything that will take a stream in, change the volume, and give me a stream out.
Are there any modules that exist that can do this, perhaps even even something that wasn't made specifically for it?
If not, then I could create a module, if someone can point me in the right direction for modifying a stream byte by byte.
Here is what I am trying to accomplish:
I am writing a program to receive several PCM audio streams, and mix them for several outputs with varying volume levels. Example:
inputs vol output
music 25% output 1
live audio 80% output 1
microphone 0% output 1
music 100% output 2
live audio 0% output 2
microphone 0% output 2
What type of connection are you using? (Would make it easier to give example code)
What you basically want to do, is create a connection. Then on the connection or request object add a listener for the 'data' event. If you don't set an encoding, the data parameter on the callback should be a Buffer. The data event is triggered after each chunk is delivered through the network.
The Buffer gives you byte-size access to the data-stream using regular javascript number values. You can then parse that chunk, keep them in memory over multiple data-events using a closure (in order to buffer multiple chunks). And when appropriate write the parsed and processed data to a socket (another socket or the same in case of bi-directional sockets). Don't forget to manage your closure in order to avoid memory leaks!
This is just an abstract description. Let me know if anything needs clarification.
I just have this issue with reading from a network stream in C#. Since am more of a Java developer I came across this issue.
In java I have this option of knowing the length of the received packet using the following code
int length = dataiInputStream.read(rcvPacket);
eventhough the size of the byte array rcvPacket assigned is larger than the amount of elements contained in it. which will allow me to read only the required length of elements so that i do not have elements in the byte array containing zeros.
While i was trying to use a similar thing on C# which was
long len = networkStream.length;
but the docucmentation says that this property is not supported. is there a workaround for this?
Thank you
The Java code doesn't show you the length of a packet. It shows you the amount of data read within that read call, which could have come from multiple packets.
NetworkStream.Read returns the same information, except using 0 to indicate the end of the stream instead of -1.
My code using NAudio to read one particular MP3 gets different results than several other commercial apps.
Specifically: My NAudio-based code finds ~1.4 sec of silence at the beginning of this MP3 before "audible audio" (a drum pickup) starts, whereas other apps (Windows Media Player, RealPlayer, WavePad) show ~2.5 sec of silence before that same drum pickup.
The particular MP3 is "Like A Rolling Stone" downloaded from Amazon.com. Tested several other MP3s and none show any similar difference between my code and other apps. Most MP3s don't start with such a long silence so I suspect that's the source of the difference.
Debugging problems:
I can't actually find a way to even prove that the other apps are right and NAudio/me is wrong, i.e. to compare block-by-block my code's results to a "known good reference implementation"; therefore I can't even precisely define the "error" I need to debug.
Since my code reads thousands of samples during those 1.4 sec with no obvious errors, I can't think how to narrow down where/when in the input stream to look for a bug.
The heart of the NAudio code is a P/Invoke call to acmStreamConvert(), which is a Windows "black box" call which I can't think how to error-check.
Can anyone think of any tricks/techniques to debug this?
The NAudio ACM code was never originally intended for MP3s, but for decoding constant bit rate telephony codecs. One day I tried setting up the WaveFormat to specify MP3 as an experiment, and what came out sounded good enough. However, I have always felt a bit nervous about decoding MP3s (especially VBR) with ACM (e.g. what comes out if ID3 tags or album art get passed in - could that account for extra silence?), and I've never been 100% convinced that NAudio does it right - there is very little documentation on how exactly you are supposed to use the ACM codecs. Sadly there is no managed MP3 decoder with a license I can use in NAudio, so ACM remains the only option for the time being.
I'm not sure what approach other media players take to playing back MP3, but I suspect many of them have their own built-in MP3 decoders, rather than relying on the operating system.
I've found some partial answers to my own Q:
Since my problem boils down to consuming too much MP3 w/o producing enough PCM, I used conditional-on-hit-count breakpoints to find just where this was happening, then drilled into that.
This showed me that some acmStreamConvert() calls are returning success, consuming 417 src bytes, but producing 0 "dest bytes used".
Next I plan to try acmStreamSize() to ask the codec how many src bytes it "wants" to consume, rather than "telling" it to consume 417.
Edit (followup): I fixed it!
It came down to passing acmStreamConvert() enough src bytes to make it happy. Giving it its acmStreamSize() requested size fixed the problem in some places but then it popped up in others; giving it its requested size times 3 seems to cure the "0 dest bytes used" result in all MP3s I've tested.
With this fix, acmStreamConvert() then sometimes returned much larger converted chunks (almost 32 KB), so I also had to modify some other NAudio code to pass in larger destination buffers to hold the results.