We're on an ARM64 Snapdragon using a Qualcomm PM8916 codec. The goal is to sample a single channel at 48000 samps/sec, and to use high resolution time stamps. The tinyalsa docs insist that we must use PCM_MMAP. When we do, the pcm_open() does not complain, but pcm_readi() fails errno:22 Invalid argument in the ioctl().
Another symptom that I believe is related is that when pcm_open is called in this form (no PCM_MMAP):
tinyHandle = pcm_open( TINYALSA_CARD, TINYALSA_DEVICE, PCM_IN, &cfg );
Audio flows wonderfully (period=256), and contrary to the docs pcm_get_htimestamp() returns values just fine. Not sure how accurate the timing is, but tomorrow I will inject GPS 1PPS and do my standard test to figure that out.
But when I follow instructions and use PCM_MMAP, eg:
tinyHandle = pcm_open( TINYALSA_CARD, TINYALSA_DEVICE, PCM_IN | PCM_MMAP | PCM_MONOTONIC | PCM_NOIRQ, &cfg );
or
tinyHandle = pcm_open( TINYALSA_CARD, TINYALSA_DEVICE, PCM_IN | PCM_MMAP | PCM_NONBLOCK, &cfg );
The read fails as mentioned above. I haven't tried every possible combination, but that's coming. The odd thing is that the whenever PCM_MMAP is specified, the time stamps that
int err = pcm_get_htimestamp( tinyHandle, &available, ×tamp);
returns become non-sensical, and tv_sec contains values circa 4917 or 5013 instead of a reasonable time_t. I'd be tempted to conclude that the tv_sec values are elapsed time since launch, except that the timestamps happen within seconds of launch.
Oh, one other detail. The PCM_NONBLOCK option was not recognized in the version of tinyalsa supplied with the sdk I have been using. So I git cloned tinyalsa, built a static lib, and upgraded to the latest and greatest. The problem I have with PCM_MMAP persists in both versions. This, coupled with the fact that hours of searching for people with the same problem has been largely a bust, leads me to believe that it is something about our new product and not about the libs.
Any thoughts about where to look next?
Thanks!
Confirmed this morning that the ioctl inside pcm_readi() is receiving the proper file descriptor (4) and period size (256). There are no other arguments that could be invalid
OK, GPS 1PPS test shows timestamp accuracy on the order of 2-3msec, which gets us going "enough" with this usage (no PCM_MMAP):
tinyHandle = pcm_open( TINYALSA_CARD, TINYALSA_DEVICE, PCM_IN, &cfg );
Here is the comment that sent me down the rabbit hole:
/** Returns available frames in pcm buffer and corresponding time stamp.
* The clock is CLOCK_MONOTONIC if flag #ref PCM_MONOTONIC was specified in #ref pcm_open,
* otherwise the clock is CLOCK_REALTIME.
* For an input stream, frames available are frames ready for the application to read.
* For an output stream, frames available are the number of empty frames available for the application to write.
* Only available for PCMs opened with the #ref PCM_MMAP flag.
* #param pcm A PCM handle.
* #param avail The number of available frames
* #param tstamp The timestamp
* #return On success, zero is returned; on failure, negative one.
*/
int pcm_get_htimestamp(struct pcm *pcm, unsigned int *avail,
struct timespec *tstamp)
It is misleading, as it deterred me from trying the pcm_get_htimestamp() call without PCM_MMAP working. There will be further investigations, and if I can post them, I will.
Thank you!
Related
I am working on a project on a Raspberry Pi 3, and I saw this code line :
ioctl(sSpiObj.spiFd, SPI_IOC_MESSAGE(1), &sSpiTransfer);
I don't know what SPI_IOC_MESSAGE(1) does, and I tried to find on the internet its explanation, but I couldn't find it. Is there anyone who could explain what it does?
See the ioctl declaration from here: link
You see you have to pass the request type in the second argument to tell the driver what you want. Your second argument is SPI_IOC_MESSAGE which is a macro defined in Linux code here link. It creates a request type number that incorporates the number of messages you want to send and then this is passed to the driver and then driver decodes the request type and the number of messages from this and process the third argument as messages to be send and finally sends the messages.
From the linux kernel Documents:
* SPI_IOC_MESSAGE gives userspace the equivalent of kernel spi_sync().
* Pass it an array of related transfers, they'll execute together.
* Each transfer may be half duplex (either direction) or full duplex.
*
* struct spi_ioc_transfer mesg[4];
* ...
* status = ioctl(fd, SPI_IOC_MESSAGE(4), mesg);
*
* So for example one transfer might send a nine bit command (right aligned
* in a 16-bit word), the next could read a block of 8-bit data before
* terminating that command by temporarily deselecting the chip; the next
* could send a different nine bit command (re-selecting the chip), and the
* last transfer might write some register values.
I recently designed a Sound recorder on a mac using AudioUnits. It was designed to behave like a video security system, recording continuously, with a graphics display of power levels for playback browsing.
I've noticed that every 85 minutes distortion appears for 3 minutes. After a day of elimination it appears that the sound acquisition that occurs before callback is called uses a circular buffer, and the callback's audioUnitRender function extracts from this buffer but with a slightly slower speed, which eventually causes the internal buffer write to wrap around and catch up with audioUnitRender reads. The duplex operation test shows the latency ever increasing, and after 85 minutes you hear about 200-300ms of latency and the noise begins as the render buffer frame has a combination of buffer segments at end and beginning of buffer, i.e long and short latencies. as the pointers drift apart the noise disappears and you hear clean audio with original short latency, then it repeats again 85 mins later. Even with low impact callback processing this still happens. I've seen some posts regarding latency but none regarding clashes, has anyone seen this?
osx 10.9.5, xcode 6.1.1
code details:-
//modes 1=playback, 2=record, 3=both
AudioComponentDescription outputcd = {0}; // 10.6 version
outputcd.componentType = kAudioUnitType_Output;
outputcd.componentSubType = kAudioUnitSubType_HALOutput; //allows duplex
outputcd.componentManufacturer = kAudioUnitManufacturer_Apple;
AudioComponent comp = AudioComponentFindNext (NULL, &outputcd);
if (comp == NULL) {printf ("can't get output unit");exit (-1);}
CheckError (AudioComponentInstanceNew(comp, au),"Couldn't open component for outputUnit");
//tell input bus that its's input, tell output it's an output
if(mode==1 || mode==3) r=[self setAudioMode:*au :0];//play
if(mode==2 || mode==3) r=[self setAudioMode:*au :1];//rec
// register render callback
if(mode==1 || mode==3) [self setCallBack:*au :0];
if(mode==2 || mode==3) [self setCallBack:*au :1];
// if(mode==2 || mode==3) [self setAllocBuffer:*au];
// get default stream, change amt of channels
AudioStreamBasicDescription audioFormat;
UInt32 k=sizeof(audioFormat);
r= AudioUnitGetProperty(*au,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
1,
&audioFormat,
&k);
audioFormat.mChannelsPerFrame=1;
r= AudioUnitSetProperty(*au,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
1,
&audioFormat,
k);
//start
CheckError (AudioUnitInitialize(outputUnit),"Couldn't initialize output unit");
//record callback
OSStatus RecProc(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData)
{
myView * mv2=(__bridge myView*)inRefCon;
AudioBuffer buffer,buffer2;
OSStatus status;
buffer.mDataByteSize = inNumberFrames *4 ;// buffer size
buffer.mNumberChannels = 1; // one channel
buffer.mData =mv2->rdata;
buffer2.mDataByteSize = inNumberFrames *4 ;// buffer size
buffer2.mNumberChannels = 1; // one channel
buffer2.mData =mv2->rdata2;
AudioBufferList bufferList;
bufferList.mNumberBuffers = 2;
bufferList.mBuffers[0] = buffer;
bufferList.mBuffers[1] = buffer2;
status = AudioUnitRender(mv2->outputUnit, ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, &bufferList);
[mv2 recproc :mv->rdata :mv->rdata2 :inNumberFrames];
return noErr;
}
You seem to be using the HAL output unit for pulling input. There might not be a guarantee that the input device and output device sample rates are exactly locked. Any slow slight drift in the sample rate of either device could eventually cause a buffer underflow or overflow.
One solution might be to find and set an input device for a separate input audio unit instead of depending on the default output unit. Try a USB mic, for instance.
According to this article https://www.native-instruments.com/forum/threads/latency-drift-problem-on-macbook.175551/ this problem appears to be a usb audio driver bug in maverick. I didn't find a kext replacement solution anywhere.
After making a sonar type tester (1 cycle 22khz square wave click every 600 ms to speaker, display selected recorded frame number after click) and could see the 3 to 4 samples drift per second along with the distortion/latency drift reset experience after 1.5 hrs, I decided to look around and find how to access the buffer pointers to stabilise the latency drift, but also no luck.
Also api latency queries show no changes as it drifts.
I did find that you could reset the latency with audiounitstop then audiounitstart (same thread), but it worked only if only one audiounit bus system wide was active. Research also showed that the latency could be reset if you toggle the hardware device sample-rate in Audio Midi Setup. this is a bit aggressive and would be uncomfortable for some.
My design toggled the nominalsamplerate (AudioObjectSetPropertyData with kAudioDevicePropertyNominalSampleRate) every 60 minutes (48000 then back to 44100), with delay by way of waiting for change notification through a callback.
This cause a 2 second void in audio input and output every hour. Safari playing a youtube video would mute, and cause a 1-2 second video freeze during this time . VLC showed the same but video remained smooth during 2 second silence.
Like I said, it wouldn't work for all, but I chose system wide 2 second mute every hour over a recording that has 3 minutes of fuzzy audio every 1.5 hrs. Its been posted that a yosemite upgrade fixes this, although some have also found crackling after going up to yosemite.
I've never worked with bluetooth before. I have to sends data via BLE and I've found the limit of 20 bytes per chunk.
The sender is an Arduino and the receiver could be both an Android or a Node.js app on a pc.
I have to send 9 values, stored in float values, so 4 bytes * 9 = 36 bytes. I need 2 chunks for all my data via BLE. The receiving part needs both chunks to process them. If some data are lost, I don't care.
I'm not expert in network protocols and I think I have to give each message an incremental timestamp so that the receiver can glue the two chunks with the same timestamp or discard the last one if the new timestamp is higher. But I'm not sure how to do a checksum, if I really need it or not, if I really have to care about it, or if - for a simple beta version of my system - I can ignore all those problems..
Does anyone can give me some advice? Like examples of similar situations handled with BLE communication?
You can get around the size limitation using the "Read Blob Request" of ATT. It allows you to read an attribute and also give an offset. So, you can use it to read the attribute with an offset of 0, if there's more than ATT_MTU bytes than you can request again with the offset at ATT_MTU*1, if there's still more ATT_MTU*2, etc... (You can read it in 3.4.4.5 of the Bluetooth v4.1 specifications; it's in the 4.0 spec too but I don't have that in front of me right now)
If the value changes between request, I'm not sure how you could go about detecting such a change. You could have the attribute send notifications when there's a change to interrupt the process in case the value changes in the middle of reading it.
I wish to record the microphone audio stream so I can do realtime DSP on it.
I want to do so without having to use threads and without having .read() block while it waits for new audio data.
UPDATE/ANSWER: It's a bug in Android. 4.2.2 still has the problem, but 5.01 IS FIXED! I'm not sure where the divide is but that's the story.
NOTE: Please don't say "Just use threads." Threads are fine but this isn't about them, and the android developers intended for AudioRecord to be fully usable without me having to specify threads and without me having to deal with blocking read(). Thank you!
Here is what I have found:
When the AudioRecord object is initialized, it creates its own internal ring type buffer.
When .start() is called, it begins recording to said ring buffer (or whatever kind it really is.)
When .read() is called, it reads either half of bufferSize or the specified number of bytes (whichever is less) and then returns.
If there is more than enough audio samples in the internal buffer, then read() returns instantly with the data. If there is not enough yet, then read() waits till there is, then returns with the data.
.setRecordPositionUpdateListener() can be used to set a Listener, and .setPositionNotificationPeriod() and .setNotificationMarkerPosition() can be used to set the notification Period and Position, respectively.
However, the Listener seems to be never called unless certain requirements are met:
1: The Period or Position must be equal to bufferSize/2 or (bufferSize/2)-1.
2: A .read() must be called before the the Period or Position timer starts counting - in other words, after calling .start() then also call .read(), and each time the Listener is called, call .read() again.
3: .read() must read at least half of bufferSize each time.
So using these rules I am able to get the callback/Listener working, but for some reason the reads are still blocking and I can't figure out how to get the Listener to only be called when there is a full read's worth.
If I rig up a button view to click to read, then I can tap it and if tap rapidly, read blocks. But if I wait for the audio buffer to fill, then the first tap is instant (read returns right away) but subsiquent rapid taps are blocked because read() has to wait, I guess.
Greatly appreciated would be any insight on how I might make the Listener work as intended - in such a way that my listener gets called when there's enough data for read() to return instantly.
Below is the relavent parts of my code.
I have some log statements in my code which send strings to logcat which allows me to see how long each command is taking, and this is how I know that read() is blocking.
(And the buttons in my simple test app also are very doggy slow to respond when it is reading repeatedly, but CPU is not pegged.)
Thanks,
~Jesse
In my OnCreate():
bufferSize=AudioRecord.getMinBufferSize(samplerate,AudioFormat.CHANNEL_CONFIGURATION_MONO,AudioFormat.ENCODING_PCM_16BIT)*4;
recorder = new AudioRecord (AudioSource.MIC,samplerate,AudioFormat.CHANNEL_CONFIGURATION_MONO,AudioFormat.ENCODING_PCM_16BIT,bufferSize);
recorder.setRecordPositionUpdateListener(mRecordListener);
recorder.setPositionNotificationPeriod(bufferSize/2);
//recorder.setNotificationMarkerPosition(bufferSize/2);
audioData = new short [bufferSize];
recorder.startRecording();
samplesread=recorder.read(audioData,0,bufferSize);//This triggers it to start doing the callback.
Then here is my listener:
public OnRecordPositionUpdateListener mRecordListener = new OnRecordPositionUpdateListener()
{
public void onPeriodicNotification(AudioRecord recorder) //This one gets called every period.
{
Log.d("TimeTrack", "AAA");
samplesread=recorder.read(audioData,0,bufferSize);
Log.d("TimeTrack", "BBB");
//player.write(audioData, 0, samplesread);
//Log.d("TimeTrack", "CCC");
reads++;
}
#Override
public void onMarkerReached(AudioRecord recorder) //This one gets called only once -- when the marker is reached.
{
Log.d("TimeTrack", "AAA");
samplesread=recorder.read(audioData,0,bufferSize);
Log.d("TimeTrack", "BBB");
//player.write(audioData, 0, samplesread);
//Log.d("TimeTrack", "CCC");
}
};
UPDATE: I have tried this on Android 2.2.3, 2.3.4, and now 4.0.3, and all act the same.
Also: There is an open bug on code.google about it - one entry started in 2012 by someone else then one from 2013 started by me (I didn't know about the first):
UPDATE 2016: Ahhhh finally after years of wondering if it was me or android, I finally have answer! I tried my above code on 4.2.2 and same problem. I tried above code on 5.01, AND IT WORKS!!! And the initial .read() call is NOT needed anymore either. Now, once the .setPositionNotificationPeriod() and .StartRecording() are called, mRecordListener() just magically starts getting called every time there is data available now so it no longer blocks, because the callback is not called until after enough data has been recorded. I haven't listened to the data to know if it's recording correctly, but the callback is happening like it should, and it is not blocking the activity, like it used to!
http://code.google.com/p/android/issues/detail?id=53996
http://code.google.com/p/android/issues/detail?id=25138
If folks who care about this bug log in and vote for and/or comment on the bug maybe it'll get addressed sooner by Google.
It's late answear, but I think I know where Jesse did a mistake. His read call is getting blocked because he is requesting shorts which are sized same as buffer size, but buffer size is in bytes and short contains 2 bytes. If we make short array to be same length as buffer we will read twice as much data.
The solution is to make audioData = new short[bufferSize/2] If the buffer size is 1000 bytes, this way we will request 500 shorts which are 1000 bytes.
Also he should change samplesread=recorder.read(audioData,0,bufferSize) to samplesread=recorder.read(audioData,0,audioData.length)
UPDATE
Ok, Jesse. I can see where another mistake can be - the positionNotificationPeriod. This value have to be large enought so it won't call the listener too often and we need to make sure that when the listener is called the bytes to read are ready to be collected. If bytes won't be ready when the listener is called, the main thread will get blocked by recorder.read(audioData, 0, audioData.length) call until requested bytes get's collected by AudioRecord.
You should calculate buffer size and shorts array length based on time interval you set - how often you want the listener to be called. Position notification period, buffer size and shorts array length all have to be adjusted correctly. Let me show you an example:
int periodInFrames = sampleRate / 10;
int bufferSize = periodInFrames * 1 * 16 / 8;
audioData = new short [bufferSize / 2];
int minBufferSize = AudioRecord.getMinBufferSize(sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);
if (bufferSize < minBufferSize) bufferSize = minBufferSize;
recorder = new AudioRecord(AudioSource.MIC, sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, buffersize);
recorder.setRecordPositionUpdateListener(mRecordListener);
recorder.setPositionNotificationPeriod(periodInFrames);
recorder.startRecording();
public OnRecordPositionUpdateListener mRecordListener = new OnRecordPositionUpdateListener() {
public void onPeriodicNotification(AudioRecord recorder) {
samplesread = recorder.read(audioData, 0, audioData.length);
player.write(short2byte(audioData));
}
};
private byte[] short2byte(short[] data) {
int dataSize = data.length;
byte[] bytes = new byte[dataSize * 2];
for (int i = 0; i < dataSize; i++) {
bytes[i * 2] = (byte) (data[i] & 0x00FF);
bytes[(i * 2) + 1] = (byte) (data[i] >> 8);
data[i] = 0;
}
return bytes;
}
So now a bit of explanation.
First we set how often the listener have to be called to collect audio data (periodInFrames). PositionNotificationPeriod is expressed in frames. Sampling rate is expressed in frames per second, so for 44100 sampling rate we have 44100 frames per second. I divided it by 10 so the listener will be called every 4410 frames = 100 milliseconds - that's reasonable time interval.
Now we calculate buffer size based on our periodInFrames so any data won't be overriden before we collect it. Buffer size is expressed in bytes. Our time interval is 4410 frames, each frame contains 1 byte for mono or 2 bytes for stereo so we multiply it by number of channels (1 in your case). Each channel contains 1 byte for ENCODING_8BIT or 2 bytes for ENCODING_16BIT so we multiply it by bits per sample (16 for ENCODING_16BIT, 8 for ENCODING_8BIT) and divide it by 8.
Then we set audioData length to be half of the bufferSize so we make sure that when the listener gets called, bytes to read are already there waiting to be collected. That's because short contains 2 bytes and bufferSize is expressed in bytes.
Then we check if bufferSize is large enought to succesfully initialize AudioRecord object, if it's not then we set bufferSize to it's minimal size - we don't need to change our time interval or audioData length.
In our listener we read and store data to short array. That's why we use audioData.length instead buffer size, because only audioData.length can tell us the number of shorts the buffer contains.
I had it working some time ago so please let me know if it will work for you.
I'm not sure why you're avoiding spawning separate threads, but if it's because you don't want have to deal with coding them properly, you can use .schedule on a Timer object after each .read, where the time interval is set to the time it takes to get your buffer filled (number of samples in buffer / sampleRate). Yes I know this is using a separate thread, but this advice was given assuming that the reason you were avoiding using threads was to avoid having to code them properly.
This way, the longest time it can possibly block the thread for should be neglible. But I don't know why you'd want to do that.
If the above reason is not why you're avoiding using separate threads, may I ask why?
Also, what exactly do you mean by realtime? Do you intend to playback the affected audio using, let's say, an AudioTrack? Because the latency on most Android devices is pretty bad.
I have been trying to use OMXCodec through Stagefright. I have implemented the code for ICS version of Android.I have two classes CustomDataSource which derives MediaSource and another is CustomOmxCodec which calls OMXCodec::Create method and execute read operation to decode h264 frames. I have tested this implementation on a device with omx.google.video.avc software decoder and it works fine. Now, when I try to run the same implementation on an android phone with hardware h264 decode, it returns error on read call. The error is as below:
[OMX.MTK.VIDEO.DECODER.AVC] ERROR (0x80001005, 0)
0x80001005 is for OMX_ErrorBadParameter.
and I get the error code -1103 on read operation.
I tried various parameters but no success.
The complete log is as below:
[OMX.MTK.VIDEO.DECODER.AVC] mVideoInputErrorRate (0.000000)
!##!>>create tid (21087) O<XCodec mOMXLivesLocally=0, mIsVideoDecoder (1), mIsVideoEncoder (0), mime(video/avc)
[OMX.MTK.VIDEO.DECODER.AVC] video dimensions are 640X480
mSupportesPartialFrames 1 err 0
[OMX.MTK.VIDEO.DECODER.AVC] allocating 10 buffers of size 65536 on input port.
[OMX.MTK.VIDEO.DECODER.AVC] mMemHeapBase = 0x00E8C288, mOutputBufferPoolMemBase=0x51F8E000, size = 9578848
[OMX.MTK.VIDEO.DECODER.AVC] ERROR (0x80001005, 0)
OMXCodec::onEvent--OMX Bad Parameter!!
Read Error : -1103
I'd grateful for any direction on this.
From the question, the hardware codec i.e. OMX.MTK.VIDEO.DECODER.AVC is not supporting one of the parameters being passed as part of the configuration steps.
From OMXCodec::create, configureCodec will be invoked which internally invokes a lot of other functions. Since the error is coming as part of OMXCodec::onEvent, one of the possible scenarios could be that the component encountered an error while decoding the first few bytes of the first frame.
Specifically, when the component encounters SPS and PPS (part of codec specific data), the component would typically trigger a portSettingsChanged. From your response, I feel that during this process, there is some error and hence, onEvent has been triggered.
Please share more logs to analyze further.
The MTK H264 decoder need the parameter csd-0 and csd-1 to init the decoder(You can get some information at http://developer.android.com/reference/android/media/MediaCodec.html). csd-0 and csd-1 stands for SPS and PPS of H264.I have asked a MTK engineer and he said that we can use the code below to set these two parameters.
byte[] sps = {0,0,0,1,103,100,0,40,-84,52,-59,1,-32,17,31,120,11,80,16,16,31
,0,0,3,3,-23,0,0,-22,96,-108};
byte[] pps = {0,0,0,1,104,-18,60,-128};
MediaFormat mFormat = MediaFormat.createVideoFormat("video/avc", width, height);
mFormat.setByteBuffer("csd-0", ByteBuffer.wrap(sps));
mFormat.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
Maybe that's whay we got the OMX Bad Parameter error message.
From the logs and mapping the same to the implemented code, I feel that the following is happening
[OMX.MTK.VIDEO.DECODER.AVC] allocating 10 buffers of size 65536 on input port.
This step allocates the buffers on the input port of the decoder
From the flow of code, after input port buffers are allocated, the buffers on output port are allocated from nativeWindow through allocateOutputBuffersFromNativeWindow.
One of the steps as part of this method implementation is to increase the number of buffers on the output port by 2 and set the same to the OMX component as shown here.
I feel your error might be stemming from this specific point as nBufferSize is a read-only parameter of OMX_IndexParamPortDefinition index. Please refer to the OMX Standard, section 3.1.3.12.1, page 83, which clearly shows that nBufferSize is a read-only parameter.
It appears that your OMX component may be strictly OMX compliant component, whereas in Android from ICS onwards, certain deviations are expected. This could be one of the potential causes of your error.
P.S: If you could share more information, we could help further.