I have been writing Spotify support for a project using libspotify, and when I wanted to implement seeking, I noticed that there is apparently no function to get the current playback position. In other words, a counterpart to sp_session_player_seek(), which returns the current offset.
What some people seem to do is to save the offset used in the last seek() call, and then accumulate the number of frames in a counter in music_delivery. Together with the stored offset, the current position can be calculated that way, yes - but is this really the only way how to do it?
I know that the Spotify Web API has a way to get the current position, so it is strange that libspotify doesn't have one.
Keeping track of it yourself the way to do it.
The way to look at it is this: libspotify doesn't actually play music, so it can't possibly know the current position.
All libspotify does it pass you PCM data. It's the application's job to get that out to the speakers, and audio pipelines invariably have various buffers and latencies and whatnot in place. When you seek, you're not saying "Move playback to here", you're saying "start giving me PCM from this point in the track".
In fact, if you're just counting frames and then passing them to an audio pipeline, your position is likely incorrect if it doesn't take into account that pipeline's buffers etc.
You can always track the position yourself.
For example:
void SpSeek(int position)
{
sp_session_player_seek(mSession, position);
mMsPosition = position;
}
int OnSessionMusicDelivery(sp_session *session, const sp_audioformat *format, const void *frames, int numFrames)
{
return SendToAudioDriver(...)
}
In my case i'm using OpenSL (Android), each time the buffer finish i update the value of the position like this
mMsPosition += (frameCount * 1000) / UtPlayerGetSampleRate();
Where numFrames is the frames consumed by the driver.
Related
I am working with an application that uses OpenAL API quite extensively. In particular, there are multiple sound sources, non-trivial listener filters, etc.
I want to be able to run this application significantly faster than real-time. At the same time, the sound must be saved for later postprocessing. Is there a way to access the OpenAL output programmatically (virtually) without ever playing the sound on the real playback device?
Ideally, I'd like to have access that would be played during every tick of the main loop of my application. Normally one tick corresponds to one rendered frame (e.g. 1/30th of a second). But in this case we would be running the app as fast as possible.
We ended up using OpenAL Soft to do this. Example:
#include "alext.h"
LPALCLOOPBACKOPENDEVICESOFT alcLoopbackOpenDeviceSOFT;
alcLoopbackOpenDeviceSOFT = alcGetProcAddress(NULL,"alcLoopbackOpenDeviceSOFT");
replace your default device with this device
ALCcontext *context = alcCreateContext(device, attrs);
Set the attrs as you would for your default device
Then in the main loop use:
LPALCRENDERSAMPLESSOFT alcRenderSamplesSOFT;
alcRenderSamplesSOFT = alcGetProcAddress(NULL, "alcRenderSamplesSOFT");
alcRenderSamplesSOFT(device, buffer, 1024);
Here the buffer will store 1024 samples. This code runs faster than real-time, therefore you can sample frames every tick
Are you able to do your required functions with the audio data prior to its being shipped to OpenAL? I've done a lot with javax.sound.sampled when it is untethered by the blocking write() method in SourceDataLine, especially when saving to file rather than playing back.
From what little I know about OpenAL, there is also a blocking process occurs when data is shipped, with a queue of arrays that are managed. I've been meaning to look into this further...
(Probably not being very helpful here. Apologies.)
The example code for creating a version 3 AudioUnit demonstrates how the implementation needs to return a function block for rendering processing. The block will both get samples from the previous
AxudioUnit in the chain via pullInputBlock and supply the output buffers with the processed samples. It also must provide some output buffers if the unit further down the chain did not. Here is an excerpt of code from an AudioUnit subclass:
- (AUInternalRenderBlock)internalRenderBlock {
/*
Capture in locals to avoid ObjC member lookups.
*/
// Specify captured objects are mutable.
__block FilterDSPKernel *state = &_kernel;
__block BufferedInputBus *input = &_inputBus;
return Block_copy(^AUAudioUnitStatus(
AudioUnitRenderActionFlags *actionFlags,
const AudioTimeStamp *timestamp,
AVAudioFrameCount frameCount,
NSInteger outputBusNumber,
AudioBufferList *outputData,
const AURenderEvent *realtimeEventListHead,
AURenderPullInputBlock pullInputBlock) {
...
});
This is fine if the processing does not require knowing the frameCount before the call to this block, but many applications do require knowing the frameCount before this block in order to allocate memory, prepare processing parameters, etc.
One way around this would be to accumulate past buffers of output, outputting only frameCount samples each call to the block, but this only works if there is known minimum frameCount. The processing must be initialized with a size greater than this frame count in order to work. Is there a way to specify or obtain a minimum value for frameCount or force it to be a specific value?
The example code is taken from: https://github.com/WildDylan/appleSample/blob/master/AudioUnitV3ExampleABasicAudioUnitExtensionandHostImplementation/FilterDemoFramework/FilterDemo.mm
Under iOS, an audio unit callback must be able to handle variable frameCounts. You can't force it to be a constant.
Thus any processing that requires a fixed size buffer should be done outside the audio unit callback. You can pass data to a processing thread by using a lock-free circular buffer/fifo or similar structure that does not require memory management in the callback.
You can suggest that the frameCount be a certain size by setting a buffer duration using the AVAudioSession APIs. But the OS is free to ignore this, depending on other audio needs in the system (power saving modes, system sounds, etc.) In my experience, the audio driver will only increase your suggested size, not decrease it (by more than a couple samples if resampling to not a power of 2).
I'm making a game for a school project and I have a sound effect that is supposed to play whenever a laser is fired. There was a brief period of time when it worked fine, but it has since stopped. After it stopped I changed the code a bit as I wanted to store the file in a datafile.
Initializing sound in Allegro
install_sound(DIGI_AUTODETECT, MIDI_AUTODETECT, NULL);
This is the code for loading and playing the sound
//Loading sound file from datafile
DATAFILE *laserShot = NULL;
laserShot = load_datafile_object("asteroids.dat", "laser_Shot");
//Error checking
if (laserShot->dat == NULL) {
allegro_message("Error loading laser_Shot.wav");
}
else {
//Playing sound for shot
play_sample((SAMPLE*) laserShot->dat, 255, 127, 1000, 0);
}
//Freeing memory
unload_datafile_object(laserShot);
The sound itself is very short if that is of any importance, less than a second.
The sound would also be trying to play multiple times in quick succession, but there's actually more of a break now than when it was originally working so I don't think that makes a difference.
Is there something I'm getting blatantly wrong?
At first, make sure all parameters are set, which aren't if you call just install_sound. You should also call this:
set_config_int("sound", "quality", 1);
Third parameter refers to used quality of sound. This should mean highest quality, if you want another type, you should search in allegro libs reference.
Second, you should allocate voice. Voice is basically space in memory for playing samples. By default, allegro 4 can allocate 255 different voices, but real number can be far more less because of hardware. You do it like this:
int laser_voice = allocate_voice("sample.wav");
Now you can set parameters, like volume, pan, sweep and playmode. For example, if you want to play looped sample in same frequency and volume like source, you should do this:
voice_set_volume( laser_voice, 200);
voice_set_pan( laser_voice, 127);
voice_set_playmode( laser_voice, PLAYMODE_LOOP);
For other options, you should visit references.
Now, to play sample, you just call
voice_start(laser_voice);
Then you can stop it, replay it, change parameters or change sample by reallocate_voice. That's all. At end of code, you deallocate it by
deallocate_voice(laser_voice);
Turns out I was just making a stupid mistake, I was calling the unloading function in the same function that I was playing the sound file; there was not enough time to play the sound file before the file was unloaded so while technically there was no error to be picked up by the compiler or to cause a crash, the code was trying to play a sound it had already forgotten. Removing the call for unloading allows the sound to play.
I recently designed a Sound recorder on a mac using AudioUnits. It was designed to behave like a video security system, recording continuously, with a graphics display of power levels for playback browsing.
I've noticed that every 85 minutes distortion appears for 3 minutes. After a day of elimination it appears that the sound acquisition that occurs before callback is called uses a circular buffer, and the callback's audioUnitRender function extracts from this buffer but with a slightly slower speed, which eventually causes the internal buffer write to wrap around and catch up with audioUnitRender reads. The duplex operation test shows the latency ever increasing, and after 85 minutes you hear about 200-300ms of latency and the noise begins as the render buffer frame has a combination of buffer segments at end and beginning of buffer, i.e long and short latencies. as the pointers drift apart the noise disappears and you hear clean audio with original short latency, then it repeats again 85 mins later. Even with low impact callback processing this still happens. I've seen some posts regarding latency but none regarding clashes, has anyone seen this?
osx 10.9.5, xcode 6.1.1
code details:-
//modes 1=playback, 2=record, 3=both
AudioComponentDescription outputcd = {0}; // 10.6 version
outputcd.componentType = kAudioUnitType_Output;
outputcd.componentSubType = kAudioUnitSubType_HALOutput; //allows duplex
outputcd.componentManufacturer = kAudioUnitManufacturer_Apple;
AudioComponent comp = AudioComponentFindNext (NULL, &outputcd);
if (comp == NULL) {printf ("can't get output unit");exit (-1);}
CheckError (AudioComponentInstanceNew(comp, au),"Couldn't open component for outputUnit");
//tell input bus that its's input, tell output it's an output
if(mode==1 || mode==3) r=[self setAudioMode:*au :0];//play
if(mode==2 || mode==3) r=[self setAudioMode:*au :1];//rec
// register render callback
if(mode==1 || mode==3) [self setCallBack:*au :0];
if(mode==2 || mode==3) [self setCallBack:*au :1];
// if(mode==2 || mode==3) [self setAllocBuffer:*au];
// get default stream, change amt of channels
AudioStreamBasicDescription audioFormat;
UInt32 k=sizeof(audioFormat);
r= AudioUnitGetProperty(*au,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
1,
&audioFormat,
&k);
audioFormat.mChannelsPerFrame=1;
r= AudioUnitSetProperty(*au,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Output,
1,
&audioFormat,
k);
//start
CheckError (AudioUnitInitialize(outputUnit),"Couldn't initialize output unit");
//record callback
OSStatus RecProc(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList * ioData)
{
myView * mv2=(__bridge myView*)inRefCon;
AudioBuffer buffer,buffer2;
OSStatus status;
buffer.mDataByteSize = inNumberFrames *4 ;// buffer size
buffer.mNumberChannels = 1; // one channel
buffer.mData =mv2->rdata;
buffer2.mDataByteSize = inNumberFrames *4 ;// buffer size
buffer2.mNumberChannels = 1; // one channel
buffer2.mData =mv2->rdata2;
AudioBufferList bufferList;
bufferList.mNumberBuffers = 2;
bufferList.mBuffers[0] = buffer;
bufferList.mBuffers[1] = buffer2;
status = AudioUnitRender(mv2->outputUnit, ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, &bufferList);
[mv2 recproc :mv->rdata :mv->rdata2 :inNumberFrames];
return noErr;
}
You seem to be using the HAL output unit for pulling input. There might not be a guarantee that the input device and output device sample rates are exactly locked. Any slow slight drift in the sample rate of either device could eventually cause a buffer underflow or overflow.
One solution might be to find and set an input device for a separate input audio unit instead of depending on the default output unit. Try a USB mic, for instance.
According to this article https://www.native-instruments.com/forum/threads/latency-drift-problem-on-macbook.175551/ this problem appears to be a usb audio driver bug in maverick. I didn't find a kext replacement solution anywhere.
After making a sonar type tester (1 cycle 22khz square wave click every 600 ms to speaker, display selected recorded frame number after click) and could see the 3 to 4 samples drift per second along with the distortion/latency drift reset experience after 1.5 hrs, I decided to look around and find how to access the buffer pointers to stabilise the latency drift, but also no luck.
Also api latency queries show no changes as it drifts.
I did find that you could reset the latency with audiounitstop then audiounitstart (same thread), but it worked only if only one audiounit bus system wide was active. Research also showed that the latency could be reset if you toggle the hardware device sample-rate in Audio Midi Setup. this is a bit aggressive and would be uncomfortable for some.
My design toggled the nominalsamplerate (AudioObjectSetPropertyData with kAudioDevicePropertyNominalSampleRate) every 60 minutes (48000 then back to 44100), with delay by way of waiting for change notification through a callback.
This cause a 2 second void in audio input and output every hour. Safari playing a youtube video would mute, and cause a 1-2 second video freeze during this time . VLC showed the same but video remained smooth during 2 second silence.
Like I said, it wouldn't work for all, but I chose system wide 2 second mute every hour over a recording that has 3 minutes of fuzzy audio every 1.5 hrs. Its been posted that a yosemite upgrade fixes this, although some have also found crackling after going up to yosemite.
I wish to record the microphone audio stream so I can do realtime DSP on it.
I want to do so without having to use threads and without having .read() block while it waits for new audio data.
UPDATE/ANSWER: It's a bug in Android. 4.2.2 still has the problem, but 5.01 IS FIXED! I'm not sure where the divide is but that's the story.
NOTE: Please don't say "Just use threads." Threads are fine but this isn't about them, and the android developers intended for AudioRecord to be fully usable without me having to specify threads and without me having to deal with blocking read(). Thank you!
Here is what I have found:
When the AudioRecord object is initialized, it creates its own internal ring type buffer.
When .start() is called, it begins recording to said ring buffer (or whatever kind it really is.)
When .read() is called, it reads either half of bufferSize or the specified number of bytes (whichever is less) and then returns.
If there is more than enough audio samples in the internal buffer, then read() returns instantly with the data. If there is not enough yet, then read() waits till there is, then returns with the data.
.setRecordPositionUpdateListener() can be used to set a Listener, and .setPositionNotificationPeriod() and .setNotificationMarkerPosition() can be used to set the notification Period and Position, respectively.
However, the Listener seems to be never called unless certain requirements are met:
1: The Period or Position must be equal to bufferSize/2 or (bufferSize/2)-1.
2: A .read() must be called before the the Period or Position timer starts counting - in other words, after calling .start() then also call .read(), and each time the Listener is called, call .read() again.
3: .read() must read at least half of bufferSize each time.
So using these rules I am able to get the callback/Listener working, but for some reason the reads are still blocking and I can't figure out how to get the Listener to only be called when there is a full read's worth.
If I rig up a button view to click to read, then I can tap it and if tap rapidly, read blocks. But if I wait for the audio buffer to fill, then the first tap is instant (read returns right away) but subsiquent rapid taps are blocked because read() has to wait, I guess.
Greatly appreciated would be any insight on how I might make the Listener work as intended - in such a way that my listener gets called when there's enough data for read() to return instantly.
Below is the relavent parts of my code.
I have some log statements in my code which send strings to logcat which allows me to see how long each command is taking, and this is how I know that read() is blocking.
(And the buttons in my simple test app also are very doggy slow to respond when it is reading repeatedly, but CPU is not pegged.)
Thanks,
~Jesse
In my OnCreate():
bufferSize=AudioRecord.getMinBufferSize(samplerate,AudioFormat.CHANNEL_CONFIGURATION_MONO,AudioFormat.ENCODING_PCM_16BIT)*4;
recorder = new AudioRecord (AudioSource.MIC,samplerate,AudioFormat.CHANNEL_CONFIGURATION_MONO,AudioFormat.ENCODING_PCM_16BIT,bufferSize);
recorder.setRecordPositionUpdateListener(mRecordListener);
recorder.setPositionNotificationPeriod(bufferSize/2);
//recorder.setNotificationMarkerPosition(bufferSize/2);
audioData = new short [bufferSize];
recorder.startRecording();
samplesread=recorder.read(audioData,0,bufferSize);//This triggers it to start doing the callback.
Then here is my listener:
public OnRecordPositionUpdateListener mRecordListener = new OnRecordPositionUpdateListener()
{
public void onPeriodicNotification(AudioRecord recorder) //This one gets called every period.
{
Log.d("TimeTrack", "AAA");
samplesread=recorder.read(audioData,0,bufferSize);
Log.d("TimeTrack", "BBB");
//player.write(audioData, 0, samplesread);
//Log.d("TimeTrack", "CCC");
reads++;
}
#Override
public void onMarkerReached(AudioRecord recorder) //This one gets called only once -- when the marker is reached.
{
Log.d("TimeTrack", "AAA");
samplesread=recorder.read(audioData,0,bufferSize);
Log.d("TimeTrack", "BBB");
//player.write(audioData, 0, samplesread);
//Log.d("TimeTrack", "CCC");
}
};
UPDATE: I have tried this on Android 2.2.3, 2.3.4, and now 4.0.3, and all act the same.
Also: There is an open bug on code.google about it - one entry started in 2012 by someone else then one from 2013 started by me (I didn't know about the first):
UPDATE 2016: Ahhhh finally after years of wondering if it was me or android, I finally have answer! I tried my above code on 4.2.2 and same problem. I tried above code on 5.01, AND IT WORKS!!! And the initial .read() call is NOT needed anymore either. Now, once the .setPositionNotificationPeriod() and .StartRecording() are called, mRecordListener() just magically starts getting called every time there is data available now so it no longer blocks, because the callback is not called until after enough data has been recorded. I haven't listened to the data to know if it's recording correctly, but the callback is happening like it should, and it is not blocking the activity, like it used to!
http://code.google.com/p/android/issues/detail?id=53996
http://code.google.com/p/android/issues/detail?id=25138
If folks who care about this bug log in and vote for and/or comment on the bug maybe it'll get addressed sooner by Google.
It's late answear, but I think I know where Jesse did a mistake. His read call is getting blocked because he is requesting shorts which are sized same as buffer size, but buffer size is in bytes and short contains 2 bytes. If we make short array to be same length as buffer we will read twice as much data.
The solution is to make audioData = new short[bufferSize/2] If the buffer size is 1000 bytes, this way we will request 500 shorts which are 1000 bytes.
Also he should change samplesread=recorder.read(audioData,0,bufferSize) to samplesread=recorder.read(audioData,0,audioData.length)
UPDATE
Ok, Jesse. I can see where another mistake can be - the positionNotificationPeriod. This value have to be large enought so it won't call the listener too often and we need to make sure that when the listener is called the bytes to read are ready to be collected. If bytes won't be ready when the listener is called, the main thread will get blocked by recorder.read(audioData, 0, audioData.length) call until requested bytes get's collected by AudioRecord.
You should calculate buffer size and shorts array length based on time interval you set - how often you want the listener to be called. Position notification period, buffer size and shorts array length all have to be adjusted correctly. Let me show you an example:
int periodInFrames = sampleRate / 10;
int bufferSize = periodInFrames * 1 * 16 / 8;
audioData = new short [bufferSize / 2];
int minBufferSize = AudioRecord.getMinBufferSize(sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);
if (bufferSize < minBufferSize) bufferSize = minBufferSize;
recorder = new AudioRecord(AudioSource.MIC, sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, buffersize);
recorder.setRecordPositionUpdateListener(mRecordListener);
recorder.setPositionNotificationPeriod(periodInFrames);
recorder.startRecording();
public OnRecordPositionUpdateListener mRecordListener = new OnRecordPositionUpdateListener() {
public void onPeriodicNotification(AudioRecord recorder) {
samplesread = recorder.read(audioData, 0, audioData.length);
player.write(short2byte(audioData));
}
};
private byte[] short2byte(short[] data) {
int dataSize = data.length;
byte[] bytes = new byte[dataSize * 2];
for (int i = 0; i < dataSize; i++) {
bytes[i * 2] = (byte) (data[i] & 0x00FF);
bytes[(i * 2) + 1] = (byte) (data[i] >> 8);
data[i] = 0;
}
return bytes;
}
So now a bit of explanation.
First we set how often the listener have to be called to collect audio data (periodInFrames). PositionNotificationPeriod is expressed in frames. Sampling rate is expressed in frames per second, so for 44100 sampling rate we have 44100 frames per second. I divided it by 10 so the listener will be called every 4410 frames = 100 milliseconds - that's reasonable time interval.
Now we calculate buffer size based on our periodInFrames so any data won't be overriden before we collect it. Buffer size is expressed in bytes. Our time interval is 4410 frames, each frame contains 1 byte for mono or 2 bytes for stereo so we multiply it by number of channels (1 in your case). Each channel contains 1 byte for ENCODING_8BIT or 2 bytes for ENCODING_16BIT so we multiply it by bits per sample (16 for ENCODING_16BIT, 8 for ENCODING_8BIT) and divide it by 8.
Then we set audioData length to be half of the bufferSize so we make sure that when the listener gets called, bytes to read are already there waiting to be collected. That's because short contains 2 bytes and bufferSize is expressed in bytes.
Then we check if bufferSize is large enought to succesfully initialize AudioRecord object, if it's not then we set bufferSize to it's minimal size - we don't need to change our time interval or audioData length.
In our listener we read and store data to short array. That's why we use audioData.length instead buffer size, because only audioData.length can tell us the number of shorts the buffer contains.
I had it working some time ago so please let me know if it will work for you.
I'm not sure why you're avoiding spawning separate threads, but if it's because you don't want have to deal with coding them properly, you can use .schedule on a Timer object after each .read, where the time interval is set to the time it takes to get your buffer filled (number of samples in buffer / sampleRate). Yes I know this is using a separate thread, but this advice was given assuming that the reason you were avoiding using threads was to avoid having to code them properly.
This way, the longest time it can possibly block the thread for should be neglible. But I don't know why you'd want to do that.
If the above reason is not why you're avoiding using separate threads, may I ask why?
Also, what exactly do you mean by realtime? Do you intend to playback the affected audio using, let's say, an AudioTrack? Because the latency on most Android devices is pretty bad.