I have downloaded the audio-echo app from the android NDK portal for opensl. Due to the lack of documentation I'm not able to identify how to change the sampling rate and buffer size of the audio in and out.
If anybody has any idea on how to:
Change the buffer size and sampling rate on OpenSL
Read the buffers to be fed to a C code to be processed
Fed to the output module of OpenSL to be fed to the speakers
Another alternative I feel is read it at the preferred sampling rate and buffer size but downsample and upsample in the code itself and use a circular buffer to get desired data. But how are we reading and feeding the data in openSL?
In the OpenSL ES API, there are calls to create either a Player or a Recorder:
SLresult (*CreateAudioPlayer) (
SLEngineItf self,
SLObjectItf * pPlayer,
SLDataSource *pAudioSrc,
SLDataSink *pAudioSnk,
SLuint32 numInterfaces,
const SLInterfaceID * pInterfaceIds,
const SLboolean * pInterfaceRequired
);
SLresult (*CreateAudioRecorder) (
SLEngineItf self,
SLObjectItf * pRecorder,
SLDataSource *pAudioSrc,
SLDataSink *pAudioSnk,
SLuint32 numInterfaces,
const SLInterfaceID * pInterfaceIds,
const SLboolean * pInterfaceRequired
);
Note that both of these take a SLDataSource *pAudioSrc parameter.
To use a custom playback rate or recording rate, you have to set up this data source properly.
I use an 11Khz playback rate using this code:
// Configure data format.
SLDataFormat_PCM pcm;
pcm.formatType = SL_DATAFORMAT_PCM;
pcm.numChannels = 1;
pcm.samplesPerSec = SL_SAMPLINGRATE_11_025;
pcm.bitsPerSample = SL_PCMSAMPLEFORMAT_FIXED_16;
pcm.containerSize = 16;
pcm.channelMask = SL_SPEAKER_FRONT_CENTER;
pcm.endianness = SL_BYTEORDER_LITTLEENDIAN;
// Configure Audio Source.
SLDataSource source;
source.pFormat = &pcm;
source.pLocator = &bufferQueue;
To feed data to the speakers, a buffer queue is used that is filled by a callback. To set this callback, use SLAndroidSimpleBufferQueueItf, documented in section 8.12 SLBufferQueueItf of the OpenGL ES specification.
Related
Lets say that I am reading from data stream, and that stream is sending the content of an h264 video feed. Given I read from that stream and I have some amount of data consisting of an indeterminate number of frames (NAL?). Given that i know the framerate, and size of the originating video, how would I go about converting this snippet into a mp4 that i could view? The video does not contain audio.
I want to do this using nodejs? My attempts to do so have produced nothing resembling a valid h264 file to convert into mp4. My thoughts so far were to strip any data preceding the first found start code in the data and feed that into a file and use ffmpeg (currently just testing in the command line) to convert the file to mp4.
What's the correct way to go about doing this?
ie. something like this (it's in Typescript but same thing)
//We assume here that when this while loop exist at least one full frame of data will have been read and written to disk
let stream: WriteStream = fs.createWriteStream("./test.h264")
while(someDataStream.available()) { //just an example not real code
let data: Buffer = someDataStream.readSomeData() //just an example not a real method call
let file = null;
try {
file = fs.statSync("./test.h264");
} catch (error) {
console.error(error)
}
if(!stream.writable) {
console.error("stream not writable")
} else if(file == null || file.size <= 0) {
let index = data.indexOf(0x7C)
console.log("index: " + index)
if(index > 0) {
console.log("index2: " + data.slice(index).indexOf(0x7c))
stream.write(data.slice(index))
}
} else {
stream.write(data)
}
}
To handle a data stream, you'll need to emit fragmented MP4. Like all MP4, fMP4 streams begin with a preamble containing ftyp, moov, and styp boxes. Then each frame is encoded with a moof / mdat box pair.
In order to generate a useful preamble from your H.264 bitstream, you need to locate a SPS / PPS pair of NALUs in the H264 data, to set up the avc1 box within the moov box. Those two NALUs are often immediately followed by an I-frame (a key frame). The first frame in a stream must be an I-frame, and subsequent ones can be P- or B- frames. E
It's a fairly complex task involving lots of bit-banging and buffer-shuffling (those are technical terms ;-).
I've been working on a piece of js code to extract H.264 from webm and put it into fmp4. It's not yet complete. It's backed up by another piece of code to decode the parts of the H264 stream that are needed to pack it properly into fMP4.
I wish I could write, "here are the ten lines of code you need" but those formats (fMP4 and H264) aren't simple enough to make that possible.
Idk why none of those questions doesn't actually have an easy answer. Here you go, Node.js solution, i argument just in case you need to offset the search
const soi = Buffer.from([0x00, 0x00, 0x00, 0x01]);
function findStartFrame(buffer, i = -1) {
while ((i = buffer.indexOf(soi, i + 1)) !== -1) {
if ((buffer[i + 4] & 0x1F) === 7) return i
}
return -1
}
I need to convert audio data from AV_CODEC_ID_PCM_S16LE to AV_CODEC_ID_PCM_ALAW and I am using this code as an example. The example code does essentially this (error checking omitted for brevity):
const AVCodec* codec = avcodec_find_encoder(AV_CODEC_ID_MP2);
AVCodecContext* c = avcodec_alloc_context3(codec);
c->bit_rate = 64000;
c->sample_fmt = AV_SAMPLE_FMT_S16;
c->sample_rate = select_sample_rate(codec);
c->channel_layout = select_channel_layout(codec);
c->channels = av_get_channel_layout_nb_channels(c->channel_layout);
avcodec_open2(c, codec, NULL);
AVFrame* frame = av_frame_alloc();
frame->nb_samples = c->frame_size;
frame->format = c->sample_fmt;
frame->channel_layout = c->channel_layout;
The example code subsequently uses c->frame_size in a for loop.
My code is similar to the above with the following differences:
const AVCodec* codec = avcodec_find_encoder(AV_CODEC_ID_PCM_ALAW);
c->sample_rate = 8000;
c->channel_layout = AV_CH_LAYOUT_MONO;
c->channels = 1;
After calling avcodec_open2, c->frame_size is zero. The example code never sets the frame size so I assume that it expects either avcodec_alloc_context3 or avcodec_open2 to set it. Is this a correct assumption? Is the setting of the frame size based on the codec being used? If I have to set the frame size explicitly, is there a recommended size?
EDIT:
Based on #the-kamilz answer it appears that the example code is not robust. The example assumes that c->frame_size will be set but that appears to be dependent on the codec. In my case, codec->capabilities was in fact set to AV_CODEC_CAP_VARIABLE_FRAME_SIZE. So I modified my code to check c->frame_size and use it only if it is not zero. If it is zero, I just picked an arbitrary one second worth of data for frame->nb_samples.
In the FFmpeg documentation it is mentioned as:
int AVCodecContext::frame_size
Number of samples per channel in an audio frame.
encoding: set by libavcodec in avcodec_open2(). Each submitted frame except the last must contain exactly frame_size samples per channel.
May be 0 when the codec has AV_CODEC_CAP_VARIABLE_FRAME_SIZE set, then
the frame size is not restricted.
decoding: may be set by some decoders to indicate constant frame size
Hope that helps.
you don't control the frame size explicitly, it is set by the encoder depending on the codecs provided at initialization (opening) time
once avcodec_open2() is successful, you can retrieve the frame's buffer size with av_samples_get_buffer_size()
I'm writing a MOD player, trying to playback a sample using Allegro5 raw stream capabilities, I can't figure out the exact init parameters for the stream to play the loaded sample data from the mod file.
This is what I have:
xf::ModLoader ml;
ml.loadFromFile("C:\\Users\\bubu\\Downloads\\agress.mod");
// getSampleLength() returns # of data words
int sample_length = ml.getSampleLength(1) * 2;
const int8_t* sample_data = ml.getSampleData(1);
ALLEGRO_MIXER* mixer = al_get_default_mixer();
ALLEGRO_AUDIO_STREAM* stream = al_create_audio_stream(1, sample_length, 8363, ALLEGRO_AUDIO_DEPTH_INT8, ALLEGRO_CHANNEL_CONF_1);
al_attach_audio_stream_to_mixer(stream, mixer);
al_set_audio_stream_gain(stream, 0.7f);
al_set_audio_stream_playmode(stream, ALLEGRO_PLAYMODE_ONCE);
al_set_audio_stream_playing(stream, true);
al_set_audio_stream_fragment(stream, (void*)sample_data);
al_drain_audio_stream(stream);
First of all, freq param is hardcoded for the test (8363Hz), but, playing back at the specified frequency I don't get what I expect, and al_drain_audio_stream() gets stuck forever playing garbage in a loop...
Any help would be appreciated.
At the very least, you need to be calling al_get_audio_stream_fragment before you call al_set_audio_stream_fragment. Typically you'd feed these streams in a while loop, while responding to the ALLEGRO_EVENT_AUDIO_STREAM_FRAGMENT event. See the ex_saw example in the Allegro's source for some sample code: https://github.com/liballeg/allegro5/blob/master/examples/ex_saw.c
First of all, i am new to audio-programming, so bear with me.
I am trying to play spotify music with NAudio or BASS.Net or any other .net audio-library.
As far as i known, libspotify delivers music as raw PCM data. what is the sample rate of spotify stream (libspotify)?
From the spotify docs:
Samples are delivered as integers, see sp_audioformat. One frame consists of the same number of samples as there are channels. I.e. interleaving is on the sample level.
When i try to play a song, spotify makes a callback with a 8192 byte buffer
channels = 2
sample_rate = 44100
num_frames = 2048
I need a little help translating this information to NAudio terms.
I have also tried with a spotify to Bass.Net sample (BASSPlayer.cs). But i haven't heard a single note from my speakers yet.
I have tried to play an mp3-song with NAudio and Bass.NET and this works fine, so the speaker volume is ok.
https://github.com/Alxandr/SpotiFire/blob/master/SpotiFire.Server/BASSPlayer.cs
There is breakthrough with NAudio. This is what i came up with, by using the trial and error method. I'm not sure if this is the right way to calculate the parameters from sampleRate/channels...
But the song is playing :-)
IWavePlayer waveOutDevice = new WaveOut();
using (var pcmStream = new FileStream(PcmFile, FileMode.Open))
{
const int songDuration = 264000;
const int sampleRate = 44100;
const int channels = 2;
var waveFormat = WaveFormat.CreateCustomFormat(WaveFormatEncoding.Pcm, sampleRate * channels, 1, sampleRate * 2 * channels, channels, 16);
var waveStream = new RawSourceWaveStream(pcmStream, waveFormat);
waveOutDevice.Init(waveStream);
waveOutDevice.Play();
Thread.Sleep(songDuration);
waveOutDevice.Stop();
waveStream.Close();
waveOutDevice.Dispose();
}
I am using Media Foundation on client side to display live mpeg4 stream from AXIS Camera through RTSP server.
Client side video works very fine if i set GOV # camera = 1, i.e. camera will only send I-Frames. But if GOV is increased and camera starts sending P-Frames also, my video suddenly gets distorted at regular intervals. I cannot set GOV = 1 for always because it consumes a lot of bandwidth.
Following is the code for RequestSample method where i supply Samples to Media Foundation :
RTPFrame frame = null;
byte[] frameBytes = null;
frame = _VideoJitter.GetNextFrame();
frameBytes = frame.GetFrameAsBytes();
frame.FrameType= RTPFrame.PredictFrameType(frameBytes);
_videoEncapsulatedSample.ReadSampleFrom(frameBytes);
videoSample = _videoEncapsulatedSample.MfSample;
long timestamp = nextSampleTimestamp ?? 0;
videoSample.SetSampleTime(timestamp);
duration_video = (long)GetPresentationTime(frame);
videoSample.SetSampleDuration(duration_video);
nextSampleTimestamp = timestamp + duration_video;
if (frame.FrameType == FrameType.IFrame)
{
videoSample.SetUINT32(MFAttributesClsid.MFSampleExtension_CleanPoint, 1);
}
return videoSample;
Do i need to set any attribute for processing P-Frames??
Any help would be highly appreciated....
Update (2012/02/22) :
I ran some statistics and found that some times I-frames never reach client, i.e. suppose GOV = 15, so every 15th frame should be an I-Frame, but sometimes (at irregular intervals) client receives an I-Frame after 28 or 30 or 59 P-frames.
Any pointers ??
Thanks,
Prateek