No sound output with WASAPI - visual-c++

I am having trouble with WASAPI. It do not output any sound and I have been checked the data that writing to the buffer.
Because of it does not output any sound, I haven't any idea to find out the problem.
It may have some problems in following code.
SoundStream::SoundStream() : writtenCursor(0), writeCursor(0), distroy(false)
{
IMMDeviceEnumerator * pEnumerator = nullptr;
HResult(CoCreateInstance(__uuidof(MMDeviceEnumerator), NULL, CLSCTX_ALL, IID_PPV_ARGS(&pEnumerator)));
IMMDevice * pDevice = nullptr;
HResult(pEnumerator->GetDefaultAudioEndpoint(eRender, eMultimedia, &pDevice));
SafeRelease(&pEnumerator);
HResult(pDevice->Activate(__uuidof(IAudioClient), CLSCTX_ALL, NULL, (void**)&pAudioClient));
SafeRelease(&pDevice);
WAVEFORMATEXTENSIBLE * pwfx = nullptr;
hEvent = CreateEvent(NULL, FALSE, FALSE, NULL);
REFERENCE_TIME hnsRequestedDuration = REFTIMES_PER_SEC * 2;
HResult(pAudioClient->GetMixFormat((WAVEFORMATEX**)&pwfx));
HResult(pAudioClient->Initialize(
AUDCLNT_SHAREMODE_SHARED,
AUDCLNT_STREAMFLAGS_EVENTCALLBACK,
hnsRequestedDuration,
0,
(WAVEFORMATEX*)pwfx,
NULL));
pAudioClient->SetEventHandle(hEvent);
channel = (size_t)pwfx->Format.nChannels;
bits = (size_t)pwfx->Format.wBitsPerSample;
validBits = (size_t)pwfx->Samples.wValidBitsPerSample;
frequency = (size_t)pwfx->Format.nSamplesPerSec;
buffer.reshape({ 0, channel, bits >> 3 });
CoTaskMemFree(pwfx);
HResult(pAudioClient->GetBufferSize(&bufferFrameCount));
HResult(pAudioClient->Start());
if (pAudioClient)
{
thread = std::thread([&]()
{
this->Sync();
});
}
}

You could look at my WASAPI.cpp code at http://jdmcox.com (which works fine).
You should also check if the expected wave format is float:
//SubFormat 00000003-0000-0010-8000-00aa00389b71 defines KSDATAFORMAT_SUBTYPE_IEEE_FLOAT
//SubFormat 00000001-0000-0010-8000-00aa00389b71 defines KSDATAFORMAT_SUBTYPE_PCM
GUID G;
WORD V;
WAVEFORMATEX *pwfx = NULL;
bool itsfloat;
pAudioClient->GetMixFormat(&pwfx);
// Do we received a WAVEFORMATEXTENSIBLE?
if(pwfx.cbSize >= 22) {
G = ((WAVEFORMATEXTENSIBLE*)pwfx)->SubFormat;
V = ((WAVEFORMATEXTENSIBLE*)pwfx)->Samples.wValidBitsPerSample;
if (G.Data1 == 3) itsfloat = true;
else if (G.Data1 == 1) itsfloat = false;
}
You know you received a WAVEFORMATEXTENSIBLE and not a simple WAVEFORMATEX because the "pwfx.cbSize >= 22".
See more at:
IAudioClient::GetMixFormat
https://learn.microsoft.com/en-us/windows/win32/api/audioclient/nf-audioclient-iaudioclient-getmixformat
WAVEFORMATEXTENSIBLE
https://learn.microsoft.com/en-us/windows/win32/api/mmreg/ns-mmreg-waveformatextensible

You could look at my WASAPI.cpp code at http://jdmcox.com AGAIN.
Now it works in shared mode as well as exclusive mode.
I should note that no conversion of wave format or wave is necessary in shared mode -- Windows takes care of both converting to and from their format used to mix waves.

Related

Desktop duplication (DirectX) screen capture fails to deliver screen updates

I'm working on an application that would capture the screen through Desktop duplication APIs (using DirectX 11) (only the diff to the previous screen update) and render it on another window (The viewer might be running on another machine connected via LAN). The code is an improved version of the sample provided in MSDN. Everything works fine except the device did not give any screen update though there is one some times in the mid, that happens around 10% of the time on some machines (mostly on windows 8/8.1 machines and rarely on windows 10 machines). I tried all the possible ways to sort out this problem. Reduced the number of device resets, that provided me some what reliable output but not always work fine for 100%.
The device fails to provide an initial screen (a full screen) some times (This happens 60% of the time on all windows operating systems where Desktop duplication is supported), I came up with a work around that retried for an initial update from the device until it provides one but that too resulted in multiple issues, the device might not even give the initial screen ever.
I have already invested weeks of my efforts to fix the problem but did not figure out a proper solution and there are no forums I know that discusses these kind of issues. Any help would be appreciated.
Below is my code to get the screen diff to the previous one, init the device, populating the adapters and monitors.
Please bear with me for a very long code snippet, Thanks in advance.
To Get the screen update:
INT getChangedRegions(int timeout, rectangles &dirtyRects, std::vector <MOVE_RECT> &moveRects, UINT &rect_count, RECT ScreenRect)
{
UINT diffArea = 0;
FRAME_DATA currentFrameData;
bool isTimeOut = false;
TRY
{
m_LastErrorCode = m_DuplicationManager.GetFrame(&currentFrameData, timeout, &isTimeOut);
if(SUCCEEDED(m_LastErrorCode) && (!isTimeOut))
{
if(currentFrameData.FrameInfo.TotalMetadataBufferSize)
{
m_CurrentFrameTexture = currentFrameData.Frame;
if(currentFrameData.MoveCount)
{
DXGI_OUTDUPL_MOVE_RECT* moveRectArray = reinterpret_cast<DXGI_OUTDUPL_MOVE_RECT*> (currentFrameData.MetaData);
if (moveRectArray)
{
for(UINT index = 0; index < currentFrameData.MoveCount; index++)
{
//WebRTC
// DirectX capturer API may randomly return unmoved move_rects, which should
// be skipped to avoid unnecessary wasting of differing and encoding
// resources.
// By using testing application it2me_standalone_host_main, this check
// reduces average capture time by 0.375% (4.07 -> 4.055), and average
// encode time by 0.313% (8.042 -> 8.016) without other impacts.
if (moveRectArray[index].SourcePoint.x != moveRectArray[index].DestinationRect.left || moveRectArray[index].SourcePoint.y != moveRectArray[index].DestinationRect.top)
{
if(m_UseD3D11BitmapConversion)
{
MOVE_RECT moveRect;
moveRect.SourcePoint.x = moveRectArray[index].SourcePoint.x * m_ImageScalingFactor;
moveRect.SourcePoint.y = moveRectArray[index].SourcePoint.y * m_ImageScalingFactor;
moveRect.DestinationRect.left = moveRectArray[index].DestinationRect.left * m_ImageScalingFactor;
moveRect.DestinationRect.top = moveRectArray[index].DestinationRect.top * m_ImageScalingFactor;
moveRect.DestinationRect.bottom = moveRectArray[index].DestinationRect.bottom * m_ImageScalingFactor;
moveRect.DestinationRect.right = moveRectArray[index].DestinationRect.right * m_ImageScalingFactor;
moveRects.push_back(moveRect);
diffArea += abs((moveRect.DestinationRect.right - moveRect.DestinationRect.left) *
(moveRect.DestinationRect.bottom - moveRect.DestinationRect.top));
}
else
{
moveRects.push_back(moveRectArray[index]);
diffArea += abs((moveRectArray[index].DestinationRect.right - moveRectArray[index].DestinationRect.left) *
(moveRectArray[index].DestinationRect.bottom - moveRectArray[index].DestinationRect.top));
}
}
}
}
else
{
return -1;
}
}
if(currentFrameData.DirtyCount)
{
RECT* dirtyRectArray = reinterpret_cast<RECT*> (currentFrameData.MetaData + (currentFrameData.MoveCount * sizeof(DXGI_OUTDUPL_MOVE_RECT)));
if (!dirtyRectArray)
{
return -1;
}
rect_count = currentFrameData.DirtyCount;
for(UINT index = 0; index < rect_count; index ++)
{
if(m_UseD3D11BitmapConversion)
{
RECT dirtyRect;
dirtyRect.bottom = dirtyRectArray[index].bottom * m_ImageScalingFactor;
dirtyRect.top = dirtyRectArray[index].top * m_ImageScalingFactor;
dirtyRect.left = dirtyRectArray[index].left * m_ImageScalingFactor;
dirtyRect.right = dirtyRectArray[index].right * m_ImageScalingFactor;
diffArea += abs((dirtyRect.right - dirtyRect.left) *
(dirtyRect.bottom - dirtyRect.top));
dirtyRects.push_back(dirtyRect);
}
else
{
diffArea += abs((dirtyRectArray[index].right - dirtyRectArray[index].left) *
(dirtyRectArray[index].bottom - dirtyRectArray[index].top));
dirtyRects.push_back(dirtyRectArray[index]);
}
}
}
}
return diffArea;
}
CATCH_ALL(e)
{
LOG(CRITICAL) << _T("Exception in getChangedRegions");
}
END_CATCH_ALL
return -1;
}
Here is the code to init the device
//
// Initialize duplication interfaces
//
HRESULT cDuplicationManager::InitDupl(_In_ ID3D11Device* Device, _In_ IDXGIAdapter *_pAdapter, _In_ IDXGIOutput *_pOutput, _In_ UINT Output)
{
HRESULT hr = E_FAIL;
if(!_pOutput || !_pAdapter || !Device)
{
return hr;
}
m_OutputNumber = Output;
// Take a reference on the device
m_Device = Device;
m_Device->AddRef();
/*
// Get DXGI device
IDXGIDevice* DxgiDevice = nullptr;
HRESULT hr = m_Device->QueryInterface(__uuidof(IDXGIDevice), reinterpret_cast<void**>(&DxgiDevice));
if (FAILED(hr))
{
return ProcessFailure(nullptr, _T("Failed to QI for DXGI Device"), _T("Error"), hr);
}
// Get DXGI adapter
IDXGIAdapter* DxgiAdapter = nullptr;
hr = DxgiDevice->GetParent(__uuidof(IDXGIAdapter), reinterpret_cast<void**>(&DxgiAdapter));
DxgiDevice->Release();
DxgiDevice = nullptr;
if (FAILED(hr))
{
return ProcessFailure(m_Device, _T("Failed to get parent DXGI Adapter"), _T("Error"), hr);//, SystemTransitionsExpectedErrors);
}
// Get output
IDXGIOutput* DxgiOutput = nullptr;
hr = DxgiAdapter->EnumOutputs(Output, &DxgiOutput);
DxgiAdapter->Release();
DxgiAdapter = nullptr;
if (FAILED(hr))
{
return ProcessFailure(m_Device, _T("Failed to get specified output in DUPLICATIONMANAGER"), _T("Error"), hr);//, EnumOutputsExpectedErrors);
}
DxgiOutput->GetDesc(&m_OutputDesc);
IDXGIOutput1* DxgiOutput1 = nullptr;
hr = DxgiOutput->QueryInterface(__uuidof(DxgiOutput1), reinterpret_cast<void**>(&DxgiOutput1));
*/
_pOutput->GetDesc(&m_OutputDesc);
// QI for Output 1
IDXGIOutput1* DxgiOutput1 = nullptr;
hr = _pOutput->QueryInterface(__uuidof(DxgiOutput1), reinterpret_cast<void**>(&DxgiOutput1));
if (FAILED(hr))
{
return ProcessFailure(nullptr, _T("Failed to QI for DxgiOutput1 in DUPLICATIONMANAGER"), _T("Error"), hr);
}
// Create desktop duplication
hr = DxgiOutput1->DuplicateOutput(m_Device, &m_DeskDupl);
DxgiOutput1->Release();
DxgiOutput1 = nullptr;
if (FAILED(hr) || !m_DeskDupl)
{
if (hr == DXGI_ERROR_NOT_CURRENTLY_AVAILABLE)
{
return ProcessFailure(nullptr, _T("Maximum number of applications using Desktop Duplication API"), _T("Error"), hr);
}
return ProcessFailure(m_Device, _T("Failed to get duplicate output in DUPLICATIONMANAGER"), _T("Error"), hr);//, CreateDuplicationExpectedErrors);
}
return S_OK;
}
Finally to get the current frame and difference to the previous one:
//
// Get next frame and write it into Data
//
_Success_(*Timeout == false && return == DUPL_RETURN_SUCCESS)
HRESULT cDuplicationManager::GetFrame(_Out_ FRAME_DATA* Data, int timeout, _Out_ bool* Timeout)
{
IDXGIResource* DesktopResource = nullptr;
DXGI_OUTDUPL_FRAME_INFO FrameInfo;
try
{
// Get new frame
HRESULT hr = m_DeskDupl->AcquireNextFrame(timeout, &FrameInfo, &DesktopResource);
if (hr == DXGI_ERROR_WAIT_TIMEOUT)
{
*Timeout = true;
return S_OK;
}
*Timeout = false;
if (FAILED(hr))
{
return ProcessFailure(m_Device, _T("Failed to acquire next frame in DUPLICATIONMANAGER"), _T("Error"), hr);//, FrameInfoExpectedErrors);
}
// If still holding old frame, destroy it
if (m_AcquiredDesktopImage)
{
m_AcquiredDesktopImage->Release();
m_AcquiredDesktopImage = nullptr;
}
if (DesktopResource)
{
// QI for IDXGIResource
hr = DesktopResource->QueryInterface(__uuidof(ID3D11Texture2D), reinterpret_cast<void **>(&m_AcquiredDesktopImage));
DesktopResource->Release();
DesktopResource = nullptr;
}
if (FAILED(hr))
{
return ProcessFailure(nullptr, _T("Failed to QI for ID3D11Texture2D from acquired IDXGIResource in DUPLICATIONMANAGER"), _T("Error"), hr);
}
// Get metadata
if (FrameInfo.TotalMetadataBufferSize)
{
// Old buffer too small
if (FrameInfo.TotalMetadataBufferSize > m_MetaDataSize)
{
if (m_MetaDataBuffer)
{
delete [] m_MetaDataBuffer;
m_MetaDataBuffer = nullptr;
}
m_MetaDataBuffer = new (std::nothrow) BYTE[FrameInfo.TotalMetadataBufferSize];
if (!m_MetaDataBuffer)
{
m_MetaDataSize = 0;
Data->MoveCount = 0;
Data->DirtyCount = 0;
return ProcessFailure(nullptr, _T("Failed to allocate memory for metadata in DUPLICATIONMANAGER"), _T("Error"), E_OUTOFMEMORY);
}
m_MetaDataSize = FrameInfo.TotalMetadataBufferSize;
}
UINT BufSize = FrameInfo.TotalMetadataBufferSize;
// Get move rectangles
hr = m_DeskDupl->GetFrameMoveRects(BufSize, reinterpret_cast<DXGI_OUTDUPL_MOVE_RECT*>(m_MetaDataBuffer), &BufSize);
if (FAILED(hr))
{
Data->MoveCount = 0;
Data->DirtyCount = 0;
return ProcessFailure(nullptr, L"Failed to get frame move rects in DUPLICATIONMANAGER", L"Error", hr);//, FrameInfoExpectedErrors);
}
Data->MoveCount = BufSize / sizeof(DXGI_OUTDUPL_MOVE_RECT);
BYTE* DirtyRects = m_MetaDataBuffer + BufSize;
BufSize = FrameInfo.TotalMetadataBufferSize - BufSize;
// Get dirty rectangles
hr = m_DeskDupl->GetFrameDirtyRects(BufSize, reinterpret_cast<RECT*>(DirtyRects), &BufSize);
if (FAILED(hr))
{
Data->MoveCount = 0;
Data->DirtyCount = 0;
return ProcessFailure(nullptr, _T("Failed to get frame dirty rects in DUPLICATIONMANAGER"), _T("Error"), hr);//, FrameInfoExpectedErrors);
}
Data->DirtyCount = BufSize / sizeof(RECT);
Data->MetaData = m_MetaDataBuffer;
}
Data->Frame = m_AcquiredDesktopImage;
Data->FrameInfo = FrameInfo;
}
catch (...)
{
return S_FALSE;
}
return S_OK;
}
Update :
Failed to acquire next frame in DUPLICATIONMANAGER is getting printed whenever the device has hung (That is in the mid of streaming the screens, Ex: Continuously capturing a video and sending it to the other end)
// Get new frame
HRESULT hr = m_DeskDupl->AcquireNextFrame(timeout, &FrameInfo, &DesktopResource);
if (hr == DXGI_ERROR_WAIT_TIMEOUT)
{
*Timeout = true;
return S_OK;
}
*Timeout = false;
if (FAILED(hr))
{
return ProcessFailure(m_Device, _T("Failed to acquire next frame in DUPLICATIONMANAGER"), _T("Error"), hr);//, FrameInfoExpectedErrors);
}
here is the detailed error info :
Id3d11DuplicationManager::ProcessFailure - Error: Failed to acquire next frame in DUPLICATIONMANAGER, Detail: The keyed mutex was abandoned.
Update 2 :
I have got the error code whenever the device failed to give screen updates forever, And here is the same
Id3d11DuplicationManager::ProcessFailure - Error: Failed to get duplicate output in DUPLICATIONMANAGER, Detail: Access is denied.
The error code is E_ACCESSDENIED.
I do not understand why I am getting this error as I am running in SYSTEM mode already and the SetThreadDesktop had been executed twice (One during the init and another after detecting a failure)
This is what the explanation of the error on MSDN : E_ACCESSDENIED if the application does not have access privilege to the current desktop image. For example, only an application that runs at LOCAL_SYSTEM can access the secure desktop.
Is there anything else that would result in this kind of issue?
It's always good to check the return codes and immediately fall back to GDI or any other available screen capturing approach in case of non-recoverable errors. Retrying doesn't work most of the time for certain hardware errors like max limit reached, out of memory, device removed, etc, I learned it in a hard way. Furthermore, DirectX device takes a few iterations before producing an initial frame on rare occasions. It wouldn't be useful to retry more than 10 times, you can safely fallback or try re-initializing the device to check one more time before falling back.
Here are some basic checks to do:
Handle DXGI_ERROR_NOT_CURRENTLY_AVAILABLE error:
_pOutput->GetDesc(&m_OutputDesc);
// QI for Output 1
IDXGIOutput1* DxgiOutput1 = nullptr;
hr = _pOutput->QueryInterface(__uuidof(DxgiOutput1), reinterpret_cast<void**>(&DxgiOutput1));
if (FAILED(hr))
{
return ProcessFailure(nullptr, _T("Failed to QI for DxgiOutput1 in DUPLICATIONMANAGER"), _T("Error"), hr);
}
// Create desktop duplication
hr = DxgiOutput1->DuplicateOutput(m_Device, &m_DeskDupl);
DxgiOutput1->Release();
DxgiOutput1 = nullptr;
if (FAILED(hr) || !m_DeskDupl)
{
if (hr == DXGI_ERROR_NOT_CURRENTLY_AVAILABLE)
{
return ProcessFailure(nullptr, _T("Maximum number of applications using Desktop Duplication API"), _T("Error"), hr);
}
return ProcessFailure(m_Device, _T("Failed to get duplicate output in DUPLICATIONMANAGER"), _T("Error"), hr);//, CreateDuplicationExpectedErrors);
}
Check for device removed(DXGI_ERROR_DEVICE_REMOVED) or Device reset(DXGI_ERROR_DEVICE_RESET) & Out of memory(E_OUTOFMEMORY) error codes (I have received E_OUTOFMEMORY sometimes, though it's uncommon):
HRESULT ProcessFailure(_In_opt_ ID3D11Device* Device, _In_ LPCWSTR Str, _In_ LPCWSTR Title, HRESULT hr)//, _In_opt_z_ HRESULT* ExpectedErrors = NULL)
{
HRESULT TranslatedHr;
// On an error check if the DX device is lost
if (Device)
{
HRESULT DeviceRemovedReason = Device->GetDeviceRemovedReason();
switch (DeviceRemovedReason)
{
case DXGI_ERROR_DEVICE_REMOVED:
case DXGI_ERROR_DEVICE_RESET:
case static_cast<HRESULT>(E_OUTOFMEMORY) :
{
// Our device has been stopped due to an external event on the GPU so map them all to
// device removed and continue processing the condition
TranslatedHr = DXGI_ERROR_DEVICE_REMOVED;
break;
}
case S_OK:
{
// Device is not removed so use original error
TranslatedHr = hr;
break;
}
default:
{
// Device is removed but not a error we want to remap
TranslatedHr = DeviceRemovedReason;
}
}
}
else
{
TranslatedHr = hr;
}
_com_error err(TranslatedHr);
LPCTSTR errMsg = err.ErrorMessage();
return TranslatedHr;
}
Furthermore, Desktop duplication requires a real graphics device to be active in order to work. You may get E_ACCESSDENIED otherwise.
There are also other scenarios you may get this error, like, Desktop switch cases, abandoned keyed mutex. You can try reinitializing the device in such cases.
I have also uploaded my sample project here.

Android OpenSL ES - issue with .wav file sampled at 44.1Khz

I'm trying to convert some of my OpenAL code to OpenSL ES for my Android usage (Kitkat 4.4.4) on Genymotion and encountered an issue with .wav files sampled at 44.1Khz. My application is a native one (glue).
I've followed the /native-audio sample of Android NDK samples and fragments from the excellent book Android NDK Beginners Guide, so my code behaves correctly on most of wav/PCM data, except those sampled at 44.1Khz. My specific code is this:
Engine init
// create OpenSL ES engine
SLEngineOption EngineOption[] = {(SLuint32) SL_ENGINEOPTION_THREADSAFE, (SLuint32) SL_BOOLEAN_TRUE};
const SLInterfaceID lEngineMixIIDs[] = {SL_IID_ENGINE};
const SLboolean lEngineMixReqs[] = {SL_BOOLEAN_TRUE};
SLresult res = slCreateEngine(&mEngineObj, 1, EngineOption, 1, lEngineMixIIDs, lEngineMixReqs);
res = (*mEngineObj)->Realize(mEngineObj, SL_BOOLEAN_FALSE);
res = (*mEngineObj)->GetInterface(mEngineObj, SL_IID_ENGINE, &mEngine); // get 'engine' interface
// create output mix (AKA playback; this represents speakers, headset etc.)
res = (*mEngine)->CreateOutputMix(mEngine, &mOutputMixObj, 0,NULL, NULL);
res = (*mOutputMixObj)->Realize(mOutputMixObj, SL_BOOLEAN_FALSE);
Player init
SLresult lRes;
// Set-up sound audio source.
SLDataLocator_AndroidSimpleBufferQueue lDataLocatorIn;
lDataLocatorIn.locatorType = SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE;
lDataLocatorIn.numBuffers = 1; // 1 buffer for a one-time load
// analyze and set correct PCM format
SLDataFormat_PCM lDataFormat;
lDataFormat.formatType = SL_DATAFORMAT_PCM;
lDataFormat.numChannels = audio->wav.channels; // etc. 1,2
lDataFormat.samplesPerSec = audio->wav.sampleRate * 1000; // etc. 44100 * 1000
lDataFormat.bitsPerSample = audio->wav.bitsPerSample; // etc. 16
lDataFormat.containerSize = audio->wav.bitsPerSample;
lDataFormat.channelMask = SL_SPEAKER_FRONT_CENTER;
lDataFormat.endianness = SL_BYTEORDER_LITTLEENDIAN;
SLDataSource lDataSource;
lDataSource.pLocator = &lDataLocatorIn;
lDataSource.pFormat = &lDataFormat;
SLDataLocator_OutputMix lDataLocatorOut;
lDataLocatorOut.locatorType = SL_DATALOCATOR_OUTPUTMIX;
lDataLocatorOut.outputMix = mOutputMixObj;
SLDataSink lDataSink;
lDataSink.pLocator = &lDataLocatorOut;
lDataSink.pFormat = NULL;
const SLInterfaceID lSoundPlayerIIDs[] = { SL_IID_PLAY, SL_IID_ANDROIDSIMPLEBUFFERQUEUE };
const SLboolean lSoundPlayerReqs[] = { SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE };
lRes = (*mEngine)->CreateAudioPlayer(mEngine, &mPlayerObj, &lDataSource, &lDataSink, 2, lSoundPlayerIIDs, lSoundPlayerReqs);
if (lRes != SL_RESULT_SUCCESS) { return; }
lRes = (*mPlayerObj)->Realize(mPlayerObj, SL_BOOLEAN_FALSE);
if (lRes != SL_RESULT_SUCCESS) { return; }
lRes = (*mPlayerObj)->GetInterface(mPlayerObj, SL_IID_PLAY, &mPlayer);
if (lRes != SL_RESULT_SUCCESS) { return; }
lRes = (*mPlayerObj)->GetInterface(mPlayerObj, SL_IID_ANDROIDSIMPLEBUFFERQUEUE, &mPlayerQueue);
if (lRes != SL_RESULT_SUCCESS) { return; }
// register callback on the buffer queue
lRes = (*mPlayerQueue)->RegisterCallback(mPlayerQueue, bqPlayerQueueCallback, NULL);
if (lRes != SL_RESULT_SUCCESS) { return; }
lRes = (*mPlayer)->SetCallbackEventsMask(mPlayer, SL_PLAYEVENT_HEADATEND);
if (lRes != SL_RESULT_SUCCESS) { return; }
// ..fetch the data in 'audio->data' from opened FILE* stream and set 'datasize'
// feed the buffer with data
lRes = (*mPlayerQueue)->Clear(mPlayerQueue); // remove any sound from buffer
lRes = (*mPlayerQueue)->Enqueue(mPlayerQueue, audio->data, datasize);
The above works good for 8000, 22050 and 32000 samples/sec but on 41100 samples, 4 out of 5 times it will repeat itself lots of time on first play. It's like having a door knocking sound effect that actually loops many times (about 50 times) by a single ->SetPlayState(..SL_PLAYSTATE_PLAYING); and in speed. Any obvious error on my code? a multi-threaded issue with these sampling? Anyone else have this kind of problem? Should i downsample on 41.1Khz cases ? Could it be a Genymotion problem? tx
I solved this by downsampling from 44Khz to 22Khz. Interestingly, this only happens on sounds containing 1 channel and 44,100 samples; in all other cases there's no problem.

How to monitor playback on ALSA via asoundlib?

I'm building an application that allows for ALSA configuration and in the GUI there is a peek meter that allows the client to see playback levels in realtime. I'm having a hard time determining what device to connect to because I don't know if ALSA has a default "loopback" or not and what it's called. I am also having trouble converting the read data into a sample, then finding said sample's amplitude. Here is what I have built so far:
Grab device and set hardware params
if (0 == snd_pcm_open(&pPcm, "default", SND_PCM_STREAM_CAPTURE, SND_PCM_NONBLOCK))
{
if (0 == snd_pcm_set_params(pPcm, SND_PCM_FORMAT_S16_LE, SND_PCM_ACCESS_RW_INTERLEAVED, 1, 96000, 1, 1)) // This last argument confuses me because I'm not given a unit of measurement (second, millisecond, mircosecond, etc.)
{
return snd_pcm_start(pPcm);
}
}
pPcm = nullptr;
return -1;
Read from device and return the peek of the audio signal
int iRtn = -1;
if (nullptr == pPcm)
{
if (-1 == SetupListener())
{
return iRtn;
}
}
// Check to make the state is sane for reading.
if (SND_PCM_STATE_PREPARED == snd_pcm_state(pPcm) ||
SND_PCM_STATE_RUNNING == snd_pcm_state(pPcm))
{
snd_pcm_resume(pPcm); // This might be superfluous.
// The state is sane, read from the stream.
signed short iBuffer = 0;
int iNumRead = snd_pcm_readi(pPcm, &iBuffer, 1);
if (0 < iNumRead)
{
// This calculates an approximation.
// We have some audio data, acquire it's peek in dB. (decibels)
float nSample = static_cast<float>(iBuffer);
float nAmplitude = nSample / MAX_AMPLITUDE_S16; // MAX_AMPLITUDE_S16 is defined as "32767"
float nDecibels = (0 < nAmplitude) ? 20 * log10(nAmplitude) : 0;
iRtn = static_cast<int>(nDecibels); // Cast to integer for GUI element.
}
}
return iRtn;
The ALSA documentation seems very barren and so I apologize if I'm misusing the API.

Strange code behavior adding OpenCV to VC++ Kinect SDK samples

I'm new to C++ and trying to add OpenCV into Microsoft's Kinect samples. I was able to do it for the ColorBasics-D2D sample by modifying this function
void CColorBasics::ProcessColor()
{
HRESULT hr;
NUI_IMAGE_FRAME imageFrame;
// Attempt to get the color frame
hr = m_pNuiSensor->NuiImageStreamGetNextFrame(m_pColorStreamHandle, 0, &imageFrame);
if (FAILED(hr))
{
return;
}
INuiFrameTexture * pTexture = imageFrame.pFrameTexture;
NUI_LOCKED_RECT LockedRect;
// Lock the frame data so the Kinect knows not to modify it while we're reading it
pTexture->LockRect(0, &LockedRect, NULL, 0);
// Make sure we've received valid data
if (LockedRect.Pitch != 0)
{
BYTE * pBuffer = (BYTE*) LockedRect.pBits;
cvSetData(img,(BYTE*) pBuffer, img->widthStep);
Mat &m = Mat(img);
Mat &hsv = Mat();
vector<Mat> mv = vector<Mat>(3,Mat(cvSize(640,480),CV_8UC1));
cvtColor(m,hsv,CV_BGR2HSV);
cvtColor(hsv,m,CV_HSV2BGR);//*/
IplImage iplimg(m);
cvNamedWindow("rgb",1);
cvShowImage("rgb",&iplimg);
// Draw the data with Direct2D
m_pDrawColor->Draw(static_cast<BYTE *>(LockedRect.pBits), LockedRect.size);
// If the user pressed the screenshot button, save a screenshot
if (m_bSaveScreenshot)
{
WCHAR statusMessage[cStatusMessageMaxLen];
// Retrieve the path to My Photos
WCHAR screenshotPath[MAX_PATH];
GetScreenshotFileName(screenshotPath, _countof(screenshotPath));
// Write out the bitmap to disk
hr = SaveBitmapToFile(static_cast<BYTE *>(LockedRect.pBits), cColorWidth, cColorHeight, 32, screenshotPath);
if (SUCCEEDED(hr))
{
// Set the status bar to show where the screenshot was saved
StringCchPrintf( statusMessage, cStatusMessageMaxLen, L"Screenshot saved to %s", screenshotPath);
}
else
{
StringCchPrintf( statusMessage, cStatusMessageMaxLen, L"Failed to write screenshot to %s", screenshotPath);
}
SetStatusMessage(statusMessage);
// toggle off so we don't save a screenshot again next frame
m_bSaveScreenshot = false;
}
}
// We're done with the texture so unlock it
pTexture->UnlockRect(0);
// Release the frame
m_pNuiSensor->NuiImageStreamReleaseFrame(m_pColorStreamHandle, &imageFrame);
}
This works fine. However, when I wanted to add something like this to the SkeletalViewer example, it is just displaying an empty window.
/// <summary>
/// Handle new color data
/// </summary>
/// <returns>true if a frame was processed, false otherwise</returns>
bool CSkeletalViewerApp::Nui_GotColorAlert( )
{
NUI_IMAGE_FRAME imageFrame;
bool processedFrame = true;
HRESULT hr = m_pNuiSensor->NuiImageStreamGetNextFrame( m_pVideoStreamHandle, 0, &imageFrame );
if ( FAILED( hr ) )
{
return false;
}
INuiFrameTexture * pTexture = imageFrame.pFrameTexture;
NUI_LOCKED_RECT LockedRect;
pTexture->LockRect( 0, &LockedRect, NULL, 0 );
if ( LockedRect.Pitch != 0 )
{
BYTE * pBuffer = (BYTE*) LockedRect.pBits;
cvSetData(img,(BYTE*) pBuffer, img->widthStep);
Mat m(img);
IplImage iplimg(m);
cvNamedWindow("rgb",1);
cvShowImage("rgb",&iplimg);
m_pDrawColor->Draw( static_cast<BYTE *>(LockedRect.pBits), LockedRect.size );
}
else
{
OutputDebugString( L"Buffer length of received texture is bogus\r\n" );
processedFrame = false;
}
pTexture->UnlockRect( 0 );
m_pNuiSensor->NuiImageStreamReleaseFrame( m_pVideoStreamHandle, &imageFrame );
return processedFrame;
}
I'm not sure why the same code doesn't work in this example. I'm using Visual Studio 2010 and OpenCV 2.4.2.
Thanks
Figured it out. Changed it to this
if ( LockedRect.Pitch != 0 )
{
BYTE * pBuffer = static_cast<BYTE *>(LockedRect.pBits);
cvSetData(img,(BYTE*) pBuffer, img->widthStep);
Mat m(img);
IplImage iplimg(m);
cvNamedWindow("rgb",1);
cvShowImage("rgb",&iplimg);
waitKey(1);
m_pDrawColor->Draw( static_cast<BYTE *>(LockedRect.pBits), LockedRect.size );
}

How to get a list of all Microsoft Media Foundation Transforms (MFTs) available on a system

I'm a newbie to native development on Windows, but I've been tasked with creating a small app that will list out all the transformers for various video+audio codecs.
Looking at the MSDN documentation, there doesn't seem to be much direct documentation on doing this. Docs that I've found indicate that this information is stored in the registry (not sure where) so that could be a vector.
Is this possible?
Generally how should I do it?
Thanks
EDIT:
It does seem that a call to MFTEnumEx with the parameters of type MFT_REGISTER_TYPE_INFO set to NULL returns a count of 8
MFTEnumEx(MFT_CATEGORY_VIDEO_DECODER,MFT_ENUM_FLAG_ALL,NULL, NULL, &ppActivate, &count);
assert(count > 0);
Still have to get the actual values though. But the passed ppActivate param should contain an enumeration of them.
EDIT:
It's surprising, but while the count above == 8, there are no video or audio attributes(the video/audio IMFAttributes object is NULL)
IMFAttributes* videoAttributes = NULL;
if(SUCCEEDED(hr)){
hr = pProfile->GetVideoAttributes(&videoAttributes);
//If there are no container attributes set in the transcode profile, the GetVideoAttributes method succeeds and videoAttributes receives NULL.
}
assert(videoAttributes != NULL); //FAILS!
EDIT:
This is a method that pulls all the IMFMediaTypes from the machine(modified call from the book Developing Microsoft® Media Foundation Applications); I then enumerate over them in the caller:
HRESULT CTranscoder::GetVideoOutputAvailableTypes(
DWORD flags,
CComPtr<IMFCollection>& pTypeCollection)
{
HRESULT hr = S_OK;
IMFActivate** pActivateArray = NULL;
MFT_REGISTER_TYPE_INFO outputType;
UINT32 nMftsFound = 0;
do
{
// create the collection in which we will return the types found
hr = MFCreateCollection(&pTypeCollection);
BREAK_ON_FAIL(hr);
// initialize the structure that describes the output streams that the encoders must
// be able to produce. In this case we want video encoders - so major type is video,
// and we want the specified subtype
outputType.guidMajorType = MFMediaType_Video;
outputType.guidSubtype = MFVideoFormat_WMV3;
// get a collection of MFTs that fit the requested pattern - video encoders,
// with the specified subtype, and using the specified search flags
hr = MFTEnumEx(
MFT_CATEGORY_VIDEO_ENCODER, // type of object to find - video encoders
flags, // search flags
NULL, // match all input types for an encoder
&outputType, // get encoders with specified output type
&pActivateArray,
&nMftsFound);
BREAK_ON_FAIL(hr);
// now that we have an array of activation objects for matching MFTs, loop through
// each of those MFTs, extracting all possible and available formats from each of them
for(UINT32 x = 0; x < nMftsFound; x++)
{
CComPtr<IMFTransform> pEncoder;
UINT32 typeIndex = 0;
// activate the encoder that corresponds to the activation object
hr = pActivateArray[x]->ActivateObject(IID_IMFTransform,
(void**)&pEncoder);
// while we don't have a failure, get each available output type for the MFT
// encoder we keep looping until there are no more available types. If there
// are no more types for the encoder, IMFTransform::GetOutputAvailableTypes[]
// will return MF_E_NO_MORE_TYPES
while(SUCCEEDED(hr))
{
IMFMediaType* pType;
// get the avilable type for the type index, and increment the typeIndex
// counter
hr = pEncoder->GetOutputAvailableType(0, typeIndex++, &pType);
if(SUCCEEDED(hr))
{
// store the type in the IMFCollection
hr = pTypeCollection->AddElement(pType);
}
}
}
} while(false);
// possible valid errors that may be returned after the previous for loop is done
if(hr == MF_E_NO_MORE_TYPES || hr == MF_E_TRANSFORM_TYPE_NOT_SET)
hr = S_OK;
// if we successfully used MFTEnumEx() to allocate an array of the MFT activation
// objects, then it is our responsibility to release each one and free up the memory
// used by the array
if(pActivateArray != NULL)
{
// release the individual activation objects
for(UINT32 x = 0; x < nMftsFound; x++)
{
if(pActivateArray[x] != NULL)
pActivateArray[x]->Release();
}
// free the memory used by the array
CoTaskMemFree(pActivateArray);
pActivateArray = NULL;
}
return hr;
}
Caller:
hr=transcoder.GetVideoOutputAvailableTypes( MFT_ENUM_FLAG_ALL, availableTypes);
if (FAILED(hr)){
wprintf_s(L"didn't like the printVideoProfiles method");
}
DWORD availableInputTypeCount =0;
if(SUCCEEDED(hr)){
hr= availableTypes->GetElementCount(&availableInputTypeCount);
}
for(DWORD i = 0; i< availableInputTypeCount && SUCCEEDED(hr); i++)
{
//really a IMFMediaType*
IMFAttributes* mediaInterface = NULL;
if(SUCCEEDED(hr)){
hr = availableTypes->GetElement(i, (IUnknown**)&mediaInterface) ;}
if(SUCCEEDED(hr)){
//see http://msdn.microsoft.com/en-us/library/aa376629(v=VS.85).aspx for a list of attributes to pull off the media interface.
GUID majorType;
hr = mediaInterface->GetGUID(MF_MT_MAJOR_TYPE, &majorType);
LPOLESTR majorGuidString = NULL;
hr = StringFromCLSID(majorType,&majorGuidString);
wprintf_s(L"major type: %s \n", majorGuidString);
wprintf_s(L"is a video? %i \n", IsEqualGUID(MFMediaType_Video,majorType));
GUID subType;
if(SUCCEEDED(mediaInterface->GetGUID(MF_MT_SUBTYPE, &subType))){
LPOLESTR minorGuidString = NULL;
if(SUCCEEDED(StringFromCLSID(subType,&minorGuidString)))
wprintf_s(L"subtype: %s \n", minorGuidString);
}
//Contains a DirectShow format GUID for a media type: http://msdn.microsoft.com/en-us/library/dd373477(v=VS.85).aspx
GUID formatType;
if(SUCCEEDED(mediaInterface->GetGUID(MF_MT_AM_FORMAT_TYPE, &formatType))){
LPOLESTR formatTypeString = NULL;
if(SUCCEEDED(StringFromCLSID(formatType,&formatTypeString)))
wprintf_s(L"format type: %s \n", formatTypeString);
}
UINT32 numeratorFrameRate = 0;
UINT32 denominatorFrameRate = 0;
if(SUCCEEDED(MFGetAttributeRatio(mediaInterface, MF_MT_FRAME_RATE, &numeratorFrameRate, &denominatorFrameRate)))
wprintf_s(L"framerate: %i/%i \n", numeratorFrameRate, denominatorFrameRate);
UINT32 widthOfFrame = 0;
UINT32 heightOfFrame = 0;
if(SUCCEEDED(MFGetAttributeSize(mediaInterface, MF_MT_FRAME_SIZE, &widthOfFrame, &heightOfFrame)))
wprintf_s(L"height of frame: %i width of frame: %i \n", heightOfFrame, widthOfFrame);
UINT32 isCompressedP = 0;
if(SUCCEEDED(mediaInterface->GetUINT32(MF_MT_COMPRESSED, &isCompressedP)))
wprintf_s(L"is media compressed? %iu \n", (BOOL)isCompressedP);
BOOL isCompressedP2 = 0;
if(SUCCEEDED((((IMFMediaType*)mediaInterface)->IsCompressedFormat(&isCompressedP2))))
wprintf_s(L"is media compressed2? %i \n", isCompressedP2);
UINT32 fixedSampleSizeP = 0;
if(SUCCEEDED(mediaInterface->GetUINT32(MF_MT_FIXED_SIZE_SAMPLES, &fixedSampleSizeP)))
wprintf_s(L"is fixed sample size? %iu \n", fixedSampleSizeP);
UINT32 sampleSize = 0;
if(SUCCEEDED(mediaInterface->GetUINT32(MF_MT_SAMPLE_SIZE, &sampleSize)))
wprintf_s(L"sample size: %iu \n", sampleSize);
UINT32 averateBitrate = 0;
if(SUCCEEDED(mediaInterface->GetUINT32(MF_MT_AVG_BITRATE, &averateBitrate)))
wprintf_s(L"average bitrate: %iu \n", averateBitrate);
UINT32 aspectRatio = 0;
if(SUCCEEDED(mediaInterface->GetUINT32(MF_MT_PAD_CONTROL_FLAGS, &aspectRatio)))
wprintf_s(L"4 by 3? %i 16 by 9? %i None? %i \n", aspectRatio == MFVideoPadFlag_PAD_TO_4x3, MFVideoPadFlag_PAD_TO_16x9 == aspectRatio, MFVideoPadFlag_PAD_TO_None == aspectRatio);
UINT32 drmFlag = 0;
if(SUCCEEDED(mediaInterface->GetUINT32(MF_MT_DRM_FLAGS, &drmFlag)))
wprintf_s(L"requires digital drm: %i requires analog drm: %i requires no drm: %i", drmFlag == MFVideoDRMFlag_DigitallyProtected, drmFlag == MFVideoDRMFlag_AnalogProtected, MFVideoDRMFlag_None == drmFlag);
UINT32 panScanEnabled = 0;
if(SUCCEEDED(mediaInterface->GetUINT32(MF_MT_PAN_SCAN_ENABLED, &panScanEnabled)))
wprintf_s(L"pan/scan enabled? %i", panScanEnabled);
UINT32 maxFrameRateNumerator = 0;
UINT32 maxFrameRateDenominator = 0;
if(SUCCEEDED(MFGetAttributeRatio(mediaInterface, MF_MT_FRAME_RATE_RANGE_MAX, &maxFrameRateNumerator, &maxFrameRateDenominator)))
wprintf_s(L"max framerate range: %i/%i \n", maxFrameRateNumerator, maxFrameRateDenominator);
}
}
It's getting some attributes from the IMFMediaInterface, but not many attributes are set and
the call to mediaInterface->GetUINT32(MF_MT_COMPRESSED, &isCompressedP) isn't successful but the call to (IMFMediaType*)mediaInterface)->IsCompressedFormat(&isCompressedP2) is, which makes me wonder if I'm doing it wrong.
This is an old question, but noone should go away unanswered.
As you discovered, MFTEnumEx can give you the list of MFTs, either bulk list, or filtered with a criteria. Now once you have the collection of transforms, you have IMFActivate for every transform available.
Having IMFActivate on hands, see this code snippet how you can obtain information about this transform: you list attributes or access attribute of interest using its key, you can obtain the category, input and output media types (MFT_INPUT_TYPES_Attributes, MFT_OUTPUT_TYPES_Attributes).
Here is sample code and MFT dump samples:
How to enumerate Media Foundation transforms on your system
Enumerating Media Foundation Transforms (MFTs)

Resources