I have an Android app which uses OpenSL ES for audio. It runs correctly in API17 through API 22. While developing the app, I read somewhere that although OpenSL supports up to 32 objects (platform dependent), Android supports only 11. And so, I designed the app accordingly.
For example I create multiple buffer objects in JNI
// buffer queue player interfaces
// FIRST BUFFER
static SLObjectItf buffer_obj_mainBeat_1 = NULL;
static SLPlayItf buffer_itf_mainBeat_1;
static SLAndroidSimpleBufferQueueItf buf_q_mainBeat_1;
// SECOND BUFFER
static SLObjectItf buffer_obj_mainBeat_2 = NULL;
static SLPlayItf buffer_itf_mainBeat_2;
static SLAndroidSimpleBufferQueueItf buf_q_mainBeat_2;
// THIRD BUFFER
static SLObjectItf buffer_obj_mainBeat_3 = NULL;
static SLPlayItf buffer_itf_mainBeat_3;
static SLAndroidSimpleBufferQueueItf buf_q_mainBeat_3;
Then, I also define the required player interfaces etc...
In the Java code I play each of the defined objects sequentially in loop:
public void playMainBeat {
switch (mainBeat_buf_counter) {
case 1:
setPlayStateMainBeat1(true); // play this sound
setPlayStateMainBeat2(false); // stop next bufQ if not done and rewind
mainBeat_buf_counter = 2;
break;
case 2:
setPlayStateMainBeat2(true);
setPlayStateMainBeat3(false);
mainBeat_buf_counter = 3;
break;
case 3:
setPlayStateMainBeat3(true);
setPlayStateMainBeat1(false);
mainBeat_buf_counter = 1;
break;
}
}
Each of the sound samples that I am playing are about 300 milliseconds. I don't want to stop one sound while playing the next as I loop through the sequence. Notice that I stop the 1st sound after starting the 3rd. Then the loop starts at the top again. The loop in the Java code looks something like the following code:
mainBeat_buf_counter = 1; // initialize counter
while (hasStarted){
playMainBeat();
try {
Thread.sleep(soundInterval);
} catch (InterruptedException e) {; }
}
This all works great until API23 (Marshmallow). Marshmallow will not play all the buffers. It simply fails to play. Sounds are absent in a regular pattern.
I cannot find any documentation regarding the number of objects supported by Android for OpenSL ES. But, if I reduce the number of objects to 7 it does not skip sounds. However, the reduced number of buffers is not adequate for my application. I suspect that the number of supported objects was reduced from 11 objects to 7 objects at API23.
Can anyone confirm the number of objects supported in Android? Or, maybe I am going about this all wrong and there is a better way?
Thanks,
Related
I'm writing to ask you some advices for a particular problem regarding SAPI engine. I have an application that can speak both to the speakers and to a WAV file. I also need some events to be aware, i.e. word boundary and end input.
m_cpVoice->SetNotifyWindowMessage(m_hWnd, TTS_MSG, 0, 0);
hr = m_cpVoice->SetInterest(SPFEI_ALL_EVENTS, SPFEI_ALL_EVENTS);
Just for test I added all events! When the engine speaks to speakers all events are triggered and sent to the m_hWnd window, but when I set output to the WAV file, none of them are sent
CSpStreamFormat fmt;
CComPtr<ISpStreamFormat> pOld;
m_cpVoice->GetOutputStream(&pOld);
fmt.AssignFormat(pOld);
SPBindToFile(file, SPFM_CREATE_ALWAYS, &m_wavStream, &fmt.FormatId(), fmt.WaveFormatExPtr());
m_cpVoice->SetOutput(m_wavStream, false);
m_cpVoice->Speak(L"Test", SPF_ASYNC, 0);
Where file is a path passed as argument.
Really this code is taken from the TTS samples found on the SAPI SDK. It seems a little bit obscure the part setting the format...
Can you help me in finding the problem? Or does anyone of you know a better way to write TTS to WAV? I can not use manager code, it should be better to use the C++ version...
Thank you very much for help
EDIT 1
This seems to be a thread problem and searching in the spuihelp.h file, that contains the SPBindToFile helper I found that it uses the CoCreateInstance() function to create the stream. Maybe this is where the ISpVoice object looses its ability to send event in its creation thread.
What do you think about that?
I adopted an on-the-fly solution that I think should be acceptable in most of the cases, In fact when you write speech on files, the major event you would be aware is the "stop" event.
So... take a look a the class definition:
#define TTS_WAV_SAVED_MSG 5000
#define TTS_WAV_ERROR_MSG 5001
class CSpeech {
public:
CSpeech(HWND); // needed for the notifications
...
private:
HWND m_hWnd;
CComPtr<ISpVoice> m_cpVoice;
...
std::thread* m_thread;
void WriteToWave();
void SpeakToWave(LPCWSTR, LPCWSTR);
};
I implemented the method SpeakToWav as follows
// Global variables (***)
LPCWSTR tMsg;
LPCWSTR tFile;
long tRate;
HWND tHwnd;
ISpObjectToken* pToken;
void CSpeech::SpeakToWave(LPCWSTR file, LPCWSTR msg) {
// Using, for example wcscpy_s:
// tMsg <- msg;
// tFile <- file;
tHwnd = m_hWnd;
m_cpVoice->GetRate(&tRate);
m_cpVoice->GetVoice(&pToken);
if(m_thread == NULL)
m_thread = new std::thread(&CSpeech::WriteToWave, this);
}
And now... take a look at the WriteToWave() method:
void CSpeech::WriteToWav() {
// create a new ISpVoice that exists only in this
// new thread, so we need to
//
// CoInitialize(...) and...
// CoCreateInstance(...)
// Now set the voice, i.e.
// rate with global tRate,
// voice token with global pToken
// output format and...
// bind the stream using tFile as I did in the
// code listed in my question
cpVoice->Speak(tMsg, SPF_PURGEBEFORESPEAK, 0);
...
Now, because we did not used the SPF_ASYNC flag the call is blocking, but because we are on a separate thread the main thread can continue. After the Speak() method finished the new thread can continue as follow:
...
if(/* Speak is went ok */)
::PostMessage(tHwn, TTS_WAV_SAVED_MSG, 0, 0);
else
::PostMessage(tHwnd, TTS_WAV_ERROR_MSG, 0, 0);
}
(***) OK! using global variables is not quite cool :) but I was going fast. Maybe using a thread with the std::reference_wrapper to pass parameters would be more elegant!
Obviously, when receiving the TTS messages you need to clean the thread for a next time call! This can be done using a CSpeech::CleanThread() method like this:
void CSpeech::CleanThread() {
m_thread->join(); // I prefer to be sure the thread has finished!
delete m_thread;
m_thread = NULL;
}
What do you think about this solution? Too complex?
I have a client application that receives video stream from a server via UDP or TCP socket.
Originally, when it was written using .NET 2.0 the code was using BeginReceive/EndReceive and IAsyncResult.
The client displays each video in it's own window and also using it's own thread for communicating with the server.
However, since the client is supposed to be up for a long period of time, and there might be 64 video streams simultaneously, there is a "memory leak" of IAsyncResult objects that are allocated each time the data receive callback is called.
This causes the application eventually to run out of memory, because the GC can't handle releasing of the blocks in time. I verified this using VS 2010 Performance Analyzer.
So I modified the code to use SocketAsyncEventArgs and ReceiveFromAsync (UDP case).
However, I still see a growth in memory blocks at:
System.Net.Sockets.Socket.ReceiveFromAsync(class System.Net.Sockets.SocketAsyncEventArgs)
I've read all the samples and posts about implementing the code, and still no solution.
Here's how my code looks like:
// class data members
private byte[] m_Buffer = new byte[UInt16.MaxValue];
private SocketAsyncEventArgs m_ReadEventArgs = null;
private IPEndPoint m_EndPoint; // local endpoint from the caller
Initializing:
m_Socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
m_Socket.Bind(m_EndPoint);
m_Socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReceiveBuffer, MAX_SOCKET_RECV_BUFFER);
//
// initalize the socket event args structure.
//
m_ReadEventArgs = new SocketAsyncEventArgs();
m_ReadEventArgs.Completed += new EventHandler<SocketAsyncEventArgs>(readEventArgs_Completed);
m_ReadEventArgs.SetBuffer(m_Buffer, 0, m_Buffer.Length);
m_ReadEventArgs.RemoteEndPoint = new IPEndPoint(IPAddress.Any, 0);
m_ReadEventArgs.AcceptSocket = m_Socket;
Starting the read process:
bool waitForEvent = m_Socket.ReceiveFromAsync(m_ReadEventArgs);
if (!waitForEvent)
{
readEventArgs_Completed(this, m_ReadEventArgs);
}
Read completion handler:
private void readEventArgs_Completed(object sender, SocketAsyncEventArgs e)
{
if (e.BytesTransferred == 0 || e.SocketError != SocketError.Success)
{
//
// we got error on the socket or connection was closed
//
Close();
return;
}
try
{
// try to process a new video frame if enough data was read
base.ProcessPacket(m_Buffer, e.Offset, e.BytesTransferred);
}
catch (Exception ex)
{
// log and error
}
bool willRaiseEvent = m_Socket.ReceiveFromAsync(e);
if (!willRaiseEvent)
{
readEventArgs_Completed(this, e);
}
}
Basically the code works fine and I see the video streams perfectly, but this leak is a real pain.
Did I miss anything???
Many thanks!!!
Instead of recursively calling readEventArgs_Completed after !willRaiseEvent use goto to return to the top of the method. I noticed I was slowly chewing up stack space when I had a pattern similar to yours.
I have a couple of tasks to do with arduino but one of them takes very long time, so I was thinking to use threads to run them simultaneously.
I have an Arduino Mega
[Update]
Finally after four years I can install FreeRTOS in my arduino mega. Here is a link
In short: NO.
But you may give it a shot at:
http://www.kwartzlab.ca/2010/09/arduino-multi-threading-librar/
(Archived version: https://web.archive.org/web/20160505034337/http://www.kwartzlab.ca/2010/09/arduino-multi-threading-librar
Github: https://github.com/jlamothe/mthread
Not yet, but I always use this Library with big projects:
https://github.com/ivanseidel/ArduinoThread
I place the callback within a Timer interrupt, and voilá! You have pseudo-threads running on the Arduino...
Just to make this thread more complete: there are also protothreads which have very small memory footprint (couple bytes if I remember right) and preserve variables local to thread; very handy and time saving (far less finite state machines -> more readable code).
Examples and code:
arduino-class / ProtoThreads wiki
Just to let you know what results you may expect: serial communication # 153K6 baudrate with threads for: status diodes blinking, time keeping, requested functions evaluation, IO handling and logic and all on atmega328.
Not real threading but TimedActions are a good alternative for many uses
http://playground.arduino.cc/Code/TimedAction#Example
Of course, if one task blocks, the others will too, while threading can let one task freeze and the others will continue...
No you can't but you can use Timer interrupt.
Ref : https://www.teachmemicro.com/arduino-timer-interrupt-tutorial/
The previous answer is correct, however, the arduino generally runs pretty quick, so if you properly time your code, it can accomplish tasks more or less simultaneously.
The best practice is to make your own functions and avoid putting too much real code in the default void loop
You can use arduinos
It is designed for Arduino environment. Features:
Only static allocation (no malloc/new)
Support context switching when delaying execution
Implements semaphores
Lightweight, both cpu and memory
I use it when I need to receive new commands from bluetooth/network/serial while executing the old ones and the old ones have delay in them.
One thread is the sever thread that does the following loop:
while (1) {
while ((n = Serial.read()) != -1) {
// do something with n, like filling a buffer
if (command_was_received) {
arduinos_create(command_func, arg);
}
}
arduinos_yield(); // context switch to other threads
}
The other is the command thread that executes the command:
int command_func(void* arg) {
// move some servos
arduinos_delay(1000); // wait for them to move
// move some more servos
}
Arduino does not support multithread programming.
However there have been some workarounds, for example the one in this project (you can install it also from the Arduino IDE).
It seems you have to define the schedule time yourself while in a real multithread environment it is the OS that decides when to execute tasks.
Alternatively you can use protothreads
The straight answer is No No No!. There are some alternatives but you can't expect a perfect multi threading functionality from an arduino mega. You can use arduino due or lenado for multithreading like below-
void loop1(){
}
void loop2(){
}
void loop3(){
}
Normally, I handle those types of cases in backend. You can run the main code in a server while using Arduino to just collect inputs and show outputs. In such cases I would prefer nodemcu which has built in wifi.
Thread NO!
Concurrent YES!
You can run different tasks concurrently with FreeRTOS library.
https://www.arduino.cc/reference/en/libraries/freertos/
void TaskBlink( void *pvParameters );
void TaskAnalogRead( void *pvParameters );
// Now set up two tasks to run independently.
xTaskCreate(
TaskBlink
, (const portCHAR *)"Blink" // A name just for humans
, 128 // Stack size
, NULL
, 2 // priority
, NULL );
xTaskCreate(
TaskAnalogRead
, (const portCHAR *) "AnalogRead"
, 128 // This stack size can be checked & adjusted by reading Highwater
, NULL
, 1 // priority
, NULL );
void TaskBlink(void *pvParameters) // This is a task.
{
(void) pvParameters;
// initialize digital pin 13 as an output.
pinMode(13, OUTPUT);
for (;;) // A Task shall never return or exit.
{
digitalWrite(13, HIGH); // turn the LED on (HIGH is the voltage level)
vTaskDelay( 1000 / portTICK_PERIOD_MS ); // wait for one second
digitalWrite(13, LOW); // turn the LED off by making the voltage LOW
vTaskDelay( 1000 / portTICK_PERIOD_MS ); // wait for one second
}
}
void TaskAnalogRead(void *pvParameters) // This is a task.
{
(void) pvParameters;
// initialize serial communication at 9600 bits per second:
Serial.begin(9600);
for (;;)
{
// read the input on analog pin 0:
int sensorValue = analogRead(A0);
// print out the value you read:
Serial.println(sensorValue);
vTaskDelay(1); // one tick delay (15ms) in between reads for stability
}
}
Just take care!
When different tasks tried to reach variables at the same time, like i2c communication line or sd card module. Use Semaphores and mutexes
https://www.geeksforgeeks.org/mutex-vs-semaphore/.
Arduino does not supports threading. However, you can do the next best thing and structure your code around state machines running in interleaving.
While there are lots of ways to implement your tasks as state machines, I recommend this library (https://github.com/Elidio/StateMachine). This library abstracts most of the process.
You can create a state machine as a class like this:
#include "StateMachine.h"
class STATEMACHINE(Blink) {
private:
int port;
int waitTime;
CREATE_STATE(low);
CREATE_STATE(high);
void low() {
digitalWrite(port, LOW);
*this << &STATE(high)<< waitTime;
}
void high() {
digitalWrite(port, HIGH);
*this << &STATE(low)<< waitTime;
}
public:
Blink(int port = 0, int waitTime = 0) :
port(port),
waitTime(waitTime),
INIT_STATE(low),
INIT_STATE(high)
{
pinMode(port, OUTPUT);
*this << &STATE(low);
}
};
The macro STATEMACHINE() abstracts the class inheritances, the macro CREATE_STATE() abstracts the state wrapper creation, the macro INIT_STATE() abstracts method wrapping and the macro STATE() abstracts state wrapper reference within the state machine class.
State transition is abstracted by << operator between the state machine class and the state, and if you want a delayed state transition, all you have to do is to use that operator with an integer, where the integer is the delay in millisseconds.
To use the state machine, first you have to instantiate it. Declaring an reference to the class in global space while instantiating it with new on setup function might do the trick
Blink *led1, *led2, *led3;
void setup() {
led1 = new Blink(12, 300);
led2 = new Blink(11, 500);
led3 = new Blink(10, 700);
}
Then you run the states on loop.
void loop() {
(*led2)();
(*led1)();
(*led3)();
}
We are developing a WPF application using TDD. As we're already working on this solution for almost two years, we've written a huge bunch of tests (almost 2000 Unittests right now).
There are some classes, that need to implement functionality multithreaded and asynchronously. For example a communication-component that can both send and receive messages and parse them. The dependencies are always mocked using RhinoMocks.
Our Test-Methods targeting these classes look very similar, as following:
[TestMethod]
public void Method_Description_ExpectedResult(){
// Arrange
var myStub = MockRepository.GenerateStub<IMyStub>();
var target = new MyAsynchronousClass(myStub);
// Act
var target.Send("Foo");
Thread.Sleep(200);
//Assert
myStub.AssertWasCalled(x => x.Bar("Foo"));
}
As you can see, this test runs at least for 200 ms due to the Thread.Sleep(). We optimized the test replacing the AssertWasCalled with a active polling method, s.th. like this:
public static bool True(Func<bool> condition, int times, int waitTime)
{
for (var i = 0; i < times; i++)
{
if (condition())
return true;
Thread.Sleep(waitTime);
}
return condition();
}
We can now use this WaitFor.True(...) Method by changing the AssertWasCalled to:
var fooTriggered = false;
myStub.Stub(x => x.Bar("Foo")).Do((Action)(() => fooTriggered = true)));
WaitFor.True(() => fooTriggered, 20, 20);
Assert.IsTrue(fooTriggered);
This construct will terminate earlier if the condition matches, but anyway - this takes too long for us. Running all of our 2000 Tests need about 5 Minutes (building and running them).
Is there any smart trick how we could optimize code like this?
You can use a monitor. I'm making this up so please excuse me if it isn't quite compiling, but it'll look something like:
[TestMethod]
public void Method_Description_ExpectedResult(){
// Arrange
var waitingRoom = new object();
var myStub = MockRepository.GenerateStub<IMyStub>();
myStub.Setup(x => x.Bar("Foo")).Callback(x =>
{
Monitor.Enter(waitingRoom);
Monitor.Pulse(waitingRoom);
Monitor.Exit(waitingRoom);
}
var target = new MyAsynchronousClass(myStub);
// Act
Monitor.Enter(waitingRoom);
target.Send("Foo");
Monitor.Wait(waitingRoom);
Monitor.Exit(waitingRoom);
//Assert
myStub.AssertWasCalled(x => x.Bar("Foo"));
}
Code written within the Monitor can't run until it's free. The test will cause the acting thread to wait until Monitor.Wait has been called. Then the callback can enter and pulse the Monitor. The test then "wakes up", and once the callback has exited the monitor, it gets control back and exits too, allowing you to Assert.
The only thing I haven't covered is that if Bar("Foo") doesn't get called it will hang, so you might want to have a timer pulse the thread too.
You can create a class which does the complex monitoring bits for you if you use it a lot. This is one I wrote to deal with asynchronous checks in UI automation; adapting it for what you're doing might help you.
So I was writing a quick application to sort my wallpapers neatly into folders according to aspect ratio. Everything is going smoothly until I try to actually move the files (using FileInfo.MoveTo()). The application throws an exception:
System.IO.IOException
The process cannot access the file because it is being used by another process.
The only problem is, there is no other process running on my computer that has that particular file open. I thought perhaps that because of the way I was using the file, perhaps some internal system subroutine on a different thread or something has the file open when I try to move it. Sure enough, a few lines above that, I set a property that calls an event that opens the file for reading. I'm assuming at least some of that happens asynchronously. Is there anyway to make it run synchronously? I must change that property or rewrite much of the code.
Here are some relevant bits of code, please forgive the crappy Visual C# default names for things, this isn't really a release quality piece of software yet:
private void button1_Click(object sender, EventArgs e)
{
for (uint i = 0; i < filebox.Items.Count; i++)
{
if (!filebox.GetItemChecked((int)i)) continue;
//This calls the selectedIndexChanged event to change the 'selectedImg' variable
filebox.SelectedIndex = (int)i;
if (selectedImg == null) continue;
Size imgAspect = getImgAspect(selectedImg);
//This is gonna be hella hardcoded for now
//In the future this should be changed to be generic
//and use some kind of setting schema to determine
//the sort/filter results
FileInfo file = ((FileInfo)filebox.SelectedItem);
if (imgAspect.Width == 8 && imgAspect.Height == 5)
{
finalOut = outPath + "\\8x5\\" + file.Name;
}
else if (imgAspect.Width == 5 && imgAspect.Height == 4)
{
finalOut = outPath + "\\5x4\\" + file.Name;
}
else
{
finalOut = outPath + "\\Other\\" + file.Name;
}
//Me trying to tell C# to close the file
selectedImg.Dispose();
previewer.Image = null;
//This is where the exception is thrown
file.MoveTo(finalOut);
}
}
//The suspected event handler
private void filebox_SelectedIndexChanged(object sender, EventArgs e)
{
FileInfo selected;
if (filebox.SelectedIndex >= filebox.Items.Count || filebox.SelectedIndex < 0) return;
selected = (FileInfo)filebox.Items[filebox.SelectedIndex];
try
{
//The suspected line of code
selectedImg = new Bitmap((Stream)selected.OpenRead());
}
catch (Exception) { selectedImg = null; }
if (selectedImg != null)
previewer.Image = ResizeImage(selectedImg, previewer.Size);
else
previewer.Image = null;
}
I have a long-fix in mind (that's probably more efficient anyway) but it presents more problems still :/
Any help would be greatly appreciated.
Since you are using your selectedImg as a Class scoped variable it is keeping a lock on the File while the Bitmap is open. I would use an using statement and then Clone the Bitmap into the variable you are using this will release the lock that Bitmap is keeping on the file.
Something like this.
using ( Bitmap img = new Bitmap((Stream)selected.OpenRead()))
{
selectedImg = (Bitmap)img.Clone();
}
New answer:
I looked at the line where you do an OpenRead(). Clearly, this locks your file. It would be better to provide the file path instead of an stream, because you can't dispose your stream since bitmap would become erroneous.
Another thing I'm looking in your code which could be a bad practice is binding to FileInfo. Better create a data-transfer object/value object and bind to a collection of this type - some object which has the properties you need to show in your control -. That would help in order to avoid file locks.
In the other hand, you can do some trick: why don't you show streched to screen resolution images compressing them so image size would be extremly lower than actual ones and you provide a button called "Show in HQ"? That should solve the problem of preloading HD images. When the user clicks "Show in HQ" button, loads that image in memory, and when this is closed, it gets disposed.
It's ok for you?
If I'm not wrong, FileInfo doesn't block any file. You're not opening it but reading its metadata.
In the other hand, if you application shows images, you should move to memory visible ones and load them to your form from a memory stream.
That's reasonable because you can open a file stream, read its bytes and move them to a memory stream, leaving the lock against that file.
NOTE: This solution is fine for not so large images... Let me know if you're working with HD images.
using(selectedImg = new Bitmap((Stream)selected))
Will that do it?