I'm trying to write a program that listens on a RabbitMQ queue and display statistics using curses.
I would like to listen for keystrokes (to quit or change the display) and queue messages at the same time, reacting appropriately as soon as they arrive. I might also want to add another dissimilar input down the line, such as a socket.
What is the simplest way to accomplish something like this in Python3? selectors? Threads? asyncio?
There seems to be a plethora of options, and yet, it's never clear to me that any of them will meet my use case.
Related
I'm looking for the best way to preform ETL using Python.
I'm having a channel in RabbitMQ which send events (can be even every second).
I want to process every 1000 of them.
The main problem is that RabbitMQ interface (I'm using pika) raise callback upon every message.
I looked at Celery framework, however the batch feature was depreciated in version 3.
What is the best way to do it? I thinking about saving my events in a list, and when it reaches 1000 to copy it to other list and preform my processing. However, how do I make it thread-safe? I don't want to lose events, and I'm afraid of losing events while synchronising the list.
It sounds like a very simple use-case, however I didn't find any good best practice for it.
How do I make it thread-safe?
How about set consumer prefetch-count=1000. If a consumer's unack messages reach its prefetch limit, rabbitmq will not deliver any message to it.
Don't ACK received message, until you have 1000 messages, then copy it to other list and preform your processing. When your job done, ACK the last message, and all message before this message will be ACK by rabbitmq server.
But I am not sure whether large prefetch is the best practice.
First of all, you should not "batch" messages from RabbitMQ unless you really have to. The most efficient way to work with messaging is to process each message independently.
If you need to combine messages in a batch, I would use a separate data store to temporarily store the messages, and then process them when they reach a certain condition. Each time you add an item to the batch, you check that condition (for example, you reached 1000 messages) and trigger the processing of the batch.
This is better than keeping a list in memory, because if your service dies, the messages will still be persisted in the database.
Note : If you have a single processor per queue, this can work without any synchronization mechanism. If you have multiple processors, you will need to implement some sort of locking mechanism.
I'm using event structure and want to do some like Launchpad.
Numeric keyboard have for each number added a sound.
Problem is, that when I press number example one, the program is waiting when the music stop play and next I can press example number four.
Is it possible, to play sounds from 3 key's at the same time using event structure ?
I put the files online here and added screenshots below. Block diagram:
Front panel:
Working Solution
I think I got this working much more easily than I expected using the Play Sound File VI under the Graphics and Sound -> Sound -> Output palette. That link is the 2011 documentation (couldn't find a more recent link), but it does not look like it has changed. The working result is shown below, with two different events handled by the event structure:
Key Down? event:
Stop Button event:
You may be fine without using the Sound Output Clear VI to the right of the main event loop, but having it there won't hurt.
It turns out that the Play Sound File VI does not block, so you can play multiple overlapping sound files. If you run into blocking on your machine (one sound file plays, then the next, and so on), let me know because I have another solution that might work.
A word on events
An important thing to understand is that events are handled in a queue. When you press keys, those key presses go in order onto the event queue. Each time your event-handling loop executes, it takes the oldest event out of that queue and processes it. The event structure in LabVIEW will only handle one event per iteration of your event-handling loop. On the next iteration, if events are still in the queue that your structure is set up to process, it will take the next-oldest one for that iteration and repeat.
Now, lets say that you want to do some super complicated processing that takes 10 seconds every time you press a key, and you put that processing inside of your main event loop. Your key presses still go onto the event queue as fast as you press them, but LabVIEW has to wait the full 10 seconds before it can dequeue the next keypress and process it, so you end up with an application that seems to hang while it chugs through the queue much slower than you are adding items to the queue.
One way to get around this is to take that complicated processing and put it outside of the queue in another process. If you have the resources, you can actually call a separate instance of a processing sub-VI in its own thread for every one of those key presses. This allows the event handling loop to spawn processes as fast as you can press keys, and then your processes take whatever time they need to simultaneously (resources permitting) perform whatever actions you wanted.
Essentially that is what the Play Sound File VI is doing. It sees that you want to play a file and spawns a process to play that sound over the speakers, allowing the event-handling loop to continue immediately rather than waiting for the sound to finish playing. As you press more keys, more processes get spawn that kill themselves when they are finished. You can do this manually too, which is the other solution that I have for you if Play Sound File does not behave the same way for you as it did for me.
Update:
Thanks to #Engineero for pointing out that Play Sound File vi actually isn't blocking. The updated code shows how to play overlapping sounds. I'll leave it to the user to add the Stop Sound on Key Up code. No timeout is needed because nothing is taking place in the event structure.
Also, note that for me the Play Sound vi needed to be in a while loop to keep playing. Not sure why this is needed, but the NI examples sets it up this way (\examples\Graphics and Sound\Sound\Sound Player.vi).
Finally, you may crash the vi if your sound card gets overwhelmed as mentioned here. If that happens I would go with a better sound library to try and squeeze more performance out of your sound card.
Original:
First, I assume you a referring to this Launchpad?
I was able to press up to 4 keys at once will the following - the important thing is to set the event timeout to 1 ms. If you need more than that it will require a more sophisticated design.
I was no able to easily implement a sound because all the basic LabVIEW beeps are what's considered "blocking I/O" meaning if you call 2 Beeps simultaneously than Windows will play one after another not both at the same time. You will need to implement you instrument notes using non blocking I/O probably in a language other than LabVIEW such as this C++ library.
Okay SO is warning me about a subjective title so please let me explain. Right now I'm looking at Go, I've read the spec, watched a few IO talks, it looks interesting but I have some questions.
One of my favourite examples was this select statement that listened to a channel that came from "DoAfter()" or something, the channel would send something at a given time from now.
Something like this (this probably wont work, pseudo-go if anything!)
to := Time.DoAfter(1000 * Time.MS)
select:
case <-to:
return nil //we timed out
case d := <-waitingfor:
return d
Suppose the thing we're waiting for happens really fast, so this function returns and isn't listening to to any more, what happens in DoAfter?
I like and know that you ought not test the channel, for example
if(chanToSendTimeOutOn.isOpen()) {
chanToSendTimeOutOn<-true
}
I like how channels sync places, with this for example it is possible that the function above could return after the isOpen() test but before the sending of true. I really am against the test, this avoids what channels do - hide locks and whatnot.
I've read the spec and seen the run time panics and recovery, but in this example where do we recover? Is the thing waiting to send the timeout a go routine or an "object" of sorts? I imagined this "object" which had a sorted list of things it had to send things to after given times, and that it'd just append TimeAfter requests to the queue in the right order and go through it. I'm not sure where that'd get an opportunity to actually recover.
If it spawned go-routines each with their own timer (managed by the run-time of course, so threads don't actually block for time) what then would get the chance to recover?
The other part of my question is about the lifetime of channels, I would imagine they're ref counted, well those able to read are ref-counted, so that if nothing anywhere holds a readable reference it is destroyed. I'd call this deterministic. For the "point-to-point" topologies you can form it will be if you stick towards Go's "send stuff via channels, don't access it"
So here for example, when the thing that wants a timeout returns the to channel is no longer read by anyone. So the go-routine is pointless now, is there a way to make it return without doing work?
Example:
File-reading go routine that has used defer to close the file when it is done, can it "sense" the channel it is supposed to send stuff to has closed, and thus return without reading any more?
I'd also like to know why the select statement is "nondeterministic" I'd have quite liked it if the first case took priority if the first and second are ready (for a non-blocking operation) - I wont condemn it for that, but is there a reason? What's the implementation of this?
Lastly, how are go-routines scheduled? Does the compiler add some sort of "yielding" every so many instructions, so a thread running will switch between different goroutines? Where can I find info on the lower level stuff?
I know Go touts that "you simply don't need to worry about this" but I like to know what things I write actually hide (that could be a C++ thing) and the reasons why.
If you write to a closed channel, your program will panic (see http://play.golang.org/p/KU7MLrFQSx for example). You could potentially catch this error with recover, but being in a situation where you don't know whether the channel you are writing to is open is usually a sign of a bug in the program. The send side of the channel is responsible for closing it, so it should know the current state. If you have multiple goroutines sending on the channel, then they should coordinate in closing the channel (e.g. by using a sync.WaitGroup).
In your Time.DoAfter hypothetical, it would depend on whether the channel was buffered. If it was an unbuffered channel, then the goroutine writing to the timer channel would block until someone read from the channel. If that never happened, then the goroutine would remain blocked until the program completed. If the channel was buffered, the send would complete immediately. The channel could be garbage collected before anyone read from it.
The standard library time.After behaves this way, returning a channel with a one slot buffer.
I've recently been investigating and playing around with linux message queues and have come across something that I don't quite understand why it happens!
If we have two programs running that are both using msgrcv() in an infinite for loop to check for messages and then send two messages, the first program running will receive the 1st message, and the second program the 2nd message? If you keep sending messages it then alternates between each receiver.
Obviously, I understand that as soon as one program has read the message it is removed from the queue but who/how is it decided who will receive the message if they are all infinitely checking?
Any help would be appreciated!
The short answer is that the kernel decides.
The long answer is that this is handled by the do_msgrcv() call within the Linux kernel. If there is no message available, the caller gets put on a queue until a message is available. It's not guaranteed to go back and forth like you describe, since it all depends on the timing of each msgrcv() call, but in your case, it will probably behave that way virtually all of the time.
I'm running a long process in the background. I've managed to output the console data to gui. But the problem is that, the data is returned only after the process is finished. But I need to display the data at realtime. ie, I need to display the data, every time it produces some output on the console. I'm running the process with in my gui from a seperate thread.
I mean, it would be like building a gui for the ping command, where output is displayed on console after each packet is send, ie at realtime. I just need to redirect that to gui, in realtime. I'm implementing the gui in wxwidgets. Any help would be greatly appreciated.
Thanking You..
Jvc
Is the output you wish to display generated in a separate process from the process running the GUI? Or in a separate thread in the same process?
I ask because most people, when they ask this question, mean a a separate thread. Since you have tagged your question with "process" I will assume that is what you mean.
You need some inter-process communication. There is a bewildering variety of techniques to do this. Personally, I always use sockets.
wxWidgets has simple, easy to use socket classes wxSocketClient and wxSocketServer.
The background process is probably not running wxWidgets, so you will need something else there. I reccomend boost::asio. I know it looks intimidating, but in fact the tutorial code can be used as is.
There is a lot more to be said, but I risk straying away from the point, since there are so few details in your question.
You can have an output queue protected by a wxMutex. The thread doing the computation writes to the queue, then signals the GUI thread using wxQueueEvent with a custom event to let it know that the thread is not empty. The GUI thread then reads the queue and outputs the data.