I have a function that receives a list of integers and what each integer does is build a json, consume some api, do validations and then insert it into a database. But there is a problem, with a list of 2 values the function takes seconds but with a list of 200 values it takes minutes more than 8 minutes, is it possible to optimize that time by sending that amount of values? , it is possible to send a series of values so that the function does its job and at the same time send another amount of values and that the function does its job and thus, simultaneously send, for example, 50 values to the function and then send other 50 to the same function at the same time to optimize time? or how would be the ideal way to optimize it
Generally there are two approaches. Optimizing the process for a single call and running multiple calls at once also known as parallelization.
For optimizing a single call, you should start by profiling the function. Python has a built-in profiler, cProfile. There are other packages as well such as py-spy which can have the results formatted to be integrated into https://www.speedscope.app/ which is a nice visual tool.
Regarding parallelization, it depends on how this function is used. You could take a look at Asyncio. However if you're hitting the database, that's probably going to prove difficult and you'll need to integrate Django Channels or something similar. Django is still working on proper Async DB functionality. Alternatively, you can use something like Celery and wrap this function in a Task. Then you can schedule calls with various arguments and scale up the workers as needed.
Hopefully this helps you on your search and understanding of how to build what you need.
Related
I have python function which receives and store live tick data from a server(By API requestes). And another function which fetch the data to candle bars of 1minute and appends it to pandas Data frame. Then i want to call another function which apply some mathematical computations and manages order execution in live market.
But i am confused in which method to use between Multi-threading, Multi-processing or AsyncIO. What i want is uninterrupted flow of tick data which receives data in fractions of milliseconds to my system so that i donot miss any realtime data, And at the same time able to manage orders and perform mathematical computations.
Please advise me which option will be better to choose from the above?
I think you should use multi-threading and/or asyncIO since getting processes to work together can be a pain in the neck. Since you need to do computations, you should store the data/use a queue and use a second process to make the calculations,and you can add more math processes if one couldn't catch up with data inflow.
On second thought. You'll still have to carefully pack data fast enough so you don't spend most of your time transferring the data(pickle or process Queues don't work 'cause they're too slow)you'll need some custom way, say structs to quickly pack and unpack the data.
But at that point, you might as well use C/C++ as your second(math processing) process :)
TL;DR: Use ansycio and/or threading to receive data, custom structs to quickly pack/unpack data, and a few other Python/C/C++ etc. processes to retrieve and process the data.
I'm new with multi-threading in Matlab so I guess that what I need to do will be simple for anyone with a little experience in it.
I have two functions f1 and f2 such that:
f1 - runs about 10 seconds and returns accurate results.
f2 - return estimated results immediately.
Both functions get the same input and return the same output (but in f1 the output is more accurate).
The functions get new inputs all the time and run constantly.
I want to run the functions in two different threads so that the user will never need to wait.
and every time one of them finishes, the other one uses its results.
The goal is to get more accurate results than running f2 all the time. Running f1 all the time is the optimal solution but then the user needs to wait 10 seconds each time. So I want to combine the two functions and to get something in the middle.
What is the best way to implement the above?
I read about batch but I'm not sure it's what I need (at least I didn't find how to do the above with batch).
Can you just lead me with the relevant functions and I'll read about them and learn how to use them?
Thanks
I am currently working on an image processing object in Matlab. I am acquiring images from a webcam using the snapshot function, which are to be processed in various ways (irrelevant to the question).
I would like these snapshots to be acquired every 5 seconds. However, in these 5 seconds, I do not want my program to pause and wait, I want it to run the image processing functions. I have tried pause, but that obviously pauses the whole program. The way I imagine the processor circuitry from my basic IC knowledge, I am looking to implement an event coming from a clock counter, which would stop the machine instructions dealing with the image processing part, and prioritise the instructions involving the image acquisition.
I have stumbled on this link that talks about multithreading in Matlab using Java. Is there an easier way of implementing what I want to do?
Could you please suggest some functions that achieve what I want to do? If there is no function which does what I want, could you point me towards some articles or books which deal with the subject?
You could use a timer object
rate=5; % call every 5 s
my_timer= timer('TimerFcn',{#my_timer_callback,arguments}, 'Period', rate,'ExecutionMode', 'fixedRate'); % specify arguments for additional arguments
start(my_timer) % stop(my_timer) to end processing
and do the processing inside my_timer_callback.
function my_timer_callback(obj,event,arguments)
% do processing here
Better would be to run the callback triggered by the camera, so I would look into whether Matlab allows you to attach callbacks to the camera data acquisition (e.g. in the same way as for daq objects).
I have three Linux boxes, each running my program.
The program needs to call a certain callback at regular intervals, and each call must happen at the exact same time across the three boxes. I don't need any other synchronization except for the calls.
If it helps, the three boxes have their clocks synchronized by NTP (one of the boxes is the master).
Is there a way to accomplish this with good precision? Preferably non Linux specific. To make things simple, the callback must be called each N ms even if a previous call hasn't completed yet.
How about you send a request to execute the function far enough ahead of time including a timestamp when the function should be executed? The receiving application would sleep/wait the remaining time (some time is lost due to latency), then execute the function at the precise timestamp you requested.
If the called function itself takes longer than your interval, you should probably consider using threads. If the function executes quickly but transfer of the results takes longer then you should get away with something like select() without additional threads.
I am currently working with three matlab functions to make them run near simultaneously in single Matlab session(as I known matlab is single-threaded), these three functions are allocated with individual tasks, it might be difficult for me to explain all the detail of each function here, but try to include as much information as possible.
They are CONTROL/CAMERA/DATA_DISPLAY tasks, The approach I am using is creating Timer objects to have all the function callback continuously with different callback period time.
CONTROL will sending and receiving data through wifi with udp port, it will check the availability of package, and execute callback constantly
CAMERA receiving camera frame continuously through tcp and display it, one timer object T1 for this function to refresh the capture frame
DATA_DISPLAY display all the received data, this will refresh continuously, so another timer T2 for this function to refresh the display
However I noticed that the timer T2 is blocking the timer T1 when it is executed, and slowing down the whole process. I am working on a system using a multi-core CPU and I would expect MATLAB to be able to execute both timer objects in parallel taking advantage of the computational cores.
Through searching the parallel computing toolbox in matlab, it seems not able to deal with infinite loop or continuous callback, since the code will not finish and display nothing when execute, probably I am not so sure how to utilize this toolbox
Or can anyone provide any good idea of re-structuring the code into more efficient structure.
Many thanks
I see a problem using the parallel computing toolbox here. The design implies that the jobs are controlled via your primary matlab instance. Besides this, the primary instance is the only one with a gui, which would require to let your DISPLAY_DATA-Task control everything. I don't know if this is possible, but it would result in a very strange architecture. Besides this, inter process communication is not the best idea when processing large data amounts.
To solve the issue, I would use Java to display your data and realise the 'DISPLAY_DATA'-Part. The connection to java is very fast and simple to use. You will have to write a small java gui which has a appendframe-function that allows your CAMERA-Job to push new data. Obviously updating the gui should be done parallel without blocking.