Running 2 infinite loops simultaneously switching control - Python3 - python-3.x

I am building an image processing application that reads the current screen and does some processing. The code skeleton is as given below:
import my_img_proc as mip
import my_app_proc as map
import time
class myprocess:
def perform_image_processing(self,image_name):
while True:
mip.perform_some_image_processing_from_screen(image_name)
time.sleep(180) # Switch to perform_another_processing as next iteration is not needed for next 3 minutes. If this processing takes more than 3 minutes, complete current iteration and then switch to the other function.
def perform_another_processing(self,inp):
while True:
map.perform_some_other_image_processing_from_screen(inp)
time.sleep(180) # Switch to perform_image_processing as next iteration is not needed for next 3 minutes. If this processing takes more than 3 minutes, pause it and switch to the other function.
mp = myprocess()
mp.perform_image_processing(my_image) # Calling of image processing to identify items from the screen capture
mp.perform_another_processing(2) # Calling of application processing to calculate values based on screen capture
As of now, i am able to run only one of the function at a time.
Question here is:
How can i run both of them simultaneously (as 2 separate thread/process??) assuming both the functions may need to access/switch the same screen at the same time.
One option that i think of is both functions setting a common variable (to 1/0) and passing control of execution to each other before going on to sleep? Is it possible? How do i implement it?
Any help with this regards will help me adding multiple other similar functionalities in my application.
Thanks
Bishnu
Note:
For all who could not visualize what i wanted to achieve, here is my code that works fine. This bot code will check the screens to shield(to protect from attacks) or gather (resources in the kingdom)
def shield_and_gather_loop(self):
current_shield_sts = self.renew_shield()
num_of_gatherers = self.num_of_troops_gathering()
gather_sleep = False if num_of_gatherers < self.Max_Army_Size else True
shield_sleep = True if current_shield_sts == 'SHIELDED' else False
shield_timeout= time.time() + 60 * 3 if shield_sleep == True else None
while True:
while shield_sleep == False: #and current_shield_sts != 'SHIELDED':
self.reach_home_screen_from_anywhere()
current_shield_sts = self.renew_shield()
print("Current Shield Status # ", datetime.datetime.now(), " :", current_shield_sts)
shield_sleep = True
#self.go_to_sleep("Shield Check ", hours=0, minutes=3, seconds=0)
shield_timeout = time.time() + 60 * 3 #3 minutes from now
while num_of_gatherers < self.Max_Army_Size and gather_sleep == False:
if time.time() < shield_timeout:
self.device.remove_processed_files()
self.reach_kd_screen_from_anywhere()
self.found_rss_tile = 0
self.find_rss_tile_for_gather(random.choice(self.gather_items))
num_of_gatherers = self.num_of_troops_gathering()
print("Currently gathering: ", str(num_of_gatherers))
if gather_sleep == True and shield_sleep == True:
print("Both gather and shield are working fine.Going to sleep for 2 minutes...")
time.sleep(120)
# Wake up from sleep and recheck shield and gather status
current_shield_sts = self.renew_shield()
num_of_gatherers = self.num_of_troops_gathering()
gather_sleep = False if num_of_gatherers < self.Max_Army_Size else True
shield_sleep = True if current_shield_sts == 'SHIELDED' else False

Seems like the obvious thing to do is this, as it satisfies your criteria of having only one of the two actions running at a time:
import my_img_proc as mip
import my_app_proc as map
import time
class myprocess:
def perform_image_processing(self,image_name,inp):
while True:
mip.perform_some_image_processing_from_screen(image_name)
map.perform_some_other_image_processing_from_screen(inp)
time.sleep(180)
mp = myprocess()
mp.perform_image_processing(my_image,2)
If you need something more elaborate (such as perform_some_other_image_processing_from_screen() being able to interrupt a call to perform_some_image_processing_from_screen() before it has returned), that might require threads, but it would also require your perform_*() functions to serialize access to the data they are manipulating, which would make them slower and more complicated, and based on the comments supplied at the bottom of your example, it doesn't sound like the second function can do any useful work until the first function has completed its identification task anyway.
As a general principle, it's probably worthwhile to move the while True: [...] sleep(180) loop out of the function call and into the top-level code, so that the caller can control its behavior directly. Calling a function that never returns doesn't leave a lot of room for customization to the caller.
import my_img_proc as mip
import my_app_proc as map
import time
class myprocess:
def perform_image_processing(self,image_name,inp):
mip.perform_some_image_processing_from_screen(image_name)
map.perform_some_other_image_processing_from_screen(inp)
mp = myprocess()
while True:
mp.perform_image_processing(my_image,2)
sleep(180)

Related

Trying to make a can counter on a PI 4 in python, Strugging with code for two sensors running at the same time

So I work in the beverage industry and I decided to try and make a can counter using a Raspberry PI4. It needs to use two industrial sensors on the GPIO that detect the cans.
Full Disclosure I have just been googling most code and reading when I get errors in terminal to try and fix the issue. I have done some rudimentary C# and C++ programming doing PLC stuff but it's nothing like what i'm trying to do right now. Just some simple statements for conversions and formulas.
I have had it counting via the sensor on a very rudimentary code
import RPi.GPIO as GPIO
import time
GPIN = 16
GPIO.setmode(GPIO.BCM)
GPIO.setup(GPIN, GPIO.IN)
counting = 0
while True:
while GPIO.input(GPIN) == 0:
time.sleep(0.1)
counting = counting + 1
print(counting)
while GPIO.input(GPIN) == 1:
time.sleep(0.1)
This counts in the terminal. It is of note I need to count the on and off state with a slight delay to keep accidental double counts from happening. I have even added in a GUI with guizero that makes it count in a window. although currently I cannot replicate that from what I remember working and i foolishly didn't save that as i was trying to get to the next stage, but the rough of it was instead of the print(counting) section in the above code I had the app.display() info.
Problem is I need it to count 2 sensors at the same time, one before the can rejector and one after. So I did some reading and figured I needed to run two (or maybe 3) loops at the same time. as I have 2 sensors that need a constant loop, plus it seems like a need another loop that runs and refreshes the GUI. I got turned into threading and have been trying to implement it as that seems like what I want but haven't been able to make heads or tails of it. I can get the GUI to display, but the sensors don't read. If I switch back to my simple code it counts away. Im having trouble meshing the two together.
import threading
from guizero import App, Text, PushButton
import RPi.GPIO as GPIO
import time
GPIN1 = 16
GPIO.setmode(GPIO.BCM)
GPIO.setup(GPIN1, GPIO.IN)
GPIN2 = 15
GPIO.setmode(GPIO.BCM)
GPIO.setup(GPIN2, GPIO.IN)
counting1 = 0
counting2 = 0
counting3 = counting1 - counting2
def sensor1():
global counting1
while GPIO.input(GPIN1) == 0:
time.sleep(0.1)
counting1 = counting1 + 1
while GPIO.input(GPIN1) == 1:
time.sleep(0.1)
def sensor2():
global counting2
while GPIO.input(GPIN2) == 0:
time.sleep(0.1)
counting2 = counting2 + 1
while GPIO.input(GPIN2) == 1:
time.sleep(0.1)
x = threading.Thread(target=sensor1)
y = threading.Thread(target=sensor2)
x.start()
y.start()
while True:
app = App(title="Can Count")
message = Text(app, text="Total")
message = Text(app, text=(counting1))
message = Text(app, text="Rejected")
message = Text(app, text=(counting3))
app.display()
I'm just a bit stumped I'm sure my way isn't the best way to do this, any advice, tips or pointers in the right direction would be appreciated. I'm trying to crash course youtube python tutorials on the side but I am still coming up short.
It seems like I can get the display to show updates if i close the window via the x it restarts the window and shows the update but I have tried a few different things with guizero using a def text(): above that code and text.repeat(10, text) thinking this would redraw the screen but that doesn't work or breaks the gui or the code.
Also I know I call PushButton and don't use it, but the end goal will have a simple reset the counter button.. Just haven't got there yet.

How to update Progressbar in PyQT through QThread which calls another function

I am quite new to PyQT. I am learning how to make a Progressbar and update it as my algorithm progresses. I have been able to make a Basic Progressbar that updates itself using this link: Python pyqt pulsing progress bar with multithreading
The problem is, I need to do the same for my algorithm and that algorithm has a different function and can not be written inside the run function of QThread.
This is my code for the QThread class, (same as given in above link)
class External(QThread):
countChanged = pyqtSignal(int)
def run(self):
count = 0
while count < 100:
count += 1
time.sleep(1)
self.countChanged.emit(count)
And this is the code for my Progressbar,
def generate_timetable(self):
self.calc = External()
self.calc.countChanged.connect(self.onCountChanged)
self.calc.start()
def onCountChanged(self, value):
self.ui.progressBar.setValue(value)
Now, this is kind of a pseudocode of my algorithm function (this function will basically replace the time.sleep(1) function)
def algorithm():
fh = False
best_solution = None
count = 1
best_fitness = 100000 # any huge number
while fh is False:
fh = True
solution, solution_fitness = another_function()
if solution_fitness < best_fitness:
best_solution = solution
best_fitness = solution_fitness
fh = False
print("ITERATION #", count, " is OVER!")
print("Best Solution FItness: ", best_fitness)
count += 1
return best_solution, best_fitness
What I need to do is after every iteration, my Thread class emits a signal and the best solution's fitness value is printed on my main GUI (I have a label for that) and Progressbar's value is also updated (let's say I want it to update by 1 count only, just like it is being done in the current program)
The problem however is this that I can't return the fitness value after one iteration, because if I do that, my algorithm will need to be restarted and hence I will lose my progress. But without returning the value from the function, I don't know how can I possibly know that one iteration is over so now emit the signal, update the progressbar value and also set the label value to the fitness value calculated after the iteration.
Can anyone guide me regarding this? or show me any project that has something similar to this.
I think you can get what you want by just changing your signal from pyqtSignal(int) to pyqtSignal(int, int) and then within your algorithm loop self.countChanged.emit(count, best_fitness). You could also define a second signal for the fitness if you want them separate.

Plot a continuous graph of Number of Snort alerts against time

I have snort logging DDOS alerts to file; I use Syslog-ng to parse the logs and output in json format into redis (wanted to set it up as a buffer, I use 'setex' command with expiry of 70 secs).
The whole thing seems not to be working well; any ideas to make it easier is welcome.
I wrote a simple python script to listen to redis KA events and count the number of snort alerts per second. I tried creating two other threads; one to retrieve the json-formatted alerts from snort and the second to count the alerts. The third is supposed to plot a graph using matplotlib.pyplot
#import time
from redis import StrictRedis as sr
import os
import json
import matplotlib.pyplot as plt
import threading as th
import time
redis = sr(host='localhost', port = 6379, decode_responses = True)
#file = open('/home/lucidvis/vis_app_py/log.json','w+')
# This function is still being worked on
def do_plot():
print('do_plot loop running')
while accumulated_data:
x_values = [int(x['time_count']) for x in accumulated_data]
y_values = [y['date'] for y in accumulated_data]
plt.title('Attacks Alerts per time period')
plt.xlabel('Time', fontsize=14)
plt.ylabel('Snort Alerts/sec')
plt.tick_params(axis='both', labelsize=14)
plt.plot(y_values,x_values, linewidth=5)
plt.show()
time.sleep(0.01)
def accumulator():
# first of, check the current json data and see if its 'sec' value is same
#that is the last in the accumulated data list
#if it is the same, increase time_count by one else pop that value
pointer_data = {}
print('accumulator loop running')
while True:
# pointer data is the current sec of json data used for comparison
#new_data is the latest json formatted alert received
# received_from_redis is a list declared in the main function
if received_from_redis:
new_data = received_from_redis.pop(0)
if not pointer_data:
pointer_data = new_data.copy()
print(">>", type(pointer_data), " >> ", pointer_data)
if pointer_data and pointer_data['sec']==new_data["sec"]
pointer_data['time_count'] +=1
elif pointer_data:
accumulated_data.append(pointer_data)
pointer_data = new_data.copy()
pointer_data.setdefault('time_count',1)
else:
time.sleep(0.01)
# main function creates the redis object and receives messages based on events
#this function calls two other functions and creates threads so they appear to run concurrently
def main():
p = redis.pubsub()
#
p.psubscribe('__keyspace#0__*')
print('Starting message loop')
while True:
try:
time.sleep(2)
message = p.get_message()
# Obtain the key from the redis emmitted event if the event is a set event
if message and message['data']=='set':
# the format emmited by redis is in a dict form
# the key is the value to the key 'channel'
# The key is in '__keyspace#0__*' form
# obtain the last field of the list returned by split function
key = message['channel'].split('__:')[-1]
data_redis = json.loads(redis.get(str(key)))
received_from_redis.append(data_redis)
except Exception e:
print(e)
continue
if __name__ == "__main__":
accumulated_data = []
received_from_redis = []
# main function creates the redis object and receives messages based on events
#this function calls two other functions and creates threads so they appear to run concurrently
thread_accumulator = th.Thread(target = accumulator, name ='accumulator')
do_plot_thread = th.Thread(target = do_plot, name ='do_plot')
while True:
thread_accumulator.start()
do_plot_thread.start()
main()
thread_accumulator.join()
do_plot_thread.join()
I currently do get errors per se ; I just cant tell if the threads are created or are working well. I need ideas to make things work better.
sample of the alert formated in json and obtained from redis below
{"victim_port":"","victim":"192.168.204.130","protocol":"ICMP","msg":"Ping_Flood_Attack_Detected","key":"1000","date":"06/01-09:26:13","attacker_port":"","attacker":"192.168.30.129","sec":"13"}
I'm not sure I understand exactly your scenario, but if you want to count events that are essentially log messages, you can probably do that within syslog-ng. Either as a Python destination (since you are already working in python), or maybe even without additional programming using the grouping-by parser.

Python27 Is it able to make timer without thread.Timer?

So, basically I want to make timer but I don't want to use thread.Timer for
efficiency
Python produces thread by itself, it is not efficient and better not to use it.
I search the essay related to this. And checked It is slow to use thread.
e.g) single process was divided into N, and made it work into Thread, It was slower.
However I need to use Thread for this.
class Works(object):
def __init__(self):
self.symbol_dict = config.ws_api.get("ASSET_ABBR_LIST")
self.dict = {}
self.ohlcv1m = []
def on_open(self, ws):
ws.send(json.dumps(config.ws_api.get("SUBSCRIPTION_DICT")))
everytime I get the message form web socket server, I store in self.dict
def on_message(self,ws,message):
message = json.loads(message)
if len(message) > 2 :
ticker = message[2]
pair = self.symbol_dict[(ticker[0])]
baseVolume = ticker[5]
timestmap = time.time()
try:
type(self.dict[pair])
except KeyError as e:
self.dict[pair] = []
self.dict[pair].append({
'pair':pair,
'baseVolume' : baseVolume,
})
def run(self):
websocket.enableTrace(True)
ws = websocket.WebSocketApp(
url = config.ws_api.get("WEBSOCK_HOST"),
on_message = self.on_message,
on_open = self.on_open
)
ws.run_forever(sslopt = {"cert_reqs":ssl.CERT_NONE})
'once in every 60s it occurs. calculate self.dict and save in to self.ohlcv1m
and will sent it to db. eventually self.dict and self.ohlcv1m initialized again to store 1min data from server'
def every60s(self):
threading.Timer(60, self.every60s).start()
for symbol in self.dict:
tickerLists = self.dict[symbol]
self.ohlcv1m.append({
"V": sum([
float(ticker['baseVolume']) for ticker in tickerLists]
})
#self.ohlcv1m will go to database every 1m
self.ohlcv1 = [] #init again
self.dict = {} #init again
if __name__ == "__main__":
work=Works()
t1 = threading.Thread(target=work.run)
t1.daemon = True
t1.start()
work.every60s()
(sorry for the indention)
I am connecting to socket by running run_forever() and getting realtimedata
Every 60s I need to check and calculate the data
Is there any way to make 60s without thread in python27?
I will be so appreciate you answer If you give me any advice.
Thank you
The answer comes down to if you need the code to run exactly every 60 seconds, or if you can just wait 60 seconds between runs (i.e. if the logic takes 5 seconds, it'll run every 65 seconds).
If you're happy with just a 60 second gap between runs, you could do
import time
while True:
every60s()
time.sleep(60)
If you're really set on not using threads but having it start every 60 seconds regardless of the last poll time, you could time the last execution and subtract that from 60 seconds to get the sleep time.
However, really, with the code you've got there you're not going to run into any of the issues with Python threads you might have read about. Those issues come in when you've got multiple threads all running at the same time and all CPU bound, which doesn't seem to be the case here unless there's some very slow, CPU intensive work that's not in your provided code.

how to unit test the components connected to Rpi?

I had completed my project in raspberry pi and I found it working. I have to test my components and codes.I had gone through pytest but I am not sure how it gonna help me in my case. Is there any automated tools that could be useful to me or any others testing modules that comes with python?
I guess that you can make a try with pytest, but probably you'll need to add something like looped delays or any other kinds of waits if you're going to test asynchronous components.
For example, if you have some class that operates with pins and want to verify high voltage level on button press, you could use something like this:
def test_reading_pin_value():
# let's pretend you have some class to control pins
with InOutController() as ctrl:
ctrl.setup_pin('button', 26, mode=pi.IN, pull_down=True)
while True:
value = ctrl['button']
if value == 1:
break
time.sleep(0.2)
Note that instead of while True you can set some reasonable limit of time and call assert with some condition or pytest.fail if timeout is reached:
import pytest
from time import time
def test_something_asynchronous():
# your external components controller
controller = Controller()
start = time()
wait, timeout = 10.0, False
while not timeout:
value = controller.wait_response()
if value:
assert value == 1, "Invalid response!"
break
elapsed = time() - start
if elapsed >= wait:
timeout = True
assert not timeout, "Timeout!"
If your components have some additional stochasticity, you could try "probabilistic asserts" like:
def probabilistic_assert(condition, trials=100):
n, ok = 0, False
while n < trials and not ok:
ok = condition()
n += 1
assert ok, "Condition has been failed during %d trials" % trials
Also, you could invoke gpio command and parse its output if you want to know voltage levels on pins.
Anyway, I think you will probably need "manual" intervention into tests to activate your outer components or to wait periodically to fulfill test expectations if your components are activated by some external condition.

Resources