I am trying to setup a new server for a project in mechanical engineering. I plan on running a priority queue on the server for sorting what it will do next. Most mechanical engineers do not know how to code so I cannot find many people to help.
For reasons that are difficult to go into, I need to avoid most built in functions withing python. I want to limit myself to the following: pop, insert, if statements, and such. I have a simple one written, however, I need this in one function that computes whenever I input values. The current one essentially will sort at the end.
I want to make it into class PriorityQueue:
Then allow it to sort as items enter
class Queue:
def __init__(self, iL=[]):
self.aList = iL[:]
def enqueue(self, newItem:'item'):
"""put an item in back the queue"""
# your code...
self.aList.append(newItem)
def dequeue(self)->'item': #return the item
"""remove an item from front of the queue, return the item"""
# your code...
return self.aList.pop(0)
def empty(self):
"""Is the queue empty?"""
# your code...
return self.aList == []
def __repr__(self): # for testing only
return repr(self.aList)
# Use the Queue class as a Priority Queue by
# having a priority associated with each item, e.g.:
# (2,'code') where item 'code' have priority 2
# (1, 'eat') this entry has a priority 1 and the item 'eat'
# (3, 'sleep') 'sleep' has priority 3
# Let's consider #1 to be the highest priority, be #1, then 2, then 3
# given this type of priority queue as input
# define a function to sort the queue,
# such that highest priority item (smallest number)will be served first
# return a sorted queue
def queueSorted(qu:Queue) -> Queue:
# your code...
# must use only those 3 queue functions
# cannot use any python function for sorting
qusorted = Queue()
while not qu.empty():
# find min, enqueue it into sorted
qutemp = Queue()
imin = qu.dequeue()
while not qu.empty():
itemp = qu.dequeue()
if itemp < imin:
itemp, imin = imin, itemp
qutemp.enqueue(itemp)
qusorted.enqueue(imin)
# do rest
qu = qutemp
return qusorted
Related
I am simulating people movements and their elevator usage. An elevator can take up multiple persons before moving to another floor. The default process has a capacity parameter, however, these indicate the number of processes and not the number of people using the elevator at the same time.
I have tried to use multiple of the resources available, such as Container, Store, and Base. The elevator should be requested and these objects do not have the functionality to be requested. Hence, the only suitable solution is to inherent from the base.Resource class. I have tried to create a subclass Elevator, implementing from base.Resource and adjusting the function _do_get to take multiple elements from the queue. I am pretty confident that this is not the proper way to implement it and it gives an error as well: RuntimeError: <Request() object at 0x1ffb4474be0> has already been triggered. I have no clue which files to adjust to make Simpy happy. Could someone point me in the right direction?
#dataclass
class Elevator(simpy.Resource):
current_floor: int = 0
available_floors: List[int] = field(default_factory=lambda: [0, 1])
capacity: int = 3
# load_carriers: List[LoadCarrier] = field(default_factory=list)
move_time: int = 5
def __init__(self, env: Environment, capacity: int = 1, elevator_capacity: int = 1):
self.elevator_capacity = elevator_capacity
if capacity <= 0:
raise ValueError('"capacity" must be > 0.')
super().__init__(env, capacity)
self.users: List[Request] = []
"""List of :class:`Request` events for the processes that are currently
using the resource."""
self.queue = self.put_queue
"""Queue of pending :class:`Request` events. Alias of
:attr:`~simpy.resources.base.BaseResource.put_queue`.
"""
#property
def count(self) -> int:
"""Number of users currently using the resource."""
return len(self.users)
if TYPE_CHECKING:
def request(self) -> Request:
"""Request a usage slot."""
return Request(self)
def release(self, request: Request) -> Release:
"""Release a usage slot."""
return Release(self, request)
else:
request = BoundClass(Request)
release = BoundClass(Release)
def _do_put(self, event: Request) -> None:
if len(self.users) < self.capacity:
self.users.append(event)
event.usage_since = self._env.now
event.succeed()
def _do_get(self, event: Release) -> None:
for i in range(min(self.elevator_capacity, len(self.users))):
try:
event = self.users.pop(0)
event.succeed()
# self.users.remove(event.request) # type: ignore
except ValueError:
pass
# event.succeed()
So here is the solution I came up with. The tricky bit is I chained two events together. When you queue up for the elevator you get a event that fires when the elevator arrives. This event also returns a second event that fires when you get to your destination floor. This second event is a common event shared by all the passengers that are on the elevator and going to the same floor. Firing this one event notifies a bunch of passengers. This subscribe broadcast pattern can greatly reduce the number of events the model needs to process which in turn improves performance. I use the chained events because if you are in a queue, and the guy in front of you gets on and you do not, then that guy is also going to get off before you, requiring a different destination arrive event. Put another way, I do not know when you will get off until you get on, so I need to defer that part till you actually get onto the elevator.
"""
Simple elevator demo using events to implements a subscribe, broadcast pattern to let passengers know when
they have reached there floor. All the passengers getting off on the same floor are waiting on the
same one event.
Programmer: Michael R. Gibbs
"""
import simpy
import random
class Passenger():
"""
simple class with unique id per passenger
"""
next_id = 1
#classmethod
def get_next_id(cls):
id = cls.next_id
cls.next_id += 1
return id
def __init__(self):
self.id = self.get_next_id()
class Elevator():
""""
Elevator that move people from floor to floor
Has a max compatity
Uses a event to notifiy passengers when they can get on the elevator
and when they arrive at their destination floor
"""
class Move_Goal():
"""
wrapps passengers so we can track where they are going to
"""
def __init__(self, passenger, start_floor, dest_floor, onboard_event):
self.passenger = passenger
self.start_floor = start_floor
self.dest_floor = dest_floor
self.onboard_event = onboard_event
self.arrive_event = None
def __init__(self,env, passenger_cap, floors):
self.env = env
self.passenger_cap = passenger_cap
self.floors = floors
self.on_floor = 0
self.move_inc = 1
# list of passengers on elevator, one per floor
self.on_board = {f:[] for f in range(1,floors + 1)}
# queue for passengers waitting to get on elevator, one queue per floor
self.boarding_queues = {f:[] for f in range(1,floors + 1)}
# events to notify passengers when they have arrived at their floor, one per floor
self.arrive_events = {f: simpy.Event(env) for f in range(1, floors + 1)}
# start elevator
env.process(self._move_next_floor())
def _move_next_floor(self):
"""
Moves the elevator up and down
Elevator stops at every floor
"""
while True:
# move time to next floor
yield self.env.timeout(5)
# update floor elevator is at
self.on_floor = self.on_floor + self.move_inc
# check if elevator needs to change direction
if self.on_floor == self.floors:
self.move_inc = -1
elif self.on_floor == 1:
self.move_inc = 1
# unload and notify passengers that want to get of at this floor
arrive_event = self.arrive_events[self.on_floor]
self.arrive_events[self.on_floor] = simpy.Event(self.env)
arrive_event.succeed()
self.on_board[self.on_floor] = []
# load new passengers
# get open capacity
used_cap = 0
for p in self.on_board.values():
used_cap += len(p)
open_cap = self.passenger_cap - used_cap
# get boarding passengers
boarding = self.boarding_queues[self.on_floor][:open_cap]
self.boarding_queues[self.on_floor] = self.boarding_queues[self.on_floor][open_cap:]
# sort bording into dest floors
for p in boarding:
# give passenger common event for arriving at destination floor
p.arrive_event = self.arrive_events[p.dest_floor]
# notify passeger that they are onboard the elevator
p.onboard_event.succeed()
self.on_board[p.dest_floor].append(p)
def move_to(self, passenger, from_floor, to_floor):
"""
Return a event that fires when the passenger gets on the elevator
The event returns another event that fires when the passager
arrives at their destination floor
(uses the env.process() to convert a process to a event)
"""
return self.env.process(self._move_to(passenger, from_floor, to_floor))
def _move_to(self, passenger, from_floor, to_floor):
"""
Puts the passenger into a queue for the elevator
"""
# creat event to notify passenger when they can get onto the elemator
onboard_event = simpy.Event(self.env)
# save move data in a wrapper and put passenger into queue
move_goal = self.Move_Goal(passenger, from_floor, to_floor, onboard_event)
self.boarding_queues[from_floor].append(move_goal)
# wait for elevator to arrive, and have space for passenger
yield onboard_event
# get destination arrival event
dest_event = self.arrive_events[to_floor]
move_goal.arrive_event = dest_event
return dest_event
def use_elevator(env, elevator, passenger, start_floor, end_floor):
"""
process for using a elevator to move from one floor to another
"""
print(f'{env.now:.2f} passenger {passenger.id} has queued on floor {start_floor}')
arrive_event = yield elevator.move_to(passenger, start_floor, end_floor)
print(f'{env.now:.2f} passenger {passenger.id} has boarded on floor {start_floor}')
yield arrive_event
print(f'{env.now:.2f} passenger {passenger.id} has arrived on floor {end_floor}')
def gen_passengers(env, elevator):
"""
creates passengers to use a elevatore
"""
floor_set = {f for f in range(1, elevator.floors + 1)}
while True:
# time between arrivals
yield env.timeout(random.uniform(0,5))
# get passenger and where they want to go
passenger = Passenger()
start_floor, end_floor = random.sample(floor_set, 2)
# use the elevator to get there
env.process(use_elevator(env, elevator, passenger, start_floor, end_floor))
# boot up
env = simpy.Environment()
elevator = Elevator(env, 20, 3)
env.process(gen_passengers(env, elevator))
env.run(100)
I am quite new to PyQT. I am learning how to make a Progressbar and update it as my algorithm progresses. I have been able to make a Basic Progressbar that updates itself using this link: Python pyqt pulsing progress bar with multithreading
The problem is, I need to do the same for my algorithm and that algorithm has a different function and can not be written inside the run function of QThread.
This is my code for the QThread class, (same as given in above link)
class External(QThread):
countChanged = pyqtSignal(int)
def run(self):
count = 0
while count < 100:
count += 1
time.sleep(1)
self.countChanged.emit(count)
And this is the code for my Progressbar,
def generate_timetable(self):
self.calc = External()
self.calc.countChanged.connect(self.onCountChanged)
self.calc.start()
def onCountChanged(self, value):
self.ui.progressBar.setValue(value)
Now, this is kind of a pseudocode of my algorithm function (this function will basically replace the time.sleep(1) function)
def algorithm():
fh = False
best_solution = None
count = 1
best_fitness = 100000 # any huge number
while fh is False:
fh = True
solution, solution_fitness = another_function()
if solution_fitness < best_fitness:
best_solution = solution
best_fitness = solution_fitness
fh = False
print("ITERATION #", count, " is OVER!")
print("Best Solution FItness: ", best_fitness)
count += 1
return best_solution, best_fitness
What I need to do is after every iteration, my Thread class emits a signal and the best solution's fitness value is printed on my main GUI (I have a label for that) and Progressbar's value is also updated (let's say I want it to update by 1 count only, just like it is being done in the current program)
The problem however is this that I can't return the fitness value after one iteration, because if I do that, my algorithm will need to be restarted and hence I will lose my progress. But without returning the value from the function, I don't know how can I possibly know that one iteration is over so now emit the signal, update the progressbar value and also set the label value to the fitness value calculated after the iteration.
Can anyone guide me regarding this? or show me any project that has something similar to this.
I think you can get what you want by just changing your signal from pyqtSignal(int) to pyqtSignal(int, int) and then within your algorithm loop self.countChanged.emit(count, best_fitness). You could also define a second signal for the fitness if you want them separate.
I am trying to loop over a set of graphs as shown in the snippet from the main script below:
import pickle
for timeOfDay in range(1440): # total number of minutes in a day=1440
with open("G_" + timeOfDay +'.pickle', 'rb') as handle:
G = pickle.load(handle)
## do something with G
Please note that the loop above cannot be parallelized because the graph at G_i.pickle must be processed before G_{i+1}.pickle. The loading time of one graph is around 1 minute while the processing time for one graph is 30 seconds. So, my whole algorithm is bound by the loading time of graphs. The RAM is sufficient for holding at least 10 graphs and the number of threads available is more than 30, so I am wondering if I can leverage parallel processing to keep updating a buffer dictionary which holds my Graphs from time timeOfDay to timeOfDay+10.
I could load all the Graphs G_1.pickle to G_1440.pickle in the RAM but it exhausts my RAM. So, I tried to create a dictionary G_RAM which has 1440 keys with G_RAM[i] = the graph loaded from G_{i}.pickle. But it runs into a whole lot of issues, sometimes getting stuck sometimes not deleting the older graphs at the right time. I would like to know if there is a standard way to do this without having to re-invent the wheel. Any help is appreciated! Thanks in advance!
To better explain my effort, I have a background thread which does the following:
reads the global value of timeOfDay variable
uses N processes to populate the G_RAM for keys timeOfDay to timeOfDay + N and deletes all keys G_RAM[i] for i < timeOfDay
My attempt is reproduced below:
import multiprocessing
import pickle
import gc
HORIZON_PARALLEL_LOADING_OF_GRAPHS = 10
THREADS_PARALLEL_LOADING_OF_GRAPHS = 5
G_RAM = {}
##### GLOBAL VARIABLES
# global variable to keep track of which timeOfDays are already being worked at by multiple process
# Simply checking for the keys in the G_RAM is not enough because it is possible that some process is
# working to populate it already but it is not complete, hence we keep track of which timeOfDays are already
# being worked at
already_being_worked_at = []
timeOfDay = 0
# Inspired by from https://amalgjose.com/2018/07/18/run-a-background-function-in-python/
def background(f):
'''
a threading decorator
use #background above the function you want to run in the background
'''
def backgrnd_func(*a, **kw):
threading.Thread(target=f, args=a, kwargs=kw).start()
return backgrnd_func
def mp_worker(key):
# load graph at time "key"
with open('G' + str(key) + '.pickle', 'rb') as handle:
read_data = pickle.load(handle)
return (key, read_data)
def mp_handler(data):
p = multiprocessing.Pool(THREADS_PARALLEL_LOADING_OF_GRAPHS)
res = p.map(mp_worker, data)
# populating the global dict here
global G_RAM
for k_v in res:
G_RAM[k_v[0]] = k_v[1]
p.close()
p.join()
#background
def updateDict():
# This will print the count for every second
# G_RAM is a global variable consisting of the graphs at different times of day
global timeOfDay
time.sleep(0.01)
gd_keys = list(G_RAM.keys())
# remove the used graphs (before time timeOfDay)
for key in gd_keys:
if key < timeOfDay or (key > min(timeOfDay + HORIZON_PARALLEL_LOADING_OF_GRAPHS, 1440)):
del G_RAM[key]
gc.collect()
# check which keys (upto t+HORIZON_PARALLEL_LOADING_OF_GRAPHS) are missing from the G_RAM
data = []
for key in range(t, min(t + HORIZON_PARALLEL_LOADING_OF_GRAPHS, 288)):
if key not in G_RAM:
data.append(key)
# removing already processed t to avoid creating duplicate processes
global already_being_worked_at
# we want to call the worker thread again only for the timeOfDays which are not being worked at by worker processes
temp = []
for key in data:
if key not in already_being_worked_at:
temp.append(key)
data = temp
# if the missing timeOfDays are already being worked at by the processes, we have no action item
# otherwise, we call the worker threads with the new timeOfDays which we need but are not being
# worked at
if len(data) > 0:
mp_handler(data)
# we update our list of already_being_worked_at
already_being_worked_at = already_being_worked_at + data
### main script follows
updateDict()
for dayNumber in range(100):
already_being_worked_at = [] # reset the list of populated keys at the start of the new day
for timeOfDay in range(1440): # total number of minutes in a day=1440
with open("G_" + timeOfDay +'.pickle', 'rb') as handle:
G = pickle.load(handle)
## do something with G
I have snort logging DDOS alerts to file; I use Syslog-ng to parse the logs and output in json format into redis (wanted to set it up as a buffer, I use 'setex' command with expiry of 70 secs).
The whole thing seems not to be working well; any ideas to make it easier is welcome.
I wrote a simple python script to listen to redis KA events and count the number of snort alerts per second. I tried creating two other threads; one to retrieve the json-formatted alerts from snort and the second to count the alerts. The third is supposed to plot a graph using matplotlib.pyplot
#import time
from redis import StrictRedis as sr
import os
import json
import matplotlib.pyplot as plt
import threading as th
import time
redis = sr(host='localhost', port = 6379, decode_responses = True)
#file = open('/home/lucidvis/vis_app_py/log.json','w+')
# This function is still being worked on
def do_plot():
print('do_plot loop running')
while accumulated_data:
x_values = [int(x['time_count']) for x in accumulated_data]
y_values = [y['date'] for y in accumulated_data]
plt.title('Attacks Alerts per time period')
plt.xlabel('Time', fontsize=14)
plt.ylabel('Snort Alerts/sec')
plt.tick_params(axis='both', labelsize=14)
plt.plot(y_values,x_values, linewidth=5)
plt.show()
time.sleep(0.01)
def accumulator():
# first of, check the current json data and see if its 'sec' value is same
#that is the last in the accumulated data list
#if it is the same, increase time_count by one else pop that value
pointer_data = {}
print('accumulator loop running')
while True:
# pointer data is the current sec of json data used for comparison
#new_data is the latest json formatted alert received
# received_from_redis is a list declared in the main function
if received_from_redis:
new_data = received_from_redis.pop(0)
if not pointer_data:
pointer_data = new_data.copy()
print(">>", type(pointer_data), " >> ", pointer_data)
if pointer_data and pointer_data['sec']==new_data["sec"]
pointer_data['time_count'] +=1
elif pointer_data:
accumulated_data.append(pointer_data)
pointer_data = new_data.copy()
pointer_data.setdefault('time_count',1)
else:
time.sleep(0.01)
# main function creates the redis object and receives messages based on events
#this function calls two other functions and creates threads so they appear to run concurrently
def main():
p = redis.pubsub()
#
p.psubscribe('__keyspace#0__*')
print('Starting message loop')
while True:
try:
time.sleep(2)
message = p.get_message()
# Obtain the key from the redis emmitted event if the event is a set event
if message and message['data']=='set':
# the format emmited by redis is in a dict form
# the key is the value to the key 'channel'
# The key is in '__keyspace#0__*' form
# obtain the last field of the list returned by split function
key = message['channel'].split('__:')[-1]
data_redis = json.loads(redis.get(str(key)))
received_from_redis.append(data_redis)
except Exception e:
print(e)
continue
if __name__ == "__main__":
accumulated_data = []
received_from_redis = []
# main function creates the redis object and receives messages based on events
#this function calls two other functions and creates threads so they appear to run concurrently
thread_accumulator = th.Thread(target = accumulator, name ='accumulator')
do_plot_thread = th.Thread(target = do_plot, name ='do_plot')
while True:
thread_accumulator.start()
do_plot_thread.start()
main()
thread_accumulator.join()
do_plot_thread.join()
I currently do get errors per se ; I just cant tell if the threads are created or are working well. I need ideas to make things work better.
sample of the alert formated in json and obtained from redis below
{"victim_port":"","victim":"192.168.204.130","protocol":"ICMP","msg":"Ping_Flood_Attack_Detected","key":"1000","date":"06/01-09:26:13","attacker_port":"","attacker":"192.168.30.129","sec":"13"}
I'm not sure I understand exactly your scenario, but if you want to count events that are essentially log messages, you can probably do that within syslog-ng. Either as a Python destination (since you are already working in python), or maybe even without additional programming using the grouping-by parser.
I am trying to asynchronously read and write from a pandas df with an apply function. For this purpose I am using the multithreading.dummy package. Since I am doing read and write simultaneously (multithreaded) on my df, I am using multiprocessing.Lock() so that no more than one thread can edit the df at the a given time. However I am a bit confused to where I should be adding a lock.acquire() and lock.release()with an apply function in pandas. I have tried doing as per below, however, it seems that doing as so the entire process becomes synchronous, so it defeats the whole purpose of multithreading.
self._lock.acquire()
to_df[col_name] = to_df.apply(lambda row: getattr(Object(row['col_1'],
row['col_2'],
row['col_3']),
someattribute), axis=1)
self._lock.release()
Note: In my case I have to be doing getattr. someattribute is simply a #property in Object. Object takes 3 arguments, which some from rows 1,2,3 from my df.
There 2 possible solutions. 1 - locks. 2 - queues. Code below is just a skeleton, it may contain typos/errors and cannot be used as is.
First. Locks where they actually needed:
def method_to_process_url(df):
lock.acquire()
url = df.loc[some_idx, some_col]
lock.release()
info = process_url(url)
lock.acquire()
# add info to df
lock.release()
Second. Queues instead of locks:
def method_to_process_url(df, url_queue, info_queue):
for url in url_queue.get():
info = process_url(url)
info_queue.put(info)
url_queue = queue.Queue()
# add all urls to process to the url_queue
info_queue = queue.Queue()
# working_thread_1
threading.Thread(
target=method_to_process_url,
kwargs={'url_queue': url_queue, 'info_queue': info_queue},
daemon=True).start()
# more working threads
counter = 0
while counter < amount_of_urls:
info = info_queue.get():
# add info to df
counter += 1
In the second case you may even start separate thread for every url without url_queue (reasonable if amount of urls is on the order of thousands or less). counter is some simple way to stop the program when all urls are processed.
I would use the second approach if you ask me. It is more flexible in my opinion.