Consider a carwash as described in the Simpy example list. Now assume that each time a car is serviced, the corresponding washing unit has to be serviced itself, before it can clean the next car. Meanwhile, the serviced car can leave.
How would I best model the above? My current solution is to have "ghost cars" with a high priority that block the carwash during the regeneration time. I don't find this solution very elegant, and guess there is a better way.
In the below example, which represents a poor copy of the above-mentioned tutorial, the serviced car cannot leave the pump during the regeneration period. How could I fix that to mimic the intended behavior? I guess the solution is straightforward; I just don't see it.
import random
import simpy
RANDOM_SEED = 42
NUM_MACHINES = 2 # Number of machines in the carwash
WASHTIME = 5 # Minutes it takes to clean a car
REGENTIME = 3
T_INTER = 7 # Create a car every ~7 minutes
SIM_TIME = 20 # Simulation time in minutes
class Carwash(object):
def __init__(self, env, num_machines, washtime):
self.env = env
self.machine = simpy.Resource(env, num_machines)
self.washtime = washtime
def wash(self, car):
yield self.env.timeout(WASHTIME)
print("Carwash removed %s's dirt at %.2f." % (car, env.now))
def regenerateUnit(self):
yield self.env.timeout(REGENTIME)
print("Carwash's pump regenerated for next user at %.2f." % (env.now))
def car(env, name, cw):
print('%s arrives at the carwash at %.2f.' % (name, env.now))
with cw.machine.request() as request:
yield request
print('%s enters the carwash at %.2f.' % (name, env.now))
yield env.process(cw.wash(name))
yield env.process(cw.regenerateUnit())
print('%s leaves the carwash at %.2f.' % (name, env.now))
def setup(env, num_machines, washtime, t_inter):
# Create the carwash
carwash = Carwash(env, num_machines, washtime)
# Create 4 initial cars
for i in range(4):
env.process(car(env, 'Car %d' % i, carwash))
# Create more cars while the simulation is running
while True:
yield env.timeout(random.randint(t_inter - 2, t_inter + 2))
i += 1
env.process(car(env, 'Car %d' % i, carwash))
# Setup and start the simulation
random.seed(RANDOM_SEED) # This helps reproducing the results
# Create an environment and start the setup process
env = simpy.Environment()
env.process(setup(env, NUM_MACHINES, WASHTIME, T_INTER))
# Execute!
env.run(until=SIM_TIME)
Thanks a lot in advance.
What you want is to model an entity that uses your resource with a very high priority so that normal entities meanwhile cannot use it. So your "ghost car" is actually not such a bad idea.
Related
I'm using a multiprocessing pool with 80 processes on a 16GB machine. The flow is as follows:
Read objects in batches from an input file
Send the entire batch to a multiprocessing pool, and record the time taken by the pool to process the batch
Write the time recorded in step 2 above to an output file
To achieve the above, I wrote code in 2 ways:
Way 1:
with open('input_file', 'r') as input_file, open('output_file', 'a') as of:
batch = read_next_batch_of_lines()
start_time = time.time()
call_api_for_each_item_in_batch(batch)
end_time = time.time()
of.write('{}\n'.format(end_time-start_time))
Way 2:
with open('input_file', 'r') as input_file:
batch = read_next_batch_of_lines()
start_time = time.time()
call_api_for_each_item_in_batch(batch)
end_time = time.time()
with open('output_file', 'a') as of:
of.write('{}\n'.format(end_time-start_time))
In the first case, nothing is being appended to the output file despite batches being processed. I'm unable to figure out the reason for this.
Details of call_api_for_each_item_in_batch():
def call_api_for_each_item_in_batch(batch):
intervals = get_intervals(batch, pool_size) #this gives intervals. Ex. if batch size is 10 and pool size is 3, then intervals would be (0, 4, 7, 10)
pool = mp.Pool(pool_size)
arguments = list(zip(intervals, intervals[1:]))
pool.starmap(call_api, arguments)
pool.close()
def call_api(start, end):
for i in range(start, end):
item = batch[i]
call_external_api(item)
How is Way 1 different from Way 2 when a pool.close() is called in the call_api_for_each_item_in_batch itself?
Also, I used pool.close() followed by pool.join(), but faced the same issue.
I run Windows 10, Python 3.7, and have a 6-core CPU. A single Python thread on my machine submits 1,000 inserts per second to grakn. I'd like to parallelize my code to insert and match even faster. How are people doing this?
My only experience with parellelization is on another project, where I submit a custom function to a dask distributed client to generate thousands of tasks. Right now, this same approach fails whenever the custom function receives or generates a grakn transaction object/handle. I get errors like:
Traceback (most recent call last):
File "C:\Users\dvyd\.conda\envs\activefiction\lib\site-packages\distributed\protocol\pickle.py", line 41, in dumps
return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
...
File "stringsource", line 2, in grpc._cython.cygrpc.Channel.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
I've never used Python's multiprocessing module directly. What are other people doing to parallelize their queries to grakn?
The easiest approach that I've found to execute a batch of queries is to pass a Grakn session to each thread in a ThreadPool. Within each thread you can manage transactions and of course do some more complex logic:
from grakn.client import GraknClient
from multiprocessing.dummy import Pool as ThreadPool
from functools import partial
def write_query_batch(session, batch):
tx = session.transaction().write()
for query in batch:
tx.query(query)
tx.commit()
def multi_thread_write_query_batches(session, query_batches, num_threads=8):
pool = ThreadPool(num_threads)
pool.map(partial(write_query_batch, session), query_batches)
pool.close()
pool.join()
def generate_query_batches(my_data_entries_list, batch_size):
batch = []
for index, data_entry in enumerate(my_data_entries_list):
batch.append(data_entry)
if index % batch_size == 0 and index != 0:
yield batch
batch = []
if batch:
yield batch
# (Part 2) Somewhere in your application open a client and a session
client = GraknClient(uri="localhost:48555")
session = client.session(keyspace="grakn")
query_batches_iterator = generate_query_batches(my_data_entries_list, batch_size)
multi_thread_write_query_batches(session, query_batches_iterator, num_threads=8)
session.close()
client.close()
The above is a generic method. As a concrete example, you can use the above (omitting part 2) to parallelise batches of insert statements from two files. Appending this to the above should work:
files = [
{
"file_path": f"/path/to/your/file.gql",
},
{
"file_path": f"/path/to/your/file2.gql",
}
]
KEYSPACE = "grakn"
URI = "localhost:48555"
BATCH_SIZE = 10
NUM_BATCHES = 1000
# Entry point where migration starts
def migrate_graql_files():
start_time = time.time()
for file in files:
print('==================================================')
print(f'Loading from {file["file_path"]}')
print('==================================================')
open_file = open(file["file_path"], "r") # Here we are assuming you have 1 Graql query per line!
batches = generate_query_batches(open_file.readlines(), BATCH_SIZE)
with GraknClient(uri=URI) as client: # Using `with` auto-closes the client
with client.session(KEYSPACE) as session: # Using `with` auto-closes the session
multi_thread_write_query_batches(session, batches, num_threads=16) # Pick `num_threads` according to your machine
elapsed = time.time() - start_time
print(f'Time elapsed {elapsed:.1f} seconds')
elapsed = time.time() - start_time
print(f'Time elapsed {elapsed:.1f} seconds')
if __name__ == "__main__":
migrate_graql_files()
You should also be able to see how you can load from a csv or any other file type in this way, but taking the values you find in that file and substitution them into Graql query string templates. Take a look at the migration example in the docs for more on that.
An alternative approach using multi-processing instead of multi-threading follows below.
We empirically found that multi-threading doesn't yield particularly large performance gains, compared to multi-processing. This is probably due to Python's GIL.
This piece of code assumes a file enumerating TypeQL queries that are independent of each other, so they can be parallelised freely.
from typedb.client import TypeDB, TypeDBClient, SessionType, TransactionType
import multiprocessing as mp
import queue
def batch_writer(database, kill_event, batch_queue):
client = TypeDB.core_client("localhost:1729")
session = client.session(database, SessionType.DATA)
while not kill_event.is_set():
try:
batch = batch_queue.get(block=True, timeout=1)
with session.transaction(TransactionType.WRITE) as tx:
for query in batch:
tx.query().insert(query)
tx.commit()
except queue.Empty:
continue
print("Received kill event, exiting worker.")
def start_writers(database, kill_event, batch_queue, parallelism=4):
processes = []
for _ in range(parallelism):
proc = mp.Process(target=batch_writer, args=(database, kill_event, batch_queue))
processes.append(proc)
proc.start()
return processes
def batch(iterable, n=1000):
l = len(iterable)
for ndx in range(0, l, n):
yield iterable[ndx:min(ndx + n, l)]
if __name__ == '__main__':
batch_size = 100
parallelism = 1
database = "<database name>"
# filePath = "<PATH TO QUERIES FILE - ONE QUERY PER NEW LINE>"
with open(file_path, "r") as file:
statements = file.read().splitlines()[:]
batch_statements = batch(statements, n=batch_size)
total_batches = int(len(statements) / batch_size)
if total_batches % batch_size > 0:
total_batches += 1
batch_queue = mp.Queue(parallelism * 4)
kill_event = mp.Event()
writers = start_writers(database, kill_event, batch_queue, parallelism=parallelism)
for i, batch in enumerate(batch_statements):
batch_queue.put(batch, block=True)
if i*batch_size % 10000 == 0:
print("Loaded: {0}/{1}".format(i*batch_size, total_batches*batch_size))
kill_event.set()
batch_queue.close()
batch_queue.join_thread()
for proc in writers:
proc.join()
print("Done loading")
I'm reading video File from opencv and store their frames in a list, Then I provide this list to face detection function which in turn store the face location in another list, the problem is that when I give an equal number of frame to multiprocessing code and single processing code, the performance is not very different. please check my code, suggest the possible solution. I am using python 3.5, the number of CPU core is 4. Multiprocessing code is supposed to give almost 4 times performance but it only gives few second gains.
My code:
import cv2,time,dlib,imutils
from multiprocessing import Pool
detector = dlib.get_frontal_face_detector()
vidcap=cv2.VideoCapture(r'/home/deeplearning/PycharmProjects
/sjtech/jurassic_park_intro.mp4')
count = 0
frame_list = []
def parallel_detection(f):
return detector(f,1)
success,image = vidcap.read()
while success:
print('Read a new frame: ', success)
frame_list.append(image)
count += 1
success,image = vidcap.read()
del frame_list[-1]
print("out of while")
p = Pool()
t1 = time.time()
#below is my multiprocessing code, on 40 frames it takes 42 seconds
face_location=p.map(parallel_detection,frame_list[900:940])
#below is single processing code, it takes 50 seconds
face_location=[detector(frame_list[x],1) for x in range(900,940)]
print(time.time()-t1)
I am trying to insert 6000 rows/entities into google cloud datastore. I am also using datastore emulator as the local server.
In the code, I created an insert function that inserts entities in batches using put_multi and set the batch size to 50. I use python multiprocessing to spawn processes that execute the function.
A slice function is also used to divide the workload based on how many CPU cores are used. e.g. if there are 3 cores, the workload (6000 entities) is divided into 3 parts with 2000 entities each, then each part is inserted by a spawned process that executes the insert function.
After insertion is done, I checked with Cloud Datastore Admin console, but couldn't find the kinds that have been inserted.
I am wondering what is the issue here and how to solve it.
code snippet is as follows,
# cores_to_use is how many cpu cores available for dividing workload
cores_to_use = 3
# a datastore client is passed in as the argument
inserter = FastInsertGCDatastore(client)
# entities is a list of datastore entities to be inserted
# the number of entities is 6000 here
input_size = len(entities)
slice_size = int(input_size / cores_to_use)
entity_blocks = []
iterator = iter(entities)
for i in range(cores_to_use):
entity_blocks.append([])
for j in range(slice_size):
entity_blocks[i].append(iterator.__next__())
for block in entity_blocks:
p = multiprocessing.Process(target=inserter.execute, args=(block,))
p.start()
class FastInsertGCDatastore:
"""
batch insert entities into gc datastore based on batch_size and number_of_entities
"""
def __init__(self, client):
"""
initialize with datastore client
:param client: the datastore client
"""
self.client = client
def execute(self, entities):
"""
batch insert entities
:param entities: a list of datastore entities need to be inserted
"""
number_of_entities = len(entities)
batch_size = 50
batch_documents = [0] * batch_size
rowct = 0 # entity count as index for accessing rows
for index in range(number_of_entities):
try:
batch_documents[index % batch_size] = entities[rowct]
rowct += 1
if (index + 1) % batch_size == 0:
self.client.put_multi(batch_documents)
index += 1
except Exception as e:
print('Unexpected error for index ', index, ' message reads', str(e))
raise e
# insert any remaining entities
if not index % batch_size == 0:
self.client.put_multi(batch_documents[:index % batch_size])
I am using simpy to create a DES with a very large numbers of objects (many millions). I am running into memory issues and have being trying to figure out how to address this. It is possible to work out which objects will not undergo anymore interactions with other processes and so I can delete these objects from the simulation in theory freeing up memory. I created the below test this.
import psutil as ps
import simpy
import random
class MemoryUse(object):
"""a class used to output memory usage at various times within the sim"""
def __init__(self, env, input_dict):
self.env = env
self.input_dict = input_dict
self.env.process(self.before())
self.env.process(self.during())
self.env.process(self.after_sr())
self.env.process(self.after())
def before(self):
yield self.env.timeout(0)
print("full object list and memory events at time: ", self.env.now, " ", ps.virtual_memory())
print(len(self.input_dict), len(self.env._queue))
def during(self):
yield self.env.timeout(2)
print("full object list and events ar time: ", self.env.now, " ", ps.virtual_memory())
print(len(self.input_dict), len(self.env._queue))
def after_sr(self):
yield self.env.timeout(4)
print("reduced object list and reduced events at time: ", self.env.now, " ", ps.virtual_memory())
print(len(self.input_dict), len(self.env._queue))
def after(self):
yield self.env.timeout(6)
print("no objects and no events at time: ", self.env.now, " ", ps.virtual_memory())
print(len(self.input_dict), len(self.env._queue))
class ExObj(object):
"""a generic object"""
def __init__(self, env, id, input_dict):
self.env = env
self.id = id
self.input_dict = input_dict
if random.randint(0, 100) < 70:
# set as SR
self.timeout = 2
else:
self.timeout = 4
def action(self):
yield self.env.timeout(self.timeout)
del self.input_dict[self.id]
class StartObj(object):
"""this enables me to create the obj events after the sim has started so as to measure memory usage before the events
associated with the object exists"""
def __init__(self, env, input_dict):
self.env = env
self.input_dict = input_dict
self.env.process(self.start_obj())
def start_obj(self):
yield self.env.timeout(1)
for k, v in self.input_dict.items():
self.env.process(v.action())
yield self.env.timeout(0)
# memory usage before we do anything
print("before all: ", ps.virtual_memory())
# create simpy env
env = simpy.Environment()
obj_dict = {}
# create memory calculation events
memory = MemoryUse(env, obj_dict)
# create objects
for i in range(2500000):
obj_dict[i] = ExObj(env, i, obj_dict)
# create process that will itself start events associated with the objects
start = StartObj(env, obj_dict)
# run
env.run()
# clear the dict if not already clear
for j in range(2500000):
obj_dict.clear()
# final memory check
print("after all: ", ps.virtual_memory())
print(len(obj_dict))
I was expecting memory usage to drop by time 4, as many objects have been removed and processes completed (around 70%). However memory usage appears to stay the same (See below). Why is this so? What is using this memory? Do completed processes stay in the simulation?
before all: svmem(total=42195423232, available=39684155392, percent=6.0, used=2246373376, free=38884859904, active=2390749184, inactive=441712640, buffers=263155712, cached=801034240, shared=28721152)
full object list and memory events at time: 0 svmem(total=42195423232, available=38834251776, percent=8.0, used=3096276992, free=38035181568, active=3241959424, inactive=441466880, buffers=263159808, cached=800804864, shared=28721152)
2500000 4
full object list and events ar time: 2 svmem(total=42195423232, available=35121584128, percent=16.8, used=6808891392, free=34322219008, active=6947561472, inactive=441761792, buffers=263163904, cached=801148928, shared=28774400)
2500000 2500002
reduced object list and reduced events at time: 4 svmem(total=42195423232, available=35120973824, percent=16.8, used=6809530368, free=34321600512, active=6948368384, inactive=441737216, buffers=263168000, cached=801124352, shared=28745728)
767416 767417
no objects and no events at time: 6 svmem(total=42195423232, available=38448134144, percent=8.9, used=3482365952, free=37648760832, active=3627053056, inactive=441733120, buffers=263172096, cached=801124352, shared=28745728)
0 0
after all: svmem(total=42195423232, available=38825793536, percent=8.0, used=3104706560, free=38026420224, active=3250180096, inactive=441733120, buffers=263172096, cached=801124352, shared=28745728)
0
Process finished with exit code 0