I would like to be able to trigger exactly one process in a list, and interrupt the other processes in the list. In the example below I want to trigger either A or B, but never both. The example below works, fine except when waitingtimeA = waitingtimeB.
import simpy
from simpy import AnyOf
def waitingprocess(env, waitingtime):
try:
yield env.timeout(waitingtime)
except:
pass
def example(env):
waitingtimeA = 1.0
waitingtimeB = 1.0
A = env.process(waitingprocess(env, waitingtimeA))
B = env.process(waitingprocess(env, waitingtimeB))
events = [A,B]
yield AnyOf(env, events)
for e in events:
if not e.processed:
e.interrupt()
print('is A Processed:', A.processed)
print('is B Processed:', B.processed)
env = simpy.Environment()
Example = env.process(example(env))
env.run()
Which should lead to either A or B being processed, but which prints:
is A Processed: True
is B Processed: True
I understand that both processes A and B are triggered and processed before the Example process continues, but is there a way around this to make processes A and B trigger mutually exclusive?
The only thing I can think of is to make a OneOf() alternative of AnyOf(), which as soon as exactly one event is triggered, interrupts all other events. For this to work the OneOf() Condition Event must be an URGENT event (an event with higher priority than normal events). However, I am kind of out of my depth. Does anyone have an idea where to start?
Thanks a lot!
Related
I have the following code (simplified):
from multiprocessing import Process, Queue
def f1(queue):
while True:
# do some stuff and get a variable called data
# ...
queue.put(data)
def f2(queue):
while True:
if not queue.empty():
data = queue.get(timeout=300)
print('queue data: ' + str(data))
if __name__ == '__main__':
q = Queue()
p1 = Process(target=f1, args=(q,))
p2 = Process(target=f2, args=(q,))
p1.start()
p2.start()
p1.join()
p2.join()
The problem I'm facing is that I don't know how to lock the queue in f1 in order to keep putting data for n seconds, before f2 is able to read it.
I tried with timeouts but of course, it didn't work. Basically, the expected behaviour would be that f1 keeps appending data into the queue and after n seconds, f2 can get what's in that queue. So, summarising, f1 should be running continuously, f2 should be running continuously too but accessing the queue every n seconds.
I can think of not so elegant ways of doing this with the time library, but I guess it has to be other way. Maybe the code's approach is wrong and I shouldn't be using Process and Queue but Pipelines or something else.
Thanks in advance!
For this particular case in which I was using the multiprocessing library, instead of threading or asyncio, I found the best way to do this by using a simple sleep, so the f2() will end up like:
def f2(queue):
while True:
time.sleep(300) # sleep for 5 minutes before POSTing
if not queue.empty():
data = queue.get(timeout=300)
print('queue data: ' + str(data))
Of course, after importing time.
As I said, maybe not the most elegant solution but I couldn't come up with anything better for the time being (and this particular use case).
This is the problem I am having. I wont share code because of condfidentiality but instead I will provide some dummy example.
Assume that we have a class as follows:
class SayHello:
def __init__(self, name, id):
self.name=name
self.id=id
#public func
def doSomething(self, arg1, arg2 ):
DoAHugeTaskWithArgument
Lets say now that in an other modules we have this:
class CallOperations:
def __init__(self):
self.dummydict={1: {"james":20, "peter":30, "victor":40, "john":45, "ali":21, "tom":41, "hector":37}, 2:{"james":23, "peter":31, "victor":44, "john":46, "ali":23, "tom":44, "hector":35} }
def runProcessors(self):
#runprocess
for _, v in self.dummydict.items():
Instances = [SayHello(g,b) for g ,b in v.items()]
with ProcessPoolExecutor(max_workers=2) as executor:
future = [executor.submit(ins.doSomething, 2, 1235) for ins in Instances]
So the problem starts here. I want to know what instances are running doSomething() funtion in their respective process. I want to set a variable = 1 when the function of that instance is running in the process and set it to zero when it is completed.
Each instance has its own name and id. Is there way to find out the name of the running instance in the process?
This problem is making me very confused and can not find a proper solution.
Thank you alot.
If I understand your question correctly, you want to know when an instance of SayHello is executing and when it is not. You can set a variable (1 or 0) by using a Manager - but the usefulness of this can be debated. You might want to use a lock instead.
I had to tweak your code a bit but this is a running example. It picks one of your tasks as the one to monitor in the while loop. It is a dummy loop that never exits but you'll get the idea. It will keep polling the variable of one of your instances and you can see it change when that task is running, and then revert back to zero.
from time import sleep
from concurrent.futures import ProcessPoolExecutor
from multiprocessing import Manager
class SayHello:
def __init__(self, name, id):
self.name=name
self.id=id
self.status = Manager().Value("i",0)
#public func
def doSomething(self, arg1, arg2 ):
self.status.value = 1
sleep(5)
self.status.value = 0
class CallOperations:
def __init__(self):
self.dummydict={1: {"james":20, "peter":30, "victor":40, "john":45, "ali":21, "tom":41, "hector":37}, 2:{"james":23, "peter":31, "victor":44, "john":46, "ali":23, "tom":44, "hector":35} }
def runProcessors(self):
#runprocess
for _, v in self.dummydict.items():
Instances = [SayHello(g,b) for g ,b in v.items()]
f = Instances[3]
executor = ProcessPoolExecutor(max_workers=2)
future = [executor.submit(ins.doSomething, 2, 1235) for ins in Instances]
while True:
print(f.status.value)
# Insert break condition here
sleep(0.5)
executor.shutdown()
foo = CallOperations()
foo.runProcessors()
The problem with this is that it can lead to a race condition depending on what you do in your main program. If you want to do any operations on the instance when it is in passive state, it might progress to active just after you check the variable but before you have completed your actions in the main program.
Locks come to rescue here, as you can also create a shared lock Manager().Lock(). If your DoSomething() tries to acquire the lock and your main process does the same when operating on a passive instance, you avoid this problem. Of course your main program could then block the executor from processing if it reserves locks for lengthy operations, as then your two workers would be stuck waiting on locks if the execution processed to those instances where locks are being held by the main program. This case would not be suitable for parallel processing implemented using executors.
EDIT: if you are only interested in the running status, you can check the Future.running() status of your future objects, in this case items in your future array.
I have been searching for days for a solution to this issue, if anyone can point me in the right direction, I will be so grateful.
I have been writing a program that allows a user to create multiple trading bots at the same time, each bot is created as an individual thread that makes an api call to Binance for the latest market data and evaluates this data with conditional statements and place trades accordingly. The code I am using for this has been trimmed to only the essential parts for simplicity.. I have a Bot class...
class Bot(threading.Thread):
def __init__(self, symbol, time, exposure, strategy):
threading.Thread.__init__(self)
self.symbol = symbol
self.time = time
self.exposure = exposure
self.strategy = strategy
self.stop_bot = False
def scanner(self):
while self.stop_bot == False:
client = Client(self.api_key, self.secret_key, {"verify": True, "timeout": 20})
price_data = client.get_klines(symbol=self.symbol, interval=self.time_interval)
self.df = pd.DataFrame(price_data)
self.modified_df = pd.DataFrame(price_data)
time.sleep(10)
def kill(self):
self.stop_bot = True
And the Bot class is called from the bot manager terminal class...
class Bot_Manager:
def __init__(self):
self.bot_list = []
bot = object
def terminal(self):
while True:
user_input = input('(1) Create new Bot (2) Stop Bot')
if user_input == '1':
symbol = 'OMGUSDT'
time = '1D'
exposure = 1000
strategy = 'SAR'
bot_name = input('name your bot: ')
bot = Bot(symbol=symbol, time=time, exposure=exposure, strategy=strategy, bot_name=bot_name)
scanner_thread = threading.Thread(target=bot.scanner, daemon=True)
scanner_thread.name = bot_name
scanner_thread.start()
self.bot_list.append(scanner_thread)
elif user_input == '2':
for thread in threading.enumerate():
print(thread.name)
print(self.bot_list)
user_input = input('Bot to stop: ')
i = int(user_input)
print(self.bot_list[i])
Now I am able to create multiple threads / bots by repeatedly selecting option 1. However the issue I am facing is stopping these threads when a user selects option 2. If for example I create 3 bots and name them Bot A, Bot B and Bot C.. when I enumerate this in a for loop, i get the following:
MainThread
Bot A
Bot B
Bot C
and when I store each thread into a list and print the list I see the following:
[<Thread(Bot A, started 8932)>, <Thread(Bot B, started 12268)>, <Thread(Bot C, started 13436)>]
I would like the user to be able to select the thread / bot they want to stop from the list, so in this example if the user types 1, it should return the thread <Thread(Bot B, started 12268)>and stop this thread by passing the variable stop_bot = True. However I haven't had much luck with this.
When I call the function bot.kill() it only stops the last thread that was created, so for this example, Bot C. When it runs again, it doesn't remove any other thread. Is there anyway to pass in the variable stop_bot = True on an already created object / thread? Or is there another method to this that I have overlooked... Any help would be greatly appreciated....
I managed to find a solution to this by adding multiple bots to a data frame and passing this into the bot class. 1 thread was created to iterate through the data frame and by removing rows of the data frame, the bots would be removed.
I have the following problem, I have 2 senders, that send one after each other, and only after 2 of them sent the message, I want a receiver to start receiving and then wait for the next two sends, e.g.
S1
S2
Rec
S1
S2 Rec...
semOne = threading.Semaphore(value = 1)
semTwo = threading.Semaphore(value = 0)
semRecNotify = threading.Semaphore(value = 0)
def senderOne():
while True:
semOne.acquire()
print("Sender One")
semTwo.release()
def senderTwo():
while True:
semTwo.acquire()
print("Sender Two")
semOne.release()
semRecNotify.release()
def receiver():
while True:
semRecNotify.acquire()
print("Receiver")
Using semOne and semTwo, I managed to synchronize 2 senders and using semRecNotify I tried (unsuccessfully) make sure that we enter receiver only after 2 consecutive sends. I am not able to get wanted functionality.
Please take a note that this is not language specific problem, I am just trying to understand myself how to properly use semaphores for synchronization.
Is there more general/mathematical approach to tackle this problem in general?
I am hoping someone can help me here.
I have an object that has the ability to have attributes that return coroutine objects. This works beautifully, however I have a situation where I need to get the results of the coroutine object from synchronous code in a separate thread, while the event loop is currently running. The code I came up with is:
def get_sync(self, key: str, default: typing.Any=None) -> typing.Any:
"""
Get an attribute synchronously and safely.
Note:
This does nothing special if an attribute is synchronous. It only
really has a use for asynchronous attributes. It processes
asynchronous attributes synchronously, blocking everything until
the attribute is processed. This helps when running SQL code that
cannot run asynchronously in coroutines.
Args:
key (str): The Config object's attribute name, as a string.
default (Any): The value to use if the Config object does not have
the given attribute. Defaults to None.
Returns:
Any: The vale of the Config object's attribute, or the default
value if the Config object does not have the given attribute.
"""
ret = self.get(key, default)
if asyncio.iscoroutine(ret):
if loop.is_running():
loop2 = asyncio.new_event_loop()
try:
ret = loop2.run_until_complete(ret)
finally:
loop2.close()
else:
ret = loop.run_until_complete(ret)
return ret
What I am looking for is a safe way to synchronously get the results of a coroutine object in a multithreaded environment. self.get() can return a coroutine object, for attributes I have set to provide them. The issues I have found are: If the event loop is running or not. After searching for a few hours on stack overflow and a few other sites, my (broken) solution is above. If the loop is running, I make a new event loop and run my coroutine in the new event loop. This works, except that the code hangs forever on the ret = loop2.run_until_complete(ret) line.
Right now, I have the following scenarios with results:
results of self.get() is not a coroutine
Returns results. [Good]
results of self.get() is a coroutine & event loop is not running (basically in same thread as the event loop)
Returns results. [Good]
results of self.get() is a coroutine & event loop is running (basically in a different thread than the event loop)
Hangs forever waiting for results. [Bad]
Does anyone know how I can go about fixing the bad result so I can get the value I need? Thanks.
I hope I made some sense here.
I do have a good, and valid reason to be using threads; specifically I am using SQLAlchemy which is not async and I punt the SQLAlchemy code to a ThreadPoolExecutor to handle it safely. However, I need to be able to query these asynchronous attributes from within these threads for the SQLAlchemy code to get certain configuration values safely. And no, I won't switch away from SQLAlchemy to another system just in order to accomplish what I need, so please do not offer alternatives to it. The project is too far along to switch something so fundamental to it.
I tried using asyncio.run_coroutine_threadsafe() and loop.call_soon_threadsafe() and both failed. So far, this has gotten the farthest on making it work, I feel like I am just missing something obvious.
When I get a chance, I will write some code that provides an example of the problem.
Ok, I implemented an example case, and it worked the way I would expect. So it is likely my problem is elsewhere in the code. Leaving this open and will change the question to fit my real problem if I need.
Does anyone have any possible ideas as to why a concurrent.futures.Future from asyncio.run_coroutine_threadsafe() would hang forever rather than return a result?
My example code that does not duplicate my error, unfortunately, is below:
import asyncio
import typing
loop = asyncio.get_event_loop()
class ConfigSimpleAttr:
__slots__ = ('value', '_is_async')
def __init__(
self,
value: typing.Any,
is_async: bool=False
):
self.value = value
self._is_async = is_async
async def _get_async(self):
return self.value
def __get__(self, inst, cls):
if self._is_async and loop.is_running():
return self._get_async()
else:
return self.value
class BaseConfig:
__slots__ = ()
attr1 = ConfigSimpleAttr(10, True)
attr2 = ConfigSimpleAttr(20, True)
def get(self, key: str, default: typing.Any=None) -> typing.Any:
return getattr(self, key, default)
def get_sync(self, key: str, default: typing.Any=None) -> typing.Any:
ret = self.get(key, default)
if asyncio.iscoroutine(ret):
if loop.is_running():
fut = asyncio.run_coroutine_threadsafe(ret, loop)
print(fut, fut.running())
ret = fut.result()
else:
ret = loop.run_until_complete(ret)
return ret
config = BaseConfig()
def example_func():
return config.get_sync('attr1')
async def main():
a1 = await loop.run_in_executor(None, example_func)
a2 = await config.attr2
val = a1 + a2
print('{a1} + {a2} = {val}'.format(a1=a1, a2=a2, val=val))
return val
loop.run_until_complete(main())
This is the stripped down version of exactly what my code is doing, and the example works, even if my actual application doesn't. I am stuck as far as where to look for answers. Suggestions are welcome as to where to try to track down my "stuck forever" problem, even if my code above doesn't actually duplicate the problem.
It is very unlikely that you need to run several event loops at the same time, so this part looks quite wrong:
if loop.is_running():
loop2 = asyncio.new_event_loop()
try:
ret = loop2.run_until_complete(ret)
finally:
loop2.close()
else:
ret = loop.run_until_complete(ret)
Even testing whether the loop is running or not doesn't seem to be the right approach. It's probably better to give explicitly the (only) running loop to get_sync and schedule the coroutine using run_coroutine_threadsafe:
def get_sync(self, key, loop):
ret = self.get(key, default)
if not asyncio.iscoroutine(ret):
return ret
future = asyncio.run_coroutine_threadsafe(ret, loop)
return future.result()
EDIT: Hanging problems can be related to tasks being scheduled in the wrong loop (e.g. forgetting about the optional loop argument when calling a coroutine). This kind of problem should be easier to debug with the PR 303 (now merged): a RuntimeError is raised instead when the loop and the future don't match. So you might want to run your tests with the latest version of asyncio.
Ok, I got my code working, by taking a different approach to it. The problem was tied with using something that had file IO, which I was converting into a coroutine using loop.run_in_executor() on the file IO components. Then, I was trying to use this in a sync function being called from another thread, processed using another loop.run_in_executor() on that function. This is a very important routine in my code (called probably a million times or more during the execution of my short-running code), and I made a decision that my logic was just getting too complicated. So... I uncomplicated it. Now, if I want to use the file IO components asynchronously, I explicitly use my "get_async()" method, otherwise, I use my attribute through normal attribute access.
By removing the complexity of my logic, it made the code cleaner, easier to understand, and even more importantly, it actually works. While I am not 100% certain that I know the root cause of the issue (I believe it has something to do with a thread processing an attribute, which then in turn starts another thread that tries to read the attribute before it is processed, which caused something like a race condition and halting my code, but I could never duplicate the error outside of my application unfortunately to completely prove it out), I was able to get past it and continue with my development efforts.