I am trying to make a code that will ask for a user input and if that input is not given in a set amount of time will assume a default value and continue through the rest of the code without requiring the user to hit enter. I am running in Python 3.5.1 on Windows 10.
I have looked through: Keyboard input with timeout in Python, How to set time limit on raw_input, Timeout on a function call, and Python 3 Timed Input black boxing the answers but none of the answers are suitable as they are not usable on Windows (principally use of signal.SIGALRM which is only available on linux), or require a user to hit enter in order to exit the input.
Based upon the above answers however i have attempted to scrap together a solution using multiprocessing which (as i think it should work) creates one process to ask for the input and creates another process to terminate the first process after the timeout period.
import multiprocessing
from time import time,sleep
def wait(secs):
if secs == 0:
return
end = time()+secs
current = time()
while end>current:
current = time()
sleep(.1)
return
def delay_terminate_process(process,delay):
wait(delay)
process.terminate()
process.join()
def ask_input(prompt,term_queue,out_queue):
command = input(prompt)
process = term_queue.get()
process.terminate()
process.join()
out_queue.put(command)
##### this doesn't even remotly work.....
def input_with_timeout(prompt,timeout=15.0):
print(prompt)
astring = 'no input'
out_queue = multiprocessing.Queue()
term_queue = multiprocessing.Queue()
worker1 = multiprocessing.Process(target=ask_input,args=(prompt,term_queue,out_queue))
worker2 = multiprocessing.Process(target=delay_terminate_process,args=(worker1,timeout))
worker1.daemon = True
worker2.daemon = True
term_queue.put(worker2)
print('Through overhead')
if __name__ == '__main__':
print('I am in if statement')
worker2.start()
worker1.start()
astring = out_queue.get()
else:
print('I have no clue what happened that would cause this to print....')
return
print('returning')
return astring
please = input_with_timeout('Does this work?',timeout=10)
But this fails miserably and yields:
Does this work?
Through overhead
I am in if statement
Traceback (most recent call last):
File "C:\Anaconda3\lib\multiprocessing\queues.py", line 241, in _feed
obj = ForkingPickler.dumps(obj)
File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
File "C:\Anaconda3\lib\multiprocessing\queues.py", line 58, in __getstate__
context.assert_spawning(self)
File "C:\Anaconda3\lib\multiprocessing\context.py", line 347, in assert_spawning
' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance
Does this work?
Through overhead
I have no clue what happened that would cause this to print....
Does this work?Process Process-1:
Traceback (most recent call last):
File "C:\Anaconda3\lib\multiprocessing\queues.py", line 241, in _feed
obj = ForkingPickler.dumps(obj)
File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
File "C:\Anaconda3\lib\multiprocessing\process.py", line 287, in __reduce__
'Pickling an AuthenticationString object is '
TypeError: Pickling an AuthenticationString object is disallowed for security reasons
Traceback (most recent call last):
File "C:\Anaconda3\lib\multiprocessing\process.py", line 254, in _bootstrap
self.run()
File "C:\Anaconda3\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\Anaconda3\saved_programs\a_open_file4.py", line 20, in ask_input
command = input(prompt)
EOFError: EOF when reading a line
Does this work?
Through overhead
I have no clue what happened that would cause this to print....
Traceback (most recent call last):
File "C:\Anaconda3\lib\multiprocessing\queues.py", line 241, in _feed
obj = ForkingPickler.dumps(obj)
File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
File "C:\Anaconda3\lib\multiprocessing\queues.py", line 58, in __getstate__
context.assert_spawning(self)
File "C:\Anaconda3\lib\multiprocessing\context.py", line 347, in assert_spawning
' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance
Process Process-2:
Traceback (most recent call last):
File "C:\Anaconda3\lib\multiprocessing\process.py", line 254, in _bootstrap
self.run()
File "C:\Anaconda3\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\Anaconda3\saved_programs\a_open_file4.py", line 16, in delay_terminate_process
process.terminate()
File "C:\Anaconda3\lib\multiprocessing\process.py", line 113, in terminate
self._popen.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'
I really don't understand the multiprocessing module well and although I have read the official docs am unsure why this error occurred or why it appears to have ran through the function call 3 times in the process. Any help on how to either resolve the error or achieve an optional user input in a cleaner manner will be much appreciated by a noob programmer. Thanks!
Related
In this video: https://www.youtube.com/watch?v=HGOBQPFzWKo, around 4:16:00, it's talked about how to implement a lock for threading issues. But as opposed to the video, I get a debug message when trying to acquire the lock.
This is a part of the code I tried out for myself:
def increase(lock):
global database_value
lock.acquire()
local_copy = database_value
local_copy += 1
time.sleep(0.1)
database_value = local_copy
lock.release()
And this is the debug message:
Exception in thread Thread-32:
Traceback (most recent call last):
File "C:\Users\Gebruiker\miniconda3\lib\threading.py", line 973, in _bootstrap_inner
Exception in thread Thread-33:
Traceback (most recent call last):
File "C:\Users\Gebruiker\miniconda3\lib\threading.py", line 973, in _bootstrap_inner
self.run()
File "C:\Users\Gebruiker\miniconda3\lib\threading.py", line 910, in run
self.run()
File "C:\Users\Gebruiker\miniconda3\lib\threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Gebruiker\AppData\Local\Temp\ipykernel_75580\1082204269.py", line 14, in increase
self._target(*self._args, **self._kwargs)
File "C:\Users\Gebruiker\AppData\Local\Temp\ipykernel_75580\1082204269.py", line 14, in increase
AttributeError: 'builtin_function_or_method' object has no attribute 'acquire'
AttributeError: 'builtin_function_or_method' object has no attribute 'acquire'
In the Jupyter Notebook, there seems to be raised an exception when I try to run the thread.
What's going wrong here?
Thanks in advance!
I tried to do what the video did, I expected it to work, but there's something not working in the thread apparently.
I am a noob in Python multiprocessing and trying to learn it.
While trying to run a class method on a separate Multiprocessing-Process. The Python class is initialized in the main process and I am trying to start a class method in a separate process.
def __initialise_strat_class_obj(self, *, strat_name: str, strat_config: StratInput):
try:
strat_instance = strat_class_dict[strat_name](strat_args=self.strat_args, strat_input=strat_config) # dynamically fetching & calling the class
strat_instance.init()
except Exception as e:
self._logger.error(f"unable to init strategy `{strat_config.id}`: {e}")
return False
strat_process = Process(target=strat_instance.run)
strat_process.start()# starting the class method in a separate process
def start(self):
self.init()
if self.is_active():
self.run()
else:
self._logger.warning(f"{self.id} -> strat is inactive, returning from thread")
return
However, I encountered the cannot pickle '_io.TextIOWrapper' object error. Below is the stack trace of the error.
Traceback (most recent call last):
File "D:\repo\StockTradeBotV2\src\strategy\__init__.py", line 237, in __init__
self.__initialise_strat_class_obj(strat_name=strat_name, strat_config=strat_inp)
File "D:\repo\StockTradeBotV2\src\strategy\__init__.py", line 263, in __initialise_strat_class_obj
strat_process.start()
File "C:\Program Files\Python311\Lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\multiprocessing\popen_spawn_win32.py", line 94, in __init__
reduction.dump(process_obj, to_child)
File "C:\Program Files\Python311\Lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object
To overcome the serialization issue, I made use of the multiprocess package which utilizes dill for serializing objects. This time I got a completely different error:
Traceback (most recent call last):
File "D:\repo\StockTradeBotV2\src\strategy\__init__.py", line 237, in __init__
self.__initialise_strat_class_obj(strat_name=strat_name, strat_config=strat_inp)
File "D:\repo\StockTradeBotV2\src\strategy\__init__.py", line 263, in __initialise_strat_class_obj
strat_process.start()
File "C:\PythonEnvironments\StockTradeBotV2\Lib\site-packages\multiprocess\process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
TypeError: BaseProcess._Popen() takes 1 positional argument but 2 were given
I have tried calling the raw class without initializing and passing the self-argument which didn’t help. On changing the Process to Thread, the code works as intended without any errors, but I want this to run on its own process instead of parallelism so that the additional cores on my PC are used.
I am doing all this on Python-3.11.1 with multiprocess-0.70.14 package.
Edit: I have to use multiprocess.Process rather than from multiprocess.process import BaseProcess as Process which solved the issue
I'm beginner for multiprocessing,
i would like to use multiprocessing for parallel code running with streaming data.
To start good, I have coded below and got error.
Could you please tell me correct way to print on the screen.
Code:
import sys
from multiprocessing import Process, Manager
import time
def producer(dic, name):
for i in range(10000):
dic["A"] = i
time.sleep(2)
def consumer(dic, name):
for i in range(10000):
aval = dic.get("A")
#print(f" {name} - Val = {aval}")
sys.stdout.write(f" {name} - Val = {aval}")
sys.stdout.flush()
time.sleep(2.2)
if __name__ == '__main__':
manager = Manager()
dic = manager.dict()
Process(target=producer, args=(dic,"TT")).start()
time.sleep(1)
Process(target=consumer, args=(dic,"Con1")).start()
Process(target=consumer, args=(dic,"Con2")).start()
When I run the same in the windows console, I got below error, how can I print Consumer's print function in the console.Thanks
(base) PS D:\> python .\mulpro.py
Process Process-3:
Process Process-4:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\multiprocessing\managers.py", line 811, in
_callmethod
conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _
bootstrap
self.run()
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in ru
n
self._target(*self._args, **self._kwargs)
File "D:\mulpro.py", line 19, in consumer
aval = dic.get("A")
File "<string>", line 2, in get
File "C:\ProgramData\Anaconda3\lib\multiprocessing\managers.py", line 815, in
_callmethod
self._connect()
File "C:\ProgramData\Anaconda3\lib\multiprocessing\managers.py", line 802, in
_connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\connection.py", line 490, i
n Client
c = PipeClient(address)
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\multiprocessing\connection.py", line 691, i
n PipeClient
_winapi.WaitNamedPipe(address, 1000)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\managers.py", line 811, in
_callmethod
conn = self._tls.connection
FileNotFoundError: [WinError 2] The system cannot find the file specified
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _
bootstrap
self.run()
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in ru
n
self._target(*self._args, **self._kwargs)
File "D:\mulpro.py", line 19, in consumer
aval = dic.get("A")
File "<string>", line 2, in get
File "C:\ProgramData\Anaconda3\lib\multiprocessing\managers.py", line 815, in
_callmethod
self._connect()
File "C:\ProgramData\Anaconda3\lib\multiprocessing\managers.py", line 802, in
_connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\connection.py", line 490, i
n Client
c = PipeClient(address)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\connection.py", line 691, i
n PipeClient
_winapi.WaitNamedPipe(address, 1000)
FileNotFoundError: [WinError 2] The system cannot find the file specified
Process Process-2:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _
bootstrap
self.run()
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in ru
n
self._target(*self._args, **self._kwargs)
File "D:\mulpro.py", line 13, in producer
dic["A"] = i
File "<string>", line 2, in __setitem__
File "C:\ProgramData\Anaconda3\lib\multiprocessing\managers.py", line 818, in
_callmethod
conn.send((self._id, methodname, args, kwds))
File "C:\ProgramData\Anaconda3\lib\multiprocessing\connection.py", line 206, i
n send
self._send_bytes(_ForkingPickler.dumps(obj))
File "C:\ProgramData\Anaconda3\lib\multiprocessing\connection.py", line 280, i
n _send_bytes
ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
BrokenPipeError: [WinError 232] The pipe is being closed
The reason might be that the main process doesn't wait for two children process, so make your code like this could work:
def run():
manager = Manager()
dic = manager.dict()
Process(target=producer, args=(dic,"TT")).start()
time.sleep(1)
Process(target=consumer, args=(dic,"Con1")).start()
Process(target=consumer, args=(dic,"Con2")).start()
while True:
pass
if __name__ == '__main__':
run()
However it's very strange that if I append a dead loop in main instead of using another function, it still raise that exception. Anyway, the code above could help.
When I try to make the HTTP client a class variable of my tcp server class I see the following error:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "cisco-dial-out-mdt-async-client.py", line 208, in <module>
tcp_server.start(0)
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/tcpserver.py", line 244, in start
self.add_sockets(sockets)
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/tcpserver.py", line 166, in add_sockets
sock, self._handle_connection
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/netutil.py", line 279, in add_accept_handler
io_loop.add_handler(sock, accept_handler, IOLoop.READ)
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 99, in add_handler
self.asyncio_loop.add_reader(fd, self._handle_events, fd, IOLoop.READ)
File "/usr/lib/python3.7/asyncio/selector_events.py", line 324, in add_reader
return self._add_reader(fd, callback, *args)
File "/usr/lib/python3.7/asyncio/selector_events.py", line 254, in _add_reader
child 13 (pid 3260) killed by signal 2, restarting
(handle, None))
File "/usr/lib/python3.7/selectors.py", line 359, in register
Traceback (most recent call last):
File "cisco-dial-out-mdt-async-client.py", line 208, in <module>
self._selector.register(key.fd, poller_events)
Traceback (most recent call last):
FileExistsError: [Errno 17] File exists
File "/usr/lib/python3.7/asyncio/selector_events.py", line 251, in _add_reader
tcp_server.start(0)
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/tcpserver.py", line 241, in start
process.fork_processes(num_processes, max_restarts)
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/process.py", line 175, in fork_processes
key = self._selector.get_key(fd)
raise RuntimeError("Too many child restarts, giving up")
File "/usr/lib/python3.7/selectors.py", line 192, in get_key
RuntimeError: Too many child restarts, giving up
raise KeyError("{!r} is not registered".format(fileobj)) from None
KeyError: '9 is not registered'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "cisco-dial-out-mdt-async-client.py", line 208, in <module>
tcp_server.start(0)
Traceback (most recent call last):
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/tcpserver.py", line 244, in start
File "/usr/lib/python3.7/asyncio/selector_events.py", line 251, in _add_reader
self.add_sockets(sockets)
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/tcpserver.py", line 166, in add_sockets
key = self._selector.get_key(fd)
File "/usr/lib/python3.7/selectors.py", line 192, in get_key
sock, self._handle_connection
File "/home/lab/venv3.7/lib/python3.7/site-packages/tornado/netutil.py", line 279, in add_accept_handler
raise KeyError("{!r} is not registered".format(fileobj)) from None
KeyError: '9 is not registered'
When I remove the line that creates the object it works fine
class TelemetryTCPDialOutServer(TCPServer):
def __init__(self, elasticsearch_server):
super().__init__()
self.elastic_server = elasticsearch_server
self.lock = Lock()
log_name = __file__.strip('.py')
self.log = self.init_log(f"{log_name}.log")
self.lock = Lock()
self.http_client = AsyncHTTPClient() <== This line
Is it possible to put the http client in the class or make it a local object to some of the class functions. (This way works too).
Thansk
It looks like something has really messed up your stack trace, but I think the problem is that you're calling tcp_server.start(0) too late. start(0) forks multiple child processes, and it's invalid to fork multiple processes once the event loop has been created.
If you can, move this start(0) call earlier (typically it's the first thing you do after processing your command line or config file). If you can't move it earlier, you'll have to split up the process creation from starting the server. This is the "advanced multi-process" pattern from https://www.tornadoweb.org/en/stable/tcpserver.html:
# This part comes very early, just after processing configs
sockets = bind_sockets(8888)
tornado.process.fork_processes(0)
# This part can come later
server = TCPServer()
server.add_sockets(sockets)
IOLoop.current().start()
I'm trying to make dictionary attack on zip file using Pool to increase speed.
But I face next error in Python 3.6, while it works in Python 2.7:
Traceback (most recent call last):
File "zip_crack.py", line 42, in <module>
main()
File "zip_crack.py", line 28, in main
for result in results:
File "/usr/lib/python3.6/multiprocessing/pool.py", line 761, in next
raise value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 450, in _ handle_tasks
put(task)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot serialize '_io.BufferedReader' object
I tried to search for same errors but couldn't find answer that can help here.
Code looks like this
def crack(pwd, f):
try:
key = pwd.strip()
f.extractall(pwd=key)
return True
except:
pass
z_file = zipfile.ZipFile("../folder.zip")
with open('words.dic', 'r') as passes:
start = time.time()
lines = passes.readlines()
pool = Pool(50)
results = pool.imap_unordered(partial(crack, f=z_file), lines)
pool.close()
for result in results:
if result:
pool.terminate()
break
pool.join()
I also tried another approach using map
with contextlib.closing(Pool(50)) as pool:
pool.map(partial(crack, f=z_file), lines)
which worked great and found passwords quickly in Python 2.7 but it throws same exception in python 3.6