How can i catch exception from a processpool executer? i have used result() but it slows down the proccess i think is there any other method?
with concurrent.futures.ProcessPoolExecutor(50) as genrator:
try:
for fl in self.finallist:
print("FIKLENAME",fl)
futures=[genrator.submit(insta.load_instantel_ascii, fl,None,',')]
results = [f.result() for f in futures]
except Exception as e:
print("EXCE", e)
print("FILENAME IS",fl)
Use concurrent.futures.as_completed to get an iterator that yields futures as they complete. There should be minimal gains here should you use this API.
Also build an iterable of futures to be used outside of the for loop.
def submit_task(executor, args):
print('FILENAME', args[1])
executor.submit(*args)
with concurrent.futures.ProcessPoolExecutor(50) as executor:
try:
futures = [
submit_task(executor, (insta.load_instantel_ascii, fl, None, ','))
for fl in self.finallist
]
futures_as_completed = concurrent.futures.as_completed(futures)
results = [future.result() for future in futures_as_completed]
except Exception as exc:
print("EXCEPTION: ", exc)
EDIT
To get fl inside of the exception block e.g. print('EXCEPTION: ', exc.fl), you can either catch the exception in the caller and re-raise it with a name binding to fl or implement a custom exception with an attribute binding to fl that is raised in the caller.
For the former, decorate insta.load_instantel_ascii if you don't have access to modify it.
def try_except_fl(method, *args, **kwargs):
try:
return method(*args, **kwargs)
except Exception as exc:
fl = args[0]
exc.fl = fl
raise exc
If you have access to modify insta.load_instantel_ascii, wrap the procedure in a try/except.
def load_instantel_ascii(self, fl, *args, **kwargs):
try:
# do some compute intensive tasks here
except Exception as exc:
exc.fl = fl
raise exc
For the later, raise a custom exception with an attribute bound to fl. e.g.
class ProcessFailure(Exception): # you may want to subclass a specialized exception class.
message = 'Alas! This task had a lot of errors.'
def __init__(fl):
self.fl = fl
Raise this in failure point.i.e. load_instantel_ascii.
def load_instantel_ascii(self, fl, *args, **kwargs):
try:
# do some compute intensive tasks here
except Exception as exc:
raise ProcessFailure(fl)
Related
I have a main_etl.py and watcher.py
main_etl.py
def test_function():
try:
do sometihng....
except Exception as e:
raise e
def main(arg1='', arg2='', arg3=''):
try:
test_function()
except Exception as e:
raise e
watcher.py
import main_etl
def main_function():
try:
main_etl.main(arg1='', arg2='', arg3='')
except Exception as e:
raise e
else:
do something....
And I am running - python watcher.py
I tried the above code, But it didn't catch any exception in the watcher.py file
I wanted to catch the exception in the calling function script (watcher.py).
Can anyone suggest me like how to catch this error or is there any other way of implementing this.
I know os.system() and subprocess.call(), But how can I call the main() function inside main_etl.py ?
I wondered wheather it is actually impossible to make a protected python class, there always seems to be a way of getting around that, but i can't find one for this:
I attempted to code out this properly encapsulated class. Challenge:
Attempt setting somevalue to the value 69, without:
changing the code from line 1 - 32
polymorphism
from sbNative.debugtools import log # module not neccessary, only for logging and debugging purposes imported
from inspect import stack
import traceback
class Protected:
def __init__(self):
self._somevalue = "Unset"
log(self._somevalue)
def __setattr__(self, name, value):
if isinstance(stack()[1][0].f_locals.get("self"), Protected) or not name.startswith("_"):
super(Protected, self).__setattr__(name, value)
else:
raise AttributeError("Protected class from setting")
def __getattribute__(self, name):
if isinstance(stack()[1][0].f_locals.get("self"), Protected) or not name.startswith("_"):
return super(Protected, self).__getattribute__(name)
else:
raise AttributeError("Protected class from getting")
#property
def somevalue(self):
return self._somevalue
#somevalue.setter
def somevalue(self, value):
if value == 69:
raise ValueError(f"{value} is an illegal value.")
self._somevalue = value
log("Instantiates without a problem:")
p = Protected()
print("\n")
log("Fails because it is not allowed to set to this value:")
try:
p.somevalue = 69
except ValueError:
traceback.print_exc()
print("\n")
log("Fails because it attemps setting and getting directly:")
try:
p._somevalue = 69
except AttributeError:
traceback.print_exc()
print("")
try:
log(p._somevalue)
except AttributeError:
traceback.print_exc()
print("\n")
log("Succeeds because it is allowed to set and get this value:")
p.somevalue = 420
log(p.somevalue)
print("ⁿᶦᶜᵉ ˡᶦⁿᵉ ⁿᵘᵐᵇᵉʳ ᵇᵗʷ")
I have a line in my code that has the potential to raise an exception, i would like to handle this execption and continue execution.
def foo():
#good code
if thingThatHappensSometimes:
raise CustomException
#code i want to execute
return resultIwant
def bar():
resultIwant = None
try:
#good code
resultIwant = foo()
except CustomException:
#code that should run if an exception was raised
finally:
print(resultIwant)
print('All done!')
My issue here is that there are situations where foo will raise an exception but there is nothing in the code logic preventing it from generating a result for resultIwant. Currently, if I handle the exception in bar I will not reach the end of execution for foo but If I handle the exception in foo the exception will already be handled and I will not reach the except block in bar. is there a way to solve this issue?
raising an exception stops execution at this point, so you cannot both raise an exception and return a value. But you can return a (value, exception_or_none) tuple:
def foo():
error = None
#good code
if thingThatHappensSometimes:
error = CustomException()
#code i want to execute
return resultIwant, error
def bar():
#good code
resultIwant, error = foo()
if error:
#code that should run if an exception was raised
print(resultIwant)
print('All done!')
Using side_effect, I am trying to raise an exception when a mock is called but I get a DID NOT RAISE EXCEPTION error that I do not understand.
Based largely on this answer, I have created a simple example where there is a Query class with a class method make_request_and_get_response which can raise several exceptions. These exceptions are being handled within the get_response_from_external_api method in main.py.
query.py
from urllib.request import urlopen
import contextlib
import urllib
class Query:
def __init__(self, a, b):
self.a = a
self.b = b
self.query = self.make_query()
def make_query(self):
# create query request using self.a and self.b
return query
def make_request_and_get_response(self): # <--- the 'dangerous' method that can raise exceptions
with contextlib.closing(urlopen(self.query)) as response:
return response.read().decode('utf-8')
main.py
from foo.query import *
def get_response_from_external_api(query):
try:
response = query.make_request_and_get_response()
except urllib.error.URLError as e:
print('Got a URLError: ', e)
except urllib.error.HTTPError as e:
print('Got a HTTPError: ', e)
# {{various other exceptions}}
except Exception:
print('Got a generic Exception!')
# handle this exception
if __name__ == "__main__":
query = Query('input A', 'input B')
result = get_response_from_external_api(query)
return result
Using pytest, I am trying to mock that 'dangerous' method (make_request_and_get_response) with a side effect for a specific exception. Then, I proceed with creating a mocked Query object to use when calling the make_request_and_get_response with the expectation that this last call give a 'URLError' exception.
test_main.py
import pytest
from unittest.mock import patch
from foo.query import Query
from foo.main import get_response_from_external_api
class TestExternalApiCall:
#patch('foo.query.Query')
def test_url_error(self, mockedQuery):
with patch('foo.query.Query.make_request_and_get_response', side_effect=Exception('URLError')):
with pytest.raises(Exception) as excinfo:
q= mockedQuery()
foo.main.get_response_from_external_api(q)
assert excinfo.value = 'URLError'
# assert excinfo.value.message == 'URLError' # this gives object has no attribute 'message'
The test above gives the following error:
> foo.main.get_response_from_external_api(q)
E Failed: DID NOT RAISE <class 'Exception'> id='72517784'>") == 'URLError'
Pytest cannot detect an exception because in get_response_from_external_api you are catching all of them. Depending on your requirements, you have these options:
Don't catch the exceptions in get_response_from_external_api.
Catch the exceptions, do whatever you want and then re-raise them.
Instead of detecting an exception with pytest.raises, use capsys fixture to capture what is being printed and make assertions on that output.
class A:
def __init__(self, data):
self.data = data
def first():
# Some functionality
# raise Exception on some condition
return self
def second():
#Some functionality
a = A('test string').first().second()
In this case I want that if first() raises an error then the chaining breaks and that error is raised. Currently it just fails silently.
How can I fix this?
Probably the best you can do is localize the type of errors that are possible in the chain and then ladder the exception handlers:
class A:
def __init__(self, data):
self.data = data
def first(self):
# Some functionality
# raise Exception on some condition
if self.data=='A':
1/0
return self
def second(self):
next(0)
pass
#Some functionality
You would know that first is numerical, and second is an iterable function that might have a TypeError.
So try this:
try:
A('A').first().second()
except ZeroDivisionError as e:
print('Likely first:', e)
except TypeError as e:
print('Likely second:', e)
Would print Likely first: division by zero with A('A') and print Likely second: 'int' object is not an iterator if you use A('something_else'). You would use this construct if you wanted to stop the chain as executed.
You can also add raise to re-raise the error and get Python's diagnosis of when are where:
try:
A('B').first().second()
except ZeroDivisionError as e:
print('Likely first:', e)
raise
except TypeError as e:
print('Likely second:', e)
raise
Prints:
Likely second: 'int' object is not an iterator
Traceback (most recent call last):
File "Untitled 79.py", line 19, in <module>
A('B').first().second()
File "Untitled 79.py", line 14, in second
next(0)
TypeError: 'int' object is not an iterator
Or, better still, use the try and except in each method:
class A:
def __init__(self, data):
self.data = data
def first(self):
try:
1/0
except ZeroDivisionError as e:
print('DEFINITELY first:', e)
return self
def second(self):
try:
next(0)
except TypeError as e:
print('DEFINITELY second:', e)
return self
>>> A('').first().second()
DEFINITELY first: division by zero
DEFINITELY second: 'int' object is not an iterator
I don't think it could fail silently if its actually failing, as this would error out python. However, you might want something like this, maybe it will jog your thinking.
class A:
def __init__(self, data):
self.data = data
def first():
try:
# Some functionality
raise ERROR_YOU WANT_RAISE:
# raise Exception on some condition
except ERROR_TO_CATCH_FROM_RAISE:
# This is where you would handle the exception after its caught
# and break your chaining possibly. Set a flag for second()
# for example that handles the error.
return self
def second():
#Some functionality