I'm executing a set of very heavy queries remotely to the company databases from our central server. Unfortunately, python logging raises this error from the database when trying to execute certain INSERT query.
/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py:329: Warning: (1592, 'Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it uses a UDF which may not return the same value on the slave.')
self._do_get_result()
/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py:329: Warning: (1592, 'Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave.')
self._do_get_result()
/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py:329: Warning: (1592, 'Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it invokes a trigger or a stored function that inserts into an AUTO_INCREMENT column. Inserted values cannot be logged correctly.')
self._do_get_result()
wihout python logging, the error es expressed differently, and does not locks the python program execution:
[2019-09-27 13:24:47,228 root ERROR] Fallo ejecucion de query: INSERT INTO movimiento (numero_sala, tipo_movimiento, fec_movimiento, hora_movimiento, nro_tarjeta, id_maquina, monto, lugar_im, fecha_hora_im) VALUES ( 3, 20, current_date, current_time, 157299522, 0, 40.000000, 2, current_timestamp)
Traceback (most recent call last):
File "/opt/cruciscripts/crucidmcs/connectionManager.py", line 58, in alter
cursor.execute(query)
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/usr/local/lib/python3.5/dist-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 684, in _read_packet
packet.check_error()
File "/usr/local/lib/python3.5/dist-packages/pymysql/protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "/usr/local/lib/python3.5/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')
Depending on the SQL engine, there are limitations for the size of a query.
The obvious solution was to partition the query in smaller chunks.
Related
I have been digging through questions/answers for the BadZipFile exception when calling read_excel() using the openpyxl engine. I looked at my error stack and dug into the Python files and it looks like ZipFile.py is being very strict when validating an archive. It is looking for an EOCD (end of central directory) signature in my XLSX archive file.
When unzipping an archive, if the EOCD cannot be found or validated, there is supposed to be an error thrown when calling unzip in Linux, but I am not seeing it. I am unsure whether the EOCD is there/correct or not (anyone know of a tool to check?).
However, from looking through my stack (below) I am examining what is happening in openpyxl/reader/excel.py. At line 67, the _validate_archive function is defined. I am wondering about the examination for a "file like object".
My use case is an AWS Lambda function which has an HTTP endpoint. I POST an Excel file (I am testing with Postman and using the binary body for the request where I select my Excel file) to the endpoint. The function needs to handle both CSV and XLSX. I include a custom header in which I specify the original file name. I split the filename, look at the extension, and either call read_csv or read_excel. read_csv is working great.
Either way, the file is coming in as base64. For an XLSX file, Pandas handles this OK - up until we get to _validate_archive... What I am unsure of is how the "file like object" check at line 76...
is_file_like = hasattr(filename, 'read')
... interacts with the type by which the Base64 is handled. I am trying straight string (event["body"]), the bytes() object, the BytesIO class, and the StringIO class, all to the same BadZipFile exception.
So... is it possible in Pandas/Openpyxl to suppress the validation of the archive? I want to be able to call read_excel() but not have the archive validated and see what happens.
My error stack:
"Error: (<class 'zipfile.BadZipFile'>, BadZipFile('File is not a zip file'),
<traceback object at 0x7f1019589dc0>)\r\n<class 'zipfile.BadZipFile'>\r\n
File is not a zip file\r\nTraceback (most recent call last):\n
File "/var/task/lambda_function.py", line 20, in lambda_handler\n inv = pd.read_excel( bufferedString, engine='openpyxl', index_col=0 )\n
File "/opt/python/pandas/util/_decorators.py", line 311, in wrapper\n return func(*args, **kwargs)\n
File "/opt/python/pandas/io/excel/_base.py", line 457, in read_excel\n io = ExcelFile(io, storage_options=storage_options, engine=engine)\n
File "/opt/python/pandas/io/excel/_base.py", line 1419, in init\n self._reader = self._engines[engine](self._io, storage_options=storage_options)\n
File "/opt/python/pandas/io/excel/_openpyxl.py", line 525, in init\n super().init(filepath_or_buffer, storage_options=storage_options)\n
File "/opt/python/pandas/io/excel/_base.py", line 518, in init\n self.book = self.load_workbook(self.handles.handle)\n
File "/opt/python/pandas/io/excel/_openpyxl.py", line 536, in load_workbook\n return load_workbook(\n
File "/opt/python/openpyxl/reader/excel.py", line 315, in load_workbook\n reader = ExcelReader(filename, read_only, keep_vba,\n
File "/opt/python/openpyxl/reader/excel.py", line 124, in init\n self.archive = _validate_archive(fn)\n
File "/opt/python/openpyxl/reader/excel.py", line 96, in _validate_archive\n archive = ZipFile(filename, 'r')\n
File "/var/lang/lib/python3.9/zipfile.py", line 1264, in init\n self._RealGetContents()\n
File "/var/lang/lib/python3.9/zipfile.py", line 1331, in _RealGetContents\n
raise BadZipFile("File is not a zip file")\n
zipfile.BadZipFile: File is not a zip file\n"
I installed the snowflake connector via the command: pip3 install snowflake-connector-python[pandas]==2.3.3 asn1crypto==1.3.0 --user
I attempted to connect via:
from snowflake import connector
con = connector.connect(
host='.snowflakecomputing.com',
user='THE USER I USE FOR LOGGING IN TO MY TRIAL ACCOUNT',
password='THE PASSWORD I USE FOR LOGGING IN TO MY TRIAL ACCOUNT',
account='zka81761.us-east-1',
warehouse='COMPUTE_WH',
database='DEMO_DB',
schema='PUBLIC',
protocol='https',
port=443)
When executing the above code it just hangs for several minutes then I get an error:
snowflake.connector.errors.OperationalError: 250003: Failed to execute request: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
The longer version is:
File "tests/integration_tests/data_sources/test_snowflake_ds.py", line 6, in test_snowflake_ds
ds = SnowflakeDS(query='SELECT * FROM HEALTHCARE_COSTS', host='.snowflakecomputing.com', user='GEORGE3D6', password='a passwordd', account='zka81761.us-east-1', warehouse='COMPUTE_WH', database='DEMO_DB', schema='PUBLIC', protocol='https', port=443)
File "/home/george/mindsdb_native/mindsdb_native/libs/data_types/data_source.py", line 13, in __init__
df, col_map = self._setup(*args, **kwargs)
File "/home/george/mindsdb_native/mindsdb_native/libs/data_sources/snowflake_ds.py", line 21, in _setup
port=port)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/__init__.py", line 52, in Connect
return SnowflakeConnection(**kwargs)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/connection.py", line 219, in __init__
self.connect(**kwargs)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/connection.py", line 414, in connect
self.__open_connection()
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/connection.py", line 613, in __open_connection
self._authenticate(auth_instance)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/connection.py", line 839, in _authenticate
self.__authenticate(self.__preprocess_auth_instance(auth_instance))
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/connection.py", line 869, in __authenticate
session_parameters=self._session_parameters,
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/auth.py", line 209, in authenticate
socket_timeout=self._rest._connection.login_timeout)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/network.py", line 509, in _post_request
_include_retry_params=_include_retry_params)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/network.py", line 586, in fetch
**kwargs)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/network.py", line 676, in _request_exec_wrapper
conn, full_url, cause)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/network.py", line 706, in handle_invalid_certificate_error
'errno': ER_FAILED_TO_REQUEST,
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 128, in errorhandler_wrapper
connection.errorhandler(connection, cursor, error_class, error_value)
File "/home/george/.local/lib/python3.7/site-packages/snowflake/connector/errors.py", line 90, in default_errorhandler
done_format_msg=error_value.get('done_format_msg'))
snowflake.connector.errors.OperationalError: 250003: Failed to execute request: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
This error message tells me nothing, any help would be appreicated
According to the documentation on the python API the host field is no longer used so try removing that. Also, even if it was used, you haven't enclosed it properly in quotes:
You have: host='.snowflakecomputing.com,
Should be: host='.snowflakecomputing.com',
First, I'd see if removing the host completely fixes your issue since it shouldn't be used anyway.
Googling the error, and the error message itself, suggests that the issue is due to the URL being too long so I'd say the error is down to the fact that you haven't enclosed it properly.
I used to connect snowflake from python like below:
import snowflake.connector as sf
sfconnection = sf.connect(
user='THE USER I USE FOR LOGGING IN TO MY TRIAL ACCOUNT',
password='THE PASSWORD I USE FOR LOGGING IN TO MY TRIAL ACCOUNT',
account='zka81761.us-east-1',
warehouse='COMPUTE_WH',
database='DEMO_DB',
schema='PUBLIC')
Apparently the docs are wrong and the host should now be the full URL (so, in my case zka81761.us-east-1.snowflakecomputing.com). That is to say, it should include the account.
While using jira python module to add comments, is there a possibility to insert comments using WIKI markup?
I noticed some example showed REST API calls with representation:wiki can do this.
But I noticed current python Jira only supports plain text as comments. Is this a limitation or I am missing something?
I checked jira source code and noticed data is represented as dictionary and also being dumped using json.dumps, I tried pushing body as dict "{'storage': {'value': '== Activity: == error-rate-percentage-is-at-acceptable-limits .. Started', 'representation': 'wiki'}}"
But got below error back from Jira API call I guess
[2019-12-02 01:07:22 DEBUG] [__init__:386] Before-control 'jira-integration' failed
Traceback (most recent call last):
File "<<HIDDEN BY ME>>>/lib/python3.7/site-packages/chaoslib/control/__init__.py", line 377, in apply_controls
settings=settings)
File "<<HIDDEN BY ME>>>/python3.7/site-packages/chaoslib/control/python.py", line 147, in apply_python_control
func(context=context, **arguments)
File "<<HIDDEN BY ME>>>/python3.7/site-packages/<<HIDDEN BY ME>>/controls/jira/tickets.py", line 220, in before_activity_control
add_comment(os.environ["SUB_TASK_TICKET"], content_as_wiki(formatting.format_as_heading2("Activity: ") + str(context["name"]) + " .. Started"))
File "<<HIDDEN BY ME>>>/python3.7/site-packages/<<HIDDEN BY ME>>/controls/jira/tickets.py", line 58, in add_comment
test = JIRA_CLIENT.add_comment(issue, comment)
File "<<HIDDEN BY ME>>/python3.7/site-packages/jira/client.py", line 126, in wrapper
result = func(*arg_list, **kwargs)
File "<<HIDDEN BY ME>>/python3.7/site-packages/jira/client.py", line 1367, in add_comment
url, data=json.dumps(data)
File "<<HIDDEN BY ME>>/python3.7/site-packages/jira/resilientsession.py", line 154, in post
return self.__verb('POST', url, **kwargs)
File "<<HIDDEN BY ME>>/python3.7/site-packages/jira/resilientsession.py", line 147, in __verb
raise_on_error(response, verb=verb, **kwargs)
File "<<HIDDEN BY ME>>/python3.7/site-packages/jira/resilientsession.py", line 57, in raise_on_error
r.status_code, error, r.url, request=request, response=r, **kwargs)
jira.exceptions.JIRAError: JiraError HTTP 400 url: https://<<HIDDEN BY ME>>>
text: Can not deserialize instance of java.lang.String out of START_OBJECT token
at [Source: com.enhancera.auditor.common.filter.RestReadingServletRequest$1#4e08280a; line: 1, column: 2] (through reference chain: com.atlassian.jira.issue.fields.rest.json.beans.CommentJsonBean["body"])
I was able to create a comment using markup by passing a string formatted according to the documentation here. For a test I created an issue and added a comment to add a two column 1 row (+ header row) table with:
comment = jira.add_comment(issue, "||header1||header2||\n|one|two|")
Which produced:
Make sure to wrap your comment string in double quotes.
I tried to use Rasa's ReminderScheduled as specified in the docs. I'm using Windows 10 with the Ubuntu subsystem to run the code. The code that calls the reminder is the following:
export_timeout = datetime.timedelta(seconds=30)
class ActionGiveListProducts(Action):
def name(self):
return 'action_give_list_products'
def run(self, dispatcher, tracker, domain):
s = getInfo("listeproduits")
dispatcher.utter_message(s)
return [ReminderScheduled("action_export_logs", datetime.datetime.now() + export_timeout)]
Executing this Action causes the following error:
Traceback (most recent call last):
File "bot.py", line 136, in <module>
run()
File "bot.py", line 108, in run agent.handle_channel(ConsoleInputChannel())
File "/usr/local/lib/python3.5/dist-packages/rasa_core/agent.py", line 126, in handle_channel
processor.handle_channel(input_channel)
File "/usr/local/lib/python3.5/dist-packages/rasa_core/processor.py", line 60, in handle_channel
input_channel.start_sync_listening(self.handle_message)
File "/usr/local/lib/python3.5/dist-packages/rasa_core/channels/console.py", line 52, in start_sync_listening
self._record_messages(message_handler)
File "/usr/local/lib/python3.5/dist-packages/rasa_core/channels/console.py", line 45, in _record_messages
self.sender_id))
File "/usr/local/lib/python3.5/dist-packages/rasa_core/processor.py", line 83, in handle_message
self._predict_and_execute_next_action(message, tracker)
File "/usr/local/lib/python3.5/dist-packages/rasa_core/processor.py", line 262, in _predict_and_execute_next_action
dispatcher)
File "/usr/local/lib/python3.5/dist-packages/rasa_core/processor.py", line 312, in _run_action
self._schedule_reminders(events, dispatcher)
File "/usr/local/lib/python3.5/dist-packages/rasa_core/processor.py", line 296, in _schedule_reminders
replace_existing=True)
File "/usr/local/lib/python3.5/dist-packages/apscheduler/schedulers/base.py", line 413, in add_job
'trigger': self._create_trigger(trigger, trigger_args),
File "/usr/local/lib/python3.5/dist-packages/apscheduler/schedulers/base.py", line 907, in _create_trigger
return self._create_plugin_instance('trigger', trigger, trigger_args)
File "/usr/local/lib/python3.5/dist-packages/apscheduler/schedulers/base.py", line 892, in _create_plugin_instance
return plugin_cls(**constructor_kwargs)
File "/usr/local/lib/python3.5/dist-packages/apscheduler/triggers/date.py", line 20, in __init__
timezone = astimezone(timezone) or get_localzone()
File "/usr/local/lib/python3.5/dist-packages/apscheduler/util.py", line 86, in astimezone
'Unable to determine the name of the local timezone -- you must explicitly '
ValueError: Unable to determine the name of the local timezone -- you must explicitly
specify the name of the local timezone.
Please refrain from using timezones like EST to prevent problems with daylight saving time.
Instead, use a locale based timezone name (such as Europe/Helsinki).
I tried to set the timezone in the launch code as following:
os.environ['TZ'] = 'Europe/London'
time.tzset()
but this didn't change anything. I also searched for other solutions, but found nothing relevant.
Does someone know what causes this error exactly and if there is way to eliminate it?
I assume you run this in a Linux environment (as I had the same error), so try to set the following:
keep as you had it in the python code:
keepos.environ['TZ'] = 'Europe/London'
and also set the timezone in the os:
sudo cp /usr/share/zoneinfo/Europe/London /etc/localtime
It worked for me.
The following works on the Ubuntu subsystem for Windows:
sudo cp /usr/share/zoneinfo/America/New_York /etc/localtime
TZ=America/New_York rasa x
You can also make the TZ environment variable permanent by adding export TZ=America/New_York in your ~/.bashrc file.
I am trying to export all tasks from all of my asana work-spaces using python-asana API. But at some point it exists after giving the following error message.
Traceback (most recent call last):
File "export.py", line 56, in <module>
for index, task in enumerate(tasks):
File "build\bdist.win32\egg\asana\page_iterator.py", line 58, in items
File "build\bdist.win32\egg\asana\page_iterator.py", line 54, in next
File "build\bdist.win32\egg\asana\page_iterator.py", line 43, in __next__
File "build\bdist.win32\egg\asana\page_iterator.py", line 74, in get_next
File "build\bdist.win32\egg\asana\client.py", line 104, in get
File "build\bdist.win32\egg\asana\client.py", line 75, in request
asana.error.InvalidRequestError: Invalid Request: Your pagination token has expired.
I read that to solve this we need to make paginated requests. But I tried passing only offset to my request as following:
tasks = client.tasks.find_all({'project' : project['id']}, limit=50)
But, there was no difference as I was not getting any 'next_page' value even though there was more than 50 tasks in the project.
So my question is:
How can I do paginated request using python-asana API? An explanation with an example would be best!
EDIT:
I am fetching the tasks as below:
tasks = client.tasks.find_all({'project' : project['id']}, item_limit=1)
print "Tasks", tasks # Prints generator object
for index, task in enumerate(tasks):
complete_task = client.tasks.find_by_id(task["id"])
print complete_task #Prints complete task dictionary
Now My question is where will I get the next_page content for the remaining tasks and how do I access it.