I have upgraded boto3 from boto3==1.7.48 to 1.13.11, and this has broken all of my tests that use Moto. It looks like (worryingly) the mock has stopped working altogether and is trying to actually access s3, here is an example test function that was previously working:
def upload_video(self, video):
s3 = boto3.client("s3")
s3.create_bucket(Bucket=settings.AWS_STORAGE_BUCKET_NAME)
for media_key in video.upload_media_keys:
s3.upload_file(
os.path.join(
os.path.dirname(os.path.realpath(__file__)), "assets/test.mp4"
),
settings.AWS_STORAGE_BUCKET_NAME,
media_key,
)
But it now gives this error
File "{path}", line 52, in upload_video
s3.create_bucket(Bucket=settings.AWS_STORAGE_BUCKET_NAME)
File "{path}/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "{path}/lib/python3.7/site-packages/botocore/client.py", line 635, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
Any help would be greatly appreciated. Here is the list of upgrades:
Before:
boto3 == 1.7.48
botocore == 1.10.84
moto == 1.3.6
After:
boto3==1.13.11
botocore==1.16.11
moto==1.3.14
I have no idea what changed, but I also ran into this. Here's my workaround:
Previously I had:
conn = boto3.resource("s3", region_name="ca-central-1")
conn.create_bucket(Bucket=os.environ["S3_BUCKET_NAME"]
but I changed the region to us-east-1 and everything worked
conn = boto3.resource("s3", region_name="us-east-1")
conn.create_bucket(Bucket=os.environ["S3_BUCKET_NAME"]
Since this is just a fake bucket for testing, I see no harm in using a different region if it makes things work
Related
I have already created an account in picovoice and recieved an access key, but when I try to put the path of my ppn file, it shows an error:
`
[ERROR] loading keyword file at 'C:\Users\Priyam\Desktop\hey keyazip' failed with 'IO_ERROR'
Traceback (most recent call last):
File "e:\Personal Project\import struct.py", line 13, in <module>
porcupine = pvporcupine.create(access_key='access key',
File "C:\Users\Priyam\AppData\Roaming\Python\Python310\site-packages\pvporcupine\__init__.py", line 77, in create
return Porcupine(
File "C:\Users\Priyam\AppData\Roaming\Python\Python310\site-packages\pvporcupine\porcupine.py", line 158, in __init__
raise self._PICOVOICE_STATUS_TO_EXCEPTION[status]()
pvporcupine.porcupine.PorcupineIOError
the code is:
`
porcupine=None
paud=None
audio_stream=None
try:
access_key="access key"
porcupine = pvporcupine.create(access_key='access key',
keyword_paths=['C:\\Users\Priyam\Desktop\hey keyazip'],keywords=['hey keya']) #pvporcupine.KEYWORDS for all keywords
paud=pyaudio.PyAudio()
audio_stream=paud.open(rate=porcupine.sample_rate,channels=1,format=pyaudio.paInt16,input=True,frames_per_buffer=porcupine.frame_length)
while True:
keyword=audio_stream.read(porcupine.frame_length)
keyword=struct.unpack_from("h"*porcupine.frame_length,keyword)
keyword_index=porcupine.process(keyword)
if keyword_index>=0:
print("hotword detected")
finally:
if porcupine is not None:
porcupine.delete()
if audio_stream is not None:
audio_stream.close()
if paud is not None:
paud.terminate()
`
I tried the code above and the code provided by picovoice itself, yet I am facing the same issues
It looks like Porcupine is having trouble finding your keyword file: [ERROR] loading keyword file at 'C:\Users\Priyam\Desktop\hey keyazip' failed with 'IO_ERROR'.
The Picovoice console provides the keyword inside a compressed .zip file. You will need to decompress the file and in your code update the path to lead to the .ppn file located inside. For example: C:\\Users\Priyam\Desktop\hey-keya_en_windows_v2_1_0\hey-keya_en_windows_v2_1_0.ppn
I am fetching a secret from secret manager on a lambda. The request fails sometimes. Which is totally strange, it is working fine and couple of hours later I check and I am getting time out.
def get_credentials(self):
"""Retrieve credentials from the Secrets Manager service."""
boto_config = BotoConfig(connect_timeout=3, retries={"max_attempts": 3})
secrets_client = self.boto_session.client(
service_name="secretsmanager",
region_name=self.boto_session.region_name,
config=boto_config,
)
secret_value = secrets_client.get_secret_value(SecretId=self._secret_name)
secret = secret_value["SecretString"]
I try to debug the lambda and later seems to be working again, without any change, those state changes happen in hours. Any hint why that could happen?
Traceback (most recent call last):
File "/opt/python/botocore/endpoint.py", line 249, in _do_get_response
http_response = self._send(request)
File "/opt/python/botocore/endpoint.py", line 321, in _send
return self.http_session.send(request)
File "/opt/python/botocore/httpsession.py", line 438, in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://secretsmanager.eu-central-1.amazonaws.com/"
You are using the legacy retry mode (is the default one in boto3), which has very limited functionality as it only works for a very limited number of errors.
You should try changing your retry strategy to something like Standard retry mode or Adaptative. In that case your code would look like:
from botocore.config import Config as BotoConfig
boto_config = BotoConfig(
connect_timeout=3,
retries={
"max_attempts": 3,
"mode":"standard"
}
)
secrets_client = self.boto_session.client(
service_name="secretsmanager",
region_name=self.boto_session.region_name,
config=boto_config,
)
I have defined this SNS-triggered Lambda in Chalice:
#app.on_sns_message(topic='arn:aws:sns:us-west-1:XXXXXXXX:MyTopic')
def step1_photo_url_preload(event, retry = 3):
try:
js = json.loads(event.message)
... some logic here, event object is never modified ...
except:
if retry:
print("WARNING: failed, %d retries remaining" % retry)
return step1_photo_url_preload(event, retry-1)
else:
raise
When an exception is raised, the function should retry up to 3 times.
Instead, what I get is the exception below. Look closely at the trace: Line 56 shows the error occurs when attempting the recursive call:
[ERROR] TypeError: 'SNSEvent' object is not subscriptable
Traceback (most recent call last):
File "/var/task/chalice/app.py", line 1459, in __call__
return self.func(event_obj)
File "/var/task/app.py", line 56, in step1_photo_url_preload
return step1_photo_url_preload(event, retry-1)
File "/var/task/chalice/app.py", line 1458, in __call__
event_obj = self.event_class(event, context)
File "/var/task/chalice/app.py", line 1486, in __init__
self._extract_attributes(event_dict)
File "/var/task/chalice/app.py", line 1532, in _extract_attributes
first_record = event_dict['Records'][0]
Mysteriously, the function can't work with the event object that it received the first time.
What could cause this?
I suspect this might have something to do with the magic behind #app.on_sns_message, but I'm not sure where to look next.
The problem is the fact that the function is decorated, the failure is in the code the decorator is running. Pull the functionality you want to run recursively into a separate function and the problem should go away.
I have defined the following function. The purpose is to make batch create entity request with dialogflow client. I am using this method after sending many individual tests did not scale well.
The problem seems to be the line that defines EntityType. Seems like "entityType" is not valid but that is what is in the dialogflow v2 documentation which is the current version I am using.
Any ideas on what the issue is?
def create_batch_entity_types(self):
client = self.get_entity_client()
print(DialogFlowClient.batch_list)
EntityType = {
"entityTypes": DialogFlowClient.batch_list
}
response = client.batch_update_entity_types(parent=AGENT_PATH, entity_type_batch_inline=EntityType)
def callback(operation_future):
# Handle result.
result = operation_future.result()
print(result)
response.add_done_callback(callback)
After running the function I received this error
Traceback (most recent call last):
File "df_client.py", line 540, in <module>
create_entity_types_from_database()
File "df_client.py", line 426, in create_entity_types_from_database
df.create_batch_entity_types()
File "/Users/andrewflorial/Documents/PROJECTS/curlbot/dialogflow/dialogflow_accessor.py", line 99, in create_batch_entity_types
response = client.batch_update_entity_types(parent=AGENT_PATH, entity_type_batch_inline=EntityType)
File "/Users/andrewflorial/Documents/PROJECTS/curlbot/venv/lib/python3.7/site-packages/dialogflow_v2/gapic/entity_types_client.py", line 767, in batch_update_entity_types
update_mask=update_mask,
ValueError: Protocol message EntityTypeBatch has no "entityTypes" field.
The argument for entity_type_batch_inline must have the same form as EntityTypeBatch.
Look how that type looks like: https://dialogflow-python-client-v2.readthedocs.io/en/latest/gapic/v2/types.html#dialogflow_v2.types.EntityTypeBatch
It has to have entity_types field, not entityTypes.
I'm trying to query large amount of data in BigQuery and then upload the table in the desired dataset (datasetxxx) using "datalab" in PyCharm as the IDE. Below is my code:
query = bq.Query(sql=myQuery)
job = query.execute_async(
output_options=bq.QueryOutput.table('datasetxxx._tmp_table', mode='overwrite', allow_large_results=True))
job.result()
However, I ended up with "No project ID found". Project Id is imported through a .jason file as os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = path to the file. I also tried to explicitly declare project Id above as follows.
self.project_id = 'xxxxx'
query = bq.Query(sql=myQuery, context = self.project_id)
This time I ended up with the following error:
TypeError: init() got an unexpected keyword argument 'context'.
It's also an up-to-date version. Thanks for your help.
Re: The project Id is specified in the "FROM" clause and I'm also able to see the path to the .json file using "echo" command. Below is the stack-trace:
Traceback (most recent call last):
File "xxx/Queries.py", line 265, in <module>
brwdata._extract_gbq()
File "xxx/Queries.py", line 206, in _extract_gbq
, allow_large_results=True))
File "xxx/.local/lib/python3.5/site packages/google/datalab/bigquery/_query.py", line 260, in execute_async
table_name = _utils.parse_table_name(table_name, api.project_id)
File "xxx/.local/lib/python3.5/site-packages/google/datalab/bigquery/_api.py", line 47, in project_id
return self._context.project_id
File "xxx/.local/lib/python3.5/site-packages/google/datalab/_context.py", line 62, in project_id
raise Exception('No project ID found. Perhaps you should set one by running'
Exception: No project ID found. Perhaps you should set one by running"%datalab project set -p <project-id>" in a code cell.
So, if you do "echo $GOOGLE_APPLICATION_CREDENTIALS" you can see the path of your JSON.
So, could you make sure if the "FROM" from the query has specified the right external project?
Also, if your QueryOutput destination is your very same project, you are doing it right,
table('dataset.table'.....)
But in order case you should specify:
table('project.dataset.table'....)
I don't exactly know how are you doing the query but the error might be there.
I reproduced this and it worked fine to me:
import google.datalab
from google.datalab import bigquery as bq
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] ="./bqauth.json"
myQuery="SELECT * FROM `MY_EXAMPLE_PROJECT.MY_EXAMPLE_DATASET.MY_EXAMPLE_TABLE` LIMIT 1000"
query = bq.Query(sql=myQuery)
job = query.execute_async(
output_options=bq.QueryOutput.table('MY_EXAMPLE_PROJECT.MY_EXAMPLE_DATASET2.MY_EXAMPLE_TABLE2', mode='overwrite', allow_large_results=True))
job.result()
Here's the updated way if someone in need:
Now you can use the Context in latest version as:
from google.datalab import bigquery as bq
from google.datalab import Context as ctx
ctx.project_id = 'PROJECT_ID'
df = bq.Query(query).execute()
...