I am trying to read data from my dynamo db table using alexa, but it seems that my lambda function is not able to access my table.
Here is the code to get data from dynamodb :
# Gets the data from dynamodb based on userid
def GetData(session):
userId = session['user']['userId'].split('.')
userId = userId[3]
try:
response = table.query(
KeyConditionExpression=Key('guid').eq(str(userId))
)
# print ("Got data: " + str(len(response)))
print(response)
for item in response['Items']:
final_response = item["command"]
tstamp = item["tstamp"]
if (response['Count'] == 0):
final_response = "No Data with this userid. You can ask to get the userid"
else:
now = datetime.datetime.utcnow()
timestamp = int(round((now - datetime.datetime(2016, 1, 1)).total_seconds()))
if ((timestamp - int(tstamp)) > 60):
final_response = "No Data received from device in past 1 minute"
return final_response
except ClientError as e:
print(e.response['Error']['Message'])
When I ask Alexa my custom question, the only response I get is No Data with this userid. You can ask to get the userid.
When I test run my lambda function, it runs successfully. But it not querying the database.
If you want to store and retrieve based on userId, you can use the persistenceAdapter interface provided in the ASK SDK. It looks like you are not currently using that. I would definitely recommend it as it makes managing of handlers for different Intents much easier (and provides useful abstractions like this one). ASK SDK docs https://developer.amazon.com/en-US/docs/alexa/alexa-skills-kit-sdk-for-python/overview.html
The abstraction will allow you to use S3 or Dynamo as an implementer. Here is the dynamoDB library. https://github.com/alexa/alexa-skills-kit-sdk-for-python/tree/master/ask-sdk-dynamodb-persistence-adapter
Create and register the Dynamo Adapter using the SkillBuilder constructor (ex): https://github.com/alexa/skill-sample-python-highlowgame/blob/master/lambda/py/lambda_function.py#L16
Use your adapter (ex): https://github.com/alexa/skill-sample-python-highlowgame/blob/master/lambda/py/lambda_function.py#L130
When using the adapter, remember that the dynamo write call does not happen until you save, so you can modify the object at will until you call save. All handler_input objects have a reference to the persistent_attributes through the attributes_manager (this also lets you easily work with session_attributes)
Related
I've been playing around with some "weirder" querying of DynamoDB with Reserved Words using boto3.resource method, and came across a pretty annoying issue which I can't resolve for quite some time (Always the same error sigh), and can't seem to find the answer anywhere.
My code is the following:
import logging
import boto3
from boto3.dynamodb.conditions import Key
logger = logging.getLogger()
logger.setLevel(logging.INFO)
TABLE_NAME = "some-table"
def getItems(record, table=None):
if table is None:
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(TABLE_NAME)
record = str(record)
get_item = table.query(
KeyConditionExpression=Key("#pk").eq(":pk"),
ExpressionAttributeNames={"#pk": "PK"},
ExpressionAttributeValues={":pk": record},
)
logger.info(
f"getItem parameters\n{json.dumps(get_item, indent=4,sort_keys=True, default=str)}"
)
return get_item
if __name__ == "__main":
record = 5532941
getItems(record)
It's nothing fancy, as I mentioned I'm just playing around, but I'm constantly getting the following error no matter what I try:
"An error occurred (ValidationException) when calling the Query operation: Value provided in ExpressionAttributeNames unused in expressions: keys: {#pk}"
As far as I understand in order to "replace" the reserved keys/values with something arbitrary you put it into ExpressionAttributeNames and ExpressionAttributeValues, but I can't wrap my head around as to why it's telling me that this key is not used.
I should mention that this Primary Key exists with this value in the record var in DynamoDB.
Any suggestions?
Thanks
If you're using Key then just provide the string values and don't be fancy with substitution. See this example:
https://github.com/aws-samples/aws-dynamodb-examples/blob/master/DynamoDB-SDK-Examples/python/WorkingWithQueries/query_equals.py
If you're writing an equality expression as a single string, then you need the substitution. See this example:
https://github.com/aws-samples/aws-dynamodb-examples/blob/master/DynamoDB-SDK-Examples/python/WorkingWithQueries/query-consistent-read.py
I have a dynamodb table configured using CDK (i used python)in one script which is created using cdkstack. Now , I have another .py script which helps me insert data into the db table created using cdk.
My boto3 script is a minimal script which looks like this -
import boto3
ddb = boto3.resource('dynamodb')
table = ddb.Table(db_stream_table.table_arn)
table.put_item(Item={'timestamp': "123456", 'current_status': "response received", "responded_with": "Yes",'ticket_id': 100})
I am not able to figure out how i can reference the dynamodb table created using cdk here in this boto3 script because the line db_stream_table.table_arn returns a token and that infact does not refer to the name of the table.
Also the name of the table looks somewhat like this: InfratestcdkStack-TableCD557FA0-1S1M37VCP51C9. (This table name is not as exactly depicted here , i have made a few changes to the numbers and alphabets, but it's the same format)
Am I missing something? or is my approach wrong? is there a simple way to do it?
I think there are more easier approaches to this , but I figured out a solution which is to set ssm parameters in your cdk script like:
aws_ssm.StringParameter(self, "SsmParameterforDynamoDB",
allowed_pattern=".*",
description="The name of the DynamoDB table",
parameter_name="dbname",
string_value=db_stream_table.table_name, # this is the name of your db table as referenced in your cdk code
tier=aws_ssm.ParameterTier.ADVANCED
)
and then if you have a seperate script running boto to put an item into this table , you can have
import boto3
client = boto3.client('ssm')
response = client.get_parameter(
Name='dbname',
WithDecryption=False
)
tbl_Transaction_table_name = response['Parameter']['Value']
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(tbl_Transaction_table_name)
table.put_item(Item={'timestamp': "123456", 'current_status': "response received", "responded_with": "Yes",
'ticket_id': 100})
Maybe I'd try having an iam role to restrict access to this as an additional step.
I tried to find a way through the Glue API docs, but there is no attribute or method related to the functions get_table(**kwargs) or get_tables(**kwargs).
I imagine something akin to the following (pseudo-)code:
client = boto3.client('glue')
paginator = client.get_paginator('get_tables')
for response in paginator.paginate(DatabaseName=db_input_shared):
for table in response['TableList']:
files = table["files"] # NOTE: the keyword "files" is invented
# Do something else
...
As far as I can see from the docs, the table from the reponse["TableList"] should be a dictionary; yet none of its keys seem to give access to the files stored in it.
The solution to the problem was using awswrangler.
The following functions checks all AWS Glue Tables within a database for a specific list of recently uploaded files. Whenever the filename matches, it is going to yield the associated table dictionary. These yielded tables are those which have been recently updated.
def _yield_recently_updated_glue_tables(upload_path_list: List[str],
db_name: str) -> Union(dict, None):
"""Check which tables have been updated recently.
Args:
upload_path_list (List[str]): contains all S3-filepaths of recently uploaded files
db_name (str): name of the AWS Glue database
Yields:
Union(dict, None): AWS Glue table dictionaries recently updated
"""
client = boto3.client('glue')
paginator = client.get_paginator('get_tables')
for response in paginator.paginate(DatabaseName=db_name):
for table_dict in response['TableList']:
table_name = table_dict['Name']
s3_bucket_path = awswrangler.catalog.get_table_location(
database=db_name, table=table_name)
s3_filepaths = list(
awswrangler.s3.describe_objects(s3_bucket_path).keys())
table_was_updated = False
for upload_file in upload_path_list:
if upload_file in s3_filepaths:
table_was_updated = True
break
if table_was_updated:
yield table_dict
I manage to make a connection to a Google Cloud Datastore databased. Now I want to get some entities given their Key/Id. Right now I am doing the following:
from google.cloud import datastore
client = datastore.Client()
query = client.query(kind='City')
query.key_filter("325899977574122") -> Exception here
I get "Invalid key: '325899977574122'".
What could be the cause of error? That Id exist, a city does have that key/Id.
It looks like it needs to be of type google.cloud.datastore.key.Key
https://googleapis.dev/python/datastore/latest/queries.html#google.cloud.datastore.query.Query.key_filter
Also, 325899977574122 is probably supposed to be cast to a long
So something like this:
client = datastore.Client()
query = client.query(kind='City')
query.key_filter(Key('City', 325899977574122L, project=project))
EDIT:
Also if youre trying to retrieve a single id, you should probably use this:
https://googleapis.dev/python/datastore/latest/client.html#google.cloud.datastore.client.Client.get
client = datastore.Client()
client.get(Key('City', 325899977574122L, project=project))
Fetching by ID is faster than doing a query
I'm rather new to the whole ORM topic, and I've already searched forums and docs.
The question is about a flask application with SQLAlchemy as ORM for the PostgreSQL.
The __init__.py contains the following line:
db = SQLAlchemy()
the created object is referenced in the other files to access the DB.
There is a save function for the model:
def save(self):
db.session.add(self)
db.session.commit()
and also an update function:
def update(self):
for var_name in self.__dict__.keys():
if var_name is not ('_sa_instance_state' or 'id' or 'foreign_id'):
# Workaround for JSON update problem
flag_modified(self, var_name)
db.session.merge(self)
db.session.commit()
The problem occurs when I'm trying to save a new object. The save function writes it to DB, it's visible when querying the DB directly (psql, etc.), but a following ORM query like:
model_list = db.session.query(MyModel).filter(MyModel.foreign_id == this_id).all()
gives an empty response.
A call of the update function does work as expected, new data is visible when requesting with the ORM.
I'm always using the same session object for example this:
<sqlalchemy.orm.scoping.scoped_session object at 0x7f0cff68fda0>
If the application is restarted everything works fine until a new object was created and tried to get with the ORM.
An unhandsome workaround is using raw SQL like:
model_list = db.session.execute('SELECT * FROM models_table WHERE
foreign_id = ' + str(this_id))
which gives a ResultProxy with latest data like this:
<sqlalchemy.engine.result.ResultProxy object at 0x7f0cf74d0390>
I think my problem is a misunderstanding of the session. Can anyone help me?
It figured out that the problem has nothing to do with the session, but the filter() method:
# Neccessary import for string input into filter() function
from sqlalchemy import text
# Solution or workaround
model_list = db.session.query(MyModel).filter(text('foreign_key = ' + str(this_id))).all()
I could not figure out the problem with:
filter(MyModel.foreign_id == this_id) but that's another problem.
I think this way is better than executing raw SQL.