My AWS dynamoDB has id as Partition key and there is no Sort key. The following does not return the existing record from the table:
response = producttable.scan(FilterExpression=Attr('title').eq("My Product"))
response['ScannedCount'] is less than the total count of the table.
You need to pagínate.
For example code look at https://github.com/aws-samples/aws-dynamodb-examples/blob/master/DynamoDB-SDK-Examples/python/WorkingWithScans/scan_paginate.py
Related
I have a list of items and would like to use each item as the pk (Primary Key) to query Dynamo DB, using Python.
I have tried using a for loop but I dont get any results, If I try the same query with the actual value from the group_id list it does work which means my query statement is correct.
group_name_query = []
for i in group_id:
group_name_query = config_table.query(
KeyConditionExpression=Key('pk').eq(i) & Key('sk').eq('GROUP')
)
Here is a sample group_ip = ['GROUP#6501e5ac-59b2-4d05-810a-ee63d2f4f826', 'GROUP#6501e5ac-59b2-4d05-810a-ee63d2sfdgd']
not answering your issue but got a suggestion, if you're querying base table with pk and sk instead of query gsi, i would suggest you Batch Get Item API to get multiple items in one shot
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/example_dynamodb_BatchGetItem_section.html
I am trying to query the traces Cassandra table which is part of the Jaeger architecture. As you can see the refs field is a list:
cqlsh:jaeger_v1_dc1> describe traces
CREATE TABLE jaeger_v1_dc1.traces (
trace_id blob,
span_id bigint,
span_hash bigint,
duration bigint,
flags int,
logs list<frozen<log>>,
operation_name text,
parent_id bigint,
process frozen<process>,
refs list<frozen<span_ref>>,
start_time bigint,
tags list<frozen<keyvalue>>,
PRIMARY KEY (trace_id, span_id, span_hash)
)
from the python code:
traces = session.execute('SELECT span_id,refs from traces')
for t in traces:
if t.refs is not None:
parentTrace=t['refs'][0].trace_id
My first question is it possible to directly select the parent trace without iterating through the result? Is there a way i can get the first element in the list and then get the elements inside from the select statment?
From the terminal using cqlsh ,I am getting this result: trace_id: 0x00000000000000003917678c73006f57. However, from a python cassandra client I got this trace_id=b'\x00\x00\x00\x00\x00\x00\x00\x009\x17g\x8cs\x00oW' any idea what transformation happened to it? How can decode it since I want to use to query the table again.
To my knowledge, there is no easy way as there is no guarantee that the spans are stored in a specific order. Worth noting though, is if by parentTrace, you mean the root span of the trace (the first span), then you can search for spans where refs is null because a root span has no parent. Another way to identify a root span is if the trace_id == span_id.
trace_id is stored as a binary blob. What you see from cassandra client is an array of 16 bytes with each octet element represented as two hexadecimal values. To convert it to the hex string you see in cqlsh, you'll need to convert the entire array to a single hex string. See the following python example that does this:
from cassandra.cluster import Cluster
cluster = Cluster(['127.0.0.1'])
session = cluster.connect()
rows = session.execute("select * from jaeger_v1_test.traces")
trace = rows[0]
hexstr = ''.join('{:02x}'.format(x) for x in trace.trace_id)
print("hex=%s, byte_arr=%s, len(byte_arr)=%d" % (hexstr, trace.trace_id, len(trace.trace_id)))
cluster.shutdown()
I have an html table that is filled from a DynamoDB table. Clicking a row pops up an edit form in a modal. The data inputted is sent to a flask server to update the item - using AWS DynamoDB - that was edited in the modal form. Upon reading the AWS documentation for this, the correct method is to use update_item. However, when doing so the item is added again instead of updating the item. I used the AWS here to script the below. In my DynamoDB table, the primary partition key is KEY1 and the primary sort key is KEY2 in the below reference.
table = dynamodb.Table('table_name') #define DynamoDB table
key1 = account_id #string value of account id
key2 = request.form["KEY2"] #this is a read only field in the form, so the key does not get updated here
form_val1 = request.form["input1"]
form_val2 = request.form["input2"]
form_val3 = request.form["input3"]
form_val4 = request.form["input4"]
form_val5 = request.form["input5"]
form_val6 = request.form["input6"]
form_val7 = request.form["input7"]
form_val8 = request.form["input8"]
form_val9 = request.form["input9"]
#update item in dynamo
table.update_item(
Key={
'KEY1': key1, #partition key
'KEY2': key2 #sort key
},
UpdateExpression='SET dbField1 = :val1, dbField2 = :val2, dbField3 = :val3, dbField4 = :val4, dbField5 = :val5, dbField6 = :val6, dbField7 = :val7, dbField8 = :val8, dbField9 = :val9',
ExpressionAttributeValues={
':val1': form_val1,
':val2': form_val2,
':val3': form_val3,
':val4': form_val4,
':val5': form_val5,
':val6': form_val6,
':val7': form_val7,
':val8': form_val8,
':val9': form_val9
}
)
You can't and I will explain to you for what that not is possible.
When you create a table on dynamo DB with key and a order key you automatically create an index between key and sort key. We know an index is inmutable, that means you can't update the keys. Is for that reason that when you update dynamo create a new element.
It's a problem of the definition of your table because you never need to change the key or the sort key. Recreate your table only with the index and not with the sort index (because if your app can change the sort index that make not sense).
Is this the full query? the update_item docs say that TableName is required, which I don't see in your snippet.
From the updateitem docs:
Edits an existing item's attributes, or adds a new item to the table
if it does not already exist.
Make sure that the primary key (partition key and sort key) are unique in your table. If they are not, updateitem will create a new item in the database.
Are you absolutely certain that the primary key for the item already exists in the database?
I have partition key in azure storage table as below, partition key is string here.
for each day, the partition key is Date in string format. if i take count of records based on partiotion key (03042018) and rowkey as (1) . i get 2 counts.
partitionkey rowkey timestamp desc
02042018 1 2018-11-26T01:23:57.149Z 'abc'
02042018 1 2018-11-26T23:46:57.149Z 'def'
03042018 1 2018-11-27T01:46:57.149Z 'fff'
03042018 1 2018-11-27T01:47:57.149Z 'ggg'
03042018 2 2018-11-27T01:48:01.149Z 'ggg'
How to pass the partition key as variable in the below query.
Here, partitionkey is 'taskseattle', but i want to pass 02042014 on today and when run the python script it should pass 03042018 in the below query in the partitionkey.
a='02042018' (how to pass a in below partition key)
tasks = table_service.query_entities('tasktable', "PartitionKey eq 'tasksSeattle'")
If you just want to pass a variable to the query, please use the code below:
from azure.cosmosdb.table.tableservice import TableService
from azure.cosmosdb.table.models import Entity
table_service = TableService(account_name='your_account',account_key='your_key')
a='03042018'
tasks = table_service.query_entities('tasktable',filter='PartitionKey eq \'' + a + '\'')
for task in tasks:
print(task.description)
The test result as below:
I have one table TestTable and partition Key TestColumn.
Inputs Dates:
from_date= "2017-04-20T16:31:54.451071+00:00"
to_date = "2018-04-20T16:31:54.451071+00:00"
when I use equal query the date then it is working.
key_expr = Key('TestColumn').eq(to_date)
query_resp = table.query(KeyConditionExpression=key_expr)
but when I use between query then is not working.
key_expr = Key('TestColumn').between(from_date, to_date)
query_resp = table.query(KeyConditionExpression=key_expr)
Error:
Unknown err_msg while querying dynamodb: An error occurred (ValidationException) when calling the Query operation: Query key condition not supported
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html
DynamoDB Query will return data from one and only one partition, meaning you have to supply a single partition key in the request.
KeyConditionExpression
The condition that specifies the key value(s)
for items to be retrieved by the Query action.
The condition must perform an equality test on a single partition key
value.
You can optionally use a BETWEEN operator on a sort key (but you still have to supply a single partition key).
If you use a Scan you can use an ExpressionFilter and use the BETWEEN operator on TestColumn