I am trying to query a dynamo table with latitude and longitude for various locations. I want to get the values between certain coordinates as a user pans on the map.
The primary key for the table is city and the sort key is id. I created a global secondary index with lat as the partition key and lon as the sort key (to query for locations between two points in latitude and longitude).
I am trying to use this query:
let doc = require('dynamodb-doc');
let dynamo = new doc.DynamoDB();
...
var params = {
TableName : "locations-dev",
IndexName: "lat-lon-index",
KeyConditionExpression: "lon between :lon2 and :lon1 AND lat between :lat1 and :lat2",
ExpressionAttributeValues: {
":lat1": JSON.stringify(event.bodyJSON.east),
":lat2": JSON.stringify(event.bodyJSON.west),
":lon1": JSON.stringify(event.bodyJSON.north),
":lon2": JSON.stringify(event.bodyJSON.south)
}
};
dynamo.query(params, function (err, data) {
if (err) {
console.error('Error with ', err);
context.fail(err);
} else {
context.succeed(data);
}
});
But I am getting this error:
{
"errorMessage": "Query key condition not supported",
"errorType": "ValidationException",
"stackTrace": [
...
]
}
Here is an example item in Dynamo:
{
"id": "18",
"lat": "39.923070",
"lon": "-86.036178",
"name": "Home Depot",
"phone": "(317)915-8534",
"website": "https://corporate.homedepot.com/newsroom/battery-recycling-one-million-pounds"
}
Primary keys (even in secondary indices) in DynamoDB can only be queried with equals criteria. This constraint is derived from its internal representation since it is stored as hashed value to identify its item partition. Those hashed values cannot be queried by range.
Choosing the Right DynamoDB Partition Key
Except for scan, DynamoDB API operations require an equal operator
(EQ) on the partition key for tables and GSIs. As a result, the
partition key must be something that is easily queried by your
application with a simple lookup (for example, using key=value, which
returns either a unique item or fewer items).
Related
I want to filter results inside DynamoDB by data that is not PartitionKey or Sort KEY (name = RowKey).
I can limit the result by PartitionKey.
So I tried it in according to the documentation to make a query (without Sort Key) but I get this error:
Unable to query. Error: {
"message": "Query condition missed key schema element: RowKey",
"code": "ValidationException",
"time": "2020-04-03T13:09:17.635Z",
"requestId": "ECD4VQBGFB0IUM6G9TUSPKA3LRVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 43.542922337630074
}
The following image is the configuration of the table
And this is the code that I used:
async function FilterUsersAwsQUERY (_tenant_id,_filter) {
//debug('Enter in function');
var params = {
TableName : "zzzuser",
ProjectionExpression:"#pk, comapnyrole",
KeyConditionExpression: "#pk = :pk and #cr = :cr",
ExpressionAttributeNames:{
"#pk": "PartitionKey",
"#cr": "companyrole"
},
ExpressionAttributeValues: {
":pk": _tenant_id,
":cr": _filter
}
};
return new Promise (function(resolve, reject) {
//debug('Start query');
docClient.query(params, function(err, data) {
if (err) {
console.error("Unable to query. Error:", JSON.stringify(err, null, 2));
reject(err);
} else {
//console.log("AWS ALL Query succeeded.");
resolve(data.Items);
//data.Items.forEach(function(item) {
// console.log(" -", item.year + ": " + item.title);
//});
}
});
});
};
PS. I know that there is a .scan options but it takes over 10 seconds to do it because I have 10.000 records inside it and every PartitionKey is about 400 records.
There is any way to do it efficently?
Thanks to all
The key condition expression has to be the partition key or the partition key and sort key. Then you can narrow it down with a FilterExpression. That said, refining with a filter expression can be expensive if you have to do it often AND the result set to filter is large.
This is a time where a LSI or GSI might be a better bet if this is an access pattern you have to do often and with high performance. if it is not something you do often, it might be cheaper just to take the hit with the FilterExpression as that would be cheaper. You have not given enough info for me to know one way or the other.
I am trying to return data from a DynamoDB table with results ordered numerically by the Primary Sort Key. I am using a Lambda scan function to return the data but it is not returning in numerical order.
The Primary Sort Key is 'time', how can i achieve this?
const AWS = require('aws-sdk');
const dynamodb = new AWS.DynamoDB({region: 'eu-west-2', apiVersion: '2012-08-10'});
exports.handler = (event, context, callback) => {
const params = {
TableName: "finalTrickstar",
};
dynamodb.scan(params, function(err, data){
if (err) {
console.log(err);
callback(err);
} else {
console.log(data);
const items = data.Items.map(
(dataField) => {
return {time: dataField.time.S, day: dataField.day.S, show: dataField.show.S, showID: dataField.showID.S};
}
);
callback(null, items);
}
});
};
I thought having a Primary Sort Key would return results ordered by the key but instead they are seemingly not ordered at all.
I don't know what you mean by "Primary Sort Key" but DynamoDB supports two types of primary keys:
partition key
partition key + sort key (composite primary key)
If your table has a primary key composed of a partition key and a sort key then the data will be kept and retrieved sorted by the sort key. If the data type of the sort key is Number, the results are returned in numeric order; otherwise, the results are returned in order of UTF-8 bytes. By default, the sort order is ascending. To reverse the order, set the ScanIndexForward parameter to false.
Make sure your table is configured correctly and that you're not expecting the data to be sorted in any other way than by the UTF-8 bytes of the sort key if it's not a number.
I am trying to get max value from id item. I am using ScanIndexForward:false which returns desc order of id values which is not working as expected now its giving the same insertion order from the table.
Please see my code below and please suggest if any changes required.
var params1 = {
TableName: "Test",
Limit:3,
ProjectionExpression: 'id',
ScanIndexForward:false,
};
docClient.scan(params1, function (err, data) {
if (err) {
console.error("Unable to query. Error:", JSON.stringify(err, null, 2));
} else {
console.log("Query succeeded."+JSON.stringify(data, null, 2));
}
});
Thanks,
You are performing a scan, which does not have a parameter called ScanIndexForward. The Query operation does have a parameter called ScanIndexForward.
The only way you can order results in a DynamoDB query is by the sort key attribute - results are ordered by default. If you want to order by any other attribute, or order scan results, you have to do it in your client application.
I'm writing a nodejs 5.7.1 application with aws-sdk for DynamoDB.
I have a table of events that I created with the following code:
var statsTableName='bingodrive_statistics';
var eventNameColumn = 'event_name';
var eventTimeColumn = 'event_time';
var eventDataColumn = 'event_data';
var params = {
TableName: statsTableName,
KeySchema: [ // The type of of schema. Must start with a HASH type, with an optional second RANGE.
{ // Required HASH type attribute
AttributeName: eventNameColumn,
KeyType: 'HASH',
},
{ // Optional RANGE key type for HASH + RANGE tables
AttributeName: eventTimeColumn,
KeyType: 'RANGE',
}
],
AttributeDefinitions: [ // The names and types of all primary and index key attributes only
{
AttributeName: eventNameColumn,
AttributeType: 'S', // (S | N | B) for string, number, binary
},
{
AttributeName: eventTimeColumn,
AttributeType: 'N'
}
],
ProvisionedThroughput: { // required provisioned throughput for the table
ReadCapacityUnits: 1,
WriteCapacityUnits: 1,
}
};
dynamodbClient.createTable(params, callback);
as you can see, I have a Hash + Range index. the range is on event_time.
now I want to scan or query for all the items between two specific dates.
so i'm sending the following params to the query function of dynamoDb:
{
"TableName": "bingodrive_statistics",
"KeyConditionExpression": "event_time BETWEEN :from_time and :to_time",
"ExpressionAttributeValues": {
":from_time": 1457275538691,
":to_time": 1457279138691
}
and i'm getting this error:
{
"message": "Query condition missed key schema element",
"code": "ValidationException",
"time": "2016-03-06T15:46:06.862Z",
"requestId": "5a672003-850c-47c7-b9df-7cd57e7bc7fc",
"statusCode": 400,
"retryable": false,
"retryDelay": 0
}
I'm new to dynamoDb. I don't know what's the best method, Scan or Query in my case. any information regarding the issue would be greatly appreciated.
You should use query. You can't use only range key if you want to query for values between two range keys, you need to use hash key as well since range key. It's because hash key (partition key) is used to select a physical partition where the data is stored, sorted by range key (sort key). From DynamoDB developer guide:
If the table has a composite primary key (partition key and sort key), DynamoDB calculates the hash value of the partition key in the same way as described in Data Distribution: Partition Key—but it stores all of the items with the same partition key value physically close together, ordered by sort key value.
Also, you should choose partition key that distributes well your data. If evenName has small total number of values, it might not be the best option (See Guidelines For Tables]
That said, if you already have eventName as your hash key and eventTime as your range Key, you should query (sorry for pseudo code, I use DynamoDBMapper normally):
hashKey = name_of_your_event
conditions = BETWEEN
attribute_values (eventTime1, eventTime2)
You don't need additional Local Secondary Index or Global Secondary Index for that. Note that GSI let's you query for columns that are not indexed with the table hash and range key, but to query data between the timestamps, you will still need a range key or will need to do a Scan otherwise.
Use this query
function getConversationByDate(req , cb) {
var payload = req.all; //05/09/2017
var params = {
TableName: "message",
IndexName: "thread_id-timestamp-index",
KeyConditionExpression: "#mid = :mid AND #time BETWEEN :sdate AND :edate",
ExpressionAttributeNames: {
"#mid": "thread_id",
"#time": "timestamp"
},
ExpressionAttributeValues: {
":mid": payload.thread_id,
":sdate": payload.startdate,
":edate": payload.enddate
}
};
req.dynamo.query(params, function (err, data) {
cb(err, data);
});
}
In my CouchDB database, all keys have the form "A_xxxxxxxx" where xxxxxxxx is zero-padded decimal number (e.g. "A_00000001" or "A_12345678")
I want to get only the number of keys in a given key range.
For example, to get the keys from A_10000000 to A_30000000, I can query something like:
GET DATABASE/_all_docs?startkey="A_00001000"&endkey="A_30000000"&include_docs=false
But the result contains all keys, and I need to count the elements in "docs" field of the output.
Since the number of keys in my query will be huge, and all I want to know is the number of keys, not the actual list of the keys.
The range start and range end value can be vary, which is not fixed.
Is is possible to get only the number of keys of the given range, without retrieving actual key list?
Thanks,
You cannot get the number of keys in a given key range using the built-in _all_docs view. But you can get the desired result using a custom map reduce view such as this one described in the CouchDB Definitive Guide
map.js
function(doc) {
emit(doc._id, 1);
}
reduce.js
function(keys, values, rereduce) {
return sum(values)
}
You can add these views to your CouchDB database using the Futon admin utility by creating a new document with these contents:
{
"_id": "_design/test",
"views": {
"count": {
"map": "function(doc) {\n emit(doc._id, 1);\n}",
"reduce": "function(keys, values, rereduce) {\n return sum(values)\n}"
}
}
}
_design/test/count can then be queried like instead of _all_docs and will return the number of documents between the start and end keys.
When I run this query again my database without a start and end key I get this result:
{
"rows":[
{
"key": null,
"value": 185
}
]
}
Running the query again with the start and end keys populated I get this result:
{
"rows":[
{
"key": null,
"value": 11
}
]
}