I tracked down this package. Generally it's pretty nice. But it seems to lack support for Projection Expressions. What is your tool of choice for dynamodb in node/typescript?
I'm not fan of the data mappers listed here because they tend to wrap the table data, or are abandoned as projects.
If typescript is an option, we use https://github.com/shiftcode/dynamo-easy. Which also does not support Projection Expression, but the underlying params can always be accessed and manipulated, so adding some non-supported feature is easy.
import { DynamoStore } from '#shiftcoders/dynamo-easy'
const queryRequest = new DynamoStore(PersonModel)
.query()
.wherePartitionKey('2018-01')
.whereSortKey().beginsWith('a')
.limit(1)
const queryParams = queryRequest.params
queryParams.ProjectionExpression = 'projectionExpression'
// also add expression attribute names if required
queryParams.ExpressionAttributeNames = {'#someExpressionAttributeName': 'someExpressionAttributeName'}
// you can also use new DynamoDB().query(queryParams), but we just use the preconfigured wrapped client
queryRequest.dynamoDBWrapper.makeRequest('query', queryParams)
.then(r => console.log('first found item with projection expression:', r))
full disclosure: I am one of the authors of this library
We use dynogels, it is maintained until to date.
https://github.com/clarkie/dynogels
If you need a GUI to construct your query, try using the "DynamoDB Visual Query Builder" I've built: https://dynobase.dev/dynamodb-query-builder/
Related
I am learning DynamoDB and AWS serverless stack. I see that a lot of tutorials suggest using AWS.DynamoDB.DocumentClient. For example, to create an item:
const dynamodb = new AWS.DynamoDB.DocumentClient();
and then
try {
await dynamodb
.put({
TableName: process.env.DISHES_TABLE_NAME,
Item: dish,
})
.promise();
} catch (error) {
console.error(error);
throw new createError.InternalServerError(error);
}
But the doc says that put
Creates a new item, or replaces an old item with a new item by
delegating to AWS.DynamoDB.putItem()
I am confused why not use AWS.DynamoDB.putItem in the first place or when to use which one.
Thank you!
From the documentation the DocumentClient is just an abstract class to make it easier to implement.
The document client simplifies working with items in Amazon DynamoDB by abstracting away the notion of attribute values. This abstraction annotates native JavaScript types supplied as input parameters, as well as converts annotated response data to native JavaScript types.
You're free to choose whichever method you want, however by using the DocumentClient class you would be having less control over the processing or manipulation of your data.
DynamoDB does not have the option to automatically generate a unique key for you.
In examples I see people creating a uid out of a combination of fields, but is there a way to create a unique ID for data which does not have any combination of values that can act as a unique identifier? My questions is specifically aimed at lambda functions.
One option I see is to create a uuid based on the timestamp with a counter at the end, insert it (or check if it exists) and in case of duplication retry with an increment until success. But, this would mean that I could potentially run over the execution time limit of the lambda function without creating an entry.
If you are using Node.js 8.x, you can use uuid module.
var AWS = require('aws-sdk'),
uuid = require('uuid'),
documentClient = new AWS.DynamoDB.DocumentClient();
[...]
Item:{
"id":uuid.v1(),
"Name":"MyName"
},
If you are using Node.js 10.x, you can use awsRequestId without uuid module.
var AWS = require('aws-sdk'),
documentClient = new AWS.DynamoDB.DocumentClient();
[...]
Item:{
"id":context.awsRequestId,
"Name":"MyName"
},
The UUID package available on NPM does exactly that.
https://www.npmjs.com/package/uuid
You can choose between 4 different generation algorithms:
V1 Timestamp
V3 Namespace
V4 Random
V5 Namespace (again)
This will give you:
"A UUID [that] is 128 bits long, and can guarantee uniqueness across
space and time." - RFC4122
The generated UUID will look like this: 1b671a64-40d5-491e-99b0-da01ff1f3341
If it's too long, you can always encode it in Base64 to get G2caZEDVSR6ZsAAA2gH/Hw but you'll lose the ability to manipulate your data through the timing and namespace information contained in the raw UUID (which might not matter to you).
awsRequestId looks like its actually V.4 UUID (Random), code snippet below:
exports.handler = function(event, context, callback) {
console.log('remaining time =', context.getRemainingTimeInMillis());
console.log('functionName =', context.functionName);
console.log('AWSrequestID =', context.awsRequestId);
callback(null, context.functionName);
};
In case you want to generate this yourself, you can still use https://www.npmjs.com/package/uuid or Ulide (slightly better in performance) to generate different versions of UUID based on RFC-4122
For Go developers, you can use these packages from Google's UUID, Pborman, or Satori. Pborman is better in performance, check these articles and benchmarks for more details.
More Info about Universal Unique Identifier Specification could be found here.
We use idgen npm package to create id's. There are more questions on the length depending upon the count to increase or decrease the size.
https://www.npmjs.com/package/idgen
We prefer this over UUID or GUID's since those are just numbers. With DynamoDB it is all characters for guid/uuid, using idgen you can create more id's with less collisions using less number of characters. Since each character has more ranges.
Hope it helps.
EDIT1:
Note! As of idgen 1.2.0, IDs of 16+ characters will include a 7-character prefix based on the current millisecond time, to reduce likelihood of collisions.
if you using node js runtime, you can use this
const crypto = require("crypto")
const uuid = crypto.randomUUID()
or
import { randomUUID } from 'crypto'
const uuid = randomUUID()
Here is a better solution.
This logic can be build without any library used because importing a lambda function layer can get difficult sometimes. Below you can find the link for the code which will generate the unique id and save it in the SQS queue, rather than DB which will incur the cost for writing, fetching, and deleting the ids.
There is also a cloudformation template provided, which you can go and deploy in your account, and it will setup the whole application. A detailed explanation is provided in the link.
Please refer to the link below.
https://github.com/tanishk97/UniqueIdGeneration_AWS_CFT/wiki
I am building a REST API which connects to a NEO4J instance. I am using the koa-neo4j library as the basis (https://github.com/assister-ai/koa-neo4j-starter-kit). I am a beginner at all these technologies but thanks to some help from this forum I have the basic functionality working. For example the below code allows me to create a new node with the label "metric" and set the name and dateAdded propertis.
URL:
/metric?metricName=Test&dateAdded=2/21/2017
index.js
app.defineAPI({
method: 'POST',
route: '/api/v1/imm/metric',
cypherQueryFile: './src/api/v1/imm/metric/createMetric.cyp'
});
createMetric.cyp"
CREATE (n:metric {
name: $metricName,
dateAdded: $dateAdded
})
return ID(n) as id
However, I am struggling to know how I can approach more complicated examples. How can I handle situations when I don't know how many properties will be added when creating a new node beforehand or when I want to create multiple nodes in a single post statement. Ideally I would like to be able to pass something like JSON as part of the POST which would contain all of the nodes, labels and properties that I want to create. Is something like this possible? I tried using the below Cypher query and passing a JSON string in the POST body but it didn't work.
UNWIND $props AS properties
CREATE (n:metric)
SET n = properties
RETURN n
Would I be better off switching tothe Neo4j Rest API instead of the BOLT protocol and the KOA-NEO4J framework. From my research I thought it was better to use BOLT but I want to have a Rest API as the middle layer between my front and back end so I am willing to change over if this will be easier in the longer term.
Thanks for the help!
Your Cypher syntax is bad in a couple of ways.
UNWIND only accepts a collection as its argument, not a string.
SET n = properties is only legal if properties is a map, not a string.
This query should work for creating a single node (assuming that $props is a map containing all the properties you want to store with the newly created node):
CREATE (n:metric $props)
RETURN n
If you want to create multiple nodes, then this query (essentially the same as yours) should work (but only if $prop_collection is a collection of maps):
UNWIND $prop_collection AS props
CREATE (n:metric)
SET n = props
RETURN n
I too have faced difficulties when trying to pass complex types as arguments to neo4j, this has to do with type conversions between js and cypher over bolt and there is not much one could do except for filing an issue in the official neo4j JavaScript driver repo. koa-neo4j uses the official driver under the hood.
One way to go about such scenarios in koa-neo4j is using JavaScript to manipulate the arguments before sending to Cypher:
https://github.com/assister-ai/koa-neo4j#preprocess-lifecycle
Also possible to further manipulate the results of a Cypher query using postProcess lifecycle hook:
https://github.com/assister-ai/koa-neo4j#postprocess-lifecycle
I recently realized that DocumentDB supports stand alone update operations via ReplaceDocumentAsync.
I've replaced the Upsert operation below with the Replace operation.
var result = _client
.UpsertDocumentAsync(_collectionUri, docObject)
.Result;
So this is now:
var result = _client
.ReplaceDocumentAsnyc(_collectionUri, docObject)
.Result;
However, now I get the exception:
Microsoft.Azure.Documents.BadRequestException : ResourceType Document is unexpected.
ActivityId: b1b2fd71-3029-4d0d-bd5d-87d8d0a2fc95
No idea why, upsert and replace are of the same vein and the object is the same that worked for upsert, so I would expect it to work without problems.
All help appreciated.
Thanks
Update: Have tried to implement this using the SelfLink approach, and it works for Replace, but selflink does not work with Upsert. The behavior is quite confusing. I don't like that I have to build a self link in code using string concatenation.
I'm afraid that building the selflink with string concatenation is your only option here because ReplaceDocument(...) requires a link to the document. You show a link to the collection in your example. It won't suck the id out and find the document as you might wish.
The NPM module, documentdb-utils, has library functions for building these links but it's just using string concatenation. I have seen an equivalent library for .NET but I can't remember where. Maybe it was in an Azure example or even in the SDK now.
You can build a document link for a replace using the UriFactory helper class:
var result = _client
.ReplaceDocumentAsync(UriFactory.CreateDocumentUri(databaseId, collectionId, docObject.Id), docObject)
.Result;
Unfortunately it's not very intuitive, as Larry has already pointed out, but a replace expects a document to already be there, while an upsert is what it says on the tin. Two different use-cases, I would say.
In order to update a document, you need to provide the Collection Uri. If you provide the Document Uri it returns the following:
ResourceType Document is unexpected.
Maybe the _collectionUri is a Document Uri, the assignment should look like this:
_collectionUri = UriFactory.CreateDocumentCollectionUri(DatabaseName, CollectionName);
Maybe the question does not apply to dynamoDB due to it not being Relational Db.
However, I'm looking for a good object mapper which can be used in nodejs and aws sdk to map existing model classes to dynamoDB tables. Does anyone have experience with this issue/question, or have you used such a module/library?
If you are looking for schema:
https://github.com/clarkie/dynogels (well supported forked from vogels which has been abandoned)
https://github.com/automategreen/dynamoose (inspired by Mongoose)
If you are looking for something to throw javascript objects (even circular graphs) to:
https://github.com/aaaristo/dyngodb (alpha)
https://github.com/aaaristo/angular-gson-express-dyngodb
dyngodb has experimental support for full-text search, and transactions too.
Both are based on aws-sdk.
Also worth considering is simple marshallers, which just translate between the dynamoDB format and regular js objects or JSON.
DynamoDb-Data-Types
https://github.com/kayomarz/dynamodb-data-types
https://www.npmjs.com/package/dynamodb-data-types
"This utility helps represent AWS DynamoDb data types. It maps (marshalls) JavaScript data into the format required by DynamoDb."
dynamoDb-marshaler
https://github.com/CascadeEnergy/dynamoDb-marshaler
https://www.npmjs.com/package/dynamodb-marshaler
"Translates sane javascript objects (and JSON) into DynamoDb format and vice versa." [does not support B type.]
Update 2016-06:
Just discovered that the AWS SDK now does this for you. Their documentation is only partially converted so I guess this is a recent addition. Read about it here.
But these marshallers are still useful because there are circumstances where you can't use the new document client, eg. when processing a dynamoDB stream.
You could also try: https://dynamoosejs.com/. It is inspired by mongoose again.
If you are using Typescript, dynamo-easy might be a good option. Just add some decorators to your model and start using it.
import { Model, PartitionKey, DynamoStore } from '#shiftcoders/dynamo-easy'
#Model()
export class Person {
#PartitionKey()
id: string
name: string
yearOfBirth: number
}
const personStore = new DynamoStore(Person)
personStore
.scan()
.whereAttribute('yearOfBirth').equals(1958)
.exec()
.then(res => console.log('ALL items with yearOfBirth == 1958', res))
It uses the AWS DynamoDB sdk but takes care of the mapping between JS and DynamoDB types and provides a simple to use fluent API.
full disclosure: I am one of the authors of this library
After looking over all the posts I landed on https://github.com/awspilot/dynamodb-oop
It doesn't hide the API but instead just wraps it in a nice, fluent way with promises even and you inject your version of the aws-sdk. It's similar to dynamodb-data-types but also wraps the methods too (not just the data types).
Extra bonus, the same author has https://github.com/awspilot/dynamodb-sql Didn't use the sql wrapper but I can see how some people may prefer that.
Dynamoose is obviously inspired by mongoose and is a good choice if you have a well-defined schema and/or want to be abstracted away from the DynamoDB details.
Have you seen dynasaur? It seems to be the type of thing you're looking for, but I have not used it myself. There's also dynamodb-data-types which is not an ORM, but makes it easy to convert to/from standard JavaScript objects.