** Edited **
I am trying to insert data to Azure cosmos DB if its not exist but can't find a way to do it in one transaction.
the current way I'm doing it is by first getting it from cosmos and if it's throwing Not found exception then insert, otherwise update it.
the issue with that is that between the get and insert someone can push an object so this approach is unsafe.
code snippet:
static async fetchOrgConfigurationTableByQuery(orgId: string): Promise<OrgConfigurationTable> {
const { container } = await getContainer(Constants.CosmosRegional, Constants.OrgConfigurationTableName);
const orgConfigurations = await container.items.query(`SELECT * from c WHERE c.OrgId = "${orgId}"`).fetchAll();
const orgConfiguration = orgConfigurations.resources.find(() => true) as OrgConfigurationTable;
if (!orgConfiguration) {
return;
}
const orgConfigurationWithDefaults = new OrgConfigurationTable();
for (const [key] of Object.entries(orgConfiguration)) {
orgConfigurationWithDefaults[key] = orgConfiguration[key];
}
return orgConfigurationWithDefaults;
}
static async upsertOrgConfiguration(
orgConfiguration: OrgConfigurationTable,
failOnEtagMismatch = false
): Promise<OrgConfigurationTable> {
const requestOptions = getRequestOptions(orgConfiguration, failOnEtagMismatch);
const { container } = await getContainer(Constants.CosmosRegional, Constants.OrgConfigurationTableName);
const returnValue = await container.items.upsert<OrgConfigurationTable>(orgConfiguration, requestOptions);
return returnValue?.resource;
}
what I have:
let orgConfig: Cosmos.OrgConfigurationTable = null;
try {
orgConfig = await fetchOrgConfigurationTableByQuery(orgId);
} catch (error) {
if (!error instanceof Cosmos.OrgNotFoundException) {
log(error)
}
}
orgConfig = {
...orgConfig,
attribute: value,
...(!orgConfig && { Key: { Value: orgId } }),
};
await upsertOrgConfiguration(orgConfig)
what I want - to dont get before upsert:
let orgConfig = {
attribute: value,
Key: { Value: orgId },
};
await upsertOrgConfiguration(orgConfig) // if exist update, else insert - in single transuction.
or some way to do somthing like this:
transaction {
let orgConfig: Cosmos.OrgConfigurationTable = null;
try {
orgConfig = await fetchOrgConfigurationTableByQuery(orgId);
} catch (error) {
if (!error instanceof Cosmos.OrgNotFoundException) {
log(error)
}
}
await upsertOrgConfiguration(orgConfig)
}
commit
any ideas?
Related
So my issue is simple yet challenging. I am trying to transfer data from one ledger to another. Now for that, I am reading a whole table of data then creating an object for every document than inserting these documents one by one into the other ledger table
Ledger#1 Table1 -> get all data -> convert all data to array of objects -> transfer to Ledger#2 Table1 one by one
The problem is that I cannot create the object from the document. I do this manually by using prototype functions and reading field type and then creating a thing that is messy and causes some data to become null. So I was wondering if there was a better way that is less prone to errors.
I asked a question of migrating ledger but had no luck in getting any response. Please help me in this.
Following is my code. Please copy and paste it inside an IDE so you can better understand
const getValueOfField = (field) => {
const name = field.getType().name;
switch (name) {
case "string":
return field.stringValue();
case "int":
return field.numberValue();
case "null":
return null;
default:
return null;
}
};
const enterDataInNewLedger = async (tableData, tableName) => {
const awsProductionDriver = awsProduction();
console.log(`Starting to insert data inside table ${tableName}`);
try {
for (const data of tableData) {
await awsProductionDriver.executeLambda(async (txn) => {
await txn.execute(`INSERT INTO ${tableName} ?`, data);
});
}
console.log(`Done inserting data inside ${tableName}`);
return { success: true };
} catch (err) {
console.log(err.message);
return { success: false, message: err.message };
}
};
const dataTransferOfTable = async (table) => {
const prodDriver = awsProd();
try {
const allTableData = await prodDriver.executeLambda(async (txn) => {
const result = await txn.execute(`SELECT * FROM ${table.name}`);
const resultList = result.getResultList();
let completeResults = [];
for (const doc of resultList) {
let newDoc = {};
const fields = doc.fields();
for (const field of fields) {
newDoc[field[0]] = getValueOfField(field[1]);
}
completeResults.push(newDoc);
}
return completeResults;
});
const response = await enterDataInNewLedger(allTableData, table.name);
checkForErrors(response);
return { success: true };
} catch (err) {
console.log(err.message);
return { success: false, message: err.message };
}
};
const startDataTransferFromOneLedgerToAnother = async () => {
try {
for (let table of tableName) {
const response = await dataTransferOfTable(table);
checkForErrors(response);
}
} catch (err) {
console.log(err.message);
}
};
startDataTransferFromOneLedgerToAnother();
So apparently I could've done this easily. I was just checking it and figured out the solution.
I can insert the whole fetched document and it will be same so my converted code is as follows
const { awsMainFunction: awsProd } = require("./awsProdConfig");
const { awsMainFunction: awsProduction } = require("./awsProductionConfig");
const { tableNamesAndIndeces: tableName, checkForErrors } = require("./utils");
const enterDataInNewLedger = async (tableData, tableName) => {
const awsProductionDriver = awsProduction();
console.log(`Starting to insert data inside table ${tableName}`);
try {
for (const data of tableData) {
await awsProductionDriver.executeLambda(async (txn) => {
await txn.execute(`INSERT INTO ${tableName} ?`, data);
});
}
console.log(`Done inserting data inside ${tableName}`);
return { success: true };
} catch (err) {
console.log(err.message);
return { success: false, message: err.message };
}
};
const dataTransferOfTable = async (table) => {
const prodDriver = awsProd();
try {
const allTableData = await prodDriver.executeLambda(async (txn) => {
const result = await txn.execute(`SELECT * FROM ${table.name}`);
return result.getResultList();
});
const response = await enterDataInNewLedger(allTableData, table.name);
checkForErrors(response);
return { success: true };
} catch (err) {
console.log(err.message);
return { success: false, message: err.message };
}
};
const startDataTransferFromOneLedgerToAnother = async () => {
try {
for (let table of tableName) {
const response = await dataTransferOfTable(table);
checkForErrors(response);
}
} catch (err) {
console.log(err.message);
}
};
startDataTransferFromOneLedgerToAnother();
I need help to get a simple query for MongoDB (SQL equivalent: "select speed from values"). But I can't find a solution for that.
Node.js backend for Vue.js
// a Part of Init-function:
await collection.replaceOne({}, {
speed: {
value: 65,
type: 2,
},
eightSpeed: {
value: 0.0043,
type: 2,
},
oSpeed: {
value: 0.31,
type: 2,
},
minusSpeed: {
value: 2.42,
type: 2,
}
}
//...
Now I need the query like this: http://192.168.220.220:3000/settings/getValue/speed to get an object speed.
const express = require("express");
const mongodb = require("mongodb");
const SettingsRouter = new express.Router();
SettingsRouter.get("/getValue/:value", async function (req, res) {
try {
const val = req.params.value; // req.body.text;
const select = {};
select[`${val}.value`] = 1;
console.log("settings.js:", "/getValue/:name", select);
const collection = await loadMongoCollection();
const value = await collection.findOne({},select)//.toArray();
res.status(201).send(value);
} catch (error) {
res.status(500).send("Failed to connect Database");
console.error("settings.js:", "/getValue/:name:", error);
}
});
async function loadMongoCollection() {
const dbUri = process.env.MONGODB_CONNSTRING || config.dbUri;
const MongoDB = process.env.MONGODB_DB || config.db;
try {
const client = await mongodb.MongoClient.connect(dbUri, {
useNewUrlParser: true,
});
const conn = client.db(MongoDB).collection("values");
return conn;
} catch (error) {
console.error("settings.js:", "loadMongoCollection:", error);
}
}
When I try it, I get all (not only the Speed Object) what I expected, or nothing.
What I do wrong?
EDIT 2022.01.06:
I try it to change my database:
data = {
speed: {
value: 65,
type: 2,
},//...
}
//...
await collection.replaceOne({}, {values:data}, { upsert: true });
And then the query:
const select = {[`values.${val}.value`]:"1"};
const where = {}; //{"values":val};
const value = await collection.find(where,select,).toArray();
But it will not work in rest... is there an issue in mongo package?
When I do it in https://cloud.mongodb.com - it works:
But my request returns all values...
Log shows:"'settings.js:', '/getValue/:name', { 'values.speed.value': '1' }"
In the screenshot you show the projection is values.speed.value, but in your endpoint you have:
const select = {};
select[`${val}.value`] = 1;
Which would evaluate to speed.value. To mimic what you have on the MongoDB console this should be:
const select = {};
select[`values.${val}.value`] = 1;
So, I changed my Frondend (vue) to spred all Values. I maked some Helper Functions:
CheckForChangedObj
check two object for changes
updateObj
copy all data from object into another
export function CheckForChangedObj(targetObject, obj) {
if (targetObject === typeof undefined || targetObject == {}) {
return true
}
if (JSON.stringify(targetObject) !== JSON.stringify(obj)) {
return true
}
return false
}
export function updateObj(targetObject, obj) {
let keys = Object.keys(obj)
for (let k in keys) {
try {
let key = keys[k]
if (!targetObject.hasOwnProperty(key)) {
//define target, if not exist
targetObject[key] = {}
}
//Workaround for References
if (obj[key].hasOwnProperty("__v_isRef")) {
if (targetObject[key].hasOwnProperty("__v_isRef")) {
targetObject[key].value = obj[key].value
} else {
targetObject[key] = obj[key].value
}
} else {
//Source i a Norm Value
//check for an Object inside, then go in...
if ("object" === typeof obj[key] && !Array.isArray(obj[key])) {
updateObj(targetObject[key], obj[key]) // recurse
} else {
//not else-if!!! __v_isRef: true
if (targetObject[key].hasOwnProperty("__v_isRef")) {
targetObject[key].value = obj[key]
} else {
targetObject[key] = obj[key]
}
}
}
} catch (error) {
console.log(
"key:",
keys[k],
"Error:",
error
)
}
}
}
Then spreading...
import {updateObj,CheckForChangedObj } from "../tools/tools"
beforeMount() {
this.getAllValuesAPI()
setInterval(() => {
this.checkForChanges()
// ml("TABLE", this.values)
}, 10000)
},
setup() {
let prevValues = {}
let values = {
speed: {
id:1,
type: SettingType.Slider,
value: ref(0)
}
//...
}
function checkForChanges() {
//intervaled function to get changes from API
let changed = CheckForChangedObj(this.values, this.prevValues)
if (changed) {
updateObj(this.prevValues, this.values)
setValuesToAPI()
}
else{
getAllValuesFromAPI()
}
}
async function getAllValuesFromAPI() {
//get ALL Settings-Data from API, and put them in values
const res = await axios.get(`settings/getAll`)
return updateObj(this.values, res.data[0].values) u
}
}
When you have a better solution, please help...
I have a function that will take an array of jobs as a parameter in it. This function will check the existence of each job in the database through its id.
If a job is not to present in the database, that particular job needs to be pushed into an array called latestJobs. I'm calling this function in my main.js file. But the code breaks and stops.
Below is my main.js code:
module.exports.app = async () => {
try {
...
const jobs = await getJobsForCountries(body);
const latestJobs = await filterPreDraftedJobs(jobs);
console.log('latestJobs', latestJobs);
} catch (e) {
console.error('Error:- ', e); // Comes to here
}
};
My checker function looks like:
module.exports = async (jobs) => {
let latestJobs = [];
for (const job of jobs) {
const params = {
TableName: process.env.DYNAMODB_TABLE,
Key: {
id: job.Id
}
};
await dynamoDb.get(params, (err, data) => {
if (err) {
latestJobs.push(job);
console.log('Job not found in DB');
}
}).promise();
}
return latestJobs;
};
How can I fix this issue? I want the latestJobs which will not present in the database. Is there a function for dynamodb which can do this for me?
You are mixing callback, promise and await style. I would do it like this
module.exports = async (jobs) => {
let latestJobs = [];
for (const job of jobs) {
const params = {
TableName: process.env.DYNAMODB_TABLE,
Key: {
id: job.Id
}
};
try {
const result = await dynamoDb.get(params).promise();
if (result) {
return;
}
} catch(err) {
latestJobs.push(job);
}
}
return latestJobs;
};
Also, make sure that table is created and the region and name you are passing is correct.
I am not much familiar with dynamoDB but looking at the above conversation code must be something like this. I have tried to improve performance and making sure the code is modular and readable.
async function addUpdateJobs(jobs)
{
let paramsArray = [];
for (const job of jobs)
{
const jobParams = {
params:{
TableName: process.env.DYNAMODB_TABLE,
Key: {
id: job.Id
}
},
job:job
};
paramsArray.push(jobParams );
}
return await this.getJobs(paramsArray);
}
function getJobs(paramsArray)
{
let latestJobs = [];
paramsArray.each(async (jobParam)=>
{
try
{
const result = await dynamoDb.get(jobParam.params).promise();
if (result)
{
return;
}
} catch (err)
{
latestJobs.push(jobParam.job);
}
});
return latestJobs;
}
PS: Also I was gonig through error handling in amazondynamodb.
I have two classes as below
class AClient
class AClient {
constructor(AHeaderObj) {
logger.info("inside tenant client constructor");
this.tenant_uri = tenantHeaderObj.tenant_uri || config.TENANT_URI;
this.refresh_token = tenantHeaderObj.refresh_token;
this.tenant_header = tenantHeaderObj.tenant_header;
}
Class BClient
class BClient {
constructor(b_client) {
logger.info("Inside mongoClient/constructor ");
this.tenant_client = tenant_client;
}
async find(db_name, collection_name, selectorQuery) {
try {
logger.info("Inside mongoClient find records");
let dbName = await getDB(db_name, this.tenant_client);
let db = await getConnection(dbName, this.tenant_client);
logger.info("selectorQuery"+JSON.stringify(selectorQuery));
let doc = db.collection(collection_name).find(selectorQuery).toArray();
logger.info("****************doc**********"+JSON.stringify(doc));
return await promiseWrapper(null, doc);
} catch (err) {
logger.info("Caught error in mongoClient find records ", err);
return await promiseWrapper(err, null);
}
}
async findSome(db_name, collection_name, query, projection, sort, limit){
logger.info("Inside mongoClient findSome records");
try{
let dbName = await gettDB(db_name, this.tenant_client);
let db = await getConnection(dbName, this.tenant_client);
var selector = [{ $match : query},
{ $project : projection},
{ $sort : sort}];
if(limit > 0){
selector.push({ $limit : limit})
}
let docs = db.collection(collection_name).aggregate(selector).toArray();
return await promiseWrapper(null, docs);
} catch (err) {
logger.info("Caught error in mongoClient findSome records ", err);
return await promiseWrapper(err, null);
}
}
Now I am using it in below 2 ways . One way it works another it is not.
Case1: When it is not working
var searchAll = function (type, text, sortBy, sort, Code, AHeaderObj) {
logger.info("Inside searchAll function.");
var selector = mongoQuery.searchAll(type, text, sortBy, sort, Code);
var AObj = new AClient(AHeaderObj);
logger.info("123");
var mongoObj = new BClient(AObj);
logger.info("1235");
var findSome = callbackify(mongoObj.findSome)
logger.info("12356");
return new Promise((resolve1, reject1) => {
logger.info("123567");
findSome(dBName, collectionName, selector.query, selector.projection, selector.sort, 999999999, function (err1, res1) {
if (err1) {
logger.error("Failed to find");
reject1(err1);
} else {
logger.info("Successfully found");
resolve1(res1)
}
});
})
}
Case2: Where it is working as charm.
var getId = function (id, headerObj, callback) {
logger.info("inside getAccountById() START:");
let selector = mongoQuery.GetAccByIdQueryWithIdModelVersion("Id", id, “version", "v2.1")
let index = 0;
var AObj = new AClient(AHeaderObj);
logger.info("123");
var mongoObj = new BClient(AObj);
logger.info("1235");
mongoObj.find(dBName, collectionName, selector).then((result) => {
logger.info("123568");
if (result && result.length > 0) {
logger.info("a");
logger.info("Found an with id: " + id);
if (result[index].credentials.length > 0) {
logger.info("getById() END:"+JSON.stringify(result));
callback(null, result);
}else {
logger.info("b");
let error = {
"message": "Unable to fetch the with id: " + id
};
logger.warn(error.message);
logger.info("getById() END:");
callback(error, null);
}
}).catch((err) => {
logger.info("c");
logger.info("error" + JSON.stringify(err));
callback(err, null);
});
}
Can someone help what am I doing wrong in Case1 . I observed that ion Case 1 object creation is not awaited and db function is been called where I get below error.
Caught error in mongoClient findSome records TypeError: Cannot read property 'tenant_client' of undefined
Instead of trying to await on class initialization, make the constructor empty and provide and async init method which does the same thing as what the constructor would do.
Make sure that other methods check if the object has been instantiated via the init method and throw an error if not :)
I am getting an error that "The provided key element does not match the schema". uuid is my primary partition key. I also have a primary sort key for version. I figured I can use batchWrite (docs) to delete all items with same uuid.
My ES6 code is as follows:
delete(uuid) {
const promise = new Promise();
const params = {
RequestItems: {
[this.TABLE]: [
{
DeleteRequest: {
Key: { uuid: uuid }
}
}
]
}
};
// this._client references the DocumentClient
this._client.batchWrite(params, function(err, data) {
if (err) {
// this gets hit with error
console.log(err);
return promise.reject(err);
}
console.log(result);
return promise.resolve(result);
});
return promise;
}
Not sure why it is erroring on the key that is the primary. I have seen posts about needing other indexes for times when I am searching by something that isn't a key. But I don't believe that's the case here.
Here is the batch write delete request sample. This code has been tested and working fine. If you change this code for your requirement, it should work.
Table Definition:-
Bag - Table Name
bag - Hash Key
No partition key in 'Bag' table
Batch Write Code:-
var AWS = require("aws-sdk");
AWS.config.update({
region : "us-west-2",
endpoint : "http://localhost:8000"
});
var documentclient = new AWS.DynamoDB.DocumentClient();
var itemsArray = [];
var item1 = {
DeleteRequest : {
Key : {
'bag' : 'b1'
}
}
};
itemsArray.push(item1);
var item2 = {
DeleteRequest : {
Key : {
'bag' : 'b2'
}
}
};
itemsArray.push(item2);
var params = {
RequestItems : {
'Bag' : itemsArray
}
};
documentclient.batchWrite(params, function(err, data) {
if (err) {
console.log('Batch delete unsuccessful ...');
console.log(err, err.stack); // an error occurred
} else {
console.log('Batch delete successful ...');
console.log(data); // successful response
}
});
Output:-
Batch delete successful ...
{ UnprocessedItems: {} }
This is doable with Node lambda, but there are a few things you need to consider to address concurrency while processing large databases:
Handle paging while querying all of the matching elements from a secondary index
Split into chunks of 25 requests as per BatchWrite/Delete requirements https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html
Above 40,000 matches you might need a 1 second delay between cycles https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html
Here a snipped that I wrote:
const AWS = require("aws-sdk");
const dynamodb = new AWS.DynamoDB.DocumentClient();
const log = console.log;
exports.handler = async (event) => {
log(event);
let TableName = event.tableName;
let params = {
let TableName,
FilterExpression: "userId = :uid",
ExpressionAttributeValues: {
":uid": event.userId,
},
};
let getItems = async (lastKey, items) => {
if (lastKey) params.ExclusiveStartKey = lastKey;
let resp = await dynamodb.scan(params).promise();
let items = resp.Items.length
? items.concat(resp.Items.map((x) => x.id))
: items;
if (resp.LastEvaluatedKey)
return await getItems(resp.LastEvaluatedKey, items);
else return items;
};
let ids = await getItems(null, []);
let idGroups = [];
for (let i = 0; i < ids.length; i += 25) {
idGroups.push(ids.slice(i, i + 25));
}
for (const gs of idGroups) {
let delReqs = [];
for (let id of gs) {
delReqs.push({ DeleteRequest: { Key: { id } } });
}
let RequestItems = {};
RequestItems[TableName] = delReqs;
let d = await dynamodb
.batchWrite({ RequestItems })
.promise().catch((e) => log(e));
}
log(ids.length + " items processed");
return {};
};
Not sure why nobody provided a proper answer.
Here's a lambda I did in nodeJS. It will perform a full scan on the table, then batch delete every 25 items per request.
Remember to change TABLE_NAME.
const AWS = require('aws-sdk');
const docClient = new AWS.DynamoDB.DocumentClient({ apiVersion: '2012-08-10' });
//const { TABLE_NAME } = process.env;
TABLE_NAME = "CHANGE ME PLEASE"
exports.handler = async (event) => {
let params = {
TableName: TABLE_NAME,
};
let items = [];
let data = await docClient.scan(params).promise();
items = [...items, ...data.Items];
while (typeof data.LastEvaluatedKey != 'undefined') {
params.ExclusiveStartKey = data.LastEvaluatedKey;
data = await docClient.scan(params).promise();
items = [...items, ...data.Items];
}
let leftItems = items.length;
let group = [];
let groupNumber = 0;
console.log('Total items to be deleted', leftItems);
for (const i of items) {
const deleteReq = {
DeleteRequest: {
Key: {
id: i.id,
},
},
};
group.push(deleteReq);
leftItems--;
if (group.length === 25 || leftItems < 1) {
groupNumber++;
console.log(`Batch ${groupNumber} to be deleted.`);
const params = {
RequestItems: {
[TABLE_NAME]: group,
},
};
await docClient.batchWrite(params).promise();
console.log(
`Batch ${groupNumber} processed. Left items: ${leftItems}`
);
// reset
group = [];
}
}
const response = {
statusCode: 200,
// Uncomment below to enable CORS requests
// headers: {
// "Access-Control-Allow-Origin": "*"
// },
body: JSON.stringify('Hello from Lambda!'),
};
return response;
};
Be aware, that you need to follow instructions:
src: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html
DeleteRequest - Perform a DeleteItem operation on the specified item. The item to be deleted is identified by a Key subelement:
Key - A map of primary key attribute values that uniquely identify the item. Each entry in this map consists of an attribute name and an attribute value. For each primary key, you must provide all of the key attributes. For example, with a simple primary key, you only need to provide a value for the partition key. For a composite primary key, you must provide values for both the partition key and the sort key.
For batch delete, we can use batchWrite with DeleteRequest. Here is an example, in this, we are providing tableName whose data is to be deleted, the payload is an array of ids which we need to remove.
In single request 25 items can be deleted.
const AWS = require('aws-sdk');
const dynamodb= new AWS.DynamoDB.DocumentClient({ apiVersion: '2012-08-10' });
const tableName = "PlayerData";
const payload = [{id=101}, {id=105}, {id=106}];
const deleteBatchData = async (tableName, payload, dynamodb) => {
try {
await dynamodb.batchWrite({
RequestItems: {
[tableName]: payload.map(item => {
return {
DeleteRequest: {
Key: {
id: item.id
}
}
};
})
}
}).
promise().
then((response) => {
return response;
})
.catch((err) => {
console.log("err ::", JSON.stringify(err))
});
} catch (err) {
console.log('Error in deleteBatchData ', err);
}
}
Why not use PartiQL. This approach is much more readable. (This too has a limit of 25 items per request just like BatchWriteITems)
// Import required AWS SDK clients and commands for Node.js.
import { BatchExecuteStatementCommand } from "#aws-sdk/client-dynamodb";
import { ddbDocClient } from "../libs/ddbDocClient.js";
const tableName = process.argv[2];
const movieYear1 = process.argv[3];
const movieTitle1 = process.argv[4];
const movieYear2 = process.argv[5];
const movieTitle2 = process.argv[6];
export const run = async (
tableName,
movieYear1,
movieTitle1,
movieYear2,
movieTitle2
) => {
try {
const params = {
Statements: [
{
Statement: "DELETE FROM " + tableName + " where year=? and title=?",
Parameters: [{ N: movieYear1 }, { S: movieTitle1 }],
},
{
Statement: "DELETE FROM " + tableName + " where year=? and title=?",
Parameters: [{ N: movieYear2 }, { S: movieTitle2 }],
},
],
};
const data = await ddbDocClient.send(
new BatchExecuteStatementCommand(params)
);
console.log("Success. Items deleted.", data);
return "Run successfully"; // For unit tests.
} catch (err) {
console.error(err);
}
};
run(tableName, movieYear1, movieTitle1, movieYear2, movieTitle2);