Azure cosmos db continuation token not working with WHERE - azure

I have a collection of lubes. A lube can either be available or out-of-stock. I would like to query max of 5 lubes at a time.
When I'm not using a filter, the continuation token is returned and works fine.
// <Get all lubes / first page (max: 5) >
async getAll(token?: string, limit = 5) {
if (!token) this.store.lube.removeAll();
const {
resources: items,
continuationToken,
hasMoreResults,
} = await this.dbContainer.items
.readAll<ILube>({
maxItemCount: limit,
continuationToken: token,
// partitionKey: "HFO",
})
.fetchNext();
this.store.lube.load(items);
return { hasMoreResults, continuationToken };
}
However, when I want to filter by available lubes, or out-of-stock, using WHERE, the continuation token is returned, however, when I retrieved the data, it doesn't work - no error.
// <Get by availability status>
async getByAvailability(availability: boolean, token?: string, limit = 5) {
if (!token) this.store.lube.removeAll();
this.store.lube.removeAll();
const querySpec = {
query: "SELECT * FROM c WHERE c.isAvailable = #availability",
parameters: [
{
name: "#availability",
value: availability,
},
],
};
const {
resources: items,
continuationToken,
hasMoreResults,
} = await this.dbContainer.items
.query<ILube>(querySpec, { maxItemCount: limit })
.fetchNext();
this.store.lube.load(items);
return { hasMoreResults, continuationToken };
}
Bottom line, how can I use continuation token with where?

There's a bug in the code. Continuation token can be used with WHERE.
Mistake 1: I didn't include the continuationToken in the query of getByAvailability
Mistake 2: I removed the items from the store even when the token!=null.
this.store.lube.removeAll();
// <Get by availability status>
async getByAvailability(availability: boolean, token?: string, limit = 5) {
if (!token) this.store.lube.removeAll();
// **Mistake 2**
// this.store.lube.removeAll(); // This line is not needed.
const querySpec = {
query: "SELECT * FROM c WHERE c.isAvailable = #availability",
parameters: [
{
name: "#availability",
value: availability,
},
],
};
**Mistake 1: Query didn't have continuationToken in it, only had the one being returned.**
const {
resources: items,
continuationToken,
hasMoreResults,
} = await this.dbContainer.items
.query<ILube>(querySpec, { maxItemCount: limit, continuationToken: token })
.fetchNext();
this.store.lube.load(items);
return { hasMoreResults, continuationToken };
}

Related

NodeJS insert voucher code to first person who calls API

I don't know if this has a solution already but I can't find it or I don't know what to search.
I have a rest api which returns a list of products and I want to add a voucher code to the response of the first person who calls the api. I'm using redis to cache the information of the user who received the code, that expires within 15 mins.
async function addVoucherCode(response, userId) {
try {
const key = "KEY_VOUCHER_CODE";
let cachedData = await redis.get(key);
if (cachedData) {
if (cachedData.userId === userId) response.voucherCode = cachedData.voucherCode;
return;
}
const voucherCode = await createVoucherCode(userId); //call to create voucher code and save to db
if (!voucherCode) return;
await redis.setEx(key, 15 * 60, {userId, voucherCode});
response.voucherCode = cachedData.voucherCode;
} catch (err) {
console.error("[Error] addVoucherCode: ", err);
}
}
I created a function that mimics a simultaneous request, and when I checked the response, all them have a voucher code, not just the first.
async function getProducts(url, params) {
try {
const customers = [
{ id: 1, token: "Bearer eyJhbGciOi....1" },
{ id: 2, token: "Bearer eyJhbGciOi....2"},
{ id: 3, token: "Bearer eyJhbGciOi....3"},
{ id: 4, token: "Bearer eyJhbGciOi....4"}
];
const data = await Promise.all(customers.map( async customer => {
return await fetch(url + "?" + params.toString(), {
headers: {
Authorization: customer.token
},
}).then(res => res.json());
}));
data.forEach((item, indx) => {
if(item.voucherCode) {
const id = customers[indx].id;
console.log(`Customer ${id} has a voucher!!!!!!!!!!!!!`)
}
})
} catch (err) {
console.error("[Error] getProducts: ", err);
}
}
Result
Customer 1 has a voucher!!!!!!!!!!!!!
Customer 2 has a voucher!!!!!!!!!!!!!
Customer 3 has a voucher!!!!!!!!!!!!!
Customer 4 has a voucher!!!!!!!!!!!!!
I tried adding a 200ms delay inside addVoucherCode but same result. Thanks in advance for the help.
You are calling addVoucherCode in a sync loop, so it'll run 4 times in parallel (and the 4 GET commands will be issued at the same time, it'll reply with null to all of them, and all of them will call createVoucherCode).
There are 2 things you can do to fix it:
Cache the promise of createVoucherCode:
const createVoucherCodePromises = new Map();
function createVoucherCode(userId) {
if (!createVoucherCodePromises.has(userId)) {
createVoucherCodePromises.set(
userId,
_createVoucherCode(userId)
.finally(() => createVoucherCodePromises.delete(userId))
);
}
return createVoucherCodePromises.get(userId);
}
async function _createVoucherCode(userId) {
// ...
}
NOTE: this will not solve the problem if you have multiple node processes running at the same time.
Use SET with NX (won't override existing values) and GET (return existing/old value)
> SET key voucher1 NX GET
OK
> SET key voucher2 NX GET # will return the existing value without overriding it
"voucher1"
> GET key
"voucher1"

NodeJs parallel request in a loop too slow

so I have this feature that I am working on that takes an object (Over 10k items) picks an item from this object, sends it to an api, which process it then give a response, then proceeds to the next item.
Currently using the async library, the mapLimit method and it works as aspected.
But my problem is that it’s takes too long to loop through the entire dataset cause of the length.
This feature is suppose to be a continual process, once the entire object is iterated, wait for few seconds then do the same thing again.
I tried forking a child_process for this, broke the objects into chunks and created a process for each chunks till it was completed. This worked well as intended but the memory consumption was massive other processes on the server failed as a result of lack of available memory, even after exiting the process after it was completed.
Please how do I achieve this at a faster rate?
I use this to get the list of wallets.
getListofWallet = async () => {
try {
const USDT = await usdt.query(sql`
SELECT * FROM USDT ORDER BY id DESC;
`);
let counter = 0;
let completion = 0;
async.mapLimit(USDT, 6, async (user) => {
let userDetail = {
email: user.email,
id: user.user_id,
address: user.address
}
try {
await this.getWalletTransaction(userDetail);
completion++;
} catch (TronGridException) {
completion++;
console.log(":: A TronGrid Exception Occured");
console.log(TronGridException);
}
if (USDT.length == completion || USDT.length == (completion-5)) {
setTimeout(() => {
this.getListofWallet();
}, 60000);
console.log('~~~~~~~ Finished Wallet Batch ~~~~~~~~~');
}
}
);
} catch (error) {
console.log(error);
console.log('~~~~~~~Restarting TronWallet File after Crash ~~~~~~~~~');
this.getListofWallet();
}
}
Then I use this to process to data sent and perform the neccessary action.
getWalletTransaction = async (walletDetail) => {
const config = {
headers: {
'TRON-PRO-API-KEY': process.env.api_key,
'Content-Type': 'application/json'
}
};
const getTransactionFromAddress = await axios.get(`https://api.trongrid.io/v1/accounts/${walletDetail.address}/transactions/trc20`, config);
const response = getTransactionFromAddress.data;
const currentTimeInMillisecond = 1642668127000; //1632409548000
response.data.forEach(async (value) => {
if (value.block_timestamp >= currentTimeInMillisecond && value.token_info.address == "TR7NHqjeKQxGTCi8q8ZY4pL8otSzgjLj6t") {
let doesHashExist = await transactionCollection.query(sql`SELECT * FROM transaction_collection WHERE txhash=${value.transaction_id};`);
if (doesHashExist.length == 0) {
if (walletDetail.address == value.to) {
const checkExistence = await CryptoTransactions2.query(sql`
SELECT * FROM CryptoTransaction2 WHERE txHash=${value.transaction_id};
`);
if (checkExistence.length == 0) {
const xCollection = {
collection: "CryptoTransaction2",
queryObject: {
currency: "USDT",
sender: ObjectID("60358d21ec2b4b33e2fcd62e"),
receiver: ObjectID(walletDetail.id),
amount: parseFloat(tronWeb.fromSun(value.value)),
txHash: value.transaction_id,
description: "New wallet Deposit " + "60358d21ec2b4b33e2fcd62e" + " into " + value.to,
category: "deposit",
createdAt: new Date(),
updatedAt: new Date(),
},
};
await new MongoDbService().createTransaction(xCollection);
//create record inside cryptotransactions db.
await CryptoTransactions2.query(sql`INSERT INTO CryptoTransaction2 (txHash) VALUES (${value.transaction_id})`)
});
}

AWS Sdk response not showing in Lambda Function

I am working on lambda function and creating a method for AWS-SDK historical metric report using node, js. The method is running successful but in response showing nothing. Have a look at the response.
Here is my code
function getKeyByValue(object, value) {
return Object.keys(object).find(key =>
object[key] === value);
}
exports.handler = async (event) => {
const AWS = require('aws-sdk');
var connect = new AWS.Connect({ apiVersion: '2017-08-08' });
let queueARN = event.queueARN || null;
const connectInstanceId = process.env.instanceID;
let flag =0, nextToken = null;
let queueARNsObject = {}, queueARNsArray=[], queueTypeObject={},listQueuesResult;
console.log('At line 12 entring do while loop....')
do{
console.log('How many times do I stay here???')
let listQueuesParams = {
InstanceId: connectInstanceId, /* required */
QueueTypes: [
"STANDARD",
],
NextToken: nextToken,
};
let listQueuesPromise = connect.listQueues(listQueuesParams).promise();
listQueuesResult = await listQueuesPromise;
// console.log(listQueuesResult);
listQueuesResult.QueueSummaryList.forEach(queue => {
if(queueARN != null){
if (queue.Arn == queueARN){
queueARNsArray = [queue.Arn];
queueARNsObject[queue.Name]= queue.Arn;
queueTypeObject[queue.QueueType]= queue.Arn;
flag = 1;
return;
}
}else{
queueARNsObject[queue.Name]= queue.Arn;
queueTypeObject[queue.QueueType]= queue.Arn;
queueARNsArray.push(queue.Arn);
nextToken = listQueuesResult.NextToken;
}
});
}while (flag=0 && nextToken != null);
const HistoricalMetrics = [
{
Name : "CONTACTS_HANDLED",
Unit : "COUNT",
Statistic : "SUM"
},
{
Name : "CONTACTS_ABANDONED",
Unit : "COUNT",
Statistic : "SUM"
},
];
// Metrics params
var getHistoricalMetricsParams = {
InstanceId: connectInstanceId,
StartTime: 1593099900,
EndTime: 1593129300,
Filters: {
Channels: ["VOICE"],
Queues: queueARNsArray
},
HistoricalMetrics: HistoricalMetrics,
Groupings: ["QUEUE"]
};
// console.log(getHistoricalMetricsParams);
// get current metrics by queues
var getHistoricalMetricsPromise = connect
.getMetricData(getHistoricalMetricsParams)
.promise();
var getHistoricalMetricsResult = await getHistoricalMetricsPromise;
console.log("historical metrics",getHistoricalMetricsResult);
// console.log("current |||||||| 1 metrics:", JSON.stringify(getCurrentMetricsResult));
let queueMetricsArray = [];
if(getHistoricalMetricsResult.MetricResults.length){
getHistoricalMetricsResult.MetricResults.forEach(queue => {
let queueMetrics = {
"Queue_Name" : getKeyByValue(queueARNsObject ,queue.Dimensions.Queue.Arn),
"CallsHandled": queue.Collections[0].Value,
"CallsAbanoded": queue.Collections[1].Value,
}
queueMetricsArray.push(queueMetrics);
console.log("TYPE||||", getKeyByValue(queueTypeObject ,queue.Dimensions.Queue.Arn))
});
}
const response = {
responseCode: 200,
metricResults: queueMetricsArray
};
return response;
};
I don't have any idea why it is not showing anything. if anyone of you knows please help me to fix it Thanks. I don't know what is Missing I've almost checked everything but I didn't get anything.
There are a few general areas you can look at:
Specify the region.
AWS.Connect({ apiVersion: '2017-08-08', region:'xxxxx' });
use Await directly with listQueues method
let listQueuesPromise = await connect.listQueues(listQueuesParams).promise();
Check Permissions - make sure there is sufficient authority
Lambda Configuration - increase timeout and memory size
PS: What did console log listQueuesPromise return?

DynamoDB scan shows fewer items than DynamoDB console

Why does a scan with nodejs shows just 3 result, while the dynamodb admin tool shows 9
var params = {
TableName: process.env.DYNAMODB_TABLE_LIGHTHOUSE,
FilterExpression: '#blickArticleId = :valblickArticleId AND #firstFolder = :valfirstFolder',
ExpressionAttributeNames: {
'#blickArticleId': 'blickArticleId',
'#firstFolder': 'firstFolder'
},
ExpressionAttributeValues: {
':valblickArticleId': 'null',
':valfirstFolder': 'null'
},
};
const queryResponse = await dynamoDb.scan(params).promise()
isnt that the same
Are you sure , your scanned content is not more than 1MB ?.
If the total number of scanned items exceeds the maximum data set size
limit of 1 MB, the scan stops and results are returned to the user
with a LastEvaluatedKey
Then you can scan the remaining items by using LastEvaluatedKey.
as jarmod mentioned pagination is the solution:
const getLalalalalal = async () => {
var params = {
TableName: process.env.DYNAMODB_TABLE_LIGHTHOUSE,
FilterExpression: '#blickArticleId = :valblickArticleId AND #firstFolder = :valfirstFolder',
ExpressionAttributeNames: {
'#blickArticleId': 'blickArticleId',
'#firstFolder': 'firstFolder'
},
ExpressionAttributeValues: {
':valblickArticleId': 'null',
':valfirstFolder': 'null'
},
};
return await scanTable(params)
}
const scanTable = async (params) => {
let scanResults = [];
let items;
do {
items = await dynamoDb.scan(params).promise();
items.Items.forEach((item) => scanResults.push(item));
params.ExclusiveStartKey = items.LastEvaluatedKey;
} while(typeof items.LastEvaluatedKey != "undefined");
return scanResults;
};
Your code needs to scan for the remaining items. The presence of a non-null LastEvaluatedKey indicates that DynamoDB results are paginated. The AWS DynamoDB console is presumably doing the pagination for you.
Here is an example of code to fetch all items if paginated.

Sequelize Transaction inside forEach loop

I'm trying to use transaction inside forEach loop using async/await syntax of Node 7.0+
When I try to print committed transaction response in console, I'm able to see the values but those same values are not committed in to DB.
Below is the code :
documentInfo.forEach(async (doc) => { // array of documentInfo
var frontImgName = await module.exports.uploadImage(docFiles, doc.front, req, res )
var backImgName = await module.exports.uploadImage(docFiles, doc.back, req, res )
var checkKycDoc = await KYCDocument.findOne({
where: {
kyc_id: checkUserKyc.dataValues.kyc_id,
user_id: checkUserKyc.dataValues.user_id
}
})
if (checkKycDoc) { //update
var updateDocument = await KYCDocument.update({
document_name: doc.document_name,
front_image: frontImgName,
back_image: backImgName
}, {
where: {
kyc_id: checkUserKyc.dataValues.kyc_id,
user_id: checkUserKyc.dataValues.user_id
},
}, {transaction})
log('updateDocument', updateDocument.dataValues)
} else { // insert
var newKycDocument = await new KYCDocument({
kyc_id: checkUserKyc.dataValues.kyc_id,
user_id: checkUserKyc.dataValues.user_id,
document_name: doc.document_name,
front_image: frontImgName,
back_image: backImgName,
status: true
}, {transaction})
log('newKycDocument', newKycDocument.dataValues)
}
if (rowCount === documentInfo.length) {
await transaction.commit() // transaction is printed on this line
log('KYC has been uploaded successfully')
helpers.createResponse(res, constants.SUCCESS,
messages.KYC_UPLOAD_SUCCESS,
{'error': messages.KYC_UPLOAD_SUCCESS}
)
} else {
rowCount++
}
})
The issue was in the create method.
To resolve the issue I had to create a new row using:
var newKycDocument = await KYCDocument.create({
kyc_id: checkUserKyc.dataValues.kyc_id,
user_id: checkUserKyc.dataValues.user_id,
document_name: doc.document_name,
front_image: frontImgName,
back_image: backImgName
}, {transaction})
I was missing the .create method.

Resources