I'm learning how to use Sequelize to persist data to a MySql database from a NodeJS application. The following script is my attempt to insert data into a table which has some records in it.
The issue is I am attempting to debug the script in various locations using console.log but I get no output.
const db = require('../engage/db');
conn = db.conn
function init() {
const Reductions = conn.sequelize.define('Reductions', {
pid : {
type: conn.Sequelize.DataTypes.INTEGER,
allowNull: false,
primaryKey: true,
},
code : {
type: conn.Sequelize.STRING
},
duration : {
type: conn.Sequelize.DataTypes.INTEGER
}
});
return Reductions
}
async function insert(p) {
console.log('Testing # insert() entrance : ', p)
let table = await init(conn.sequelize, conn.Sequelize)
// conn.sequelize.sync({force:true}).then( async () => {
conn.sequelize.sync().then( async () => {
conn.sequelize.query('SELECT * FROM Reductions WHERE pid = :pid',{
replacements: {pid : p.id},
type: conn.Sequelize.QueryTypes.SELECT,
}).then((data) => {
console.log('TESTING DATA: ', data)
return data
}).then((results) => {
console.log('Checking results: ', results)
})
}).finally(() => {
console.log('Reductions update complete')
})
}
module.exports = { init, insert }
I an performing an SQL search and chaining .then() functions to debug the output.
But my problem is non of the console.log calls in the insert function are responding except for the first one.
I am very certain the required data is in the Reductions table of the database but even if it is not, the console.log in the finally section does not output either.
How can I understand and resolve why the console.logs are not printing?
Update
Insert function call code, as requested.
.
.
.
properties.forEach(async(property) => {
if( property.reduced !== false ) {
await reductions.insert(property)
} else {
uninvestibles += 1
}
})
.
.
.
Special thanks to Rinkesh P for helping me find the answer to my problem.
Apparently the foreach loop does not wait for promises to resolve. So my await related code wouldn't execute as intended. Not so sure why and curiously my foreach solution had been working.
But apparently, foreach loops should not be used with asynchronous code. So I refactored my foreach into the following for loop:
for (let index = 0; index < properties.length; index++) {
if( properties[index].reduced !== false ) {
await reductions.insert(properties[index], location)
} else {
uninvestibles += 1
}
}
That got my code working again.
Related
This one has had me spinning my tires for about 2 days now, I'm ready to reach out for help :)
I have a google firebase functions app, running as a middle-ware to an angular SPA. Hoping to avoid some of the pay-by-use cost of Azure SQL, I wanted to implement a caching option for the most common queries.
I thought I knew redis, I've worked with it before. There's a simple enough example on the repo: https://www.npmjs.com/package//redis
Everything works fine, if it is top level.
But the way my application is built, i need the ability to set a cache value from within the .then of a Promise, and when I try to do that, all operation just stops, with no identifiable error logging, or even response from redis. Even in Azure insights, i'm not getting much feed-back, only that the 'set' operation isn't being counted in metrics.
So, just to clarify, this works:
// "cache", the object, set globally
export const testCache = functions.https.onRequest(
async (req: any, res) => {
await cache.connect();
var redisKey = 'testing_global';
var result = await cache.get(redisKey);
await cache.set(redisKey, 'testing new class')
console.log("\nDone");
cache.disconnect();
res.send('done');
})
But, this does not:
import { createClient } from 'redis';
const cache = createClient({
url: "rediss://" + process.env.REDIS_HOST_NAME + ":6380",
password: process.env.REDIS_KEY,
});
export const getValues = functions.https.onRequest(
(req: any, response) => {
cors(req, response, async () => {
response.set('Access-Control-Allow-Origin', origin);
var searchText = req.body['search'];
var offset = req.body['offset'];
var fetch = req.body['fetch'];
var x = req.body['x'];
var y = req.body['y'];
var districts = req.body['districtFilter'];
var sort = req.body['sort'];
if (offset) {
if (!/^\d+$/.test(offset))
throw new Error('bad number');
} else {
offset = 0;
}
if (fetch) {
if (!/^\d+$/.test(fetch))
throw new Error('bad number');
} else {
fetch = 200;
}
if (x) {
if (!/^-?\d+$/.test(x))
throw new Error('bad number');
} else {
x = null;
}
if (y) {
if (!/^-?\d+$/.test(y))
throw new Error('bad number');
} else {
y = null;
}
if (districts && districts.length > 0) {
districts = sanitizeStringArray(districts);
} else {
districts = null;
}
if (sort) {
switch (sort) {
case "scoreAsc":
case "scoreDesc":
case "priceAsc":
case "priceDesc":
break;
default:
sort = '';
}
}
var redisKey = `get_values_${searchText}_${offset}_${fetch}_${x}_${y}_${districts}_${sort}`;
var cacheResult: any = null;
await cache.connect();
// because I want to end up in the 'else', for testing
cacheResult = await cache.getFromCache('someOtherKey');
if (null !== cacheResult) {
response.send({
"status": "success",
"totalCount": cacheResult.totalCount,
"data": cacheResult.result
});
cache.disconnect();
} else {
var connection = new Connection(sqlConfig);
var totalCount: number = 0;
connection.on('connect', function (err: any) {
// If no error, then good to proceed.
console.log("Connected");
var sql = `EXEC SomeSPC;`;
const sqlRequest = new Request(sql, function (err: any) {
if (err) {
console.log(err);
}
});
const countRequest = new Request(
`EXEC SomeOtherSPC;`
, function (err: any) {
if (err) {
console.log(err);
}
}
)
sqlRequest.connection = connection;
countRequest.connection = connection;
var result: any[] = [];
sqlRequest.on('row', function (columns: any[]) {
var rowResult: any = {};
columns.forEach(function (column: any) {
rowResult[column['metadata']['colName']] = column['value'];
});
result.push(rowResult);
});
sqlRequest.on("requestCompleted", function (rowCount: any, more: any) {
console.log(rowCount + ' rows returned');
connection.execSql(countRequest);
countRequest.on('row', function (columns: any[]) {
totalCount = columns[0]['value'];
});
countRequest.on('requestCompleted', async function (rowCount: any, more: any) {
connection.close();
cacheResult = {
totalCount: totalCount
, result: result
};
// ******************************************************************
// Does Not Work, Just Fails, Without Much to Go On
await cache.set(redisKey, 'in Promise')
response.send({
"status": "success",
"totalCount": totalCount,
"data": result
});
})
});
connection.execSql(sqlRequest);
});
connection.on('infoMessage', infoError);
connection.on('errorMessage', infoError);
connection.on('end', end);
connection.on('debug', debug);
connection.connect();
console.log("Reading rows from the Table...");
}
})
}
)
There's **update- no longer ** a fair amount of psuedo-code here, so please **update- No Need to ** disregard any inconsistent lines. I went ahead and put in the full function, including all the fluff, since trimming the fat seems to make things difficult for others to understand what is being asked.
The sql stuff all works, if i take out the cache.set() everything is fine, but that line, in the result of the Promise, just fails, and I can't figure out why.
I've tried using cache locally and globally, extracting the cache operations to a function, and then to a separate class, and in all cases, i'm getting the same result.
Is there a known reason this wouldn't work?
As far as I can understand, given you didn't provide a reproducible example, your code is binding to the requestCompleted event for the second request only after it runs execSql(): I would suggest moving the binding block before that, otherwise the event may be skipped.
Using loopback I have a simple for loop in which I perform a findOne:
let my_data = [];
Orders.forEach(function(order,idx) {
let postalcode = order.toJSON().customer.postal_code;
let ps4 = postalcode.slice(0,4);
app.models.postalcode.findOne({where: {postal_code: parseInt(ps4)},include: ['depot']}, function (err, Postalcode) {
if (err) {
winston.error('Could not load postalcode %s due to error %s: ', ps4, err.message);
} else {
if (Postalcode) {
let depot = Postalcode.toJSON().depot;
if (!depot) {
//
} else {
let depot_city = depot.city;
if (cities_to_process.indexOf(depot_city) > -1) {
my_data.push(order);
} else {
}
}
} else {
winston.warn('Could not find postal code %s', ps4)
}
}
});
});
console.log(my_data);
After the for loop I would like to do something with the collected data in my_data. Since findOne appears to be asynchronous, what is the preferred way of doing this?
Use async library. It will be easy for you to use async.eachSearies() to loop the async functions in series and get the desired output. You can take a reference here link
You can create new promises in forEach like:
let Y = X.forEach((item) => {
return new Promise((resolve, reject) => {
resolve('done')
})
})
Promise.all(y, callback)
result.forEach(element => {
//Get each element
console.log("LOOP");
dbo.collection("users").findOne({email: emailGiven, "friends.email": element.email},function(errT, resultT) {
if (errT){
console.log("Query Error Inside!");
res.status(errT.status); // or use err.statusCode instead
console.log(errT);
//db.close();
//return res.send(errT.message);
}
else {
if (resultT) {
var oneUser = {
email: element.email,
username: element.username,
fullName: element.fullName,
status: resultT
};
//console.log(resultT);
foundUsers.push(oneUser);
} else {
//Not found means not added or pending
var oneUser = {
email: element.email,
username: element.username,
fullName: element.fullName,
status: 0
};
foundUsers.push(oneUser);
//console.log(emailGiven + " " + element.email)
console.log(oneUser);
}
}
});
});
i have an object array for each elemant i would like to do mongoDB call for each element and depending on the results i wanna push the results in an array as im doing, the problem is that mongoDb is async so my main thread finished before i can push results to the array foundUsers, how may i fix this issue?
As you said, need to do handle an asynchronous operation into a synchronous loop. For doing this, you can use async library. It is so useful in such operatinos.
Just install async module in your project first
npm install --save async
Afterwards, you can do sth like this:
// for use with Node-style callbacks...
var async = require("async");
var obj = {dev: "/dev.json", test: "/test.json", prod: "/prod.json"};
var configs = {};
async.forEachOf(obj, (value, key, callback) => {
fs.readFile(__dirname + value, "utf8", (err, data) => {
if (err) return callback(err);
try {
configs[key] = JSON.parse(data);
} catch (e) {
return callback(e);
}
callback();
});
}, err => {
if (err) console.error(err.message);
// configs is now a map of JSON data
doSomethingWith(configs);
});
For working with this library, it uses async.forEachOf function instead of simple forEach loop. Three parameters is sent to this function.
The 1st parameter that is passed to async.forEachOf is an array to iterate over it (obj).
The 2nd parameter is a callback function that apply over each item in obj.
The 3rd or the last parameter that is passed to async.forEachOf function, is another callback function too. It is called when iteration process over every item in obj has finished.
I am trying to validate the array of objects before inserting them into the mongodb.
what i am trying to do is, lets say i have an object like below
var data= { prodDetails:
[
{ measured: 'Liters',
name: 'A',
prodCode: '713',
status: true },
{ measured: 'Liters',
name: 'B',
prodCode: '713',
status: true },
{ measured: 'Liters',
name: 'C',
prodCode: '674',
status: true }
]
};
before making a bulk insert call i want check whether the given prodCode is valid DB Code or not and name duplicated or not
i am using node bluebird promises.
i tried the following code to validate prodCode
var bulkOperations = {
bulkProdInsert: function (body) {
return new Promise(function (reslv, rej) {
if (body.prodDetails.length > 0) {
common_lg.getValueById(body, "typesProd", body.prodDetails[0].prodCode).then(bulkOperations.successCallback(reslv, rej, body)).catch(bulkOperations.errCallback(reslv, rej, body));
};
reslv();
});
},
successCallback: function (reslv, rej, body) {
return function (res) {
if (res) {
body.prodDetails.splice(0, 1);
if (body.prodDetails.length > 0) {
common_lg.getValueById(body, "typesProd", body.prodDetails[0].prodCode).then(bulkOperations.successCallback(reslv, rej, body)).catch(bulkOperations.errCallback(reslv, rej, body));
}
};
};
},
errCallback: function (reslv, rej, body) {
return function (err) {
body.prodDetails.splice(0, 1);
if (body.prodDetails.length > 0) {
common_lg.getValueById(body, "typesProd", body.prodDetails[0].prodCode).then(bulkOperations.successCallback(reslv, rej, body)).catch(bulkOperations.errCallback(reslv, rej, body));
};
};
}
};
but i want to do is insert all the objects/documents into DB when name and prodCode of all the objects/documents is validated.
how to achieve this.
thanks
It sounds like you want to check the input object and then make DB calls. I would suggest the .map method of bluebird.
var promise = require('bluebird');
var checkValues = promise.method( function(prod){
if( isValid(prod.prodCode) ){
return prod;
}
//something went wrong!
throw new Error('prodCode ' + prod.prodCode + ' is invalid');
}
promise.map( data.prodDetails, checkValues )
.then(function(){
//it worked! You can call the DB now.
})
.catch(function(error){
//something went wrong, look at that error (or pass along)
})
So long as your checkValues method is a promise, you can run it against every value in your input array, and use the success of .then to know things worked and call your DB!
I created a function to:
take an array of 'labels' and look for whether they have a record in the db already
create those which don't exist,
and update those which do exist
return a json array reporting on each item, whether they were updated/created, or resulted in an error
I managed to make it work but I feel like I just made some ugly dogs' dinner!
var models = require("../models");
var Promise = models.Sequelize.Promise;
module.exports = {
addBeans: function (req, callback) {
Promise.map(req.body.beansArr, function (bean) {
return models.Portfolio.findOrCreate({where: {label: bean}}, {label: bean});
}).then(function (results) { // Array of 'instance' and 'created' for each bean "findOrCreate(where, [defaults], [options]) -> Promise<Instance>"
var promisesArr = [];
results.forEach(function (result) {
if (result[1]) { // result[1] = wasCreated
promisesArr.push(Promise.resolve([result[0].dataValues.label, "created"]));
} else {
promisesArr.push(
models.Portfolio.update({label: result[0].dataValues.label},
{where: {label: result[0].dataValues.label}}).then(function () {
return Promise.resolve([result[0].dataValues.label, "updated"])
})
);
}
});
return promisesArr;
// When it's all done create a JSON response
}).then(function (results) {
var resultObj = {items: []}; // JSON to return at the end
Promise.settle(results).then(function (promiseinstances) {
for (var i = 0; i < promiseInstances.length; i++) {
if (promiseInstances[i].isFulfilled()) {
resultObj.items.push({
item: {
label: promiseInstances[i].value()[0],
result: promiseInstances[i].value()[1],
error: ''
}
});
}
else if (promiseInstances[i].isRejected()){
resultObj.items.push({
label: promiseInstances[i].value()[0],
result: 'error',
error: promiseInstances[i].reason()
});
}
}
// Send the response back to caller
}).then(function () {
return callback(null, resultObj);
}, function (e) {
return callback(e, resultObj);
});
});
}
};
Question:
Is there an easier or more obvious way to create/update values with Sequelize?
Is my use of Promise.settle() appropriate for this case? I have the feeling I made this more complicated than it needs to be.
I am new to Sequelize and using Promises, I'd appreciate if someone could advise on this.
I feel like this would work better on CodeReview.SE but I can see a few issues.
Is there an easier or more obvious way to create/update values with Sequelize?
Well, for one thing:
.then(function(array){
var newArr = [];
array.forEach(function(elem){
newArr.push(fn(elem);
}
return newArr;
});
Is just
.map(fn)
Additionally, promises assimilate so you can return val; from a .then you don't have to return Promise.resolve(val);.
So:
).then(function (results) { // Array of 'instance' and 'created' for each bean "findOrCreate(where, [defaults], [options]) -> Promise<Instance>"
var promisesArr = [];
results.forEach(function (result) {
if (result[1]) { // result[1] = wasCreated
promisesArr.push(Promise.resolve([result[0].dataValues.label, "created"]));
} else {
promisesArr.push(
models.Portfolio.update({label: result[0].dataValues.label},
{where: {label: result[0].dataValues.label}}).then(function () {
return Promise.resolve([result[0].dataValues.label, "updated"])
})
);
}
});
return promisesArr;
})
Is just
.map(function(result){
if(result[1]) return [result[0].dataValues.label, "created"];
return models.Portfolio.update({label: result[0].dataValues.label},
{where: {label: result[0].dataValues.label}}).
return([result[0].dataValues.label, "updated"]);
});
However, since you want it to work regardless of it being resolved, you'd have to do:
.then(function(results){
return results.map(function(result){
if(result[1]) return [result[0].dataValues.label, "created"];
return models.Portfolio.update({label: result[0].dataValues.label},
{where: {label: result[0].dataValues.label}}).
return([result[0].dataValues.label, "updated"]);
});
});
Which means it'll resolve regardless, then you'd call .settle():
.settle().then(function(results){
// your settle logic here
});
Note that the last:
}).then(function () {
return callback(null, resultObj);
}, function (e) {
return callback(e, resultObj);
});
Is simply:
.nodeify(callback);
However, I recommend sticking to promises.
I use Promise.settle for sequelize.update, and can get affect rows number by _settledValueField .
promise.push(...update...)
db.sequelize.Promise.settle(promise).then(function (allresult) {
var affectcnt = 0
allresult.forEach(function (singlecnt) {
if (undefined !== singlecnt._settledValueField[1]) {
affectcnt += parseInt(singlecnt._settledValueField[1])
}
})
unfortunately, it's only work for update.
You can insert array in database using sequelize. You want to change in model like below. I am trying to add multiple languages in database through array.
language: { type: DataTypes.STRING,
allowNull: false,
get()
{
return this.getDataValue('language').split(';')
},
set(val)
{
this.setDataValue('language',Array.isArray(val) ? val.join(','):val);
}
}