I have an issue with not able to get the affected rows result from the following
During the debug I notice it always crashes at conn.querySync(query.sqlUpdate, params);
Console.log is not showing anything as well.
What did I do wrong here?
CODE
//imports
const format = require('string-format');
const query = require('../db/query');
const message = require('../common/message');
const constant = require('../common/constant');
var ibmdb = require("ibm_db");
require('dotenv').config();
// access the environment variables for this environment
const database = "DATABASE=" + process.env.DATABASE + ";";
const hostname = "HOSTNAME=" + process.env.HOSTNAME + ";";
const uid = "UID=" + process.env.UID + ";";
const pwd = "PWD=" + process.env.PWD + ";";
const dbport = "PORT=" + process.env.DBPORT + ";";
const protocol = "PROTOCOL=" + process.env.PROTOCOL;
const connString = database+hostname+uid+pwd+dbport+protocol;
function updateContact(params) {
ibmdb.open(connString, function(err, conn){
//blocks until the query is completed and all data has been acquired
var rows = conn.querySync(query.sqlUpdate, params);
console.log(rows);
});
}
module.exports.updateContact = updateContact;
I finally understand what the problem is.
The problem lies in me using the querySync function. This function not return affected row counts.
https://github.com/ibmdb/node-ibm_db/blob/master/APIDocumentation.md#querySyncApi
The proper way is to use prepare followed by executeNonQuery.
https://github.com/ibmdb/node-ibm_db/blob/master/APIDocumentation.md#executeNonQueryApi
So from the API, i modify my codes.
...
conn.prepare(query.SQL_UPDATE, function (error, stmt) {
if (err) {
console.log(err);
return conn.closeSync();
}
stmt.executeNonQuery(params, function (err, result) {
if( err ) {
console.log(err);
}
else {
console.log("Affected rows = " + result);
}
//Close the connection
conn.close();
});
});
...
Related
I'm trying to run a NodeJS code that reads some data fields from an array, use them to do a database query to check if the data is duplicate before inserting them into the corresponding table.
My NodeJS code will be called from a PHP script so I need to know when it ends this is why I need to add process.exit(0) somewhere. The problem I have is that if I add it, the script is terminated and my promise never gets the time to send back the result.
Here is my code:
var bar = new Promise((resolve, reject) => {
result.forEach((row, index, array) => {
var escaped = _.map(row, mysql.escape);
var checkQuery = "SELECT COUNT(*) as found FROM data WHERE field1 = " + escaped[0] + " AND field2 = " + escaped[1] + " AND field3 = " + escaped[2] + " AND field4 = " + escaped[3] + " AND field5 = " + escaped[4] + " AND field6 = " + escaped[5] + " AND field7 = " + escaped[6] + ";";
conn.query(checkQuery, function (err, res) {
if (err) {
console.log("Error checking row for duplicate");
console.log(checkQuery);
process.exit(1);
} else {
if (res[0].found == 0) {
var query = " (";
var escaped = _.map(row, mysql.escape);
var csv = escaped.join(',');
query += csv;
query += ")";
query += row !== _.last(result) ? ',' : ';';
console.log(query);//This will change to inserting the data to the table
}else{
console.log("Duplicate found!");
}
}
});
if (index === array.length -1) resolve();
});
});
bar.then(() => {
console.log('All done!');
process.exit(0);
});
If I remove process.exit(0); I see "All done" first then console.log(query) result.
If I add it, the script is terminated and I see "All done" only.
Is there a better approach to do this task please?
Thanks.
Here is a way to wait for a promise before the application exits.
class Waiter {
private timeout: any
constructor() {
this.waitLoop()
}
private waitLoop():void {
this.timeout = setTimeout(() => { this.waitLoop() }, 100 * 1000)
}
okToQuit():void {
clearTimeout(this.timeout)
}
}
// Your app.
const appPromise:Promise<any> = ...
const w = new Waiter()
appPromise.finally(() => {
console.log("Quitting")
w.okToQuit()
})
Running multiple asynchronous operations in a loop and tracking when everything is done is just way, way, way easier if you use promises for all the individual asynchronous operation rather than trying to track asynchronous operations that use plain callbacks.
You don't say exactly what your database is, but if it's mysql, then there is a mysql2/promise driver that natively supports promises and that would be my recommendation to switch to that. Then you can directly use a promise returned from .query(). But, without the info about your specific database driver, I've shown how to manually promisify .query().
Then, the looping code can use a for loop and await to sequence the database calls so it's easy to know when they are all complete.
const { promisify } = require('util');
async function someFunc() {
// other code here
// promisify conn.query (or use promise interface directly from the database)
conn.queryP = promisify(conn.query);
try {
for (const row of result) {
const escaped = _.map(row, mysql.escape);
const checkQuery = "SELECT COUNT(*) as found FROM data WHERE field1 = " + escaped[0] + " AND field2 = " +
escaped[1] + " AND field3 = " + escaped[2] + " AND field4 = " + escaped[3] + " AND field5 = " +
escaped[4] + " AND field6 = " + escaped[5] + " AND field7 = " + escaped[6] + ";";
let res = await con.queryP(checkQuery);
if (res[0].found == 0) {
const csv = _.map(row, mysql.escape).join(',');
const terminator = row !== _.last(result) ? ',' : ';';
const query = " (" + csv + ")" + terminator;
console.log(query); //This will change to inserting the data to the table
} else {
console.log("Duplicate found!");
}
}
} catch (e) {
console.log("Error checking row for duplicate: ", checkQuery);
console.log(e);
process.exit(1);
}
console.log('All done!');
process.exit(0);
}
The code appears to be trying to build a query inside the loop where each iteration of the loop will add-on to the next (that's what _.last(result) ? ',' : ';'; look like anyway). If that's the case, then the query variable has to be moved outside the loop so it can build from one iteration of the loop to the next. But, you don't show what you're really trying to do with that query so you're on your own for that part.
you decide how many promises will go out before hand and then count them as they resolve, then exit
in this example the same principle is applied but it has callback functions instead of promises. For promises you would call a count function from the .then() or .finally(), and the count function will decide whether it is time to exit
mongoose example from a javascript server:
let g_DB = null;
//init mongoose
const mongoose = require("mongoose");
const connectionParams = {
useNewUrlParser: true,
useUnifiedTopology: true,
};
const connStr1 = "mongodb+srv://XX:XX#clusterXX.XX.mongodb.net/XX?
retryWrites=true&w=majority";
mongoose.set("strictQuery", false);
mongoose.connect(connStr1, connectionParams)
.then(handleConnection())
.catch((err) => console.log("Error:", err));
//end script
//handleConnection - start on successful response from mongoose connection
function handleConnection(msg) {
console.log("mongoose has connected to Mongo Atlas successfully");
g_DB = mongoose.connection;
g_DB.once("open", function () {
console.log(
"mongoose has connected to Mongo Atlas Cluster using database XX"
);
doTest();
});
}
//---------------------------------------------------
function doTest() {
console.log("test-05: create 500 books");
//---- MODEL ----
const _schema = new mongoose.Schema({
name: String,
price: Number,
quantity: Number,
});
//g_DB is a mongoose connection set earlier in the script
const _model = g_DB.model("book_schema", _schema, "bookstore");
let loopcount = 500;
let waitcount = loopcount;
for (let i = 0; i < loopcount; i++) {
_m = new _model({
name: `WHY MAKE 500 BOOKS ${new Date().toISOString()}`,
price: 200,
quantity: 2000,
});
_m.save((e, x) => {
if (e) return console.error(e);
console.log(x, `waitcount: ${--waitcount}`);
if (!waitcount) doExit();
});
}
}
//--
function doExit() {
console.log("exit from server");
process.exit();
}
Use Reject/Resolve to manage promise in Nodejs
When your task fulfils your request send result with resolve(); and if its failing use reject();
In your case you are not managing promise properly that's why it's running asynchronously, better to use following way with the proper returns.
var bar = new Promise((resolve, reject) => {
return result.forEach((row, index, array) => {
var escaped = _.map(row, mysql.escape);
var checkQuery = "SELECT COUNT(*) as found FROM data WHERE field1 = " + escaped[0] + " AND field2 = " + escaped[1] + " AND field3 = " + escaped[2] + " AND field4 = " + escaped[3] + " AND field5 = " + escaped[4] + " AND field6 = " + escaped[5] + " AND field7 = " + escaped[6] + ";";
return conn.query(checkQuery, function (err, res) {
if (err) {
console.log("Error checking row for duplicate");
console.log(checkQuery);
return reject(err);
} else {
if (res[0].found == 0) {
var query = " (";
var escaped = _.map(row, mysql.escape);
var csv = escaped.join(',');
query += csv;
query += ")";
query += row !== _.last(result) ? ',' : ';';
console.log(query);//This will change to inserting the data to the table
return resolve(query)
} else {
console.log("Duplicate found!");
return reject('Duplicate Found');
}
}
});
});
});
bar.then((data) => {
console.log('All done!');
});
In above code I am returning query + resolve/reject so it makes better to run in more synchronised way.
return conn.query(checkQuery, function (err, res) {
Plus, while processing this promise I am handling with .then((data) so I can handle that resolve values here.
bar.then((data) => {
console.log('All done!');
});
Note: If you are rejecting any promise it won't be available in above .then block you'll find this reject in catch block so code will be changed in following way.
bar.then((data) => {
console.log('All done!');
}).catch(err=>{
console.log(err);
});
You can try the following:
(async () => {
await new Promise((resolve, reject) => {
result.forEach((row, index, array) => {
var escaped = _.map(row, mysql.escape);
var checkQuery = "SELECT COUNT(*) as found FROM data WHERE field1 =
" + escaped[0] + " AND field2 = " + escaped[1] + " AND field3 = " +
escaped[2] + " AND field4 = " + escaped[3] + " AND field5 = " + escaped[4] + " AND field6 = " + escaped[5] + " AND field7 = " + escaped[6] + ";";
conn.query(checkQuery, function (err, res) {
if (err) {
console.log("Error checking row for duplicate");
console.log(checkQuery);
process.exit(1);
} else {
if (res[0].found == 0) {
var query = " (";
var escaped = _.map(row, mysql.escape);
var csv = escaped.join(',');
query += csv;
query += ")";
query += row !== _.last(result) ? ',' : ';';
console.log(query);//This will change to inserting the data to the table
}else{
console.log("Duplicate found!");
}
}
});
if (index === array.length -1) resolve();
});
});
console.log('All done!');
})();
You don't even need to call the process.exit(0) because the code will always terminate when the job is done :)
I retrieved a JSON object from a local database, I want to edit a value (invItems) and add a new value to it (filed[filed.invItems]), then upload it back to the database, but it does not seem to work (the JSON does not seem to change)
async function invPut(itemID, message) {
var filed = await frenzyDB.getKey(id + "_invcache");
console.log("Before: " + filed)
newInvItems = filed.invItems + 1;
filed.invItems = newInvItems;
filed[filed.invItems] = itemID;
console.log("After: " + filed);
await frenzyDB.addKey(id + "_invcache", filed)
}
Console Output:
Before: {"invItems":0}
After: {"invItems":0}
It shows no errors, but the JSON doesnt change. Am I doing something wrong? If so, what can I do to fix it?
Thanks for all your help!
Notes:
frenzyDB is just a javascript file that deals with a standard REPL.it Database
Code of frenzyDB:
const Database = require("#replit/database")
const db = new Database()
async function addKey(key, value) {
await db.set(key, value).then(() => {return;});
}
async function getKey(key) {
return await db.get(key).then(value => {return value;});
}
function listAllKeys() {
db.list().then(keys => {return keys;});
}
async function hasKey(key) {
var keys = await listAllKeys();
if (keys.includes(key)) {
return true;
} else {
return false;
}
}
async function removeKey(key) {
await db.delete(key).then(() => {return;});
}
module.exports = {
addKey,
getKey,
listAllKeys,
hasKey,
removeKey
};
Edit: Latest code:
async function invPut(itemID, message) {
await init(message.author.id);
var filed = await frenzyDB.getKey(message.author.id + "_invcache");
console.log(filed)
const result = {};
result.invItems = (filed['invItems'] + 1) || 1;
result.hasOwnProperty(filed.invItems) ? result[filed.invItems + 1] = itemID : result[filed.invItems] = itemID;
console.log(result);
frenzyDB.addKey(message.author.id + "_invcache", result)
message.reply("A **"+ itemIDs[itemID].name + "** was placed in your inventory");
return true;
}
EDIT 2: Latest Console Output:
{ '4': 3, invItems: 5 }
{ '5': 3, invItems: 6 }
Any help will be appreciated!
Thanks
Try this
// Demo Data
const itemID = 10;
var filed = { "invItems" : 0 };
// Real function
console.log("Before: " + JSON.stringify(filed));
const result = {};
result.invItems = (filed['invItems'] + 1) || 1;
result.hasOwnProperty(filed.invItems) ? result[filed.invItems + 1] = itemID : result[filed.invItems] = itemID;
console.log("After: " + JSON.stringify(result));
The result I get is
Before: {"invItems":0}
After: {"0":10,"invItems":1}
You would then of course use result to store the data away in the DB.
async function invPut(itemID, message) {
// Typo?
var filed = await frenzyDB.getKey(itemID + "_invcache");
console.log("Before: " + filed)
const result = {};
result.invItems = (filed['invItems'] + 1) || 1;
result.hasOwnProperty(filed.invItems) ? result[filed.invItems + 1] = itemID : result[filed.invItems] = itemID;
console.log("After: " + result);
// Typo?
await frenzyDB.addKey(itemID + "_invcache", result)
}
Answer Edit:
const result = { ...filed };
result.invItems = (filed['invItems'] + 1) || 1;
result.hasOwnProperty(filed.invItems) ? result[filed.invItems + 1] = itemID : result[filed.invItems] = itemID;
console.log(JSON.stringify(result));
maybe this will help you
const json = fs.readFileSync(`${__dirname}/data/data.json`, "utf-8");
const inputData = JSON.parse(json);
inputData.push({input: 'front'}) // creates new element for data.json
-------------------------------------------
array.push({front: 'front', back: 'back'});
I am new to node, express and mongo so I need some guidance. I set up a mongo database with 3 objects. I created an application with a route so that http://localhost:3000/Employeeid will lead the page to display all the 3 objects in the database, which the page does. However for each subsequent refresh, the same data gets displayed multiple number of times based on the number of refresh. (i.e. For the first refresh, the data duplicates once. For the second refresh, the data duplicates twice.) Does anyone know what may be wrong?
var express = require('express')
var app = express()
var MongoClient = require('mongodb').MongoClient
var url = 'mongodb://localhost:27017'
var str = ''
app.route('/Employeeid').get(function(req, res) {
MongoClient.connect(url, { useNewUrlParser: true }, function(err, client) {
var db = client.db('EmployeeDB')
var cursor = db.collection('Employee').find()
cursor.forEach(function(item) {
if (item != null) {
str = str + ' Employee id  ' +
item.Employeeid + '</br>'
}
})
res.send(str)
client.close()
})
})
var server = app.listen(3000, function() {})
You are appending to str every time the /EmployeeId route is called. To fix this, move str inside the callback:
app.route('/Employeeid').get(function(req, res) {
MongoClient.connect(url, { useNewUrlParser: true }, function(err, client) {
var str = ''
var db = client.db('EmployeeDB')
var cursor = db.collection('Employee').find()
cursor.forEach(function(item) {
if (item != null) {
str = str + ' Employee id  ' +
item.Employeeid + '</br>'
}
})
res.send(str)
client.close()
})
})
Side note - you should reuse the Mongo connection instead of calling connect and close on every request.
The Reason why your getting duplicate results is you are not making str = null after sending the result. you are concatenating str for each request because str is a global variable. your code can be modified for performance as like this.
var express = require('express')
var app = express()
var MongoClient = require('mongodb').MongoClient
var url = 'mongodb://localhost:27017'
var str = ''
var db = null
MongoClient.connect(url, { useNewUrlParser: true }, function(err, client) {
if( err ) {
throw new Error( err );
}
db = client.db('EmployeeDB') // we are storing db reference in global variable db
});
app.route('/Employeeid').get(function(req, res) {
db.collection('Employee').find().toArray( function(err, cursor){
if( err ) {
res.send('');
throw new Error(err );
}
cursor.forEach(function(item) {
if (item != null) {
str = str + ' Employee id  ' +
item.Employeeid + '</br>'
}
}
res.send(str)
str = '';
})
})
})
var server = app.listen(3000, function() {})
If you don't move res.send() into callback function you may send result before getting data from db due to async nature of node.
I think I'm very close to what I want to do. I have the following api get method in node.js that is retrieving a file varbinary(MAX) from an SQL Server database. It was converted from a base64 encoded string before inserted so the Content Type information was stripped from the string.
node.js
router.get('/getFile', (req, res) => {
console.log("Calling getFile for file " + req.query.serialNumber + ".")
var serialNumber = req.query.serialNumber;
let request = new sql.Request(conn);
request.query('SELECT FileName + \'.\' + FileExtension AS \'File\', FileType, ContentType, SerialNumber, Chart ' +
'FROM dbo.ChangeFiles ' +
'WHERE SerialNumber = ' + serialNumber)
.then(function (recordset) {
log("Successfully retrieved file " + recordset[0].SerialNumber + " from database.");
log("Length of blob " + recordset[0].File + " is " + recordset[0].Chart.length)
res.status(200);
res.setHeader('Content-Type', recordset[0].ContentType);
res.setHeader('Content-Disposition', 'attachment;filename=' + recordset[0].File);
res.end(Buffer.from((recordset[0].Chart)));
}).catch(function (err) {
log(err);
res.status(500).send("Issue querying database!");
});
});
That works fine, but what to do in Angular to retrieve it and prompt for a download for the user has not been clear for me, nor has there been a lot as far as help/resources online. Here is what I have so far in my service class.
fileDownload.service.ts
downloadFile(serialNumber: string): Observable<any> {
return this.http.get(this.baseURL + '/getFile', { params: { serialNumber: serialNumber } })
.map(this.extractFile);
}
private extractFile(response: Response) {
const file = new Blob([response.blob]);
FileSaver.saveAs(file);
// const url = window.URL.createObjectURL(file);
// window.open(url);
return file;
}
As you can see I've tried a couple of approaches. The commented out portion of the extractFile method didn't work at all, and using the FileSaver.saveAs function produces a file download of an unknown type, so the headers sent from node.js didn't seem to affect the file itself.
Would someone be able to advise how to proceed in Angular with what is successfully being sent from node.js so that I can successfully download the file, regardless of type?
Thanks so much in advance.
I got it working afterall. I had to rework the api call so that it sent all of the file information separately so that the MIME type, and file name can be assigned to the file on the client side in the component class. For some reason when I tried to do so all in the api, it wouldn't work so that was my work around. So here is what works for me.
node.js api
router.get('/getFile', (req, res) => {
console.log("Calling getFile for file " + req.query.serialNumber + ".")
var serialNumber = req.query.serialNumber;
let request = new sql.Request(conn);
request.query('SELECT FileName + \'.\' + FileExtension AS \'File\', FileType, ContentType, SerialNumber, Chart ' +
'FROM dbo.ChangeFiles ' +
'WHERE SerialNumber = ' + serialNumber)
.then(function (recordset) {
log("Successfully retrieved file " + recordset[0].SerialNumber + " from database.");
log("Length of blob " + recordset[0].File + " is " + recordset[0].Chart.length)
res.send(recordset[0]);
}).catch(function (err) {
log(err);
res.status(500).send("Issue querying database!");
});
});
component class
downloadFile(serialNumber: string): void {
this.changeService.downloadFile(serialNumber).subscribe((res: any) => {
const ab = new ArrayBuffer(res.Chart.data.length);
const view = new Uint8Array(ab);
for (let i = 0; i < res.Chart.data.length; i++) {
view[i] = res.Chart.data[i];
}
const file = new Blob([ab], { type: res.ContentType });
FileSaver.saveAs(file, res.File);
console.log(res);
});
}
service class
downloadFile(serialNumber: string): Observable<any> {
return this.http.get(this.baseURL + '/getFile', { params: { serialNumber: serialNumber } })
.map(this.extractFile);
}
private extractFile(response: Response) {
// const file = new Blob([response.blob]);
// FileSaver.saveAs(file);
// const url = window.URL.createObjectURL(file);
// window.open(url);
const body = response.json();
return body || {};
}
Update your code to call subscribe instead of map
I am new to node and writing a small application. I haven't used a language as asynchronous as this on the server before and have myself in a bit of a pickle. I need to take a string, query a table for an id, then insert in a second table using the result, then return a string from the funtion two levels up. I have a custom dao I use for the db stuff. Here is the function where it all happens:
function generateToken(data, userId, client) {
var random = Math.floor(Math.random() * 100001);
var sha256 = crypto.createHmac("sha256", random );
var token = sha256.update(data).digest("base64");
var query = dao.select(
'auth.apps',
{
name: client.name,
version: client.version,
subversion: client.subversion,
patch: client.patch
}
).done(
function(result) {
dao.insert(
'auth.tokens',
{
user_id:userId,
app_id: result.rows[0].id,
token:token
}
);
return "mmmm yellllo";
}
);
var ret_val = await(query);
console.log("Token return: " + ret_val);
return ret_val;
}
and here is the relevant part of my dao for select:
dbo.prototype.select = function(table, where, order_by) {
var where_clause = this.construct_where(where);
var sql = 'SELECT * FROM ' + table + ' WHERE ' + where_clause;
if(order_by !== undefined) {
sql = sql + ' ORDER BY ' + order_by;
};
var result = this.pool.query(sql);
return result;
};
and insert:
dbo.prototype.insert= function(table, values) {
var key_list='', value_list = '';
for( var k in values)
{
key_list = key_list + ', ' + k;
value_list = value_list + ", '" + values[k] + "'";
}
// chop off comma space
key_list = key_list.substring(2);
value_list = value_list.substring(2);
var sql = 'INSERT INTO ' + table + '(' + key_list + ') VALUES(' + value_list + ') RETURNING id';
var result = this.pool.query(sql).catch(function(error) {
console.log("SQL:" + sql + " error:" + error);
});
return result;
};
How do unwind the double promise. I want the generateToken function to return the token variable but only after the insert query has finished.
There is a library named deasync.
And the motivation to create it was to solve the situations when
API cannot be changed to return merely a promise or demand a callback
parameter
So this is the primary and probably the only use case. Because in general Node.js should stay async.
To do the trick you basically should write a function that accepts a callback and then wrap it with deasync as follows:
var deasync = require('deasync');
//It can still take the params before the callback
var asyncGenerateToken = function (data, userId, client, callback) {
var token = 'abc';
//Async operation starts here
setTimeout(function () {
//Async operation is finished, now we can return the token
//Don't forget that the error is 1st arg, data is the 2nd
callback(null, token);
}, 1000);
};
var generateToken = deasync(asyncGenerateToken);
//We'll retrieve a token only after a second of waiting
var token = generateToken('my data', 'my user id', 'my client');
console.log(token);
Hope this helps.