Async communicating with serialport gives mixed data - node.js

I'm trying to communicating with an arduino that has sensors.
So i have an object called motherboard who has sensors and each sensor has metric which may have threshold and/or polling which has a methode called getValue that sends data to the arduino and returns data with a promise. The problem is that if i async the sensors to get their values all sensors get the same value.
I don't know why this is happening. I only have programmed with javascript for 1 year and with angular for 5 months. I checked the post async/await using serialport in node.js but i checked my code and i did the something that was suggested in the post.
The communicate method is inside a service.
Can anyone help?
tl;dr :
send data to arduino get data back in a promise.
Metric A and B get the same promise.
The component polling also gets the promise of threshold. (metric has threshold and polling)
Me
communication.service.ts
communicate(cmd: string, serialPort: SerialPort, expectedResponse: string, notExpectedResponse){
const parser = serialPort.pipe(new Delimiter({delimiter: '\n'}));
return new Promise((resolve, reject) => {
serialPort.write(cmd, () => {
console.log('message written');
parser.on('data', data => {
const dataString = data.toString();
if (dataString != null && dataString.includes(expectedResponse)) {
let responseRemoved = dataString.replace(expectedResponse + ' ', '');
resolve(responseRemoved);
} else {
let response;
if (dataString != null && dataString.includes(notExpectedResponse)) {
response = dataString.replace(notExpectedResponse + ' ', '');
}
reject(response);
}
});
setTimeout(() => {
reject('');
}, this.timeOutTime);
});
});
}
threshold.component.ts
private getValuesThreshold(): void{
console.log(this.metricId);
this.motherboardInUse.getValues(this.metricId, GlobalVariableCMD.GET_THRESHOLD_VALUES,
GlobalVariableResponse.GET_THRESHOLD_VALUES, GlobalVariableResponse.GET_THRESHOLD_VALUES).then(data => {
let dataString = data.toString();
if(dataString){
console.log(dataString);
let responseSplit = dataString.toString().split(' ');
let minimumValue = parseInt(responseSplit[1]);
let maximumValue = parseInt(responseSplit[2]);
minimumValue < this.floor ? this.minimumThreshold = this.floor : this.minimumThreshold = minimumValue;
maximumValue > this.ceil ? this.maximumThreshold = this.ceil : this.maximumThreshold = 90;
this.enabled = responseSplit[0].includes('1');
console.log(this.minimumThreshold);
console.log(this.maximumThreshold);
console.log(this.enabled);
}
}).catch(err => {
let errString = err.toString();
if(errString){
console.log(errString);
}
});
}
motherboard.component.ts
getValuesThreshold(metricId: string, ATcmd: string, expectedResponse: string, notExpectedResponse: string) {
let command = this.communicateBuilder.BuildCommandGetMetricValue(ATcmd, this.usedSensorId, metricId);
console.log('motherboard get values' + command);
let responseOk = this.commandBuilderService.respondsSuccess(expectedResponse);
let responseNotOk = this.commandBuilderService.respondsFail(notExpectedResponse);
return this.communicateService.communicate(command, this.motherboard.serialPort, responseOk, responseNotOk);
}

Maybe you can try this
async getValuesThreshold(metricId: string, ATcmd: string, expectedResponse: string, notExpectedResponse: string) {
let command = this.communicateBuilder.BuildCommandGetMetricValue(ATcmd, this.usedSensorId, metricId);
console.log('motherboard get values' + command);
let responseOk = this.commandBuilderService.respondsSuccess(expectedResponse);
let responseNotOk = this.commandBuilderService.respondsFail(notExpectedResponse);
return await this.communicateService.communicate(command, this.motherboard.serialPort, responseOk, responseNotOk);
}

The problem was sending async data with serialport.
Serialport node had no way of knowing what response was linked to what data that was send.
So it acted on the first response it got and returned that.
Only way to solve this was to ask them sync.

Related

How to loop many http requests with axios in node.js

I have an array of users where each user has an IP address.
I have an API that I send an IP as a request and it returns a county code that belongs to this IP.
In order to get a country code to each user I need to send separate request to each user.
In my code I do async await but it takes about 10 seconds until I get all the responses, if I don't do the async await, I don’t get the country codes at all.
My code:
async function getAllusers() {
let allUsersData = await usersDao.getAllusers();
for (let i = 0; i < allUsersData.length; i++) {
let data = { ip: allUsersData[i].ip };
let body = new URLSearchParams(data);
await axios
.post("http://myAPI", body)
.then((res) => {
allUsersData[i].countryCode = res.data.countryCode;
});
}
return allUsersData;
}
You can use Promise.all to make all your requests once instead of making them one by one.
let requests = [];
for (let i = 0; i < allUsersData.length; i++) {
let data = { ip: allUsersData[i].ip };
let body = new URLSearchParams(data);
requests.push(axios.post("http://myAPI", body)); // axios.post returns a Promise
}
try {
const results = await Promise.all(requests);
// results now contains each request result in the same order
// Your logic here...
}
catch (e) {
// Handles errors
}
If you're just trying to get all the results faster, you can request them in parallel and know when they are all done with Promise.all():
async function getAllusers() {
let allUsersData = await usersDao.getAllusers();
await Promise.all(allUsersData.map((userData, index) => {
let body = new URLSearchParams({ip: userData.ip});
return axios.post("http://myAPI", body).then((res) => {
allUsersData[index].countryCode = res.data.countryCode;
});
}));
return allUsersData;
}
Note, I would not recommend doing it this way if the allUsersData array is large (like more than 20 long) because you'll be raining a lot of requests on the target server and it may either impeded its performance or you may get rate limited or even refused service. In that case, you'd need to send N requests at a time (like perhaps 5) using code like this pMap() here or mapConcurrent() here.

How to convert NodeJS form-data object into JSON string

I am using form-data package in my NodeJS application to send formdata. I am using Axios interceptor for logging the requests in a file. In axiosIns.config.data, I need the JSON string corresponding to the formdata set but currently it's FormData Object.
This library provides a toString method but on using that I have found that it returns a static string [object FormData] instead of the stringified input. I have opened an issue regarding that but seems unattended.
I have created a repl for regenerating that.
Is there any way so that I can convert my formdata object into a readable, loggable, preferably JSO string?
I solved It
const FormData = require("form-data");
var data = new FormData();
data.append("modid", "IM");
data.append("token", "provider\nagg");
data.append("cat_type", "3");
var boundary = data.getBoundary();
var data_row = data.getBuffer().toString();
console.log(rawFormDataToJSON(data_row,boundary));
function rawFormDataToJSON(raw_data,boundary){
var spl = data_row.split(boundary);
var data_out = [];
spl.forEach(element => {
let obj = {};
let ll = element.split("\n");
if(ll[1]){
let key = ll[1].split("=")[1].replace('"',"").replace('"\r',"");
let val = "";
if(ll.length > 3){
for (let i = 3; i < ll.length; i++) {
val += ll[i]+"\n";
}
}
obj[key] = val.replace("--","").replace("\r\n\n","");
data_out.push(obj);
}
});
return data_out;
}
Expected Output
[ { modid: 'IM' }, { token: 'provider\nagg' }, { cat_type: '3' } ]
Edit: I got reply on the github issue mentioned and as per that "this package doesn't intend to implement the toString() in a way to return stringified data. If I want spec-compliant FormData, I'll need to install the other packages mentioned. So it's not an issue but an intended unimplemented feature."
formdata-node
formdata-polyfill
undici
I tried below code, seems fine but not recommended as it's based on text splitting and filtering, if text format changes, it might create issue. This is the sandbox for the same.
const FormData = require("form-data");
var data = new FormData();
data.append("modid", "IM");
data.append("token", "provider");
data.append("cat_type", "3");
const objectifyFormdata = (data) => {
return data
.getBuffer()
.toString()
.split(data.getBoundary())
.filter((e) => e.includes("form-data"))
.map((e) =>
e
.replace(/[\-]+$/g, "")
.replace(/^[\-]+/g, "")
.match(/\; name\=\"([^\"]+)\"(.*)/s)
.filter((v, i) => i == 1 || i == 2)
.map((e) => e.trim())
)
.reduce((acc, cur) => {
acc[cur[0]] = cur[1];
return acc;
}, {});
};
console.log(objectifyFormdata(data));
// { modid: 'IM', token: 'provider', cat_type: '3' }

Correct way to organise this process in Node

I need some advice on how to structure this function as at the moment it is not happening in the correct order due to node being asynchronous.
This is the flow I want to achieve; I don't need help with the code itself but with the order to achieve the end results and any suggestions on how to make it efficient
Node routes a GET request to my controller.
Controller reads a .csv file on local system and opens a read stream using fs module
Then use csv-parse module to convert that to an array line by line (many 100,000's of lines)
Start a try/catch block
With the current row from the csv, take a value and try to find it in a MongoDB
If found, take the ID and store the line from the CSV and this id as a foreign ID in a separate database
If not found, create an entry into the DB and take the new ID and then do 6.
Print out to terminal the row number being worked on (ideally at some point I would like to be able to send this value to the page and have it update like a progress bar as the rows are completed)
Here is a small part of the code structure that I am currently using;
const fs = require('fs');
const parse = require('csv-parse');
function addDataOne(req, id) {
const modelOneInstance = new InstanceOne({ ...code });
const resultOne = modelOneInstance.save();
return resultOne;
}
function addDataTwo(req, id) {
const modelTwoInstance = new InstanceTwo({ ...code });
const resultTwo = modelTwoInstance.save();
return resultTwo;
}
exports.add_data = (req, res) => {
const fileSys = 'public/data/';
const parsedData = [];
let i = 0;
fs.createReadStream(`${fileSys}${req.query.file}`)
.pipe(parse({}))
.on('data', (dataRow) => {
let RowObj = {
one: dataRow[0],
two: dataRow[1],
three: dataRow[2],
etc,
etc
};
try {
ModelOne.find(
{ propertyone: RowObj.one, propertytwo: RowObj.two },
'_id, foreign_id'
).exec((err, searchProp) => {
if (err) {
console.log(err);
} else {
if (searchProp.length > 1) {
console.log('too many returned from find function');
}
if (searchProp.length === 1) {
addDataOne(RowObj, searchProp[0]).then((result) => {
searchProp[0].foreign_id.push(result._id);
searchProp[0].save();
});
}
if (searchProp.length === 0) {
let resultAddProp = null;
addDataTwo(RowObj).then((result) => {
resultAddProp = result;
addDataOne(req, resultAddProp._id).then((result) => {
resultAddProp.foreign_id.push(result._id);
resultAddProp.save();
});
});
}
}
});
} catch (error) {
console.log(error);
}
i++;
let iString = i.toString();
process.stdout.clearLine();
process.stdout.cursorTo(0);
process.stdout.write(iString);
})
.on('end', () => {
res.send('added');
});
};
I have tried to make the functions use async/await but it seems to conflict with the fs.openReadStream or csv parse functionality, probably due to my inexperience and lack of correct use of code...
I appreciate that this is a long question about the fundamentals of the code but just some tips/advice/pointers on how to get this going would be appreciated. I had it working when the data was sent one at a time via a post request from postman but can't implement the next stage which is to read from the csv file which contains many records
First of all you can make the following checks into one query:
if (searchProp.length === 1) {
if (searchProp.length === 0) {
Use upsert option in mongodb findOneAndUpdate query to update or upsert.
Secondly don't do this in main thread. Use a queue mechanism it will be much more efficient.
Queue which I personally use is Bull Queue.
https://github.com/OptimalBits/bull#basic-usage
This also provides the functionality you need of showing progress.
Also regarding using Async Await with ReadStream, a lot of example can be found on net such as : https://humanwhocodes.com/snippets/2019/05/nodejs-read-stream-promise/

Twilio Node JS - filter sms per phone number

I would like to filter sms per phone number and date the SMS was sent using REST API, however the output of the following code is not available outside of client.messages.each() block.
Please advise how I can use the latest sms code sent to the filtered number:
const filterOpts = {
to: '+13075550185',
dateSent: moment().utc().format('YYYY-MM-DD')
};
let pattern = /([0-9]{1,})$/;
let codeCollection = [];
client.messages.each(filterOpts, (record) => {
codeCollection.push(record.body.match(pattern)[0]);
console.log(record.body.match(pattern)[0], record.dateSent);
});
console.log(codeCollection,'I get an empty array here');//how to get
the latest sms and use it
doSomethingWithSMS(codeCollection[0]);
Twilio developer evangelist here.
The each function doesn't actually return a Promise. You can run a callback function after each has completed streaming results by passing it into the options as done like this:
const codeCollection = [];
const pattern = /([0-9]{1,})$/;
const filterOpts = {
to: '+13075550185',
dateSent: moment().utc().format('YYYY-MM-DD'),
done: (err) => {
if (err) { console.error(err); return; }
console.log(codeCollection);
doSomethingWithSMS(codeCollection[0]);
}
};
client.messages.each(filterOpts, (record) => {
codeCollection.push(record.body.match(pattern)[0]);
console.log(record.body.match(pattern)[0], record.dateSent);
});
Let me know if that helps at all.
Do you have access to the length of the array of messages? If so, you can do something like this
const filterOpts = {
to: '+13075550185',
dateSent: moment().utc().format('YYYY-MM-DD')
};
let pattern = /([0-9]{1,})$/;
let codeCollection = [];
var i = 0
client.messages.each(filterOpts, (record) => {
if (i < messages.length){
codeCollection.push(record.body.match(pattern)[0]);
console.log(record.body.match(pattern)[0], record.dateSent);
i++;
else {
nextFunction(codeCollection);
}
});
function nextFunction(codeCollection){
console.log(codeCollection,'I get an empty array here');
doSomethingWithSMS(codeCollection[0]);
}
messages.each() is running asynchronously, so your main thread moves on to the next call while the client.messages() stuff runs on a background thread. So, nothing has been pushed to codeCollection by the time you've tried to access it. You need to somehow wait for the each() to finish before moving on. Twilio client uses backbone style promises, so you can just add another .then() link to the chain, like below. You could also use a library like async which lets you use await to write asynchronous code in a more linear looking fashion.
const filterOpts = {
to: '+13075550185',
dateSent: moment().utc().format('YYYY-MM-DD')
};
let pattern = /([0-9]{1,})$/;
let codeCollection = [];
client.messages.each(filterOpts, (record) => {
codeCollection.push(record.body.match(pattern)[0]);
console.log(record.body.match(pattern)[0], record.dateSent);
}).then(
function() {
console.log(codeCollection,'I get an empty array here');
if( codeCollection.count > 0 ) doSomethingWithSMS(codeCollection[0]);
}
);

http call in backbone promise

Hi I have a backbone web app using Jquery and NodeJs/mongo as the server side framework. I'm having problems with making a http get call with a foreah loop and the results of the get call being iteratively added to each row of the loop.
var eventid = this.model.get("_id");
var inPromise = $.get("/registrants/list?eventid="+eventid,null,null,"json").then(
function (result){
var temp;
var finalVal = '';
var tempfinalVal = "";
var loop = 0
percentage = 0;
$.each(result.registrants,function(index,registrant){
temp = JSON.parse(registrant.fields);
for (var key in temp) {
if(key =="Email"){
if(temp[key] != ""){
$.get("/stats/registrant?userid="+temp[key]+"&eventid="+eventid,null,null,"json").then(function(result2){
percentage = (result2.Stats.type ===undefined || result2.Stats.type ==null) ? "0": result2.Stats.type;
finalVal +=percentage+"\n";
}).fail(function(){
percentage = "0";
});
}
}else if(key =="eventid"){
loop++;
finalVal = finalVal.slice(0, - 1);
finalVal +='\n';
}
finalVal +=temp[key] + ',';
}
});
//promises.push(inPromise);
}
).done(function(finalVal){
$("#webcast-download-registrants-tn").attr("href",'data:text/csv;charset=utf-8;filename=registration.csv",'+encodeURIComponent(finalVal));
console.log("DONE");
}).fail(function(){
console.log("fail");
});
// promise.done(function () {
// console.log(" PROMISE DONE");
// });
So I have the loop through a collection and the last item of the docuemnt gets a content froma nother http call and when all is fone it will create a CSV file. The problem is that THE "DONE" text echos firts then the "CALL" text is displayed
Rick, your problem is not the simplest due to :
the need for nested asynchronous gets
the need to build each CSV data row partly synchronously, partly asynchronously.
the need for a mechanism to handle the fulfilment of multiple promises generated in the inner loop.
From what you've tried, I guess you already know that much.
One important thing to note is that you can't rely on for (var key in temp) to deliver properties in any particular order. Only arrays have order.
You might try something like this :
var url = "/stats/registrant",
data = { 'eventid': this.model.get('_id') },
rowTerminator = "\n",
fieldNames = ['firstname','lastname','email','company','score'];
function getScore(email) {
return $.get(url, $.extend({}, data, {'userid':email}), null, "json").then(function(res) {
return res.Stats ? res.Stats.type || 0 : 0;
}, function() {
//ajax failure - assume score == 0
return $.when(0);
});
}
$.get("/registrants/list", data, null, "json").then(function(result) {
var promises = [];//An array in which to accumulate promises of CSV rows
promises.push($.when(fieldNames)); //promise of CSV header row
if(result.registrants) {
$.each(result.registrants, function(index, registrant) {
if(registrant.fields) {
// Synchronously initialize row with firstname, lastname, email and company
// (omitting score for now).
var row = fieldNames.slice(0,-1).map(function(fieldName, i) {
return registrant.fields[fieldName] || '';
});
//`row` remains available to inner functions due to closure
var promise;
if(registrant.fields.Email) {
// Fetch the registrant's score ...
promise = getScore(registrant.fields.Email).then(function(score) {
//... and asynchronously push the score onto row
row.push(score);
return row;
});
} else {
//or synchronously push zero onto row ...
row.push(0);
//... and create a resolved promise
promise = $.when(row);
}
promises.push(promise);//Accumulate promises of CSV data rows (still in array form), in the correct order.
}
});
}
return $.when.apply(null, promises).then(function() {
//Join all the pieces, in nested arrays, together into one long string.
return [].slice.apply(arguments).map(function(row) {
return row.join(); //default glue is ','
}).join(rowTerminator);
});
}).done(function(str) {
$("#webcast-download-registrants-tn").attr("href",'data:text/csv;charset=utf-8;filename=registration.csv",'+encodeURIComponent(str));
console.log("DONE");
}).fail(function() {
console.log("fail");
});
partially tested
See comments in code for explanation and please ask if there's anything you don't follow.

Resources