I am trying to:
Poll a public API every 5 seconds
Store the resulting JSON in a variable
Store the next query to this same API in a second variable
Compare the first variable to the second
Print the second variable if it is different from the first
Else: Print the phrase: 'The objects are the same' if they haven't changed
Unfortunately, the comparison part appears to fail. I am realizing that this implementation is probably lacking the appropriate variable scoping but I can't put my finger on it. Any advice would be highly appreciated.
data: {
chatters: {
viewers: {
},
},
},
};
//prints out pretty JSON
function prettyJSON(obj) {
console.log(JSON.stringify(obj, null, 2));
}
// Gets Users from Twitch API endpoint via axios request
const getUsers = async () => {
try {
return await axios.get("http://tmi.twitch.tv/group/user/sixteenbitninja/chatters");
} catch (error) {
console.error(error);
}
};
//Intended to display
const displayViewers = async (previousResponse) => {
const usersInChannel = await getUsers();
if (usersInChannel.data.chatters.viewers === previousResponse){
console.log("The objects are the same");
} else {
if (usersInChannel.data.chatters) {
prettyJSON(usersInChannel.data.chatters.viewers);
const previousResponse = usersInChannel.data.chatters.viewers;
console.log(previousResponse);
intervalFunction(previousResponse);
}
}
};
// polls display function every 5 seconds
const interval = setInterval(function () {
// Calls Display Function
displayViewers()
}, 5000);```
The issue is that you are using equality operator === on objects. two objects are equal if they have the same reference. While you want to know if they are identical. Check this:
console.log({} === {})
For your usecase you might want to store stringified version of the previousResponse and compare it with stringified version of the new object (usersInChannel.data.chatters.viewers) like:
console.log(JSON.stringify({}) === JSON.stringify({}))
Note: There can be issues with this approach too, if the order of property changes in the response. In which case, you'd have to check individual properties within the response objects.
May be you can use npm packages like following
https://www.npmjs.com/package/#radarlabs/api-diff
Related
I need some advice on how to structure this function as at the moment it is not happening in the correct order due to node being asynchronous.
This is the flow I want to achieve; I don't need help with the code itself but with the order to achieve the end results and any suggestions on how to make it efficient
Node routes a GET request to my controller.
Controller reads a .csv file on local system and opens a read stream using fs module
Then use csv-parse module to convert that to an array line by line (many 100,000's of lines)
Start a try/catch block
With the current row from the csv, take a value and try to find it in a MongoDB
If found, take the ID and store the line from the CSV and this id as a foreign ID in a separate database
If not found, create an entry into the DB and take the new ID and then do 6.
Print out to terminal the row number being worked on (ideally at some point I would like to be able to send this value to the page and have it update like a progress bar as the rows are completed)
Here is a small part of the code structure that I am currently using;
const fs = require('fs');
const parse = require('csv-parse');
function addDataOne(req, id) {
const modelOneInstance = new InstanceOne({ ...code });
const resultOne = modelOneInstance.save();
return resultOne;
}
function addDataTwo(req, id) {
const modelTwoInstance = new InstanceTwo({ ...code });
const resultTwo = modelTwoInstance.save();
return resultTwo;
}
exports.add_data = (req, res) => {
const fileSys = 'public/data/';
const parsedData = [];
let i = 0;
fs.createReadStream(`${fileSys}${req.query.file}`)
.pipe(parse({}))
.on('data', (dataRow) => {
let RowObj = {
one: dataRow[0],
two: dataRow[1],
three: dataRow[2],
etc,
etc
};
try {
ModelOne.find(
{ propertyone: RowObj.one, propertytwo: RowObj.two },
'_id, foreign_id'
).exec((err, searchProp) => {
if (err) {
console.log(err);
} else {
if (searchProp.length > 1) {
console.log('too many returned from find function');
}
if (searchProp.length === 1) {
addDataOne(RowObj, searchProp[0]).then((result) => {
searchProp[0].foreign_id.push(result._id);
searchProp[0].save();
});
}
if (searchProp.length === 0) {
let resultAddProp = null;
addDataTwo(RowObj).then((result) => {
resultAddProp = result;
addDataOne(req, resultAddProp._id).then((result) => {
resultAddProp.foreign_id.push(result._id);
resultAddProp.save();
});
});
}
}
});
} catch (error) {
console.log(error);
}
i++;
let iString = i.toString();
process.stdout.clearLine();
process.stdout.cursorTo(0);
process.stdout.write(iString);
})
.on('end', () => {
res.send('added');
});
};
I have tried to make the functions use async/await but it seems to conflict with the fs.openReadStream or csv parse functionality, probably due to my inexperience and lack of correct use of code...
I appreciate that this is a long question about the fundamentals of the code but just some tips/advice/pointers on how to get this going would be appreciated. I had it working when the data was sent one at a time via a post request from postman but can't implement the next stage which is to read from the csv file which contains many records
First of all you can make the following checks into one query:
if (searchProp.length === 1) {
if (searchProp.length === 0) {
Use upsert option in mongodb findOneAndUpdate query to update or upsert.
Secondly don't do this in main thread. Use a queue mechanism it will be much more efficient.
Queue which I personally use is Bull Queue.
https://github.com/OptimalBits/bull#basic-usage
This also provides the functionality you need of showing progress.
Also regarding using Async Await with ReadStream, a lot of example can be found on net such as : https://humanwhocodes.com/snippets/2019/05/nodejs-read-stream-promise/
i have an array of variable number of urls and i must merge the data get with axion
the problem is then every axios call is relative to the data of the previus
if i have a fixed number of ulrs i can nest axion calls and live with that
i think to use something like this
var urls = ["xx", "xx", "xx"];
mergeData(urls);
function mergeData(myarray, myid = 0, mydata = "none") {
var myurl = "";
if (Array.isArray(mydata)) {
myurl = myarray[myid];
// do my stuff with data and modify the url
} else {
myurl = myarray[myid];
}
axios.get(myurl)
.then(response => {
// do my stuff and get the data i need and put on an array
if (myarray.length < myid) {
mergeData(myarray, myid + 1, data);
} else {
// show result on ui
}
})
.catch(error => {
console.log(error);
});
}
but i dont like it
there is another solution?
(be kind, i'm still learning ^^)
just to be clear
i need to optain
http request to "first url", parse the the json, save some data(some needed for the output)
another http request to "second url" with one or more parameter from previous data, parse the the json, save some data(some needed for the output)
... and so on, for 5 to 10 times
If your goal is to make subsequent HTTP calls based on information you get from previous calls, I'd utilize async/await and for...of to accomplish this instead of relying on a recursive solution.
async function mergeData(urls) {
const data = [];
for (const url of urls) {
const result = await axios.get(url).then(res => res.data);
console.log(`[${result.id}] ${result.title}`);
// here, do whatever you want to do with
// `result` to make your next call...
// for now, I am just going to append each
// item to `data` and return it at the end
data.push(result);
}
return data;
}
const items = [
"https://jsonplaceholder.typicode.com/posts/1",
"https://jsonplaceholder.typicode.com/posts/2",
"https://jsonplaceholder.typicode.com/posts/3"
];
console.log("fetching...")
mergeData(items)
.then(function(result) {
console.log("done!")
console.log("final result", result);
})
.catch(function(error) {
console.error(error);
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.19.2/axios.min.js"></script>
Using async/await allows you to utilize for...of which will wait for each call to resolve or reject before moving onto the next one.
To learn more about async/await and for...of, have a look here:
developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...of
Hope this helps.
I'm new to node.js and DialogFlow. I am using DynamoDB to store data and I'm creating skills on Google. I'm trying to write a code to retrieve a specific item on that table.
I've got it working to show all items where ID is equal = 1, but how would I make it so I can just get attribute 'name'?
My idea is a user provides an id then the code will retrieve name where id was 1 and store that as a variable and use agent.add('hello $(name)'); to display it back as speech to the user.
function readdata(agent){
let dbread = new aws.DynamoDB.DocumentClient();
const id = agent.parameters.id;
let read = function(){
var parameters = {
TableName:"Dynamodb",
Key:{
"id":id
}
};
dbread.get(parameters, function(err,data){
if(err){
console.log("error",JSON.stringify(data,null,2));
}else{
console.log("success",JSON.stringify(data,null,2));
}
});
agent.add(`hello ${name}`);
};
read();
}
Once you have the data back from the get() call, the data object will contain an Item attribute. The value of this attribute will be another object that contains the attribute/value pairs for this record, or be empty if the record isn't found.
The debugging you have in place that shows JSON.stringify(data) should show this.
Assuming you knew all the fields were there, you could do something like
const name = data.Item.name;
a more robust way using current JavaScript would be to make sure everything was assigned, otherwise return undefined at any point. So something like this would work
const name = data && data.Item && data.Item.name;
However - you will have a problem doing this with Dialogflow
You don't show which Dialogflow library you're using, but most of them require you to return a Promise to indicate that it needs to wait for asynchronous calls (such as the call to DynamoDB) to complete. You're using get() with a callback function instead of a Promise. So you need to do one of the following:
Wrap the call in a Promise
Since get() returns an AWS.Request you can use the promise() method of this to get a Promise that you can return and which has then portions that generate the response - similar to how you're doing your callbacks now.
Under this scheme, your call might look something like this (untested):
return dbread.get(parameters).promise()
.then( data => {
console.log("success",JSON.stringify(data,null,2));
const name = data && data.Item && data.Item.name;
if( name ){
agent.add( `Hello ${name}` );
} else {
agent.add( "I don't know who you are." );
}
})
.catch( err => {
console.log("error",JSON.stringify(data,null,2));
agent.add( "There was an error" );
});
My removeObjects function has me stummped.The function is suppose to syncronoulsy get a list of objects in an S3 bucket then asyncronously removes the objects. Repeat if the list was truncated, until the there are no more objects to remove. (AWS doesn't provide the total count of objects in the bucket and listObjects pages the results.)
What am I doing wrong / why doesn't my function work? The solution should exploit single thread and async nature of JS. For the bounty I am hoping for an answer specific to the module. The git repo is public if you want to see the entire module.
export function removeObjects(params: IS3NukeRequest): Promise<S3.Types.DeleteObjectsOutput> {
const requests: Array<Promise<S3.Types.DeleteObjectsOutput>> = [];
let isMore;
do {
listObjectsSync(params)
.then((objectList: S3.Types.ListObjectsV2Output) => {
isMore = objectList.ContinuationToken = objectList.IsTruncated ? objectList.NextContinuationToken : null;
requests.push(params.Client.deleteObjects(listObjectsV2Output2deleteObjectsRequest(objectList)).promise());
})
.catch((err: Error) => { Promise.reject(err); });
} while (isMore);
return Promise.all(requests);
}
export async function listObjectsSync(params: IS3NukeRequest): Promise<S3.Types.ListObjectsV2Output> {
try {
return await params.Client.listObjectsV2(s3nukeRequest2listObjectsRequest(params)).promise();
} catch (err) {
return Promise.reject(err);
}
}
Thanks.
The thing is that listObjectsSync function returns a Promise, so you need to treat it as an async function and can't just use a loop with it. What you need to do is to create a chain of promises while your isMore is true, I've done it using a recursive approach (I'm not pro in TS, so please check the code before using it). I also haven't tried the code live, but logically it should work :)
const requests: Array<Promise<S3.Types.DeleteObjectsOutput>> = [];
function recursive(recursiveParams) {
return listObjectsSync(recursiveParams).then((objectList: S3.Types.ListObjectsV2Output) => {
let isMore = objectList.ContinuationToken = objectList.IsTruncated ? objectList.NextContinuationToken : null;
requests.push(params.Client.deleteObjects(listObjectsV2Output2deleteObjectsRequest(objectList)).promise());
if (isMore) {
//do we need to change params here?
return recursive(recursiveParams)
}
//this is not necessary, just to indicate that we get out of the loop
return true;
});
}
return recursive(params).then(() => {
//we will have all requests here
return Promise.all(requests);
});
i feel a bit embarrassed, can you please kindly explain parts of the code?
For example, I have no idea, what is this part? where can I read more about it?
function parsePostStory(data) {
return {
name : data.name
}
}
What is req.body? Is it json req body?
Why do we declare empty array and why do we return it? Just for the clarity?
Is Story.create just a mongoose method?
The rest of the code is here:
router.post('/stories', function(req, res) {
var validation = validatePostStory(req.body);
if(validation.length > 0) {
return res.badRequestError(validation);
}
var story = parsePostStory(req.body);
Story.create(story, function(err, story) {
if(err) {
console.log(err.message);
return res.internalServerError();
} res.send(story);
});
});
function validatePostStory(data) {
var array = [];
if (!data.name || typeof data.name !== 'String') {
return array.push('name');
}
return array;
}
function parsePostStory(data) {
return {
name : data.name
}
}
Sorry once more for that kind of a question and thanks a ton.
I'm assuming you know how the request-response cycle works with HTTP requests and the client-server interactions with it. If not, Wikipedia Request-Response and Client-Server (Two link limit, otherwise I would have posted them as links)
A request sends a lot of information to the server. If you console.log the request in NodeJS, you will see that it contains a lot of information that isn't entirely relevant to what you need.
You're using Express as your web framework. In this case, req.body is the information that you are sending to the server from the client. Using req.body will make sure that you're not using the extra information passed in to the server from the client. Req.body is your code that you want. (Note: Req.body isn't natively supported by Express v4, you'll have to use something like body-parser) See Express docs for more details
Now, let's break up this code a bit. You essentially have 3 separate functions. Let's take a look at validatePostStory.
function validatePostStory(data) {
var array = [];
if (!data.name || typeof data.name !== 'String') {
return array.push('name');
}
return array;
}
This function is a validation function. It takes one argument - an object and returns an array. Effectively, what this is doing is checking if the name is a string or not - if not, return an array that has a length of 1. The following conditional checks length and returns a 400 if greater than 0
if(validation.length > 0) {
return res.badRequestError(validation);
}
I'm not entirely sure why this needs to be a separate function. Looks like you can probably just do this instead.
if (!req.body.name || typeof req.body.name !== 'String') {
return res.badRequestError(validation);
}
The following function function essentially converts the data so that mongodb/mongoose can store it in the proper format
function parsePostStory(data) {
return {
name : data.name
}
}
It's the same as saying:
var story = {name: req.body.name}
I would assume Story.create is a custom mongoose method yes.