Stuck inside Promise - node.js

Using native node promises. Here is my code:
(req, res) =>
requestp('https://swapi.co/api/planets')
.then((planets) => Promise.all(planets.results.map(planet => {
var residents = planet.residents.map(requestp(r))
return {
planetName: planet.name,
residents: Promise.all(residents).then((r) => r.name)
}
}))
.then((planets) => res.json(planets.map(p => {
let obj = {}
return res.json(obj[p.planetName] = p.residents);
}))))
.catch((err) => res.status(400).send(err))
I am trying to return an array of objects that look like this:
{ Alderaan: ["nameofResident1","nameofResident2"]}
I seem to be getting stuck in planet.residents.map() part of my code though.
Any help would be appreciated.

If you break down the inner workings to some functions, not only does it make it easier to write, it makes it easier to read
In the following, to make it executable, I called the main function doit -
I also added a requestp function, which uses fetch - so the snippet wont work unless you use a modern browser (no Internet Explorer allowed here)
run the snippet, you should see the result you are looking for
// you wont need this function
var requestp = (url) => fetch(url).then(response=>response.json());
// code starts here
var getResidentName = (url) => requestp(url).then((resident) => resident.name);
var getAllResidentNames = (residents) => Promise.all(residents.map(r => getResidentName(r)));
var processPlanet = (planet) => getAllResidentNames(planet.residents).then((residents) => ({[planet.name]: residents}));
var processPlanets = (planets) => Promise.all(planets.results.map(planet => processPlanet(planet)));
// slight change so it can run in the snippet, ignore the var doit = for your code
var doit = (req, res) =>
requestp('https://swapi.co/api/planets')
.then(processPlanets)
.catch((err) => res.status(400).send(err));
// doit and log the result as a JSON string
doit().then(result => console.log(JSON.stringify(result)));
The code you would use from above can also be written
(req, res) => {
var getResidentName = (url) => requestp(url).then((resident) => resident.name);
var getAllResidentNames = (residents) => Promise.all(residents.map(r => getResidentName(r)));
var processPlanet = (planet) => getAllResidentNames(planet.residents).then((residents) => ({[planet.name]: residents}));
var processPlanets = (planets) => Promise.all(planets.results.map(planet => processPlanet(planet)));
return requestp('https://swapi.co/api/planets')
.then(processPlanets)
.catch((err) => res.status(400).send(err));
}
That way all the "support" functions are not polluting the global namespace
After you have it working, you can, if you must, refactor the code into one monolithic block quite easily

Related

What is the ideal way to loop API requests with fetch?

I'm relatively new to working with NodeJS, and I'm doing a practice project using the Youtube API to get some data on a user's videos. The Youtube API returns a list of videos with a page token, to successfully collect all of a user's videos, you would have to make several API requests, each with a different page token. When you reach the end of these requests, there will be no new page token present in the response, so you can move on. Doing it in a for, or while loop seemed like the way to handle this, but these are synchronous operations that do not appear to work in promises, so I had to look for an alternative
I looked at a few previous answers to similar questions, including the ones here and here. I got the general idea of the code in the answers, but I couldn't quite figure out how to get it working fully myself. The request I am making is already chained in a .then() of a previous API call - I would like to complete the recursive fetch calls with new page tokens, and then move onto another .then(). Right now, when I run my code, it moves onto the next .then() without the requests that use the tokens being complete. Is there any way to stop this from happening? I know async/await may be a solution, but I've decided to post here just to see if there are any possible solutions without having to go down that route in the hope I learn a bit about fetch/promises in general. Any other suggestions/advice about the way the code is structured is welcome too, as I'm pretty conscious that this is probably not the best way to handle making all of these API calls.
Code :
let body = req.body
let resData = {}
let channelId = body.channelId
let videoData = []
let pageToken = ''
const fetchWithToken = (nextPageToken) => {
let uploadedVideosUrlWithToken = `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${uploadedVideosPlaylistId}&pageToken=${nextPageToken}&maxResults=50&key=${apiKey}`
fetch(uploadedVideosUrlWithToken)
.then(res => res.json())
.then(uploadedVideosTokenPart => {
let {items} = uploadedVideosTokenPart
videoData.push(...items.map(v => v.contentDetails.videoId))
pageToken = (uploadedVideosTokenPart.nextPageToken) ? uploadedVideosTokenPart.nextPageToken : ''
if (pageToken) {
fetchWithToken(pageToken)
} else {
// tried to return a promise so I can chain .then() to it?
// return new Promise((resolve) => {
// return(resolve(true))
// })
}
})
}
const channelDataUrl = `https://youtube.googleapis.com/youtube/v3/channels?part=snippet%2CcontentDetails%2Cstatistics&id=${channelId}&key=${apiKey}`
// promise for channel data
// get channel data then store it in variable (resData) that will eventually be sent as a response,
// contentDetails.relatedPlaylists.uploads is the playlist ID which will be used to get individual video data.
fetch(channelDataUrl)
.then(res => res.json())
.then(channelData => {
let {snippet, contentDetails, statistics } = channelData.items[0]
resData.snippet = snippet
resData.statistics = statistics
resData.uploadedVideos = contentDetails.relatedPlaylists.uploads
return resData.uploadedVideos
})
.then(uploadedVideosPlaylistId => {
// initial call to get first set of videos + first page token
let uploadedVideosUrl = `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${uploadedVideosPlaylistId}&maxResults=50&key=${apiKey}`
fetch(uploadedVideosUrl)
.then(res => res.json())
.then(uploadedVideosPart => {
let {nextPageToken, items} = uploadedVideosPart
videoData.push(...items.map(v => v.contentDetails.videoId))
// idea is to do api calls until pageToken is non existent, and add the video id's to the existing array.
fetchWithToken(nextPageToken)
})
})
.then(() => {
// can't seem to get here synchronously - code in this block will happen before all the fetchWithToken's are complete - need to figure this out
})
Thanks to anyone who takes the time out to read this.
Edit:
After some trial and error, this seemed to work - it is a complete mess. The way I understand it is that this function now recursively creates promises that resolve to true only when there is no page token from the api response allowing me to return this function from a .then() and move on to a new .then() synchronously. I am still interested in better solutions, or just suggestions to make this code more readable as I don't think it's very good at all.
const fetchWithToken = (playlistId, nextPageToken) => {
let uploadedVideosUrlWithToken = `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${playlistId}&pageToken=${nextPageToken}&maxResults=50&key=${apiKey}`
return new Promise((resolve) => {
resolve( new Promise((res) => {
fetch(uploadedVideosUrlWithToken)
.then(res => res.json())
.then(uploadedVideosTokenPart => {
let {items} = uploadedVideosTokenPart
videoData.push(...items.map(v => v.contentDetails.videoId))
pageToken = (uploadedVideosTokenPart.nextPageToken) ? uploadedVideosTokenPart.nextPageToken : ''
// tried to return a promise so I can chain .then() to it?
if (pageToken) {
res(fetchWithToken(playlistId, pageToken))
} else {
res(new Promise(r => r(true)))
}
})
}))
})
}
You would be much better off using async/await which are basically a wrapper for promises. Promise chaining, which is what you are doing with the nested thens, can get messy and confusing...
I converted your code to use async/await so hopefully this will help you see how to solve your problem. Good luck!
Your initial code:
let { body } = req
let resData = {}
let { channelId } = body
let videoData = []
let pageToken = ''
const fetchWithToken = async (nextPageToken) => {
const someData = (
await fetch(
`https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${uploadedVideosPlaylistId}&pageToken=${nextPageToken}&maxResults=50&key=${apiKey}`,
)
).json()
let { items } = someData
videoData.push(...items.map((v) => v.contentDetails.videoId))
pageToken = someData.nextPageToken ? someData.nextPageToken : ''
if (pageToken) {
await fetchWithToken(pageToken)
} else {
// You would need to work out
}
}
const MainMethod = async () => {
const channelData = (
await fetch(
`https://youtube.googleapis.com/youtube/v3/channels?part=snippet%2CcontentDetails%2Cstatistics&id=${channelId}&key=${apiKey}`,
)
).json()
let { snippet, contentDetails, statistics } = channelData.items[0]
resData.snippet = snippet
resData.statistics = statistics
resData.uploadedVideos = contentDetails.relatedPlaylists.uploads
const uploadedVideosPlaylistId = resData.uploadedVideos
const uploadedVideosPart = (
await fetch(
`https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${uploadedVideosPlaylistId}&maxResults=50&key=${apiKey}`,
)
).json()
let { nextPageToken, items } = uploadedVideosPart
videoData.push(...items.map((v) => v.contentDetails.videoId))
await fetchWithToken(nextPageToken)
}
MainMethod()
Your Edit:
const fetchWithToken = (playlistId, nextPageToken) => {
return new Promise((resolve) => {
resolve(
new Promise(async (res) => {
const uploadedVideosTokenPart = (
await fetch(
`https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${playlistId}&pageToken=${nextPageToken}&maxResults=50&key=${apiKey}`,
)
).json()
let { items } = uploadedVideosTokenPart
videoData.push(...items.map((v) => v.contentDetails.videoId))
pageToken = uploadedVideosTokenPart.nextPageToken
? uploadedVideosTokenPart.nextPageToken
: ''
if (pageToken) {
res(fetchWithToken(playlistId, pageToken))
} else {
res(new Promise((r) => r(true)))
}
}),
)
})
}

TypeError: Cannot set property 'brief_title' of null - node and Mongoose

I have an app using Node/Express/Mongo and I'm running into trouble when I want to edit a document. I can add documents no problem but when I built out the Edit form I get the error in the title. I can fetch the document as well and see the information I've put in. The problem is when I try to save whatever changes I've made.
Here's my edit function: The error occurs at the brief.brief_title = updated_brief_title in the FindByIdAndUpdate method.
exports.postEditBrief = (req, res, next) => {
const briefId = req.body.briefId;
const updated_brief_title = req.body.brief_title;
const updated_country = req.body.country;
const updated_psg = req.body.psg;
const updated_one_year_withholding = req.body.one_year_withholding;
const updated_withholding_only = req.body.withholding_only;
const updated_practice_advisory = req.body.practice_advisory;
const updated_courthouse = req.body.courthouse;
const updated_pages = req.body.pages;
const updated_additional_psg = req.body.additional_psg;
const updated_gangs = req.body.gangs;
const updated_gang_name = req.body.gang_name;
const updated_link = req.body.link;
Brief.findByIdAndUpdate(briefId)
.then(brief => {
brief.brief_title = updated_brief_title;
brief.country = updated_country;
brief.psg = updated_psg;
brief.one_year_withholding = updated_one_year_withholding;
brief.withholding_only = updated_withholding_only;
brief.practice_advisory = updated_practice_advisory;
brief.courthouse = updated_courthouse;
brief.pages = updated_pages;
brief.additional_psg = updated_additional_psg;
brief.gangs = updated_gangs;
brief.gang_name = updated_gang_name;
brief.link = updated_link;
return brief.save();
})
.then(result => {
console.log(Brief);
res.redirect('/');
})
.catch(err => console.log(err));
};
I've played around with the various "find and update" methods in Mongoose but the results are the same.
You need to create an object of the data you want to update and pass it as a second argument in the function, when you do .then() it gets you the result of the operation you are doing and you don't need to call .save() as it does that internally
Correct version should look something like this
exports.postEditBrief = (req, res, next) => {
const briefId = req.body.briefId;
let update = {
brief_title: req.body.brief_title,
country: req.body.country,
psg: req.body.psg,
one_year_withholding: req.body.one_year_withholding,
withholding_only: req.body.withholding_only,
practice_advisory: req.body.practice_advisory,
courthouse: req.body.courthouse,
pages: req.body.pages,
additional_psg: req.body.additional_psg,
gangs: req.body.gangs,
gang_name: req.body.gang_name,
link: req.body.link
}
Brief.findByIdAndUpdate(briefId, update)
.then(result => {
console.log(result);
res.redirect('/');
})
.catch(err => console.log(err));
};
You can simplify it even further using object destructuring.
Update:
You need to use $set with findOneandUpdate if you don't want to overwrite your document
exports.postEditBrief = (req, res, next) => {
const briefId = req.body.briefId;
let update = {
brief_title: req.body.brief_title,
country: req.body.country,
psg: req.body.psg,
one_year_withholding: req.body.one_year_withholding,
withholding_only: req.body.withholding_only,
practice_advisory: req.body.practice_advisory,
courthouse: req.body.courthouse,
pages: req.body.pages,
additional_psg: req.body.additional_psg,
gangs: req.body.gangs,
gang_name: req.body.gang_name,
link: req.body.link
}
Brief.findOneAndUpdate(briefId, {$set:update})
.then(result => {
console.log(result);
res.redirect('/');
})
.catch(err => console.log(err));
};

how much time each function takes in promis.all? [duplicate]

This question already has answers here:
How to measure the execution time of a promise?
(5 answers)
Closed 3 years ago.
I have some URLs and I want to call each of them simultaneous. I want to know how much time each request takes?
my code like this:
var urls=["http://req0.com","http://req1.com","http://req2.com"];
Promis.all(urls.map(e=>return axios.post(e,{test:""test}).catch(err=>return e)).then(
(values)=>{
console.log(values[0]);
console.log(values[1]);
console.log(values[2]);
})
what I want is something like this
conosle.log(value[0].responseTime);
conosle.log(value[1].responseTime)
conosle.log(value[2].responseTime)
is there any way to get this time?
Pretty simple, your .map functor offers the opportunity for a reliable closure for the start time of each axios request, allowing calculation of time taken by subtraction in the requests' .then callback.
var urls = ["http://req0.com","http://req1.com","http://req2.com"];
Promise.all(urls.map(e => {
let start = Date.now();
return axios.post(e, {test:'test'})
.then(value => ( { value, t: Date.now() - start} ));
}))
.then((timedValues) => {
let times = timedValues.map(x => x.t);
let values = timedValues.map(x => x.value);
console.log(times);
console.log(values);
});
If you wish to include the timing of errors, then it's only slightly more complicated:
var urls=["http://req0.com","http://req1.com","http://req2.com"];
Promise.all(urls.map(e => {
let t = Date.now();
return axios.post(e, {test:"test"})
.then(value => ( { outcome:'success', value, t:Date.now() - t} ))
.catch(error => ( { outcome:'error', error, t:Date.now() - t} ));
}))
.then((timedOutcomes) => {
let times = timedOutcomes.map(x => x.t);
let values = timedOutcomes.filter(x => x.outcome === 'success').map(x => x.value);
let errors = timedOutcomes.filter(x => x.outcome === 'error').map(x => x.error);
console.log(times);
console.log(values);
console.log(errors);
});
you can use async/await and measure the time with console.time(), console.timeEnd().
async getPost(){
const url = 'https://jsonplaceholder.typicode.com/posts?_start=1';
console.time();
const post = await axios.get(url);
console.timeEnd();
return post;
};
const post = getPost();
console.log(`post ${post}`);

Why ctx.state did not pass to another middleware?

use koa2 ejs koa-router, ejs template how to use another middleware's ctx.state
localhost:3000/admin/usermsg
admin.get('/usermsg', async(ctx) => {
ctx.state.userMsg = {
page: Number(ctx.query.page),
limit: 4,
pages: 0,
count: count
}
var userMsg = ctx.state.userMsg;
ctx.state.users = await new Promise(function(resolve, reject){
userMsg.pages = Math.ceil(userMsg.count / userMsg.limit);
userMsg.page = userMsg.page > userMsg.pages ? userMsg.pages : userMsg.page;
userMsg.page = userMsg.page < 1 ? 1 : userMsg.page;
var skip = (userMsg.page - 1) * userMsg.limit;
User.find().limit(userMsg.limit).skip(skip).exec(function(err, doc){
if(doc){
resolve(doc);
}
if(err){
reject(err);
}
})
})
await ctx.render('admin/usermsg');
})
localhost:3000/damin/category
admin.get('/category', async(ctx) => {
await ctx.render('admin/category');
})
in the category templateļ¼Œcan not get ctx.state.userMsg.
how should i get ctx.state.userMsg in category template?
Well, assuming userMsg is something you use a lot in your views, you could make a dedicated middleware just to obtain that value.
Middleware work in 'stacks': by calling next(), you can pass control to the next one in the stack (with access to the modified ctx.state). A trivial example:
const setUserMsg = async (ctx, next) => {
ctx.state.userMsg = await myFuncThatReturnsAPromise()
await next()
}
router.get('/someroute',
setUserMsg,
ctx => { ctx.body = ctx.state.userMsg })
router.get('/someotherroute',
setUserMsg,
ctx => { ctx.body = ctx.state.userMsg })
Here, setUserMsg's sole purpose is to extract a value (presumably from the database) and add it to the context.

CustomPouchError 409 Conflict Document Update Conflict

I'm not too good with Promises (first time working with them). With some other help, I've modified the code some, but I'm still getting a 409 conflict error on the putAttachment() method in the readFileThenAttach() method. So, I need another set of eyes to look at this for me. I had been consistently getting this error, then I come in this morning and tried it again. It worked twice (I didn't get the error). I stopped the program then when I ran it again, I got the error again. I'm not sure what's wrong. The revisions seem to be okay, so I'm not sure if its some timing issue or what with how the Promises are in my code. The execution path of this code begins in the importProject() method and from there importInspectionPhotos() is called. Can someone take a look to see if they can see anything that stands out? Thanks.
function buildReinspectionLinks(db: InspectionDb) {
return db.allObservations()
.then(([points, lines]) => {
let observations = new Map([...points, ...lines]
.filter(obs => (<any>obs).access_original)
.map<[string, Observation]>(obs => [(<any>obs).access_id, obs]))
let changed = new Set()
for (let obs of observations.values()) {
let doc = (<any>obs).access_original
if (doc.Inspect_ID != doc.Original_ID) {
let reinspected = observations.get(doc.Original_ID)
doc.reinspected_id = reinspected._id
reinspected.reinspected = true
if (!reinspected.reinspection_ids) {
reinspected.reinspection_ids = []
}
reinspected.reinspection_ids.push(obs._id)
changed.add(obs)
changed.add(reinspected)
}
}
// TODO: Recurse the relationships?
return Promise.all([...changed].map(obs => db.post(obs)))
})
}
function importInspectionPhotos(db: InspectionDb, directoryBase: string) {
const observations = db.allObservations().then(([points, lines]) => new Map([...points, ...lines].filter(obs => (<any>obs).access_original).map<[string, Observation]>(obs => [(<any>obs).access_id, obs])))
const filenames = globP("**/*.{jpg, jpeg, gif, png}", { cwd: directoryBase })
return Promise.all([observations, filenames]).then(([obs, names]: [Map<string, Observation>, string[]]) => {
const fileObservations: FileObservation[] = names.map(file => {
const filename = basename(file)
const accessID = getAccessObservationId(filename)
return {
file,
path: `${directoryBase}/${file}`,
observation: obs.get(accessID)
} as FileObservation
}).filter((fileOb: FileObservation) => !!fileOb.observation)
return fileObservations.reduce((lastPromise, fileOb) => lastPromise.then(() => readFileThenAttach(db, fileOb)), Promise.resolve())
})
}
function getAccessObservationId(filename: string): string {
return filename.substr(0, filename.lastIndexOf('_'))
}
function readFileThenAttach(db: InspectionDb, fileOb: FileObservation): Promise<any> {
return readFileP(fileOb.path)
.then((data: Buffer) => blobUtil.arrayBufferToBlob(data.buffer, contentType(extname(fileOb.path))))
.then(blob => ({ content_type: blob.type, data: blob }) as PouchAttachment)
.then(pa => db.putAttachment(fileOb.observation._id, (fileOb.observation as any)._rev, fileOb.filename, pa.data, pa.content_type))
.then(update => ((fileOb.observation as any)._rev = update.rev))
}
function importData(filename: string, db: InspectionDb, table: string, importer: (db: InspectionDb, doc: any) => Promise<any>) {
return new Promise((resolve, reject) => {
let trader = spawn(TraderPath, ['select', `-f="${filename}"`, `-t=${table}`])
trader.stderr.on('data', reject)
trader.on('error', reject)
let outstream = []
trader.stdout.on('data', data => outstream.push(data))
var imports = []
trader.on('close', code => {
if (code > 0) reject('Trader returned non-zero exit code')
safeLoadAll(''.concat(...outstream), (doc: any) => imports.push(importer(db, doc))) // safeLoadAll is synchronous
Promise.all(imports).then(resolve)
})
})
}
export function importProject(filename: string, project_id: string) {
let db: InspectionDb
return Promise.resolve()
.then(() => {
db = new InspectionDb(project_id)
return Promise.all([
importData(filename, db, 'Project_Inspections', importInspection),
importData(filename, db, 'Inspection', importObservation),
])
})
.then(() => buildReinspectionLinks(db))
.then(() => importInspectionPhotos(db, join(dirname(filename), '../Inspection_Projects')))
}
The solution to this ended up being very simple and silly. The PouchDB typings in the project was wrong...it had the rev and filename parameters for putAttachment() reversed...filename should have been first then rev. Changing this corrected the issue.

Resources