apollo-server-micro package tries to receive request body stream that already received before(in some scenarios) and hangs on forever cause never receives events of stream(it already fully obtained)
I gone step by step through all the flow to discover the issue.
In brief the issue pops up when request stream that passed to Apollo already read before. It means we already had used for example: body.on('data', onData) & body.on('end', onEnd) or it was executed by another chain in the flow(express server, next.js server, firebase cloud function).
And if it was before what is going in apollo-server-micro code is that it tries to do it again, but it will never occur and we will fail on timeout or never get the response, because: body.on('data', or body.on('end' will never be called again(the stream already parsed before fully, these event will not happen).
So I think some way is needed to treat this situation and to give Apollo option to work with body stream already received. May be we need some way to say Apollo to not try to receive body stream if it already exists and just deliver it already prepared(buffer) by some property. So we don't need to do it if I already can provide it to the Apollo.
I found some hack I can do, but by far it needed to be done in more proper way.
Apollo uses json function from micro package (https://github.com/vercel/micro) to get the body stream. if I change this line there:
const body = rawBodyMap.get(req);
to something like:
const body = rawBodyMap.get(req) || req.rawBody;
I have rawBody because I use firebase cloud function and when it receives body stream it saves received stream buffer in rawBody property of request (and it's exactly what json function of micro tries to achieve).
full flow:
src/ApolloServer.ts ( from apollo-server-micro package )
import { graphqlMicro } from './microApollo';
const graphqlHandler = graphqlMicro(() => {
return this.createGraphQLServerOptions(req, res);
});
const responseData = await graphqlHandler(req, res);
send(res, 200, responseData);
microApollo.ts - we use here json function from 'micro' passing req as parameter
import { send, json, RequestHandler } from 'micro'; ( https://github.com/vercel/micro )
const graphqlHandler = async (req: MicroRequest, res: ServerResponse) => {
let query;
try {
query =
req.method === 'POST'
? req.filePayload || (await json(req))
: url.parse(req.url, true).query;
} catch (error) {
// Do nothing; `query` stays `undefined`
}
https://github.com/vercel/micro package
const getRawBody = require('raw-body');
exports.json = (req, opts) =>
exports.text(req, opts).then(body => parseJSON(body));
exports.text = (req, {limit, encoding} = {}) =>
exports.buffer(req, {limit, encoding}).then(body => body.toString(encoding));
exports.buffer = (req, {limit = '1mb', encoding} = {}) =>
Promise.resolve().then(() => {
const type = req.headers['content-type'] || 'text/plain';
const length = req.headers['content-length'];
// eslint-disable-next-line no-undefined
if (encoding === undefined) {
encoding = contentType.parse(type).parameters.charset;
}
// my try to hack the behavior
const body = rawBodyMap.get(req) || req.rawBody;
console.log(">>>>>>>>>>>>>>>>>>> ", body);
if (body) {
return body;
}
return getRawBody(req, {limit, length, encoding})
.then(buf => {
rawBodyMap.set(req, buf);
return buf;
})
.catch(err => {
if (err.type === 'entity.too.large') {
throw createError(413, `Body exceeded ${limit} limit`, err);
} else {
throw createError(400, 'Invalid body', err);
}
});
});
if I don't stop by my hack the code from going to receive the body
stream it calls to : getRawBody from 'raw-body' package;
raw-body package
function getRawBody (stream, options, callback) {
……
return new Promise(function executor (resolve, reject) {
readStream(stream, encoding, length, limit, function onRead (err, buf) {
if (err) return reject(err)
resolve(buf)
})
})
}
function readStream (stream, encoding, length, limit, callback) {
…….
// attach listeners
// these callbacks never called because body request stream already received before
stream.on('aborted', onAborted)
stream.on('close', cleanup)
stream.on('data', onData)
stream.on('end', onEnd)
stream.on('error', onEnd)
…….
Related
I am trying to process signup data for a uni project . I am using basic koa modules and I am not allowed to use express, ideally I want to get the data inside the variable post. I want to process the data for example to see if the password has less than 5 characters , if so i would like that the program would not redirect the user to different address but if no errors occur i would like the program to redirect to regOk.html, I tried many other ways like initializing the variable outside of ctx.req.on but none were successful . Can anyone help me ?
export async function postregister(ctx) {
let bodyString = "";
ctx.req.on("data", (chunk) => {
bodyString += chunk;
});
//let collectData = new Array();
ctx.req.on("end", () => {
var post = querystring.parse(bodyString);
var email = post["email"];
var password = post["password"];
var passbestätigen = post["passwort bestä"];
var vorname = post["vorname"];
var nachname = post["nachname"];
var adresse = post["adresse"];
var stadt = post["stadt"];
var telefonnummer = post["telefonnummer"];
var geburtsdatum = post["geburtsdatum"];
var regData = model.add(ctx.db, post);
regData.then(() => console.log("singup successful"))
});
await ctx.render("regOk.html");
}
I'm not very familiar with koa, but I believe your issue is related to the order in which your code is executed.
The event in charge of parsing the data received in the body of the request ends after the synchronic execution of your postregister method, so you never get to see the value of post in the order you'd expect.
One possible solution to go around this issue would be wrapping the parsing of data in a promise, waiting for that promise to complete, and executing then and catch functions once the processing is done.
export async function postregister(ctx) {
await new Promise((resolve) => {
let bodyString = "";
ctx.req.on("data", (chunk) => {
bodyString += chunk;
});
ctx.req.on("end", async () => {
resolve(querystring.parse(bodyString));
});
})
.then(async (post) => {
await model.add(ctx.db, post)
.then(async () => {
console.log("singup successful");
await ctx.render('regOk.html');
});
})
.catch(async (error) => {
console.error(error);
await ctx.render('error.html');
});
}
This way, you handle body parsing inside the Promise, and after that completed you get the result of querystring.parse(bodyString) as a variable named post in your then handler.
I'm trying to build a basic pub/sub event emitter and I'm having trouble understanding how I can pipe an incoming event with payload to all of the event's subscribers. This is what I have:
app.post("/events", (req, res) => {
const { type } = req.body;
if (!type) {
res.status(404).send(`no event found`);
return;
}
if (!eventTypeToUrls[type]) {
res.status(404).send(`no event ${type} subscribed to`);
return;
}
let count = 0;
const urls = eventTypeToUrls[type];
urls.forEach((url) => {
console.log("sending payload to >>", url);
const writeStream = http.request(
url + "/events",
{ method: "post" },
() => {
count += 1;
if (count === urls.length) {
res.status(201).send("ok");
}
}
);
writeStream.on("error", (e) => res.status(401).send(e));
req.pipe(writeStream);
});
});
I saw that I might in a non-piping setting would need to call res.write with the serialized payload, but since I'm piping from a readable stream to a writable stream, I'm not sure exactly how that would change. I imagine I wouldn't need to call res.end either. thanks.
The problem is that the readable req-stream will already have been consumed, once the request-handler callback is hit, thus you cannot pipe it to a writeable stream.
However, if all you want to do is to send http-requests to all subscriber urls, you can use Promise.all to await the non-sequential processing of the requests and simply do res.status(..).send(..) once all promises are fulfilled. Something like this (I'm using superagent here as the http-request library):
app.post('/events', async (req, res) => {
// ...
try {
await Promise.all(urls.map(url => {
return superagent
.post(url + '/events')
.send(req.body);
}));
res.status(201).send("ok");
} catch (err) {
res.status(500).end('something went wrong ...');
}
});
I'm trying to download a bunch of files. Let's say 1.jpg, 2.jpg, 3.jpg and so on. If 1.jpg exist, then I want to try and download 2.jpg. And if that exist I will try the next, and so on.
But the current "getFile" returns a promise, so I can't loop through it. I thought I had solved it by adding await in front of the http.get method. But it looks like it doesn't wait for the callback method to finish. Is there a more elegant way to solve this than to wrap the whole thing in a new async method?
// this returns a promise
var result = getFile(url, fileToDownload);
const getFile = async (url, saveName) => {
try {
const file = fs.createWriteStream(saveName);
const request = await http.get(url, function(response) {
const { statusCode } = response;
if (statusCode === 200) {
response.pipe(file);
return true;
}
else
return false;
});
} catch (e) {
console.log(e);
return false;
}
}
I don't think your getFile method is returning promise and also there is no point of awaiting a callback. You should split functionality in to two parts
- get file - which gets the file
- saving file which saves the file if get file returns something.
try the code like this
const getFile = url => {
return new Promise((resolve, reject) => {
http.get(url, response => {
const {statusCode} = response;
if (statusCode === 200) {
resolve(response);
}
reject(null);
});
});
};
async function save(saveName) {
const result = await getFile(url);
if (result) {
const file = fs.createWriteStream(saveName);
response.pipe(file);
}
}
What you are trying to do is getting / requesting images in some sync fashion.
Possible solutions :
You know the exact number of images you want to get, then go ahead with "request" or "http" module and use promoise chain.
You do not how the exact number of images, but will stop at image no. N-1 if N not found. then go ahed with sync-request module.
your getFile does return a promise, but only because it has async keyword before it, and it's not a kind of promise you want. http.get uses old callback style handling, luckily it's easy to convert it to Promise to suit your needs
const tryToGetFile = (url, saveName) => {
return new Promise((resolve) => {
http.get(url, response => {
if (response.statusCode === 200) {
const stream = fs.createWriteStream(saveName)
response.pipe(stream)
resolve(true);
} else {
// usually it is better to reject promise and propagate errors further
// but the function is called tryToGetFile as it expects that some file will not be available
// and this is not an error. Simply resolve to false
resolve(false);
}
})
})
}
const fileUrls = [
'somesite.file1.jpg',
'somesite.file2.jpg',
'somesite.file3.jpg',
'somesite.file4.jpg',
]
const downloadInSequence = async () => {
// using for..of instead of forEach to be able to pause
// downloadInSequence function execution while getting file
// can also use classic for
for (const fileUrl of fileUrls) {
const success = await tryToGetFile('http://' + fileUrl, fileUrl)
if (!success) {
// file with this name wasn't found
return;
}
}
}
This is a basic setup to show how to wrap http.get in a Promise and run it in sequence. Add error handling wherever you want. Also it's worth noting that it will proceed to the next file as soon as it has received a 200 status code and started downloading it rather than waiting for a full download before proceeding
I have copied the very good code from https://www.tomas-dvorak.cz/posts/nodejs-request-without-dependencies/ to make a http request in nodejs using native modules.
I want to be able to use the data value later on in the script.
I know this is a common issue with newbies and async code, i just CANNOT understand this yet and have struggled for weeks to get it.
I have coped much code, watched youtube, talked to people, its flippen hard..
const getContent = function(url) {
return new Promise((resolve, reject) => {
const https = require('https')
const request = https.get(url, (response) => {
// handle http errors
if (response.statusCode < 200 || response.statusCode > 299) {
reject(new Error('Failed to load page, status code: ' + response.statusCode));
}
// temporary data holder
const body = [];
// on every content chunk, push it to the data array
response.on('data', (chunk) => body.push(chunk));
// we are done, resolve promise with those joined chunks
response.on('end', () => resolve(body.join('')));
});
// handle connection errors of the request
request.on('error', (err) => reject(err))
})
}
getContent('https://myapi/json')
.then((data) => console.log(data))
.catch((err) => console.error(err))
// I want to use the "data" value down here in my script. I want to do things with the "data" value like JSON.parse(data)
console.log(data) //undefined
let json = JSON.parse(data) //undefined
console.log('after')
my result for data is undefined
How can i use data down here below all the code above?
You can setup a callback and access your data within this callback, this pattern should be easy enough to use.
getContent('https://myapi/json')
.then(useData)
.catch((err) => console.error(err))
// Use this callback to do what you want with your data!
function useData(data) {
console.log(data);
let json = JSON.parse(data);
}
Or using async / await ( this might be more intuitive!):
async function testAwait() {
let data = await getContent('https://myapi/json');
console.log("data: ", data);
}
testAwait();
My application uses an internal webservice for fetching data, i have a job which creates approx 500 requests which getsfired async to complete the fetch operation.
I make use of Axios, by creating an array of axios promises and then resolving them using using Axios.all();
It works fine until some 200 requests but post that i get socket hung up, however on the server side i see the requests are being processed.
How to configure axios to set custom time out, or is it a better idea to splice my promises array and then run them as multiple batches ?
Source code
let getAxiosPromiseArray = (urlList) => {
var axiosArrayofPromise = [];
return new Promise ( (resolve, reject) => {
try {
urlList.forEach ( (URL) => {
axiosArrayofPromise.push(axios.get(URL));
});
resolve(axiosArrayofPromise);
}
catch (err) {
reject("There is a problem getting Axios array of promises " + err);
}
})
}
async function processAxiosPromises (PromiseArray) {
try {
var results = []
results = await axios.all(PromiseArray);
return results;
}
catch(err) {
throw("There was a problem resolving promises array (Axios) " + err);
}
}
getallID().then ( (urlList) => {
return getAxiosPromiseArray(urlList);
}).then( (AxiosPromises) => {
return processAxiosPromises(AxiosPromises);
}).then ((resultData) => {
console.log(resultData);
});
Error
There was a problem resolving promises array (Axios) Error: socket hang up
First, that pair of functions getAxiosPromiseArray() and processAxiosPromises() needs fixing.
Your new Promise() construction is unnecessary. You can simply return Promise.all(arrayofPromise) (or axios.all(...) if you must) and do away with the other function.
Renaming the remaining function to something meaningful, you would end up with eg :
let getData = (urlList) => {
return Promise.all(urlList.map(URL => axios.get(URL)))
.catch(error => {
error.message = "There is a problem getting Axios array of promises " + error.message; // augment the error message ...
throw error; // ... and re-throw the errror.
});
};
And call as follows :
getallID().then(getData)
.then(resultData => {
console.log(resultData);
}).catch(error => {
console.error(error);
});
That will put you on solid ground but, on its own, is unlikely to fix a concurrency problem (if that's what it is), for which the simplest approach is to use Bluebird's Promise.map with the concurrency option.
The caller code can remain the same, just change getData(), as follows:
let getData = (urlList) => {
let concurrency = 10; // play with this value to find a reliable concurrency limit
return Promise.map(urlList, URL => axios.get(URL), {'concurrency': concurrency})
.catch(error => {
error.message = "There is a problem getting Axios array of promises " + error.message;
throw error;
});
};
// where `Promise` is Bluebird.
const axios = require('axios');
const axiosThrottle = require('axios-throttle');
//pass axios object and value of the delay between requests in ms
axiosThrottle.init(axios,200)
const options = {
method: 'GET',
};
const urlList = [
'https://jsonplaceholder.typicode.com/todos/1',
'https://jsonplaceholder.typicode.com/todos/2',
'https://jsonplaceholder.typicode.com/todos/3',
'https://jsonplaceholder.typicode.com/todos/4',
'https://jsonplaceholder.typicode.com/todos/5',
'https://jsonplaceholder.typicode.com/todos/6',
'https://jsonplaceholder.typicode.com/todos/7',
'https://jsonplaceholder.typicode.com/todos/8',
'https://jsonplaceholder.typicode.com/todos/9',
'https://jsonplaceholder.typicode.com/todos/10'
];
const promises = [];
const responseInterceptor = response => {
console.log(response.data);
return response;
};
//add interceptor to work with each response seperately when it is resolved
axios.interceptors.response.use(responseInterceptor, error => {
return Promise.reject(error);
});
for (let index = 0; index < urlList.length; index++) {
options.url = urlList[index];
promises.push(axiosThrottle.getRequestPromise(options, index));
}
//run when all promises are resolved
axios.all(promises).then(responses => {
console.log(responses.length);
});
https://github.com/arekgotfryd/axios-throttle