In a Firebase Function, I'm trying to accept a large string and store it.
exports.store = functions.https.onRequest(async (req, res) => {
const body : string = req.body;
await store(body);
return req.send(...);
});
I have found that using GZIP can improve performance and reduce file size by about 3-4x.
The problem is that if a client sends a GZip-encoded string with content encoding like so:
header("Content-Encoding", "gzip")
Then the call to req.body will 'automagically' decode the string to UTF-8. This both drastically reduces performance and requires me to again compress the string to efficiently store it in the server.
If I don't specify a content encoding then the endpoint will work as expected (i.e. it won't decompress the string and store it as-is, and run faster as a result). The problem is then I can't verify the user has sent a gzip-compressed string instead of a UTF-8 one, unless I implement some hacks to check for patterns in the string.
I am sure there is some simple solution, e.g.
functions.dontInflateTheRequestPleaseThanks();
exports.store = functions.https.onRequest(async (req, res) => {
if (req.headers["content-encoding"] !== "gzip") return res.status(415).send("gzip please");
const body : string = req.body;
await store(body);
return req.send(...);
});
The trick is to not pass Content-Encoding=gzip so express (the underlying framework) won't inflate (decompress) the request. Then to identify the request as a gzip, pass ContentType=application/gzip.
if (req.headers["content-encoding"] === "gzip") {
res.status(415).send("Don't specify gzip as the content-encoding. This trips up firebase.");
return;
}
if (req.headers["content-type"] !== "application/gzip") {
res.status(415).send("must be compressed using gzip");
return;
}
// Then body is guaranteed to not be fiddled with.
const body = req.body;
You have some middleware which fills req.body already and auto-decompresses it along the way. Instead, you can collect the body yourself.
Caution: The middleware that already fills req.body uses the same on("data")/on("end") technique as the code below, therefore you must make sure that this code is not executed together with the body-filling middleware.
To ensure this, pass an entire express app to functions.https.onRequest:
exports.store = functions.https.onRequest(express().use(async (req, res, next) => {
if (req.headers["content-encoding"] !== "gzip")
return res.status(415).send("gzip please");
var buffers = [];
req.on("data", function(data) {
buffers.push(data);
}).on("end", async function() {
await store(Buffer.concat(buffers));
res.send(...);
});
}));
Related
I know what is wrong with my code and I have looked into the best way of solving it, however with my lack of experience, I am having a hard time finding a good answer.
I need my first route(/data) to be fully completed before the second(/logo) express route sends the data. In short, I just need the variable symbolUrl to be completed before it goes into the second fetch call. Here is the code down below to explain
app.use(express.static('public'));
const url =
'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest';
const qString =
'?CMC_PRO_API_KEY=' + process.env.apiKey + '&start=1&limit=10&convert=USD';
let symbol = [];
app.get('/data', async (req, res) => {
const fetch_res = await fetch(url + qString);
const coinData = await fetch_res.json();
for (let i = 0; i < 9; i++) {
symbol.push(coinData.data[i]['symbol']);
};
res.json(coinData);
});
app.get('/logo', async (req, res) => {
const symbolUrl = symbol.join(',');
const url2 = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/info';
const qString2 = `?CMC_PRO_API_KEY=${apiKey}%symbol=${symbolUrl}`;
const fetch_res2 = await fetch(url2 + qString2);
const coinLogo = await fetch_res2.json();
res.json(coinLogo);
});
The issue I am trying to solve with this project is that I want to send the data(/data) to be sent to the front end first because this API call will load the majority of the page. Then my second call will load images and other larger files afterward. HOWEVER, the API I am working with to get the logos(images) of the specific crypto coins I want, I need a different endpoint as well as use %symbol=${symbolUrl} in the API call to get the correct tokens I want to call.
client code:
fetch('http://localhost:2000/data')
.then(async (response) => {
return response.json();
})
.then(async (data) => {
const parsedData = data['data'];
// console.log(data['data'][0]['name'])
await parsedData.forEach((element) => {
// this just has about 20 lines of code generating the the look of the page. It works as intended
});
fetch('http://localhost:2000/logo')
.then(async (response) => {
return response.json();
})
.then(async (logo) => {
console.log(logo)});
***I have tried putting this in an async function and awaiting the first fetch call
All I need to be done is for app.get(/data) to be fully complete before doing my second app.get. I have done testing and I know that is the issue. I apologize if it is something easy, but I couldn't find anything on making an app.get synchronous and I have tried putting both in a async function, however that did not work.
You cannot send responses in fragments like you're trying to do, it would throw an error saying Can't set headers after they are sent to client
The proper method to implement what you are trying to do is to define the first layer as middleware, and then allow the second layer to return the response. Here layer basically means a function handler.
In order to control when the execution passes to the next layer / next function handler, express has a third parameter (request, response, next). You're only using request and response, researching about next will solve your concern.
Express next function, what is it really for?
First handler
app.get('something_unique', async (req, res, next) => {
// do whatever you want to do first
// save data into res.locals
res.locals.foo = {...}
next()
})
Second Handler
app.get('something_unique', (req, res) => {
const data = res.locals.foo;
// whatever you want
return res.json({ anything })
})
More:
Express next function, what is it really for?
Error: Can't set headers after they are sent to the client
Passing variables to the next middleware using next() in Express.js
I'm not sure what client code you're really running as it sounds like you've bee trying several things, but this should work to sequence the /data request and the /logo request so that the /logo request is not run until the response from the /data request has been received.:
async function run() {
const r1 = await fetch('http://localhost:2000/data');
const data = await r1.json();
const parsedData = data.data;
parsedData.forEach((element) => {
// this just has about 20 lines of code generating
// the the look of the page. It works as intended
});
const r2 = await fetch('http://localhost:2000/logo');
const logo = await r2.json();
return logo;
}
run().then(logo => {
console.log(logo);
}).catch(err => {
// handle errors here
console.log(err);
});
If there is any asynchronous code inside the .forEach(), then we will have to see that also to properly sequence that.
As I've said in my comments, stuffing the data from the first request into a server-side variable is probably the wrong design on the server because two separate clients both issuing /data requests will conflict with one another, creating race conditions. But, you haven't explained what this data is really for or why you're stuffing it into a variable on the server for us to suggest an alternate design.
I'll try to keep it simple:
An example of what I currently have
router.post('/handleMail', async (req, res, next) => {
var x = [{a:{b:"b"}, c:{ok:"ok}] // just imagine this is convoluted array containing objects.
res.json({body: x})
})
So lets say variable "x" is HUGE, containing some base64 encoded string etc.
When I try to simply return it like this the process usually stalls.
Will streaming this as a response help this?
How would I go about streaming it to "res" in this case?
Thanks :)
You could try using a library like json-stream-stringify that creates a readable stream from your json that you can pipe into the express response-object:
const JsonStreamStringify = require('json-stream-stringify');
router.post('/handleMail', async (req, res, next) => {
const jsonObject = [{a:{b:"b"}, c:{ok:"ok}];
new JsonStreamStringify(jsonObject ).pipe(res);
});
I send JSON data to route:
const data=[{name:this.name}]
//qs.stringify(payload)
axios
.post('/users',{name:this.name})
.then(
response => {
console.log(response.data);
}
)
.catch(
// error=>console.log(error)
)
And try get data:
router.post('/', function (req, res, next) {
var data = req.body; // here is your data
var obj=JSON.parse(data);
console.log(obj.toString());
res.toString("ok");
});
And I got error 500.
Why not get the data?
Your client side code is fine, except that the constant named data is entirely unused. On server-side however, req.body almost certainly contains a parsed JSON object (provided you have included a body paring middleware already). If you haven't included any body parsing middleware then req.body would be undefined.
Additionally, the toString() method in res doesn't send any response, it simply returns a string representation of the response.
You would need to make following changes to your code:
Include a body parsing middleware (eg. body-parser) to your express middleware chain, if one isn't already included.
Don't call JSON.parse(req.body). body-parser would already have done that. Calling it again will only throw exception, and return 500.
To convert an Object to JSON string, use JSON.stringify() method. obj.toString() will probably only return [object Object].
Send response using one of .send(), .json(), .end() methods on res. Since you need to send a string back, res.send("ok") seems most appropriate.
The changed code should appear something like this:
router.post('/', function (req, res, next) {
var data = req.body; // here is your data
console.log(JSON.stringify(data));
res.send("ok");
});
Brothers and sisters, I am building an Express API Endpoint that needs to consume an external API, perform some changing of keys and values, and return to the result to the client. Here is what I have thus far:
const external_endpoint = <external_api_end_point>;
app.get('/', function (req, res, next) {
request({ url: external_endpoint}).pipe(res);
});
This returns the exact payload you would get from hitting the external_endpoint directly.
Isn't there something I can do to change res before it gets sent to the client? I tried a few things but nothings has worked. Any ideas or best practices associated with doing a transform on the incoming payload?
For the sake of simplicity. Lets say this is the payload obj.json:
{
"sad": {
"userid": 5,
"username": "jsmith",
"isAdmin": true
}
}
and I am wanting to change sad to happy.
I know outside of the request I could do something like this:
obj = JSON.parse(JSON.stringify(obj).split('"sad":').join('"happy":'));
but throwing obj in place of res will not work. I have tried assigning the value of this res and res.body but no dice.
Thanks for you help in advance!
If you're using request-promise, you can simply make a new response and send it, or modify the response you got back:
app.get('/', function (req, res, next) {
request({ url: external_endpoint, json: true})
.then(response => res.json({ happy: response.sad })))
.catch(next);
});
(of course, you need to handle errors appropriately)
If you want to process it as a stream (which makes sense if you have a massive amount of data), you can use the original request module, and use event-stream to create your pipe:
const es = require('event-stream');
const swapper = es.through(
function write(data) {
this.emit("data", data.replace("sad", "happy"));
},
function end() {
this.emit("end");
}
);
request({ url: external_endpoint})
.pipe(es.stringify())
.pipe(swapper)
.pipe(es.parse())
.pipe(res);
Here's a sandbox to test the stream processing: https://codesandbox.io/s/3wqx67pq6
Can the req object be manipulated within a HTTP request? So is there any chance a request can set a value for req.token? I know that various object properties like req.query, req.body etc. can be freely changed from the outside, but can new properties like req.token be added or is this example code safe?
var auth = function (req, res, next) {
if (isValid()) {
req.token = getToken();
return next();
}
}
app.get('/foo', auth, function(req, res) {
if (req.token) {
// valid request
} else {
// invalid request
}
});
Yes, it's safe, provided you don't overwrite a property that a special meaning (a non-exhaustive list of those can be found here).
It's a commonly used technique, and it's also shown in the Express documentation.