I'm trying to set up HTTP2 for an Express app I've built. As I understand, Express does not support the NPM http2 module, so I'm using SPDY. Here's how I'm thinking to go about it-I'd appreciate advice from people who've implemented something similar.
1) Server setup-I want to wrap my existing app with SPDY, to keep existing routes. Options are just an object with a key and a cert for SSL.
const app = express();
...all existing Express stuff, followed by:
spdy
.createServer(options, app)
.listen(CONFIG.port, (error) => {
if (error) {
console.error(error);
return process.exit(1)
} else {
console.log('Listening on port: ' + port + '.')
}
});
2) At this point, I want to enhance some of my existing routes with a conditional PUSH response. I want to check to see if there are any updates for the client making a request to the route (the client is called an endpoint, and the updates are an array of JSON objects called endpoint changes,) and if so, push to the client.
My idea is that I will write a function which takes res as one of its parameters, save the endpoint changes as a file (I haven't found a way to push non-file data,) and then add them to a push stream, then delete the file. Is this the right approach? I also notice that there is a second parameter that the stream takes, which is a req/res object-am I formatting it properly here?
const checkUpdates = async (obj, res) => {
if(res.push){
const endpointChanges = await updateEndpoint(obj).endpointChanges;
if (endpointChanges) {
const changePath = `../../cache/endpoint-updates${new Date().toISOString()}.json`;
const savedChanges = await jsonfile(changePath, endpointChanges);
if (savedChanges) {
let stream = res.push(changePath, {req: {'accept': '**/*'}, res: {'content-type': 'application/json'}});
stream.on('error', function (err) {
console.log(err);
});
stream.end();
res.end();
fs.unlinkSync(changePath);
}
}
}
};
3) Then, within my routes, I want to call the checkUpdates method with the relevant parameters, like this:
router.get('/somePath', async (req, res) => {
await checkUpdates({someInfo}, res);
ReS(res, {
message: 'keepalive succeeded'
}, 200);
}
);
Is this the right way to implement HTTP2?
Related
I have an internal HTTP Post API that generates files, size 5-10mb each. This service is not modifiable.
I want to "proxy" this file download through the public API, which is based on Node.js+Express. However, I can't figure out the best way of doing so.
I guess I can download this file with Axios into a temporary file in the Node.js API container, but that seems to be prone to issues with these temporary files potentially piling up and requiring later cleanup. Is there a way to achieve such file download -> send further to a client without creating a temporary file?
Or what would be the most efficient and "clean" way of doing so if temporary files are unavoidable?
router.post('/route/:someid',
[someRequestVerificationMiddleware],
(req, res, next) => {
const myFileId = req.params.someid;
const downloadRequestParams= {
"id": myFileId
};
let dlPromise = axios.post(`http://myinternalservice:80`,
downloadRequestParams, {responseType: "stream"});
dlPromise.then(response => {
try {
let filename = response.headers["x-result-filename"];
//
// What would be the most efficient way to return the received file
// from response data to the client calling this route without creating
// too much garbage?
//
} catch (e) {
console.error(e);
}
})
.catch(e=>{
console.error(e);
res.status(500);
})
.finally(() => {
next();
})
});
module.exports = router;
res is a stream. You can simply pipe your axios stream to the response.
res.setHeader("content-type", "...");
return dlPromise.then((response) => response.data.pipe(res));
From inside my expressJS application I have to verify that a cookie token is valid with a back-end server. So the relevant code involved in this is as follows:
app.get('*', (req, res, next) => {
console.log('GET: ' + req.path);
// ...
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
// build request
const restReq = https.request(authServerOptions, result => {
console.log('back-end response' + result.statusCode);
result.on('data', data => {
next(); // token is good now proceed.
});
result.on('error', error => {
res.redirect('somewhere'); // token is bad or timeout
});
});
restReq.write(token);
restReq.end();
}
So the main get function sets the REST request in motion and then just returns without calling next() or anything.
Questions:
Is this the right code for doing this? What happens if the callbacks are never called?
Is the application blocked from processing other requests until the back-end server returns or times out?
If so is there some way of freeing up the thread to process more requests?
Thanks in advance for any help. I haven't found many examples for this code pattern so if there is one a link would be appreciated.
Yes, I think the general idea of your implementation is correct.
I would also suggest, as done in the comments, to use a client such as axios to handle the request in a less verbose and more comprehensive manner, which would leave your code looking something like this:
const axios = require('axios');
app.get('*', (req, res, next) => {
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
axios.post(url, payload, opts)
.then(response => next())
.catch(error => {
console.error(error);
res.redirect('somewhere');
});
});
A bit more to the point, but functionally almost equivalent to your implementation. The one thing you are missing is the onerror callback for your request object, which currently may fail and never return a response as you correctly suspected. You should add:
restReq.on('error', error => {
console.error(error);
res.redirect('somewhere');
});
On the same vein, it would probably be more fitting to call next on result end, instead of doing so while reading response data:
result.on('end', () => {
next();
});
Then you'd be covered to guarantee that a callback would be invoked.
Neither implementation blocks the processing of future requests, as the call to the token validation service is done asynchronously in both cases.
I was learning to build a weather app using Node (Express) + React. I successfully fetched weather data from open weather API.
However I was directly using the open weather API key in my React app like this const weatherURL = 'http://api.openweathermap.org/data/2.5/weather?q=london,uk&APPID=1234567qwerty';. Obviously this is not safe as it exposed the API key to the client. I thought about storing the API key in .env file, but according to [this answer][1], I should never store API key in .env file or .gitignore. The right way is to make a request to backend API and make an API call to backend and send the data back. I could not find out how to do it. Can anyone help?
Following is my node js code:
const express = require('express');
const cors = require('cors');
const app = express();
const SELECT_ALL_QUERY = 'SELECT * FROM `mySchema`.`myTable`;';
app.use(cors());
app.get('/', (req, res) => {
res.send('go to /myTable to see content')
});
const pool = require('./awsPool');
pool.getConnection((err, connection) => {
if (err) {
return console.log('ERROR! ', err);
}
if(!connection) {
return console.log('No connection was found');
}
app.get('/myTable', (req, res) => {
console.log(connection);
connection.query(SELECT_ALL_QUERY, (err, results) => {
if (err) {
return res.send(err)
}
else {
return res.json({
data: results
})
};
});
});
});
let port=process.env.PORT||4000;
app.listen(port, () => {
console.log(`App running on port ${port} `);
});```
[1]: https://stackoverflow.com/a/57103663/8720421
What the linked answer was suggesting is to create a route in your Node/Express backend API that will make the call to the weather API for you, instead of the front end. This way the request and your API key are not public-facing whenever your front end makes a call.
The method for doing this would essentially be the same as what you have done in React, making an HTTP request using a built-in or 3rd party library. This resource I just found has some information on how to do both.
The simplest pure http-request in node looks like this:
const http = require('http')
const url = 'http://api.openweathermap.org/data/'
http.request(url, callback).end()
function callback (weatherResponse) {
let jsonString = ''
weatherResponse.on('data', chunk => {
jsonString += chunk
})
weatherResponse.on('end', () => {
// Now you have the complete response and can do whatever you want with it
// like return it to your user `res.send(jsonString)`
console.log(jsonString)
})
}
Many people find it bulky to having to handle chunks and the whole asynchronous thing, so there are many popular npm modules, like: https://www.npmjs.com/package/axios. (And here's a list of other contenders https://github.com/request/request/issues/3143).
Also, it is normal to store API-keys in environment variables on the backend. It makes things easy if you ever try to dockerize your app, or just scale up to using two backend servers instead of one.
I found a solution based on #ippi answer, add the following part to the original code:
const request = require('request');
const url = 'http://api.openweathermap.org/data/2.5/weather?q=london,uk&APPID=1234567';
app.get('/weather', (req, res) => {
request(url, (error, response, body) => {
if (!error && response.statusCode == 200) {
var info = JSON.parse(body)
res.send(info);
}
})
})
The url can be stored in .env file and passed into the above code. The returned weather data can be viewed in JSON format at http://localhost:4000/weather. In React the weather data can be fetched via this localhost url.
EDIT: request is deprecated, so here is a solution using axios
app.get('/weather', (req, res) => {
axios.get(url)
.then(response => {res.json(response.data)})
.catch(error => {
console.log(error);
});
})
User Passport middleware for nodeJs/Express. They provide passport-headerapikey strategy using which you can create and authorize apiKeys. http://www.passportjs.org/packages/passport-headerapikey/
I have a nuxt application in which I will need to append data from a generated configuration file when the application is first started. The reason I cannot do this in the actual build is because the configuration file does not exists at this point; it is generated just before calling npm start by a bootstrap script.
Why don't I generated the configuration file before starting the application you may ask and this is because the application is run in a docker container and the built image cannot include environment specific configuration files since it should be used on different environments such as testing, staging and production.
Currently I am trying to use a hook to solve this, but I am not really sure on how to actually set the configuration data in the application so it can be used everywhere:
# part of nuxt.config.js
hooks: {
listen(server, listener) {
# load the custom configuration file.
fs.readFile('./config.json', (err, data) => {
let configData = JSON.parse(data));
});
}
},
The above hook is fired when the application first starts to listen for connecting clients. Not sure this is the best or even a possible way to go.
I also made an attempt of using a plugin to solve this:
import axios from ‘axios’;
export default function (ctx, inject) {
// server-side logic
if (ctx.isServer) {
// here I would like to simply use fs.readFile to load the configuration, but this is not working?
} else {
// client-side logic
axios.get(‘/config.json’)
.then((res) => {
inject(‘storeViews’, res.data);
});
}
};
In the above code I have problems both with using the fs module and axios.
I was also thinking about using a middleware to do this, but not sure on how to proceed.
If someone else has this kind of problem here is the solution I came up with in the end:
// plugins/config.js
class Settings
{
constructor (app, req) {
if (process.server) {
// Server side we load the file simply by using fs
const fs = require('fs');
this.json = fs.readFileSync('config.json');
} else {
// Client side we make a request to the server
fetch('/config')
.then((response) => {
if (response.ok) {
return response.json();
}
})
.then((json) => {
this.json = json;
});
}
}
}
export default function ({ req, app }, inject) {
inject('config', new Settings(app, req));
};
For this to work we need to use a server middleware:
// api/config.js
const fs = require('fs');
const express = require('express');
const app = express();
// Here we pick up requests to /config and reads and return the
// contents of the configuration file
app.get('/', (req, res) => {
fs.readFile('config.json', (err, contents) => {
if (err) {
throw err;
}
res.set('Content-Type', 'application/json');
res.end(contents);
});
});
module.exports = {
path: '/config',
handler: app
};
I am trying to develop a MERN stack application, and I have done numerous attempts at this. So, what I am trying to achieve is have some data I am pulling from an api and dump it to a database, then query from the database to create a JSON file every 5 minutes(using jenkins and python, the best approach I can think of). Below is a method I am implementing and it does not work. If I remove the setInterval() function and un-comment the callback function, the code works but does not update the data.
const express = require('express');
const fs = require('fs');
const app = express();
// Read JSON File
function readJSON(callback) {
fs.readFile('./name.json', "utf8", function(err, result) {
if (err) callback (err);
callback(JSON.parse(result));
});
}
// Process JSON File during callback
// readJSON(function(res) {
// app.get('/api/customers', (request, response) => {
// response.json(res);
// });
// });
// Attempt to run every 5 minutes
setInterval(readJSON(function(res) {
app.get('/api/customers', (request, response) => {
response.json(res);
})}, 60000 * 5); // 5 Minutes
const port = 5000
app.listen(port, () => `Server running on port ${port}`);
I thought of using sockets, but I don't want it to be real-time, only live data on an interval. Restful API's I don't believe are a good fit here either, I don't want a 2-way communication to modify/update the data. If my approach is bad, please let me know why you'd pick another approach. I am just trying to establish a foundation in full-stack web dev. Thanks!
A logical code would be:
On server side:
function readJSON(callback) {
fs.readFile('./name.json', "utf8", function(err, result) {
if (err) callback(err);
callback(null, JSON.parse(result));
});
}
app.get('/api/customers', (request, response) => {
readJSON((err, nameContent) => {
if(err) {
response.status(500).send(err);
}
response.send(nameContent);
})
});
And in the client side ask for the data every 5 minutes:
someAjaxMethod('/api/customers', (err, nameContent) => console.log(err, nameContent));