I'm playing with node.js. Using http-proxy, I want to create a simple web proxy that makes the request and then stash the response somewhere before passing back the response. If I then detect no internet connection or some flag is set somewhere, I want to replay the response that I have stashed away. So the URL would be the "key" and the entire response would be the "value".
My question is, what's the easiest way to serialize this response object so that it can be replayed later? I was looking at mongodb and mongoosejs, but I'm put off because mongoose wants me to create a schema for my object, and I just want to dump the entire response object somewhere (with the URL as a key). Is there an easier way?
Here's my super simple node.js proxy code:
var httpProxy = require('http-proxy');
var server = httpProxy.createServer(function (req, res, proxy) {
var buffer = httpProxy.buffer(req);
proxy.proxyRequest(req, res, {
host: 'url.to.proxy.com',
port: 80,
buffer: buffer
});
});
server.proxy.on('end', function (req) {
console.log("The request was proxied.",req.url);
});
server.listen(8000);
The easiest way is to store the response object in a file; you don't need a database.
Create a response object cache directory.
Hash the URL using SHA-256 and use the result as your file name.
Stream the response object to/from the file.
Related
I want to create a nodejs server which is acting a proxy to download files i.e. user clicks
on the a download button, call get from nodejs server, nodejs server fetches link from a different
remote server and starts the download (in terabytes). This download is then forwarded to the user.
The terabyte file should not be stored on the nodejs server and then sent.
Here is my attempt:
function (request, response) {
// anything related to the remote server having the file
var options= {
path: "./bigData",
hostname:"www.hugeFiles.net"
}
// get the file from the remote server hugefiles and push to user's response
https.get(options, function(downFile)) {
downFile.pipe(response)
}
}
Before I was using res.download(file, function(err)) {} but file has to be downloaded completely from the remote server
You're very close, you're sending the right http body but with the wrong http headers.
Here's a minimal working example:
const express = require('express');
const http = require('http');
const app1 = express();
app1.get('/', function (req, res) {
res.download('server.js');
});
app1.listen(8000);
const app2 = express();
app2.get('/', function (req, res) {
http.get({ path: '/', hostname: 'localhost', port: 8000}, function (resp) {
res.setHeader('content-disposition', resp.headers['content-disposition']);
res.setHeader('Content-type', resp.headers['content-type']);
resp.pipe(res);
});
});
app2.listen(9000);
Though I would say you should take a look at modules like https://github.com/nodejitsu/node-http-proxy which take care of the header etc . . . for you.
There is no way for your server to provide a file to the client without the server downloading it first.
What you may want to do instead is provide the client with a download link to the huge file. To make it seem automatic, you can create html which starts a download from the content provider automatically and serve that to the client.
In other words, in the scenario you are describing, the server is acting as a middleman between your client and the content provider. Unless the server needs to process the data or the client isn't allowed to retrieve the data themselves, it makes more sense to cut out the middleman.
What is the difference between the request in this line of code:
http.createServer(function(request,response){. . .}
and request in
http.request()
Are both requests done to the server?
I am new to node.js and I am sorry if I sound dumb!
How does http.request() work?
In http.request() we fetch data from another site but in order to fetch data from another site we first need to go to our site and then make a request? Explain it with a simple real-life example!
http.request() makes a request to another HTTP server. Suppose for some reason I wanted to go download Stack Overflow's home page...
http.request('https://stackoverflow.com/', (res) => {
// ...
});
http.createServer()... it creates an HTTP server. That is, it binds your application to a socket to listen on. When a new connection is made from somewhere or something else, it handles the underlying HTTP protocol of that request and asks your application to deal with it by way of a callback. From the Node.js documentation:
http.createServer((req, res) => {
res.writeHead(200, { 'Content-Type': 'text/plain' });
res.end('okay');
});
These two methods have absolutely nothing to do with each other. http.request() is for making a request to an HTTP server. http.createServer() is for creating your own HTTP server. Don't get confused by the callbacks.
Based on the source code of nodejs (extract below), createServer is just a helper method to instantiate a Server.
Extract from line 1674 of http.js.
exports.Server = Server;
exports.createServer = function(requestListener) {
return new Server(requestListener);
};
The http.request() API is for when you want your server code to act as a client and request content from another site and has GET, POST, PUT, DELETE methods.
Trying to do something seemingly basic.
I'd like to create a POST request through which I'll be sending JSONs. These JSONs will be created into files, which I'd like to return to the user via download.
The use case for this is that I'm building an application which takes a form and converts it into a JSON for upload to a MongoDB database. Users can load these JSONs into the application to re-load their old records as templates.
This is how I'm approaching it as of now:
// Download JSON Previews
var jsondownload = {};
// Grabs the JSON from POST request
app.post('/api/download', function(req, res, next){
jsondownload = {};
var json = req.body;
jsondownload = json;
res.json(jsondownload);
next();
});
// Immediately downloads the JSON thereafter
app.get('/api/download', function(req, res){
res.set({"Content-Disposition":"attachment; filename='test.json'"});
res.send(jsondownload);
});
What's the right way to do this?
There is no one "right" way to do it, but a few solutions include:
Remove the GET route handler (and the jsondownload variable) completely and just respond immediately with the Content-Disposition set appropriately. This is the better of the 3 because it reduces code and keeps things simple.
Use a simple redirect in your POST route handler. Instead of responding with the JSON immediately, you would do res.redirect('/api/download').
Do more or less what currently doing, but move the logic (the res.set() and res.send()) to a separate function that gets called from both route handlers.
I have some information that is unique to each client. This is not the login information but just some generic data of the client - for example, say the address of the client.
Now, when a client connects with the server for the first time, I will read the address of the client.
Subsequently, each time the client makes requests to the server, I would like to use the address that I read from the first request.
How do I do this? I mean, how do I store some information on the client in the server itself so that with each request, I do not have to read the address again (from the DB) but instead read it directly from say, a global variable or a like a request header or something on those lines... How, do I attach the address to each client so that for future requests from the client, I can directly read the address information from the client's request itself without having to query the DB once more...
edit - this requires Express, didn't notice that your question was not tagged express. if you aren't using express you may want to look at the connect.session example, this is what express uses behind the scenes to populate req.session.
Depending on exactly what you want to do with the data, you might prefer to use req.session to store temporary information until the client closes the window or their cookie times out. Here's an example that keeps track of the address across future requests:
var express = require('express');
var app = express();
app.use(express.logger());
app.use(express.cookieParser());
app.use(express.bodyParser());
app.use(express.session({secret:'$tackoverflow-rules'}));
app.get('/', function(req, res){
if(req.session.address === undefined){
res.send(200,'<html><body><form action="address" method="post">'
+'<input type="text" name="address" placeholder="Enter address">'
+'</form></body></html>');
} else {
res.send(200,'<html><body><span>I know that you live at '+req.session.address+'!</span></body></html>');
};
});
app.post('/address', function(req, res){
if(req.body.address !== undefined && req.body.address !== ""){
req.session.address = req.body.address;
res.redirect('/');
} else {
req.session.address = undefined;
res.redirect('/');
};
});
require('http').createServer(app).listen(3000);
You can store the information in cookies or localStorage if the data is being displayed on a webpage. If you are using a jsp or php I would store it in a session. What client are you using?
You can create a global object with a clients address as the key that stores info:
var client = {}
//on connect
client[clientID] = {address: clientAddress} //so on
//To access
client[clientID].address; //or whatever you need
This is how I store client specific information with socket.io
If the data is small you can assign the data to a cookie and that way it will be passed back and forth on each request to the server (Localstorage will allow it to keep it on the client but it will NOT be be passed again to the server on the next request)
You can keep a collection in memory with this data on the server and look it up as #tymeJV indicated but this will only works if you have the data is not too big and if you run in a single server. As soon as the data grows you might run out of memory. Also if the server restarts you'll loose this information. As soon as you add a second server this approach will not work because you'll need to make sure the client connects to the same server (which I guess you could do with sticky sessions)
You might want to look into caching this data with something like Redis or another fast database like MongoDB for this kind of session information. That way you'll have a central store for it that could be optimized for performance so your site does not get bogged down while checking for this information.
I have made a couchdb design document which works perfectly on the following url
http://localhost:5984/db/_design/app/index.html
Now the problem is i am trying to fetch the page contents and display it from node js but only the html page is displayed the linked css and js files are not working and when i tried to narrow down the problem i found that the css and js files are suppose to have the login credentials of the couchdb and is not linking I even tried adding the auth header in the response parameter but still no luck
var http = require('http');
var json;
var root = new Buffer("admin:pass").toString('base64');
http.createServer(function(req, res) {
res.setHeader('Authorization', root);
res.writeHead(200, { 'Content-Type':'text/html' });
couchPage();
res.end(json);
}).listen(8080);
function couchPage() {
var options = {
hostname: 'localhost',
port: 5984,
path: '/db/_design/app/index.html',
auth: 'admin:pass',
method: 'GET'
};
var req = http.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (chunk) {
json = chunk;
});
});
req.end();
}
could any one please guide me where am i wrong
I think this has nothing to do with couchdb authorization. The problem is that you do not perform any routing on your nodejs server. That is, the browser makes a request to localhost:8080 and receives the content of /db/_design/app/index.html as an answer. Now, the browser detects a link to a stylesheet, say "style.css". It performs a request to localhost:8080/style.css but your nodejs server simply ignores the "style.css" part of the request. Instead, the client will receive the content of /db/_design/app/index.html again!
If you want to serve attachments of your design document through nodejs, you have to parse the request first and then retrieve the corresponding document from couchdb. However, I don't think that you actually want to do this. Either you want to use couchdb in a traditional way behind nodejs (not directly accessible from the client) and then you would just use it as a database with your html (or template) files stored on disk. Or you want to directly expose couchdb to the client and make nodejs listen to events via the couchdb _changes feed.