The request documentation talks about importing cookies from a file with the following example:
var FileCookieStore = require('tough-cookie-filestore');
// NOTE - currently the 'cookies.json' file must already exist!
var j = request.jar(new FileCookieStore('cookies.json'));
request = request.defaults({ jar : j })
request('http://www.google.com', function() {
request('http://images.google.com')
})
However, as noted in the comment, it expects cookies.json to already exist. The question is, if I have an exsting jar with cookies in it, how can I export it to JSON?
I am not sure to understand what you mean by "if I have an exsting jar with cookies in it", but here is how I manage persistent cookies with nodejs.
To avoid errors with FileCookieStore, I add a piece of code to create the json file if it does not exist. The file can be empty, as long as it exists:
if(!fs.existsSync(cookiepath)){
fs.closeSync(fs.openSync(cookiepath, 'w'));
}
Now, if you look closely at the FileCookieStore code, you will see it calls the saveToFile method anytime there is a change in the cookies. It means that by passing a FileCookieStore object to the request module (using the jar option as the request documentation explains), the json file will always reflect the state of the cookies.
Here is a complete example:
var FileCookieStore = require('tough-cookie-filestore');
var request = require('request');
var fs = require("fs");
var cookiepath = "cookies.json";
// create the json file if it does not exist
if(!fs.existsSync(cookiepath)){
fs.closeSync(fs.openSync(cookiepath, 'w'));
}
// use the FileCookieStore with the request package
var jar = request.jar(new FileCookieStore(cookiepath));
request = request.defaults({ jar : jar });
// do whatever you want
request('http://www.google.com', function() {
request('http://images.google.com')
});
// the cookies in 'jar' corresponds to the cookies in cookies.json
console.log(jar);
To start anew, simply delete the cookipath file.
Related
I'm working on a simple local image server that provides images to a web application with some JSON. The web application has pagination that will do a get request "/images?page=X&limit&200" to an express.js server that returns the JSON files in a single array. I want to take advantage of the browser's internal caching such that if a user goes to a previous page the express.js returns an ETAG. I was wondering how this could be achieved with express.js? For this application, I really just want the computation of the ETAG to take in three parameters the page, the directory, and the limit (It doesn't need to consider the whole JSON body). Also this application is for local use only, so I want the server to do the heavy lifting since I figured it be faster than the browser. I did see https://www.npmjs.com/package/etag which seems promising, but I'm not sure how to use it with express.js
Here's a boilerplate of the express.js code I have below:
var express = require('express');
var app = express();
var fs = require('fs');
app.get('/', async (req, res) =>{
let files = [];
let directory = fs.readdirSync("mypath");
let page = parseInt(req.query.page);
let limit = parseInt(req.query.limit);
for (let i = 0; i < limit; ++i) {
files.push(new Promise((resolve) => {
fs.readFile(files[i + page * limit].name, (err, data) => {
// format the data so easy to use for UI
resolve(JSON.parse(data));
});
});
}
let results = await Promise.all(files);
// compute an etag here and attach it the results.
res.send(results);
});
app.listen(3000);
When your server sends an ETag to the client, it must also be prepared to check the ETag that the client sends back to the server in the If-None-Match header in a subsequent "conditional" request.
If it matches, the server shall respond with status 304; otherwise there is no benefit in using ETags.
var serverEtag = "<compute from page, directory and limit>";
var clientEtag = req.get("If-None-Match");
if (clientEtag === serverEtag) res.status(304).end();
else {
// Your code from above
res.set("ETag", serverEtag);
res.send(results);
}
The computation of the serverEtag could be based on the time of the last modification in the directory, so that it changes whenever any of the images in that directory changes. Importantly, this could be done without carrying out the fs.readFile statements from your code.
I simply want to download a .gz file from a URL and save it in a folder. I would like to do this without any third party libraries if possible. Here's what I have so far, but it only downloads an empty file:
const fs = require('fs')
const https = require('https')
let file = fs.createWriteStream('./folder/filename.gz')
let request = https.get('https://someurl/somefile.gz', function(res) {
res.pipe(file)
})
you can try this, using HTTP module for nodesJS,it looks similar to downloading any other file, just remember to mention the extension of the downloaded file when calling instead....Here is an example:
NOTE: IF you are trying to download from an HTTPS link, use the HTTPS
module instead, its exactly the same, but just replace all the
HTTP in the following code with HTTPS
const http = require('http');
const fs = require('fs');
//I added './' assuming that you want to download it where the server
//file is located, just change it to your desired path, followed by the
//filename and the EXTENSION
const file = fs.createWriteStream("./result.tar.gz");
const request = http.get("http://alpha.gnu.org/gnu/gzip/gzip-1.3.6.tar.gz", (response) => {
response.pipe(file);
});
Im wondering if is posible to know what is the file name of an incomming binary request.
This is my situation I have this code that handles the file upload
router.route('/:filename')
.put(function(req,res){
var uuid = guid();
var fileExtension = req.params.filename.substring(req.params.filename.lastIndexOf("."));
if(!fs.existsSync('../files')){
fs.mkdirSync('../files')
}
var newFile = fs.createWriteStream('../files/'+uuid+fileExtension);
req.pipe(newFile);
req.on('end',function(end){
console.log("Finished")
res.send(uuid+fileExtension)
})
})
as you can see now ,I need the file name specified in the URL('/:filename'). My question is: If it is possible to take that attribute from the resquest stream, instead the url or a form key?
If you use multer middleware you can access the uploaded filename like so
var multer = require('multer')
var upload = multer()
router.route('/:filename')
.put(upload.single('fileField'),function(req,res){
var fileName = req.file.originalname
var uuid = guid();
var fileExtension = req.params.filename.substring(req.params.filename.lastIndexOf("."));
if(!fs.existsSync('../files')){
fs.mkdirSync('../files')
}
var newFile = fs.createWriteStream('../files/'+uuid+fileExtension);
req.pipe(newFile);
req.on('end',function(end){
console.log("Finished")
res.send(uuid+fileExtension)
})
})
You'll need to inspect the Content-Disposition header of the request and parse the file name information out if processing the HTTP request manually.
However, I'd recommend you look at some of the existing file upload middlewares, no point in reinventing the wheel
busboy
multer
formidable
multiparty
pez
Is it possible to fetch data from MongoDB and render a html template on the server side itself for a node-js project?
As of now in my serverside js file I've done the following.
//Failing array will be populated by a db.find later on.
var failing = [
{ name: "Pop" },
{ name: "BOB" }
];
/*Now i have to send a mail from the server for which I'm using nodemailer.
Where do i store the template ? This is what I've done in the same file */
var template = "<body>{#failing} <p>{.name}</p> {/failing}</body>"
// Add this as the body of the mail and send it.
I'm not sure how to render the data and how to get it displayed. I'm aware storing the template in the variable isn't right but I'm not sure what else to do.
If your template is that short, you can store it in a variable without problem. Obviously, you can store it in a file also.
Let's say you decide to store it in a file index.dust:
<body>{#failing} <p>{.name}</p> {/failing}</body>
Now, in your node controller you need to load the file and generate the html content from it:
const fs = require('fs');
const dust = require('dustjs-linkedin');
// Read the template
var src = fs.readFileSync('<rest_of_path>/index.dust', 'utf8');
// Compile and load it. Note that we give it the index name.
var compiled = dust.compile(src, 'index');
dust.loadSource(compiled);
// Render the template with the context. Take into account that this is
// an async function
dust.render('index', { failing: failing }, function(err, html) {
// In html you have the generated html.
console.log(html);
});
Check the documentation in order not to have to compile the template every time you have to use it.
I was under the impression that when you run a nodejs webserver the root of the web is the folder containing the js file implementing the webserver. So if you have C:\foo\server.js and you run it, then "/" refers to C:\foo and "/index.htm" maps to C:\foo\index.htm
I have a file server.js with a sibling file default.htm, but when I try to load the contents of /default.htm the file is not found. An absolute file path works.
Where is "/" and what controls this?
Working up a repro I simplified it to this:
var fs = require('fs');
var body = fs.readFileSync('/default.htm');
and noticed it's looking for this
Error: ENOENT, no such file or directory 'C:\default.htm'
So "/" maps to C:\
Is there a way to control the mapping of the web root?
I notice that relative paths do work. So
var fs = require('fs');
var body = fs.readFileSync('default.htm');
succeeds.
I believe my confusion arose from the coincidental placement of my original experiment's project files at the root of a drive. This allowed references to /default.htm to resolve correctly; it was only when I moved things into a folder to place them under source control that this issue was revealed.
I will certainly look into express, but you haven't answered my question: is it possible to remap the web root and if so how?
As a matter of interest this is server.js as it currently stands
var http = require('http');
var fs = require('fs');
var sys = require('sys');
var formidable = require('formidable');
var util = require('util');
var URL = require('url');
var QueryString = require('querystring');
var mimeMap = { htm : "text/html", css : "text/css", json : "application/json" };
http.createServer(function (request, response) {
var body, token, value, mimeType;
var url = URL.parse(request.url);
var path = url.pathname;
var params = QueryString.parse(url.query);
console.log(request.method + " " + path);
switch (path) {
case "/getsettings":
try {
mimeType = "application/json";
body = fs.readFileSync("dummy.json"); //mock db
} catch(exception) {
console.log(exception.text);
body = exception;
}
break;
case "/setsettings":
mimeType = "application/json";
body="{}";
console.log(params);
break;
case "/":
path = "default.htm";
default:
try {
mimeType = mimeMap[path.substring(path.lastIndexOf('.') + 1)];
if (mimeType) {
body = fs.readFileSync(path);
} else {
mimeType = "text/html";
body = "<h1>Error</h1><body>Could not resolve mime type from file extension</body>";
}
} catch (exception) {
mimeType = "text/html";
body = "<h1>404 - not found</h1>" + exception.toString();
}
break;
}
response.writeHead(200, {'Content-Type': mimeType});
response.writeHead(200, {'Cache-Control': 'no-cache'});
response.writeHead(200, {'Pragma': 'no-cache'});
response.end(body);
}).listen(8124);
console.log('Server running at http://127.0.0.1:8124/');
I'm not completely certain what you mean by "routes" but I suspect that setsettings and getsettings are the sort of thing you meant, correct me if I'm wrong.
Nodejs does not appear to support arbitrary mapping of the web root.
All that is required is to prepend absolute web paths with a period prior to using them in the file system:
var URL = require('url');
var url = URL.parse(request.url);
var path = url.pathname;
if (path[0] == '/')
path = '.' + path;
While you're correct that the root of the server is the current working directory Node.js won't do a direct pass-through to the files on your file system, that could be a bit of a security risk after all.
Instead you need to provide it with routes that then in turn provide content for the request being made.
A simple server like
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(1337, '127.0.0.1');
Will just capture any request and respond in the same way (but doesn't read the file system).
Now if you want to serve out file contents you need to specify some way to read that file into the response stream, this can be done a few ways:
You can use the fs API to find the file on disk, read its contents into memory and then pipe them out to the response. This is a pretty tedious approach, especially when you start getting a larger number of files, but it does allow you very direct control over what's happening in your application
You can use a middleware server like express.js, which IMO is a much better approach to do what you're wanting to do. There's plenty of questions and answers on using Express here on StackOverflow, this is a good example of a static server which is what you talk about
Edit
With the clarification of the question the reason:
var body = fs.readFileSync('/default.htm');
Results in thinking the file is at C:\default.htm is because you're using an absolute path not a relative path. If you had:
var body = fs.readFileSync('./default.htm');
It would then know that you want to operate relative to the current working directory. / is from the root of the partition and ./ is from the current working directory.