I am about to teach creating a simple web server in node.js to my students. I am doing it initially using the http module and returning a static page. The server code looks like this:
var http = require('http');
var fs = require('fs');
http.createServer(function(request, response) {
getFile(response);
}).listen(8080);
function getFile(response) {
var fileName = __dirname + "/public/index.html";
fs.readFile(fileName, function(err, contents) {
if (!err) {
response.end(contents);
} else {
response.end();
console.log("ERROR ERROR ERROR");
}
});
}
index.html looks like this:
<!DOCTYPE html>
<html>
<head>
<title>Static Page</title>
</head>
</body>
<h1>Returned static page</h1>
<p>This is the content returned from node as the default file</p>
<img src="./images/portablePhone.png" />
</body>
</html>
As I would expect, I am getting the index.html page display without the image (because I am not handling the mime-type). This is fine; what is confusing me is, when I look at the network traffic, I would expect to have the index.html returned three times (the initial request, the image request and one for favicon.ico request). This should happen, because the only thing the web server should ever return is the index.html page in the current folder. I logged the __dirname and fileName var and they came out correctly on each request and there were indeed three requests.
So my question is, what am I missing? Why am I not seeing three index.html response objects in the network monitor on Chrome? I know one of the students will ask and I'd like to have the right answer for him.
what is confusing me is, when I look at the network traffic, I would
expect to have the index.html returned three times (the initial
request, the image request and one for favicon.ico request)
When I run your app, I see exactly three network requests in the network tab in the Chrome debugger, exactly as you proposed and exactly as the HTML page and the web server are coded to do. One for the initial page request, one for the image and one for favicon.ico.
The image doesn't work because you don't actually serve an image (you are serving index.html for all requests) - but perhaps you already know that.
So my question is, what am I missing? Why am I not seeing three
index.html response objects in the network monitor on Chrome?
Here's my screenshot from the network tab of the Chrome debugger when I run your app:
The code that you actually wrote (originally, can't be sure you won't edit the question) just serves an index.html. There is nothing in there that could read any other file (like an image).
I don't think you should teach students that syntax/mechanism because it is outdated. For starters, do not teach them to indent with tabs or four spaces. Indent with 2 spaces for JavaScript. Also, it just doesn't make sense to teach ES5 at this point. They should learn ES2015 or later (ES6/ECMAScript 2016/whatever they call it). For the current version of Node out of the box (6.6 as of writing), this would be the equivalent of what you wrote:
const http = require('http');
const fs = require('fs-promise');
http.createServer( (request, response) => {
fs.readFile(`${__dirname}/public/index.html`)
.then( data => {response.end(data)} )
.catch(console.error);
}).listen(8080);
But what you seem to be trying to do is create a gallery script.
Another thing about Node is, there are more than 300,000 modules available. So it just absolutely does not make sense to start from 0 and ignore all 300,000+ modules.
Also, within about three months, 6 at the most, async/await will land in Node 7 without requiring babel. And people will argue that kids will be confused if they don't have enough time toiling with promises, but I don't think I buy that. I think you should just teach them how to set up babel and use async/await. Overall its going to make more sense and they will learn a much clearer way to do things. And then the next time you teach the class you won't need babel probably.
So this is one way I would make a simple gallery script that doesn't ignore all of the modules on npm and uses up-to-date syntax:
import {readFile} from 'fs-promise';
import listFilepaths from 'list-filepaths';
import Koa from 'koa';
const app = new Koa();
app.use(async (ctx) => {
if (ctx.request.querystring.indexOf('.jpg')>0) {
const fname = ctx.request.querystring.split('=')[1];
ctx.body = await readFile(`images/${fname}`);
} else {
let images = await listFilepaths('./images',{relative:true});
images = images.map(i=>i.replace('images/',''));
ctx.body = `${images.map(i=> `<img src = "/?i=${i}" />` )}`;
}
});
app.listen(3000);
Related
I am scraping Make My Trip Flight data for a project but for some reason it doesn't work. I've tried many selectors but none of them worked. On the other hand I also tried scraping another site with the same logic, and it worked. Can someone point out where did I go wrong?
I am using cheerio and axios
const cheerio = require('cheerio');
const axios = require('axios');
Make My Trip:
axios.get('https://www.makemytrip.com/flight/search?itinerary=BOM-DEL-14/11/2020&tripType=O&paxType=A-1_C-0_I-0&intl=false&cabinClass=E').then(urlRes => {
const $ = cheerio.load(urlRes.data);
$('.fli-list.one-way').each((i, el) => {
const airway = $(el).find('.airways-name ').text();
console.log(airway);
});
}).catch(err => console.log(err));
The other site for which the code works:
axios.get('https://arstechnica.com/gadgets/').then(urlRes => {
const $ = cheerio.load(urlRes.data);
$('.tease.article').each((i, el) => {
const link = $(el).find('a.overlay').attr('href');
console.log(link);
});
}).catch(err => console.log(err));
TLDR you should parse
https://voyager.goibibo.com/api/v2/flights_search/find_node_by_name_v2/?search_query=DEL&limit=15&v=2
instead of
https://www.makemytrip.com/flight/search?itinerary=BOM-DEL-14/11/2020&tripType=O&paxType=A-1_C-0_I-0&intl=false&cabinClass=E
Explanation (hope it is clear enough)
Cause you're trying to parse heavy web application using one plain GET request ... it is impossible in this way :)
The main difference between provided urls:
the second web page (yes just a page not js app like makemytrip) 'https://arstechnica.com/gadgets/' respond to you with a complete content
makemytrip respond to you only with a js script, which do the work - loads data and etc.
To parse such complicated web apps you should investigate (press f12 in browser -> web) all requests which are running in your browser on page load and repeat these requests in your script ... like in this case you could notice API endpoint which responds with all needed data.
I think cheerio works just fine, I will recommend go over the HTML again and find a new element, class or something else to search for.
When I went into the given url I did not find .fli-list.one-way in any combination.
Just try to find something more particular to filter on.
If you still need help I can try and scrape it by myself and send you some code
I'm trying out node and some frameworks for node atm, specifically locomotive. However, i seem to be stuck on routing using locomotive. A couple questions i can't find the answer to, so here goes:
why does the locomotive out-of-box install use index.html.ejs as a
filename? Why not just index.ejs? What's the benefit?
i'm trying to add a route to a view: searchName.html.ejs which i
added in the views folder. To achieve this i made a toolController
like this:
var locomotive = require('locomotive').Controller,
toolController = new Controller();
toolController.searchName = function() {
this.render();
}
module.exports = toolController;
I also added a route in routes.js like so:
this.match('searchName', 'tool#searchName');
However, that doesn't work (and yet it's what the documentation says ought to work). The result is a 404 error. So how do i make that route work?
Suppose i want to make a route to eg, anExample.html? How do i go
about that? I notice that in the out-of-the-box app from
locomotive, you cannot enter localhost:3000/index.html . Nor even
localhost:3000/index This seems highly impractical to me, as there
are plenty of users who'll add the specific page they want to go to.
So how can i make that work?
PS: I went through all questions regarding this on stackoverflow and searched the web, but i still can't figure this out.enter code here
The benefit is that this naming scheme allows you to specify several different formats for a single route. So you could have search_name.html.ejs and search_name.xml.ejs, then respond with either view depending on what your client is expecting.
There are a couple issues with the example code you posted. You should be seeing a more descriptive error than a 404, so I'm not sure what's happening there, but here are the fixes to your code that work in my environment.
In the controller:
//tool_controller.js
var locomotive = require('locomotive');
var toolController = new locomotive.Controller();
toolController.searchName = function() {
this.render();
};
module.exports = toolController;
In routes.js:
//routes.js
module.exports = function routes()
{
this.match('searchName', 'tool#searchName');
}
Then, you'll need to change the view to this: views/tool/search_name.html.ejs. It's not clear from the documentation, but locomotive automatically lowercases and underscores actions that are camel-cased, like searchName.
Now start the app and browse to http://localhost:3000/searchName
If you just want to serve a static html file, the easiest way is to just drop it in the public folder. This folder is specifically for serving up static content like client-side js, css, etc. And it works just fine for serving static HTML as well.
Im learning nodejs but I ran into a roadblock. Im trying to setup a simple server that will serve static files. My problem is that unless I explicitly type in the extension in the url, I cannot get the file extension. The httpheader 'content-type' comes in as undefined .
Here is my code, pretty simple:
var http = require("http"),
path = require("path"),
fs = require("fs");
var server = http.createServer(function(request, response){
console.log([path.extname(request.url), request.headers['content-type']]);
var fileName = path.basename(request.url) || "index.html",
filePath = __dirname + '/public/' + fileName;
console.log(filePath);
fs.readFile( filePath, function(err,data){
if (err) throw err
response.end(data);
});
})
server.listen(3000)
Any ideas?
FYI I dont just wanna dive into connect or other, I wanna know whats going on before I drop the grind and go straight to modules.
So static web servers generally don't do any deep magic. For example, nginx has a small mapping of file extensions to mime types here: http://trac.nginx.org/nginx/browser/nginx/conf/mime.types
There's likely also some fallback logic defaulting to html. You can also use a database of file "magic numbers" as is used by the file utility to look at the beginning of the file data and guess based on that.
But there's no magic here. It's basically
go by the file extension when available
maybe go by the beginning of the file content as next option
use a default of html because normally only html resources have URLs with no extensions, whereas images, css, js, fonts, multimedia, etc almost always do use file extensions in their URIs
Also note that browsers generally have fairly robust set of checks that will intepret files correctly even when Content-Type headers are mismatched with the actual response body data.
I am learning node and express from the simplest and when rendering views using res.render('view',{data:data}) is it only a template engine like jade that fits in view. can I not use a normal html?
You can, but this is a problem I ran into when I was learning Node. If you do not want to use a templating engine, you can still have Node just spit out the contents of your HTML file in a static way. For example (VERY BASIC EXAMLE):
var base = '/path/to/your/public_html',
fs = require('fs'),
http = require('http'),
sys = requrie('sys');
http.createServer(function (req,res) {
path = base + req.url;
console.log(path);
path.exists(path, function(exists) {
if(!exists) {
res.writeHead(404);
res.write('Bad request: 404\n');
res.end();
} else {
res.setHeader('Content-Type','text/html');
res.statusCode = 200;
var file = fs.createReadStream(path);
file.on("open",function() {
file.pipe(res);
});
file.on("error",function(err) {
console.log(err);
});
}
});
}).listen(80);
console.log('server on tcp/80');
The great thing about Node is that it forces you to separate templates from logic (to a certain level, you can squeeze a lot of logic into template anyway).
I didn't like Jade and used EJS until it turned out that client-side EJS is different from server-side and you cannot really re-use templates in the browser (as you would definitely want at some point, when you start rendering pages in the browser). You can re-use simple EJS templates but you cannot re-use templates with partials (as most of your templates will be).
After a lot of searching and trial-and-error I ended up using doT templates that are very fast (the fastest there is, actually), light-weight (only 140 lines of JavaScript), can be easily integrated in Express (by following the pattern of consolidate - you can't use consolidate directly with doT yet), can be used in the browser (the function to load partials will have to be different, but it is easy again).
doT seems to have features that I haven't seen in other templating engines, has a very elegant syntax that is the closest to handlebars (my favourite) but still allows normal JavaScript inside (that's why I chose EJS in the first place).
What I basically try to accomplish is to re-use jade partials/templates when getting data through a socket connection. Non working example:
socket.on('company_created', function(company) {
var html = include _company;
$('#companies ul').append(html);
});
Normally I had to create a new li and set the content like so (which is working as expected):
$('#companies ul').append($('<li>').text(company.name));
This is okay for a simple list, but if I had complexer list and stuff, this could get messy pretty quick, plus I had to write plain HTML again, so I figured re-using my already existing jade templates with all their goodness would be awesome, but had not luck, yet.
Any clue?
PS: Please do not tell my to use Ember, Backbone, Derby, Meteor, Angular or whatsoever.
Thanks in advance!
You can compile your jade sources to JS with jade.compile. Then include these sources in the client-side javascript, include jade's runtime.min.js, and refer to your jade templates as to normal JS functions in your client-side code.
For example,
server.js
app.get('/templates/:template.js', function (req, res) {
var template = req.params.template;
response.end([
"window.templates = window.templates || {};",
"window.templates[\"" + template + "\"] = " + jade.compile(template + ".jade", { client: true; });
].join("\r\n"));
});
client.js
$(function() { $("#placeholder").html(window.templates["content"]({user: "Daniel" })); });
content.jade
h1: Hello #{user}!
index.jade
!!!
html
head
script(src='/lib/jquery/jquery.js')
script(src='/lib/jade/runtime.min.js')
script(src='/templates/content.js')
script(src='/scripts/client.js')
body
#placeholder
Note that the code above might be syntactically incorrect and is provided solely to illustrate the idea.
we have a build step that compiles them to functions sort of like penartur mentioned. I dont use extend or include (which dont work on the client anyway ATM), but personally I find we have absolutely no need for that on the client at all since the DOM provides all the separation we need.