I'm trying to make a simple HTTP module, similar to express, to help learn how to use HTTP.
When working with express I often use parameters such as:
app.get('/id/:id' (req, res) => {console.log(req.params.id); stuff})
I was wondering-
Is it possible just using HTTP?
If it isn't able to be done with just HTTP, then how would you go about creating it?
Your question is a little bit confusing but I think what you mean is how to implement an HTTP Router in pure javascript instead of relying on a framework like express.
If that's the case I support your initiative 100%! It's really important to understand what is going on behind the scenes.
A good way to really understand a good way to do so would be by reading the source code of a good router that is already out there.
You could study the Express router's source code but I recommend you to go play with find-my-way which is a dedicated router that you can use with HTTP only without any other framework.
Yes but you need to process it manually.
Assuming you're using node's http module, the information you are looking for is in req.url but it includes the whole url path.
For example, you want to parse /id/:id, and the browser is making a request to http://your.server/id/100?something=something. Then the value of req.url will be:
/id/100?something=something
But if you are writing an HTTP module from scratch using the net module then you need to know the format of an HTTP request. Basically an HTTP request looks like a text file with the following format:
GET /id/100?something=something HTTP/1.1
Host: your.server
The header section ends with double newlines. Technically it should be \r\n\r\n but \n\n is also acceptable. You first need to get the url from the first line of the request between the protocol (GET in the example above but can be POST, PUT etc.) and the HTTP version number.
For anyone interested in writing a HTTP server from scratch I always recommend James Marshall's excellent article: https://www.jmarshall.com/easy/http/. It was originally written in the late 90s but to this day I haven't found a clearer summary of the HTTP protocol. I've used it myself to write my first HTTP server.
Now you have to write code to extract the 100 from the string.
If you are doing this manually, not trying to write a framework like Express, you can do it like this:
const matches = req.url.match(/id\/([^\/?]+)/);
const id = matches[1];
If you want to write a library to interpret the /id/:id pattern you can maybe do something like this (note: not an optimized implementation, also it behaves slightly differently to express):
function matchRoute (req, route) {
const [ urlpath, query ] = req.url.split('?'); // separate path and query
const pathParts = urlpath.split('/');
const routeParts = urlpath.split('/');
let params = {};
for (let i=0; i<pathParts.length; i++) {
if (routeParts[i].match(/^:/)) { // if route part starts with ":"
// this is a parameter:
params[routeParts[i].replace(':','')] = pathParts[i];
}
else {
if (routeParts[i] !== urlParts[i]) {
return false; // false means path does not match route
}
}
// if we get here (don't return early) it means
// the route matches the path, copy all found params to req.params:
req.params = params;
return true; // true signifies a match so we should process this route
}
}
// Test:
let req.url = '/id/100';
let result = matchRoute(req, '/id/:id');
console.log(result, req.params); // should print "true, {id:100}"
Related
The code segment is:
app.get('/api/photo/:S3ObjectKey', photo.get);
And photo.get():
const S3ObjectKey = req.params.S3ObjectKey;
if (!S3ObjectKey) {
console.log("No S3ObjectKey specified");
}
console.log("S3ObjectKey: ", S3ObjectKey);
const canShow = true;
if (canShow) {
const bucket_name = process.env.AWS_S3_BUCKET_NAME;
const data = await S3.getFile(bucket_name, S3ObjectKey);
data.Body.pipe(res);
}
Is there a way to map all the rest of url as one parameter, such that:
a GET request to https://mylovelydomain/api/photo/bucketMyPhoto/2020/07/12/home/table.jpeg would hit the api endpoint and S3ObjectKey has the value bucketMyPhoto/2020/07/12/home/table.jpeg?
Express uses path-to-regex:
Custom Matching Parameters
Parameters can have a custom regexp, which overrides the default match ([^/]+).
On your routing path you can use * to get everything overwriting that default match (note that this code also reatcs to empty requests):
app.get("/api/photos/:S3ObjectKey(*)", /*...*/)
As per express documentation, route path supports regular expression, so should be able to do it. Quoting the example from the documentation.
I am using Express with Body Parser. Given a below header key:
X-Master-Key
When I am using the below code snippet, it fails to output the value
req.headers['X-Master-Key'] // Fails
but when the above is changed to, it works
req.headers['x-master-key'] // Works
Further, when I tried to output req.headers, it turns out that Express outputs all the headers in a down-case format.
I started digging down further and tried using the below code, either of these snippets work
req.header('X-Master-Key'); // Works
// -- OR
req.header('x-master-key'); // Works
So what's the issue here? Why does Express changes all header keys to down-case? Moreover, how using req.header() is different from req.headers[]?
The problem arises because in the HTTP protocol, headers are case-insensitive. This means that content-type, Content-Type, and coNTEnt-tYPe all refer to the same header, and the Express framework needs to be able to handle any of them.
The different between req.headers (the object) and req.header (the function) is simply this:
If you want to get a property from a Javascript object, the property name is case-sensitive. So req.headers['content-type'] will work; req.headers['Content-Type'] will not. Why does the lower case version work? Because the Express framework, in an attempt to handle all the different possible cases (remember, HTTP will allow anything), converts everything to lower case.
But the developers of Express recognize that you (the developer) might be looking for Content-Type and you might not remember to convert to lower case, so they provided a function, req.header, which will take care of that for you.
So, in short:
This is recommended:
const myHeader = req.header('Content-Type');
Use whatever case you want - the function will convert it to lower case and look up the value in req.headers.
This is not recommended:
const myHeader = req.headers['Content-Type'];
If you don't use a lower-case header name, you won't get what you expect.
The problem comes down to case-sensitivity.
When you look at the documentation for req.get (which is aliased by req.header), it states:
Returns the specified HTTP request header field (case-insensitive match). The Referrer and Referer fields are interchangeable.
The w3 standard indicates that headers should be case-insensitive:
Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive.
So it would appear that node http module, which express uses, just treats them all as lower-case to "save you steps" according to this github issue
You can see that the express framework req object actually utilizes the node module http:
var accepts = require('accepts');
var deprecate = require('depd')('express');
var isIP = require('net').isIP;
var typeis = require('type-is');
var http = require('http');
var fresh = require('fresh');
var parseRange = require('range-parser');
var parse = require('parseurl');
Furthermore, in the code you can see that the req.header method converts whatever you give it to lower-case:
req.get =
req.header = function header(name) {
if (!name) {
throw new TypeError('name argument is required to req.get');
}
if (typeof name !== 'string') {
throw new TypeError('name must be a string to req.get');
}
var lc = name.toLowerCase();
switch (lc) {
case 'referer':
case 'referrer':
return this.headers.referrer
|| this.headers.referer;
default:
return this.headers[lc];
}
};
Finally, the http module parses headers using the matchKnownFields function which automatically lower-cases any and all headers that aren't "traditional headers", in which case it is case-insensitive.
Here is the responsible snippet, that implements the behavior you are seeing:
if (lowercased) {
return '\u0000' + field;
} else {
return matchKnownFields(field.toLowerCase(), true);
}
I'm still new enough with Node that HTTP requests trip me up. I have checked all the answers to similar questions but none seem to address my issue.
I have been dealt a hand in the Wild of having to go after JSON files in an API. I then parse those JSON files to separate them out into rows that populate a SQL database. The API has one JSON file with an ID of 'keys.json' that looks like this:
{
"keys":["5sM5YLnnNMN_1540338527220.json","5sM5YLnnNMN_1540389571029.json","6tN6ZMooONO_1540389269289.json"]
}
Each array element in the keys property holds the value of one of the JSON data files in the API.
I am having problems getting either type of file returned to me, but I figure if I can learn what is wrong with the way I am trying to get 'keys.json', I can leverage that knowledge to get the individual JSON data files represented in the keys array.
I am using the npm modules 'request' and 'request-promise-native' as follows:
const request = require('request');
const rp = require('request-promise-native');
My URL is constructed with the following elements, as follows (I have used the ... to keep my client anonymous, but other than that it is a direct copy:
let baseURL = 'http://localhost:3000/Users/doug5solas/sandbox/.../server/.quizzes/'; // this is the development value only
let keysID = 'keys.json';
Clearly the localhost aspect will have to go away when we deploy but I am just testing now.
Here is my HTTP call:
let options = {
method: 'GET',
uri: baseURL + keysID,
headers: {
'User-Agent': 'Request-Promise'
},
json: true // Automatically parses the JSON string in the response
};
rp(options)
.then(function (res) {
jsonKeysList = res.keys;
console.log('Fetched', jsonKeysList);
})
.catch(function (err) {
// API call failed
let errMessage = err.options.uri + ' ' + err.statusCode + ' Not Found';
console.log(errMessage);
return errMessage;
});
Here is my console output:
http://localhost:3000/Users/doug5solas/sandbox/.../server/.quizzes/keys.json 404 Not Found
It is clear to me that the .catch() clause is being taken and not the .then() clause. But I do not know why that is because the data is there at that spot. I know it is because I placed it there manually.
Thanks to #Kevin B for the tip regarding serving of static files. I revamped the logic using express.static and served the file using that capability and everything worked as expected.
I need a way to parse my JSONP object on server side to save it, due to cross domain origin issue I have shifted my way of communication from JSON to JSONP but not finding any suitable way to parse JSONP on server side to save it to the database.
Following is the Model,
define(['backbone'],function(Backbone){
'use strict';
return Backbone.Model.extend({
url:"http://crossdomain:9847/page",
defaults: {
type:'text',
position:0,
align:'left',
text:{"en":""},
color:"#000",
weight:'normal',
size:"14px",
font:"Verdana",
pageid:'askdkasdkgaskdgks'
},
idAttribute:'_id',
sync: function(method, collections, options) {
options.dataType = "jsonp";
return Backbone.sync(method, collections, options);
}
});
});
Express Server,
var express = require('/root/node_modules/express');
var page = require('./routes/page.js');
var app = express();
app.configure(function () {
app.use(express.json());
app.use(express.urlencoded());
app.set("jsonp callback", true);
})
app.get('/page', page.updatePage);
app.listen(9847);
exports.updatePage = function(req, res) {
console.log(req.query);
// Here how I can parse the req is my problem
// so I can save object to database?
}
URL is generating like,
http://crossdomain:9847/page?callback=jQuery203010156283635587138_1384408493698&{%22text%22:{%22en%22:%22Text%22},%22type%22:%22text%22,%22position%22:0,%22align%22:%22left%22,%22color%22:%22#000%22,%22weight%22:%22normal%22,%22size%22:%2214px%22,%22font%22:%22Verdana%22,%22pageid%22:%22askdkasdkgaskdgks%22}&_=1384408493700
and I am able to receive,
{ callback: 'jQuery203010156283635587138_1384408493698',
'{"text":{"en":"Text"},"type":"text","position":0,"align":"left","color":"': '' }
Now how can I parse this ? I can get callback from callback parameter, but how to get actual data ?
You can't parse the result because it's not valid JSON. Your problem is probably in this line:
app.set("jsonp callback", true);
This is where you set the JSONP callback variable, for example changing it from the default of callback to instead be callbackVariable.
Just comment out that line, and the JSONP you get back will hopefully be parseable. Or, you might also have to fix how Backbone is constructing the JSONP URL. If you instead used a URL like http://crossdomain:9847/page?callback=jQuery203010156283635587138_1384408493698&type=text&position=0&align=left&color=%23000&weight=normal&size=14px&font=Verdana&pageid=askdkasdkgaskdgks I believe it would work. Backbone seems to be adding additional encoding into the values in the URL, which makes parsing harder.
Finally, if you need help easily picking specific values out of a (valid) JSON string that has been parsed into a Javascript object, take a look at the many useful function in lodash.
Is it possible to access the NancyContext from a route definition?
What I would like to build is a single route for a legacy request that receives parameters in a way like this:
/points/upload?longitude=12.34&latitude=23.45
(And I did not find a way to specify these parameters with patterns, as Nancy seems to ignore anything behind the question mark.)
Everything after the question-mark is a query-string and you access those using the Request.Query property, inside your route
Get["/points/upload"] = x => {
var long = Request.Query.longitude;
var lang = Request.Query.latitude;
return 200;
}