I am in the midst of writing a router that uses a JSON object as a manifest. The router uses the JSON manifest as a reference for where to route incoming requested url paths. The code below belongs to main.js that is ran using the node command to start the application. Near the top of the document, you can see the router object that takes the JSON Manifest as an argument. Below this bit of code, is another bit, that shows the router document where the router class is written into it, you can see there how I am attempting to route request using this JSON Manifest.
const app = require('./serv/katanaLib');
const Router = require('./serv/router');
var router = new Router.Router({
"^/(.*)$" : "/public/view/page/:[?]",
"^/user/(.*)$" : "/public/view/account/:[?]",
"^/style/(.*)$" : "/public/style/:[?]"
});
app.Server(8888, function(request, response)
{
const $_Request = app.parseHttpReq(request);
router.onReq($_Request);
});
Below is router.js, this is where I am having a problem, which I will explain at the bottem of this question.
module.exports.Router = class Router
{
constructor(JSON_manifest)
{
this.manifest = JSON_manifest;
this.keys = Object.keys(JSON_manifest);
this.values = Object.values(JSON_manifest);
};
onReq($_Request)
{
this.keys.forEach((key, i)=>
{
key = ('^' + key + '$');
var re = new RegExp(key, 'i');
let x = $_Request.path.search(re);
console.log('\n\n' + key);
console.log(re);
console.log(x + '\n\n');
});
};
};
As anyone can obviously see, I am writing Regular Expressions into the keys of the JSON manifest object. You can see that I search the requested path using a foreach loop, looping through each manifest key. The problem I am having is that my regex values are matching multiple times. When I run the code above, and make the follow request to my server.
http://localhost:8888/index
Request Recieved:
Parsed-Request:
{ path: '/index', method: 'GET', ext: '', type: 'text/html' }
^^/(.*)$$
/^^\/(.*)$$/i
0
^^/user/(.*)$$
/^^\/user\/(.*)$$/i
-1
^^/style/(.*)$$
/^^\/style\/(.*)$$/i
-1
that is a desireable result, but when I run:
http://localhost:8888/user/abc
i get...
Parsed-Request:
{ path: '/user/abc', method: 'GET', ext: '', type: 'text/html' }
^^/(.*)$$
/^^\/(.*)$$/i
0
^^/user/(.*)$$
/^^\/user\/(.*)$$/i
0
^^/style/(.*)$$
/^^\/style\/(.*)$$/i
-1
The Regular Expression is matching both, the first and second, which is undesirable. If I requested http://localhost/style/123 the requested path will match the first key and third key. If anyone has advice on how to write the regex so I can finish this that would be great, thank you...
To solve the issue with multiple regexes matching, it's best to test them in a specific order, from most specific to least specific. Then, the first match is to be used.
In your example, the regexes are provided as object keys.
Since object keys are unordered, you'd need to use an array instead to control the order of the regexes.
In this case, the order of regexes would be as follows (first and second can be swapped):
^/user/(.*)$
^/style/(.*)$
^/(.*)$
Related
I am having difficulties in parsing an HTTP request that uses a "%26" instead of a "&" character.
Context: I am trying to receive an HTTP request made by an IP camera that has a limitation where it cannot send the '&' character, so I have to set it up using '%26' to separate parameters, so the camera is making a POST request like this:
http://192.168.45.4:8000/test?foo=me%26bar=them
To process this request on my server side, I have this simple function:
Broker.all("*", (req,res) => {
console.log("foo=" + req.query.foo);
console.log("bar=" + req.query.bar);
console.log("Received a generic put request");
res.send("Processed");
});
This code is working fine when I make a request using the '&' character.
This is the result to the request: http://192.168.45.4:8000/test?foo=me&bar=them
foo=me
bar=them
Received a generic put request
This is the result to the request: http://192.168.45.4:8000/test?foo=me%26bar=them
foo=me&bar=them
bar=undefined
Received a generic put request
I have been banging my head for the past... (ashamed)... nevermind how many hours, I am not sure what I am doing wrong.
I'm pretty sure this is easy, but I really couldn't find it on the forums, maybe because I wasn't able to search specifically for "%26" in the posts.
Any help would be appreciated.
EDIT after being solved
As suggested by GPack, I used his parser code, and it seems to be working. Just need to make sure to check beforehand if the Generic URL does have parameters or not.
I still believe there might be a more natural way to do this with express...
This is the new code with the parser which is working:
broker.all("*", (req,res) => {
const list = req.originalUrl.split("?")[1].split("%26");
const params = list.reduce((obj, str) => {
const parts = str.split("=");
obj[parts[0]] = parts[1];
return obj;
}, {});
console.log(params); // { bar: "them", foo: "me" }
console.log("foo=" + params.foo);
console.log("bar=" + params.bar);
console.log("Received a generic put request");
res.send("I'm just a simple broker");
});
I tried using decodeURI & decodeURIComponent but in vain.
So I wrote the parser myself:
// The request object should be something like this (with other attributes)
const req = {
originalUrl: "/test?foo=me%26bar=them"
}
// You may want to handle the case where a request doesn't have parameters
const list = req.originalUrl.split("?")[1].split("%26");
const params = list.reduce((obj, str) => {
const parts = str.split("=");
obj[parts[0]] = parts[1];
return obj;
}, {});
console.log(params); // { bar: "them", foo: "me" }
I am using Express with Body Parser. Given a below header key:
X-Master-Key
When I am using the below code snippet, it fails to output the value
req.headers['X-Master-Key'] // Fails
but when the above is changed to, it works
req.headers['x-master-key'] // Works
Further, when I tried to output req.headers, it turns out that Express outputs all the headers in a down-case format.
I started digging down further and tried using the below code, either of these snippets work
req.header('X-Master-Key'); // Works
// -- OR
req.header('x-master-key'); // Works
So what's the issue here? Why does Express changes all header keys to down-case? Moreover, how using req.header() is different from req.headers[]?
The problem arises because in the HTTP protocol, headers are case-insensitive. This means that content-type, Content-Type, and coNTEnt-tYPe all refer to the same header, and the Express framework needs to be able to handle any of them.
The different between req.headers (the object) and req.header (the function) is simply this:
If you want to get a property from a Javascript object, the property name is case-sensitive. So req.headers['content-type'] will work; req.headers['Content-Type'] will not. Why does the lower case version work? Because the Express framework, in an attempt to handle all the different possible cases (remember, HTTP will allow anything), converts everything to lower case.
But the developers of Express recognize that you (the developer) might be looking for Content-Type and you might not remember to convert to lower case, so they provided a function, req.header, which will take care of that for you.
So, in short:
This is recommended:
const myHeader = req.header('Content-Type');
Use whatever case you want - the function will convert it to lower case and look up the value in req.headers.
This is not recommended:
const myHeader = req.headers['Content-Type'];
If you don't use a lower-case header name, you won't get what you expect.
The problem comes down to case-sensitivity.
When you look at the documentation for req.get (which is aliased by req.header), it states:
Returns the specified HTTP request header field (case-insensitive match). The Referrer and Referer fields are interchangeable.
The w3 standard indicates that headers should be case-insensitive:
Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive.
So it would appear that node http module, which express uses, just treats them all as lower-case to "save you steps" according to this github issue
You can see that the express framework req object actually utilizes the node module http:
var accepts = require('accepts');
var deprecate = require('depd')('express');
var isIP = require('net').isIP;
var typeis = require('type-is');
var http = require('http');
var fresh = require('fresh');
var parseRange = require('range-parser');
var parse = require('parseurl');
Furthermore, in the code you can see that the req.header method converts whatever you give it to lower-case:
req.get =
req.header = function header(name) {
if (!name) {
throw new TypeError('name argument is required to req.get');
}
if (typeof name !== 'string') {
throw new TypeError('name must be a string to req.get');
}
var lc = name.toLowerCase();
switch (lc) {
case 'referer':
case 'referrer':
return this.headers.referrer
|| this.headers.referer;
default:
return this.headers[lc];
}
};
Finally, the http module parses headers using the matchKnownFields function which automatically lower-cases any and all headers that aren't "traditional headers", in which case it is case-insensitive.
Here is the responsible snippet, that implements the behavior you are seeing:
if (lowercased) {
return '\u0000' + field;
} else {
return matchKnownFields(field.toLowerCase(), true);
}
I'm still new enough with Node that HTTP requests trip me up. I have checked all the answers to similar questions but none seem to address my issue.
I have been dealt a hand in the Wild of having to go after JSON files in an API. I then parse those JSON files to separate them out into rows that populate a SQL database. The API has one JSON file with an ID of 'keys.json' that looks like this:
{
"keys":["5sM5YLnnNMN_1540338527220.json","5sM5YLnnNMN_1540389571029.json","6tN6ZMooONO_1540389269289.json"]
}
Each array element in the keys property holds the value of one of the JSON data files in the API.
I am having problems getting either type of file returned to me, but I figure if I can learn what is wrong with the way I am trying to get 'keys.json', I can leverage that knowledge to get the individual JSON data files represented in the keys array.
I am using the npm modules 'request' and 'request-promise-native' as follows:
const request = require('request');
const rp = require('request-promise-native');
My URL is constructed with the following elements, as follows (I have used the ... to keep my client anonymous, but other than that it is a direct copy:
let baseURL = 'http://localhost:3000/Users/doug5solas/sandbox/.../server/.quizzes/'; // this is the development value only
let keysID = 'keys.json';
Clearly the localhost aspect will have to go away when we deploy but I am just testing now.
Here is my HTTP call:
let options = {
method: 'GET',
uri: baseURL + keysID,
headers: {
'User-Agent': 'Request-Promise'
},
json: true // Automatically parses the JSON string in the response
};
rp(options)
.then(function (res) {
jsonKeysList = res.keys;
console.log('Fetched', jsonKeysList);
})
.catch(function (err) {
// API call failed
let errMessage = err.options.uri + ' ' + err.statusCode + ' Not Found';
console.log(errMessage);
return errMessage;
});
Here is my console output:
http://localhost:3000/Users/doug5solas/sandbox/.../server/.quizzes/keys.json 404 Not Found
It is clear to me that the .catch() clause is being taken and not the .then() clause. But I do not know why that is because the data is there at that spot. I know it is because I placed it there manually.
Thanks to #Kevin B for the tip regarding serving of static files. I revamped the logic using express.static and served the file using that capability and everything worked as expected.
/*Making http request to the api (Git hub)
create request
parse responce
wrap in a function
*/
var https = require("https");
var username = 'lynndor';
//CREATING AN OBJECT
var options = {
host: 'api.github.com',
path: ' /users/'+ username +'/repos',
method: 'GET'
};
var request = https.request(options, function(responce){
var body = ''
responce.on("data", function(chunk){
body += chunk.toString('utf8')
});
responce.on("end", function(){
console.log("Body", body);
});
});
request.end();
Im trying to create a request to the git hub api, the aim is to get the list repository for the specified you, but i keep getting the above mentioned error, please help
for other situation can be helpful
JavaScript encodeURI() Function
var uri = "my test.asp?name=ståle&car=saab";
var res = encodeURI(uri);
Your "path" variable contains space
path: ' /users/'+ username +'/repos',
Instead it should be
path: '/users/'+ username +'/repos',
Use encodeURIComponent() to encode uri
and decodeURIComponent() to decode uri
Its because there are reserved characters in your uri. You will need to encode uri using inbuilt javascript function encodeURIComponent()
var options = {
host: 'api.github.com',
path: encodeURIComponent('/users/'+ username +'/repos'),
method: 'GET'
};
to decode encoded uri component you can use decodeURIComponent(url)
Typically, you do not want to use encodeURI() directly. Instead, use fixedEncodeURI(). To quote MDN encodeURI() Documentation...
If one wishes to follow the more recent RFC3986 for URLs, which makes square brackets reserved (for IPv6) and thus not encoded when forming something which could be part of a URL (such as a host), the following code snippet may help:
function fixedEncodeURI(str) {
return encodeURI(str).replace(/%5B/g, '[').replace(/%5D/g, ']');
}
There is a similar issue with encodeURIComponent() (source: MDN encodeURIComponent() Documentation), as well as a similar fixedEncodeURIComponent() function. These should be used, rather than the actual encodeURI() or encodeURIComponent() function calls...
To be more stringent in adhering to RFC 3986 (which reserves !, ', (, ), and *), even though these characters have no formalized URI delimiting uses, the following can be safely used:
function fixedEncodeURIComponent(str) {
return encodeURIComponent(str).replace(/[!'()*]/g, function(c) {
return '%' + c.charCodeAt(0).toString(16);
});
}
I was getting this error while trying to hit Elasticsearch's API. For me, it was due to Chinese characters in the Document's Title (in the Request I was sending). Switching to all English characters fixed the issue.
Sometimes browser inspector uses abbreviation of long JSON object.
In my case, the data included unescaped characters such as '…' which should not be in http request.
I'm trying to use breeze.js with sails.js.
Therefore I'm using the breeze json uriBuilder.
I logged the req.query and got the following:
{ '{"where":{"name":"foo"},"orderBy":': { '"name"': '' } }
To query waterline objects I need to bring it into a format like this:
{ where: { name: 'foo' }, sort: 'name' }
First thing I tried:
var queryString = JSON.parse(Object.keys(req.query)[0]);
That works as long as I only put a where clause in. But with more parameters i get this strangely formatted json object.
How can I parse it to get the correct object?
Solution:
Don't parse he req.query. Parse the url and parse it. This way u will get a json uery that sails accepts. Now strip of unsupported parameters as select and you're done.
var parsedUrl = urlUtils.parse(req.url, true);
var jsonQueryString = Object.keys(parsedUrl.query)[0];
var jsonQuery = JSON.parse(jsonQueryString);
Good question! We haven't yet documented this adequately, but the basic idea is that there is a breeze-client npm package (https://www.npmjs.com/package/breeze-client) that you can use to parse the incoming json query into a breeze EntityQuery instance, from there you can interrogate the EntityQuery and turn it into whatever server side syntax you want, i.e. in your case Sails.)
// req = the HTTP request object.
var resourceName = req.params.slug; // the HTTP endpoint
var entityQuery = breeze.EntityQuery.fromUrl(req.url, resourceName);
// you would write the transform to sails below.
var sailsQuery = tranformToSails(entityQuery);
This is exactly what we do in the breeze-sequelize npm package where we take the incoming req.query and go thru the process above to create a 'Sequelize' query.
See http://www.getbreezenow.com/sequelize-mysqlpostgressql-lite