I'm using NodeJS with Express, and when I use foreign characters in the URL, they automatically get encoded.
How do I decode it back to the original string?
Before calling NodeJS, I escape characters.
So the string: אובמה
Becomes %u05D0%u05D5%u05D1%u05DE%u05D4
The entire URL now looks like: http://localhost:32323/?query=%u05D0%u05D5%u05D1%u05DE%u05D4
Now in my NodeJS, I get the escaped string %u05D0%u05D5%u05D1%u05DE%u05D4.
This is the relevant code:
var url_parts = url.parse(req.url, true);
var params = url_parts.query;
var query = params.query; // '%u05D0%u05D5%u05D1%u05DE%u05D4'
I've tried url and querystring libraries but nothing seems to fit my case.
querystring.unescape(query); // still '%u05D0%u05D5%u05D1%u05DE%u05D4'
Update 16/03/18
escape and unescape are deprecated.
Use:
encodeURIComponent('אובמה') // %D7%90%D7%95%D7%91%D7%9E%D7%94
decodeURIComponent('%D7%90%D7%95%D7%91%D7%9E%D7%94') // אובמה
Old answer
unescape('%u05D0%u05D5%u05D1%u05DE%u05D4') gives "אובמה"
Try:
var querystring = unescape(query);
You should use decodeURI() and encodeURI() to encode/decode a URL with foreign characters.
Usage:
var query = 'http://google.com';
query = encodeURI(query);
query = decodeURI(query); // http://google.com
Reference on MDN:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURI
Decoding query parameters from a URL
decodeURIComponent cannot be used directly to parse query parameters from a URL. It needs a bit of preparation.
function decodeQueryParam(p) {
return decodeURIComponent(p.replace(/\+/g, ' '));
}
console.log(decodeQueryParam('search+query%20%28correct%29'));
// 'search query (correct)'
SOURCE: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#decoding_query_parameters_from_a_url
#**update may-2020**
# how to use an encoder/decoder on node js
### I am writing my answer due to I have noisy data I spend 4 hour to fixed
data email input = myemail#gmail.com
data URL input = /us/home
```
decodeURI function that only decodes a URL special character
email output =>myemail%40gmail.com
url output => %2Fus%2F
using decodeURIComponent
email output = > myemail#gmail.com
url output => /us/
```
here some clarification where you can use decodeURI and decodeURIComponent a fucntion
Related
I tried to find and retrieve a post by a title but it failed and when I console log requestedTitle = req.params.postTitle it shows the ? is removed from http://localhost:1035/posts/Is%20Google%20Analytics%20is%20illegal? to just "Is Google Analytics is illegal" which fails to find the article with the title "Is Google Analytics is illegal?" with a ?.
app.get('/posts/:postTitle', function (req, res) {
//single blog post
// const postId = req.params.postId
const requestedTitle = req.params.postTitle
Post.findOne({title: requestedTitle}, function (err, post) {
console.log(requestedTitle);
if (post) {res.render('post', {
singleTitle: post.title,
singleContent: post.content,
singleAuthor: post.author,
uploadedImg: post.img,
postCreated: post.created_at
})} else {
res.send(`No "${requestedTitle}" article was found.`)
}
A question mark is interpreted by Express (and per the http spec) as the delimiter for the URL query string. As such, Express has already parsed out the query string and does not match that against your route. The parsed query string will be in req.query.
So, if you send a request for a URL such as:
http://localhost:1035/posts/Is%20Google%20Analytics%20is%20illegal?
Then, the ? will be seen as the delimiter for the query string and your postTitle will just be Is Google Analytics is illegal.
If you want a question mark to be part of what you get in postTitle, then the client will have to properly encode the postTitle using encodeURIComponent() before constructing the URL. That would "hide" the ? from Express (it would be encoded as %3F and it would then be left in the decoded value of postTitle as a ?.
FYI, there are many other characters which are also reserved so ANY end-user content you're trying to put into the URL piece that you want to be postTitle MUST be encoded with encodeURIComponent() when constructing the URL. Note, you don't call encodeURIComponent() on the entire URL, just on the piece of user data that becomes the postTitle. Express will then handle the decoding for you.
I'm trying to get param from request object using following code.
module.exports = function (req, res) {
var query = req.query;
var data = JSON.parse(query.param1);
}
This is working fine for most of the cases.
If param1 contains & character then query.param1 get values before & and next values are considered as new param.
eg
localhost/?param1={"url":"http://s.test.com/x?format=jsonp&id=a&callback=b"}
Edit original url is already encoded
localhost/?param1=%7B%22url%22%3A%22http%3A%2F%2Fs.test.com%2Fx%3Fformat%3Djsonp%26id%3Da%26callback%3Db%22%7D
in this case I'm getting param1 = {"url":"http://s.test.com/x?format=jsonp
which is not valid json, so I'm getting error, currently we've solved it using regex (removing localhost/?param1= part of url).
What's best way to handle this use case?
Edit : server environment
centos 6.5 server
node v0.12.7
express#4.13.3
req.query is already an object with express :
An object containing a property for each query string parameter in the route. If there is no query string, it is the empty object, {}.
req.query documentation
You have to encode your URL before calling nodejs API. And on the nodejs side, you have to decode the URL to get correct parameteres.
/*Making http request to the api (Git hub)
create request
parse responce
wrap in a function
*/
var https = require("https");
var username = 'lynndor';
//CREATING AN OBJECT
var options = {
host: 'api.github.com',
path: ' /users/'+ username +'/repos',
method: 'GET'
};
var request = https.request(options, function(responce){
var body = ''
responce.on("data", function(chunk){
body += chunk.toString('utf8')
});
responce.on("end", function(){
console.log("Body", body);
});
});
request.end();
Im trying to create a request to the git hub api, the aim is to get the list repository for the specified you, but i keep getting the above mentioned error, please help
for other situation can be helpful
JavaScript encodeURI() Function
var uri = "my test.asp?name=ståle&car=saab";
var res = encodeURI(uri);
Your "path" variable contains space
path: ' /users/'+ username +'/repos',
Instead it should be
path: '/users/'+ username +'/repos',
Use encodeURIComponent() to encode uri
and decodeURIComponent() to decode uri
Its because there are reserved characters in your uri. You will need to encode uri using inbuilt javascript function encodeURIComponent()
var options = {
host: 'api.github.com',
path: encodeURIComponent('/users/'+ username +'/repos'),
method: 'GET'
};
to decode encoded uri component you can use decodeURIComponent(url)
Typically, you do not want to use encodeURI() directly. Instead, use fixedEncodeURI(). To quote MDN encodeURI() Documentation...
If one wishes to follow the more recent RFC3986 for URLs, which makes square brackets reserved (for IPv6) and thus not encoded when forming something which could be part of a URL (such as a host), the following code snippet may help:
function fixedEncodeURI(str) {
return encodeURI(str).replace(/%5B/g, '[').replace(/%5D/g, ']');
}
There is a similar issue with encodeURIComponent() (source: MDN encodeURIComponent() Documentation), as well as a similar fixedEncodeURIComponent() function. These should be used, rather than the actual encodeURI() or encodeURIComponent() function calls...
To be more stringent in adhering to RFC 3986 (which reserves !, ', (, ), and *), even though these characters have no formalized URI delimiting uses, the following can be safely used:
function fixedEncodeURIComponent(str) {
return encodeURIComponent(str).replace(/[!'()*]/g, function(c) {
return '%' + c.charCodeAt(0).toString(16);
});
}
I was getting this error while trying to hit Elasticsearch's API. For me, it was due to Chinese characters in the Document's Title (in the Request I was sending). Switching to all English characters fixed the issue.
Sometimes browser inspector uses abbreviation of long JSON object.
In my case, the data included unescaped characters such as '…' which should not be in http request.
I'm using a node.bcrypt.js hash returning hex numbers in node.js for a password reset token.
user.reset_password_token = require('crypto').randomBytes(32).toString('hex'
);
Should I also base64 encode the token before I pass it around in urls (ie: link reset email)?
Is there any benefit to doing this?
I seem to recall base64 encoding can contain forward slashes which would mess up the path:
var token = user.reset_password_token;
//is there any benefit to doing base64 encoding?
var encoded_token = new Buffer(token).toString('base64');
var reset_link = 'http://example.com/reset/'+ encoded_token;
sendResetLink( reset_link );
You don't need a third-party library for that. You can just use base64url encoding (starting from nodejs v14.18.0)
const encoded_token = Buffer.from(token).toString('base64url');
I solved it using URLSafeBase64 nodejs LIB at https://www.npmjs.org/package/urlsafe-base64
var email =email_lines.join("\r\n").trim();
var base64EncodedEmail = URLSafeBase64.encode(new Buffer(email));
gmail.users.messages.send({userId:"me",
resource: {raw:base64EncodedEmail} }, callbackFn});
Another option is base64url library:
base64url("ladies and gentlemen we are floating in space");
// bGFkaWVzIGFuZCBnZW50bGVtZW4gd2UgYXJlIGZsb2F0aW5nIGluIHNwYWNl
base64 can indeed contain forward slashes, but base32 can't!
I have a string like that: "abcde 李". It can be any string with non latin characters.
I want to encode it to use in request, so it will be "abcde %E6%9D%8E" and can be used for http.request.
I have tried this:
str.toString("utf-8");
or
var buffer = new Buffer(str);
str = buffer.toString('utf-8');
but none of them work. what is the proper way to handle this?
That string is already UTF-8. It looks like you're trying to escape it for use in an HTTP query string, so try this:
var qs = require('querystring');
qs.escape('abcde 李'); // => 'abcde%20%E6%9D%8E'