I have a NodeJS and an API that handles get requests.
...
var apiRoutes = express.Router();
apiRoutes.get("/filter/:name",function(req, res){
// do something
res.json(result);
}
app.use('/api', apiRoutes);
Then in the client (not an important information but it is Angular2):
find(name:string): void{
name.trim();
this.http.get(encodeURI('http://server_address/api/filter/' + name))...
It works well for the parameters don't contain whitespaces, etc. In order to make it working with spaced inputs also, I used encodeURI function. However, when I give an input with whitespaces the server gives error:
undefined:0
^
SyntaxError: Unexpected end of input
at Object.parse (native)
at IncomingMessage.<anonymous> (/user/home/server/server.js:65:28)
at IncomingMessage.EventEmitter.emit (events.js:117:20)
at _stream_readable.js:920:16
at process._tickCallback (node.js:415:13)
Any idea what can I do to fix it?
I figured out the problem. I was doing something like:
apiRoutes.get("/filter/:name",function(req, res){
http.request(anotherURL + req.body.name)...
}
And thought that the name parameter is already encoded since it was encoded in the client. However I see that I have to encode it in the server again.
apiRoutes.get("/filter/:name",function(req, res){
http.request(anotherURL + encodeURI(req.body.name))...
}
Related
I am trying to PUT a file to an S3 pre-signed URL. Doing so with bash/cURL works fine but with Node I am getting the following error:
Error: write EPIPE
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) {
errno: -32,
code: 'EPIPE',
syscall: 'write'
}
Here is the code
const fs = require('fs');
const request = require('request');
stream = fs.createReadStream('/tmp/file');
r = request.put('https://s3.eu-west-2.amazonaws.com/bucketname/path?X-Amz-Content-Sha256=....&...');
stream.pipe(r).on('error', function(err) {
console.log(err);
});
EPIPE means that the writing request failed because the other end closed the connection. Looks like there might be some additional settings required inorder to work with amazon s3. I know that curl has native support for multipart/form-data. You can use this library to create readable multipart/form-data streams.
https://nicedoc.io/form-data/form-data
Or you can use third party libraries to send data
https://www.npmjs.com/package/s3-upload-stream
https://www.npmjs.com/package/streaming-s3
I'm using next.js with a custom express server when the params of a request contains % it causes this error:
URIError: Failed to decode param '%%faker'
at decodeURIComponent (<anonymous>)
at decode_param (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\layer.js:172:12)
at Layer.match (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\layer.js:148:15)
at matchLayer (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:574:18)
at next (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:220:15)
at middleware (D:\ahmed\coding\react js\with-redux-app\node_modules\http-proxy-middleware\lib\index.js:43:7)
at Layer.handle [as handle_request] (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\layer.js:95:5)
at trim_prefix (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:317:13)
at D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:284:7
at Function.process_params (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:335:12)
at next (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:275:10)
at expressInit (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\middleware\init.js:40:5)
at Layer.handle [as handle_request] (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\layer.js:95:5)
at trim_prefix (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:317:13)
at D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:284:7
at Function.process_params (D:\ahmed\coding\react js\with-redux-app\node_modules\express\lib\router\index.js:335:12)
for example if the request is http://localhost:3000/summoner/eune/%%faker the error happens but if it's http://localhost:3000/summoner/eune/^^faker the ^^ gets encoded and the url becomes http://localhost:3000/summoner/eune/%5E%5Efaker and everything works perfectly.i could fix this error by following this answer Express handling URIError: Failed to decode param like so:
server.use((err, req, res, next) => {
if (err instanceof URIError) {
err.message = "Failed to decode param: " + req.url;
err.status = err.statusCode = 400;
console.log(err);
return res.redirect(`http://${req.get("Host")}${req.url}`);
// return app.render(req, res, "/_error");
}
});
return res.redirect(`http://${req.get("Host")}${req.url}`); this will redirect the user from http://localhost:3000/summoner/eune/%%faker to http://localhost:3000/summoner/eune/%25%25faker and if i use return app.render(req, res, "/_error"); it will send the default error page provided by next.js back to user but this is not what i want. I want to handle the % like ^.
so my questions are:
why the % doesn't get encoded to %25 and if there is a way to make it happen?
who is responsible for encoding the browser or express?
what is the best way to handle this error?
I'm using node v8.9.1, express ^4.16.3.
Please, make the answer detailed i'm a beginner developer.
Thanks for your time.
Like you pointed out, urls are percent-encoded and http://localhost:3000/summoner/eune/%%faker is just not valid as an url.
When you type an invalid url, most browsers are kind enough to change it to something valid, ex: http://localhost:3000/test test is automatically changed to http://localhost:3000/test%20test, but it's just a fallback to avoid too many errors.
In your case, % is not automatically changed to %25 because browsers cannot know when to substitute % and when to leave it. Ex: when you type %25%25faker, should this url be used as is or should it be replaced to %2525%2525faker ?
In summary: You must use valid urls at any point in time and not rely on browser kindness.
I create a HTTP server that streams a video file.
http.createServer((req, res) => {
const file = 'example.mp4';
const size = fs.statSync(file).size;
res.writeHead(200, { 'Content-Length': size, 'Content-Type': 'video/mp4' });
fs.createReadStream(file).pipe(res);
}).listen(1911, '127.0.0.1');
I connect to it in my browser or video player to verify that it works. It does.
I encrypt a file:
fs.createReadStream('example.mp4')
.pipe(crypto.createCipher('aes-256-ctr', 'x'))
.pipe(fs.createWriteStream('encrypted_file'));
I decrypt it and play it back to verify that it works. It does.
Yet combining that decryption and streaming in the manner below fails silently.
const decrypt = crypto.createDecipher('aes-256-ctr', 'x');
http.createServer((req, res) => {
const file = 'encrypted_file';
const size = fs.statSync(file).size;
res.writeHead(200, { 'Content-Length': size, 'Content-Type': 'video/mp4' });
fs.createReadStream(file).pipe(decrypt).pipe(res);
}).listen(1911, '127.0.0.1');
The original and the encrypted file are the same size in bytes, and the original and the encrypted-then-decrypted file both have the same SHA-256 hash. Given that, I'd expect fs.createReadStream(original) and fs.createReadStream(encrypted).pipe(decrypt) to behave identically -- yet they don't. No video data is sent to the user, but no error is displayed to them either, and the error event never fires on the http.Server instance.
What am I missing?
Your deciphering code look ok and works in my variant correctly for short file http://techslides.com/demos/sample-videos/small.mp4 in my test (code based on https://gist.github.com/paolorossi/1993068 which is based on Video streaming with HTML 5 via node.js):
var http = require('http'),
fs = require('fs'),
util = require('util'),
crypto = require('crypto');
http.createServer(function (req, res) {
var path = 'encrypted';
var stat = fs.statSync(path);
var total = stat.size;
const decrypt = crypto.createDecipher('aes-256-ctr', 'x');
console.log('ALL: ' + total);
res.writeHead(200, { 'Content-Length': total, 'Content-Type': 'video/mp4' });
fs.createReadStream(path).pipe(decrypt).pipe(res);
}).listen(1912, '127.0.0.1');
console.log('Server running at http://127.0.0.1:1912/');
Your file may be bigger, so it can be useful to print to console request from the client. In my tests both unencrypted and encrypted servers got two requests and sent 383631 bytes.
The differences between my and your variant are: this is my first node.js server (and one of the first js programs) and it is not first for you. And I don't declare decrypt as global const but as local const. By debug prints I see two requests from the browser; and second one tries to modify global const decrypt in your variant with the error:
Server running at http://127.0.0.1:1912/
ALL: 383631
ALL: 383631
events.js:141
throw er; // Unhandled 'error' event
^
Error: write after end
at writeAfterEnd (_stream_writable.js:159:12)
at Decipher.Writable.write (_stream_writable.js:204:5)
at ReadStream.ondata (_stream_readable.js:528:20)
at emitOne (events.js:77:13)
at ReadStream.emit (events.js:169:7)
at readableAddChunk (_stream_readable.js:146:16)
at ReadStream.Readable.push (_stream_readable.js:110:10)
at onread (fs.js:1743:12)
at FSReqWrap.wrapper [as oncomplete] (fs.js:576:17)
So move decrypt inside server code. Your error is reusing of Decipher object which cant be reused (https://nodejs.org/api/crypto.html#crypto_class_decipher "Once the decipher.final() method has been called, the Decipher object can no longer be used to decrypt data."); and on second call decipher may try to decode file with wrong (non-zero) counter value.
I've got a fairly standard connect-mongo setup
mongoose is initialised / connected prior to this
app.use(express.session({
secret: "sfdgsdgsdfg",
store: new MongoSessionStore({db: mongoose.connection.db})
}));
This works fine.
However -
Assuming my mongodb connection suddenly dies (stop mongod locally in the example below)
- the next time I try to hit a route, my express app crashes too -
Error: failed to connect to [localhost:27017]
at null. (/Users/alex/Projects/MyProject/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:540:74)
at emit (events.js:106:17)
at null. (/Users/alex/Projects/MyProject/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:140:15)
at emit (events.js:98:17)
at Socket. (/Users/alex/Projects/MyProject/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection.js:478:10)
at Socket.emit (events.js:95:17)
at net.js:440:14
at process._tickCallback (node.js:419:13)
Is there a way to handle this error and (For example) redirect to a /error route?
(Obviously one that doesn't require session!)
EDIT
So now, I'm creating a separate mongoose connection.
I then use the on('error' to listen for errors
...this is where I'm getting stuck -
The process still dies because re-throwing the err doesn't pass it into the express error handler...
var sessionDbConnection = mongoose.createConnection(config.sessionDb);
sessionDbConnection.on('error', function(err) {
console.log('oops');
throw err; //instead of re-throwing the error, i think i need to pass it to the error handler below??
})
app.use(express.session({
secret: "sfdgsdgsdfg",
store: new MongoSessionStore({db: sessionDbConnection.db})
}));
app.use(function(err, req, res, next) {
res.render('error', err); //just for testing
});
Setup a general error handler at the end of your using chain ...
//function must accept 4 arguments
app.use(function(err, req, res, next) {
//use accepts to determin if you should return json, html, image?
//render the appropriate output.
});
Also, have your handlers and other modules accept three parameters (req, res, next) in the case of an error, return next(err) typically you will want to optionally use a .code property to match up your http response code to the one matching the error. using 4xx for input errors, 5xx for server errors etc.
Also, if you want to handle the general case...
process.on('uncaughtException', function(err) {
console.error('Caught exception: ' + err);
console.error(err.stack);
//interrogate the error, if it's something you can recover from, let it be.
//if the exception is fatal, exit with prejudice
setTimeout(process.exit.bind(process, 666), 1000); //exit in a second
});
This will handle your specific case, but you should only allow the process to keep running if its' something that should recover on its' own.
In response to your edit... you could have a global variable that changes when you get a DB error... unfortunately, this error doesn't necessarily happen in the context of an http request.. it could happen before/during/after ... in this way, you cannot know.
If you are using a client that will re-connect on failure, then that will alleviate some issues. The best you can do, is keep track of this variable, serve errors to all requests.. then restart your process.
There are lots of modules from pm2 to forever that will help you with this.
I'm trying to build a Twitter streaming app using node.js and the ntwitter module, here's my code :
var app = require('express').createServer(),
twitter=require('ntwitter');
app.listen(3000);
var feed = new twitter({
consumer_key: 'MY KEY',
consumer_secret:'MY SECRET KEY',
access_tocken_key:'MY ACCESS TOCKEN KEY',
access_tocken_secret:'MY ACCESS TOCKEN SECRET'
});
feed.stream('statuses/filter',{track: ['love', 'hate']}, function(stream){
stream.on('data',function(tweet){
console.log(tweet.text);
});
});
But here's what I get :
events.js:74
throw TypeError('Uncaught, unspecified "error" event.');
^
TypeError: Uncaught, unspecified "error" event.
at TypeError (<anonymous>)
at EventEmitter.emit (events.js:74:15)
at ClientRequest.<anonymous> (/Users/maximeheckel/Documents/My_Repositories/nodetwitter/node_modules/ntwitter/lib/twitter.js:251:14)
at ClientRequest.EventEmitter.emit (events.js:95:17)
at HTTPParser.parserOnIncomingClient [as onIncoming] (http.js:1628:21)
at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:119:23)
at CleartextStream.socketOnData [as ondata] (http.js:1523:20)
at CleartextStream.read [as _read] (tls.js:470:10)
at CleartextStream.Readable.read (_stream_readable.js:294:10)
at EncryptedStream.write [as _write] (tls.js:344:25)
I don't understand why I'm stuck with that as I'm following very carefully a lot of tutorials. Even when I clone the authors project I'm still getting this error.
Hope you can help.
UPDATE : I added
stream.on('error', function(error, code) {
console.log("My error: " + error + ": " + code);
});
To my stream function and I'm getting a HTTP 401 error
Any ideas ?
Everything with your code looks fine except for the spelling of "token" in the parameters passed to the constructor for twitter. You need to change access_tocken_key to access_token_key and access_tocken_secret to access_token_secret. Your error (401) is an authentication issue; that change should hopefully result in ntwitter passing the correct user authentication details to the Twitter API.
I had the same problem and it was because my servers system clock had floated by six minutes. Usually, API's give you about a five minute margin of error.