I'm hacking together some node.js code that calls an external webservice and I'm getting bad results. I'd like to log the raw request and response so that I can inspect it.
Here's the thing: I'm not consuming the http library directly, I'm consuming it through an OAuth library.
I'm already adding debug statements in the oauth library code and I don't like it. Now it looks like I'm going to have to go into http library and start messing with that? This can't be correct.
If I was on windows, I'd fire up fiddler. A friend mentioned wireshark but wireshark tells me I have to install X11. Really? I'm not going down that rabbit hole.
Then I tried node-inspector, but I think that is for server code not client code. It says your suppose to start your node process before attaching. Well my node process is a test case (vows) that ends shortly after is starts... so no luck there.
I guess this would difficult with any stack but jeez, it makes me miss .net!
So, how can I inspect what's going over the wire when using node.js as client to external webservice on mountain lion?
thanks!
Dan
Managed to install a hook on http/https request for the same reason.
function requestLogger(httpModule){
var original = httpModule.request
httpModule.request = function(options, callback){
console.log(options.href||options.proto+"://"+options.host+options.path, options.method)
return original(options, callback)
}
}
requestLogger(require('http'))
requestLogger(require('https'))
You can check out the global-request-logger module, which uses the same technique that #uiron mentioned, but with more details.
Related
I'm very new to Node.js, so I might just not be getting it, but after searching quite a bit, and trying a few different solutions, I am still not able to find a decent way to mock API responses using Node for acceptance testing.
I've got a javascript app (written in elm actually) that interacts with an API (pretty common, I imagine), and I want to write some acceptance tests... so I setup WebdriverIO with selenium and mocha, write some tests, and of course now I need to mock some API responses so that I can setup some theoretical scenarios to test under.
mock-api-server: Looked pretty nice, but there's no way to adjust the headers getting sent back from the server!
mock-http-server: Also looked pretty nice, lets me adjust headers, but there's no way to reset the mock responses without shutting down the whole server... !? And that has issues because the server won't shut down while the browser window is still open, so that means I have to close and relauch the browser just to clear the mocks!
json-server: Simple and decent way to mock some responses, but it relies entirely on files on disk for the responses. I want something I can configure from within a test run without reading and writing files to disk.
Am I missing something? Is this not how people do acceptance testing in the Node universe? Does everyone just use a fixed set of mock data for their entire test suite? That just sounds insane to me... Particularly since it seems like it wouldn't be that hard to write a good one based on express server that has all the necessary features... does it exist?
Necessary Features:
Server can be configured and launched from javascript
Responses(including headers) can be configured on the fly
Responses can also be reset easily on the fly, without shutting down the server.
I hit this problem too, so I built one: https://github.com/pimterry/mockttp
In terms of the things you're looking for, Mockttp:
Lets you start & reconfigure the server dynamically from JS during the test run, with no static files.
Lets you adjust headers
Lets you reset running servers (though I'd recommend shutting down & restarting anyway - with Mockttp that takes milliseconds, is clear & easily automatable, and gives you some nice guarantees)
On top of that, it:
Is configurable from both Node & browsers with identical code, so you can test universally
Can handle running tests in parallel for quicker testing
Can fake HTTPS servers, self-signing certificates automatically
Can mock as an intercepting proxy
Has a bunch of nice debugging support for when things go wrong (e.g. unmatched requests come back with a readable explanation of the current configuration, and example code that would make the request succeed)
Just to quickly comment on the other posts suggesting testing in-process: I really wouldn't. Partly because a whole bunch of limitations (you're tied to a specific environment, potentially even specific node version, you have to mock for the whole process, so no parallel tests, and you can't mock subprocesses), but mainly because it's not truly representative. For a very very small speed cost, you can test with real HTTP, and know that your requests & responses will definitely work in reality too.
Is this not how people do acceptance testing in the Node universe? Does everyone just use a fixed set of mock data for their entire test suite?
No. You don't have to make actual HTTP requests to test your apps.
All good test frameworks lets you fake HTTP by running the routes and handlers without making network requests. Also you can mock the functions that are making the actual HTTP requests to external APIs, which should be abstracted away in the first place, so no actual HTTP requests need to take place here as well.
And if that's not enough you can always write a trivially simple server using Express, Hapi, Restify, Loopback or some other frameworks, or plain http, or even net module (depending on how much control do you need - for example you should always test invalid responses that don't use HTTP protocol correctly, or broken connections, incomplete connections, slow connections etc. and for that you may need to use lower lever APIs in Node) to provide the mock data yourself.
By the way, you also always need to test responses with invalid JSON because people often wrongly assume that the JSON they get is always valid which it is not. See this answer to see why it is particularly important:
Calling a JSON API with Node.js
Particularly since it seems like it wouldn't be that hard to write a good one based on express server that has all the necessary features... does it exist?
Not everything that "wouldn't be that hard to write" necessarily has to exist. You may need to write it. Hey, you even have a road map ready:
Necessary Features:
Server can be configured and launched from javascript
Responses(including headers) can be configured on the fly
Responses can also be reset easily on the fly, without shutting down the server.
Now all you need is choose a name, create a repo on GitHub, create a project on npm and start coding.
You now, even "it wouldn't be that hard to write" it doesn't mean that it will write itself. Welcome to the open source world where instead of complaining that something doesn't exist people just write it.
You could try nock. https://github.com/node-nock
It supports all of your feature requests.
I'm struggling with a technical issue, and because of I'm pretty new on NodeJS world I think I don't have the proper good practise and tools to help me solve this.
Using the well known request module, I'm making a stream proxy from a remote server to the client. Almost everything is fine and working properly until a certain point, if there is too much requests at the same time the server does no longer respond. Actualy it does get the client request but is unable to go through the stream process and serve the content.
What I'm currently doing:
Creating a server with http module with http.createServer
Getting remote url from a php script using exec
Instanciate the stream
How I did it:
http://pastebin.com/a2ZX5nRr
I tried to investigate on the pooling stuff and did not understand everything, same thing the pool maxSocket was recently added, but did not helped me. I was also seting before the http.globalAgent to infinity, but I read that this was no longer limited in nodeJS from a while, so it does not help.
See here: https://nodejs.org/api/http.html#http_http_globalagent
I also read this: Nodejs Max Socket Pooling Settings but I'm wondering what is the difference between a custom agent and the global one.
I believed that it could come from the server but I tested it on a very small one and a bigger one and it was not coming from there. I think it definitely coming from my app that has to be better designed. Indeed each time I'm restarting the app instance it works again. Also if I'm starting a fork of the server meanwhile the other is not serving anything on another port it will work. So it might not be about ressources.
Do you have any clue, tools or something that may help me to understand and debug what is going on?
NPM Module that can help handle stream properly:
https://www.npmjs.com/package/pump
I made few tests, and I think I've found what I was looking for. The unpipe things more info here:
https://nodejs.org/api/stream.html#stream_readable_unpipe_destination
Can see and read this too, it leads me to understand few things about pipe remaining open when target failed or something:
http://www.bennadel.com/blog/2679-how-error-events-affect-piped-streams-in-node-js.htm
So what I've done, i'm currently unpiping pipes when stream's end event is fired. However I guess you can make this in different ways, it depends on how you want to handle the thing but you may unpipe also on error from source/target.
Edit: I still have issues, it seams that the stream is now unpiping when it does not have too. I'll have to doubile check this.
I'm getting Error: Can't set headers after they are sent in my app. I know a response is being sent before the response I intend to deliver, but I'm not sure where that's coming from. Can I log to the console when a respnose has been sent? I'm currently using morgan for logging, but it only seems to log requests. I can't find a way to log every response as well.
If not, what is another effective way to hunt this problem down?
Update: This app runs on Azure and uses the Azure SDK for node or else I would try running it through the node debugger or node-inspector. I don't think there's a way to do this while also running the app in the emulator. I could be wrong about that.
Your error says that there was a response send already, check your code with the node-inspector or any other debugging tool and search for the spot.
Here the node-inspector.
And since you work with Azure this should help you getting started with the node-inspector.
I've got problem connecting Flash client to Node.js server.
Short story:
For a first time I'm building a Node.js server that should be used by both web client (WebSocket) as well as a Flash client (Socket). The web client, of course, works like a charm, but I can't get over the Flash one. I get SECURITY_ERROR. After a day of research I think it's because of the policy file not being loaded. Ideas (primus on top of engine.io) ?
Long story:
I'm using Primus as I thought I'll need it because I have both web sockets and flash sockets to handle. Not sure if this is accurate? :)
I'm using Engine.io as a 'transformer/transporter' - the main framework that the layer uses. I won't discuss the standard web client (using Chrome and primus-client), as it's easy to setup.
I'm using simple and standard Sockets in AS3:
_socket = new Socket();
_socket.addEventListener(Event.CONNECT, onSocketConnect);
//...
_socket.addEventListener(SecurityErrorEvent.SECURITY_ERROR, onSecurityError);
_socket.addEventListener(IOErrorEvent.IO_ERROR, onIOError);
_socket.connect('localhost', '1337);
When building it within Flash IDE, it goes to the onSocketConnect function, but if I try to write anything to the socked - I get disconnected. If I run this from the web browser, I get into the onSecurityError method.
I must say that I don't get any traces in the node console!
primus.on('connection', function connection(spark) {
console.log('new connection'); // never gets logged!
As I know, security error is thrown when there is error with the policy file, so I started searching for a solution for that.
I've read a lot of things online, and most common solution was simple usage of socket.io and so called FlashSocket.IO. I tried implementing it, but it's so old, that some of the code is a kind of missing and I finally got some errors from the hurlant library - I couldn't get it working.
I also saw some node package called policy, which runs separate server to server the policy file.
I tried adding a transport array with flashsocket in it - no change. I also can't understand why all of the samples are using transports - I've searched and both index.js and primus.js are using transport (why there are two separate files, Jesus?!)
I could try using only engine.io without primus, but I don't know if this would be of any help. All the posts and samples I've found are pretty old - please help me with any up to date solution or at least some explanation what needs to be done - seems like a whole new universe to me :)
Thanks in advance!
Edit:
Thanks to the The_asMan, I figured out it has something to do with the handshake. I've tried this simple example (despite the fact it's so old) - it worked perfectly for the Flash client! Of course I cannot connect web sockets to it, as the handshake is not proper - it has some kind of protocol for it.
So I guess I just have to understand how to get the <policy-file-request/> in node - I'll be able to return the policy file. But I don't know how to get it - I don't receive any kind of data nor connect handler...
You have a cross domain policy issue.
I answered it all here.
AS3 - Flash/AIR Socket Communication writeUTFBytes only works once
just an idea:
On some operating systems, flush() is called automatically between execution frames, but on other operating systems, such as Windows, the data is never sent unless you call flush() explicitly. To ensure your application behaves reliably across all operating systems, it is a good practice to call the flush() method after writing each message (or related group of data) to the socket.
Following up on this question I was wondering is there is a way, without using socket.io, to avoid the buffering of the response that happens on most navigators. So for instance if the node server emit every 5 secondes : 'hello world' i can directly print them on a webpage as soon as the data is available.
Is there a way to do so ?
Unfortunately, this is not how web browsers work. If you want this type of functionality without using WebSockets (or a socket.io fallback) you could try with Server-Sent Events. See this gist for an example (in coffeescript). Also, here is a polyfill for older browsers.
Yes, this is doable. This is how comet streaming servers work.
See http://faye.jcoglan.com/ for an example for Node.js.