I'm having fits accomplishing something and after scouring google & SO, throwing my hands up after a few days. Trying to do something that I think is pretty common: debug / examine all HTTP traffic while developing a node.js app.
In Windows it is as simple as firing up Fiddler and I can see all HTTP & HTTPS traffic from all processes. But I've switched platforms over to OSX and trying to make the same work.
I've tried using Charles & MITMPROXY, but all I'm seeing is the traffic to, with the response, my node.js app. My node.js app is calling external services, some using the popular request package (which I have seen how to set that up) but also using other packages, like azure-storage. What's troubling me is I can't get any of the debugging proxies to show me at the azure-storage package is sending / receiving to the endpoints they are calling.
Conceptually I think I get it... I have to tell these different things (like node.js, request & azure-storage) to go through the proxy each of these tools uses... but how can you do that without modifying their source? Can't, like how Fiddler works on Windows, you do something to "all traffic goes through this proxy"?
I'd use Fiddler on OSX but it is currently not working with no ETA in sight after talking to Telerik.
So the problem I was having is what I thought... in my specific instance the module that I was using to access Azure storage was not using the default proxy. I found a package (**global-tunnel that hijacked everything that used the request package to control it going through a proxy. Now I saw stuff show up in the HTTP debuggers I was using.
The problem now is when I am trying to reach an HTTPS endpoint... using something like Charles, it used it's own SSL cert which wasn't trusted by Azure so the connections were refused. Back to the drawing board...
Related
My client has a functionality in the App where it converts an audio file into text. For this, I am using the Google Cloud Speech to text API. The client has a VM setup where there is no internet connection available, and all the network traffic should go through a proxy if it needs to connect to the internet. The SpeechToText API calls don’t go via the proxy, but directly hit the firewall, which in turns blocks it and the translation fails.
I looked for ways for using global proxies in the APP, which didn’t work as these calls are gRPC based and not REST based. Looked on gRPC code as well for the proxy settings and used one of the environment variables that they provide for the same, but even that didn’t work.
I also tried to check in the Google Speech To Text client libraries if they provide proxy related settings, but even there is no solution for that.
The Google Cloud API calls use gRPC and gRPC protocol is using HTTP/2 which doesn’t seem to provide proxy based control.
I already tried to follow the steps in the instructions, however it's still not working to send the traffic via the proxy.
Any ideas what else I can do?
https://medium.com/google-cloud/accessing-google-cloud-apis-though-a-proxy-fe46658b5f2a
https://developers.google.com/gdata/articles/proxy_setup
I found an example on how to use the twitch EventSub webhooks(https://github.com/twitchdev/eventsub-webhooks-node-sample/blob/main/index.js) but i'm struggling with finding out how to setup it up without having to install ngrok or other apps on my PC since i have a vps where i host the bot. I understood the GET method but POST is a bit difficult for me.
Hope i explained it well enough for someone to understand.
Twitch EventSub at time of writing only offers a "Webhook transport"
So you should be able to set this up no problem on your VPS, since your VPS is web accessabile.
To test this locally on your PC yes you will need a proxy/tunnel such as NGROK to make your PC web accessable.
A "webhook transport" (to over simplfy) operates in the same way a login from on a Website does. You fill in the form and hit submit, and the form is POST'ed to the server.
Webhook's it's the same thing, except the data isn't POST'ed as a form but a JSON blob in the body.
So you can use anything capable of receiving a HTTP POST. There are just a few NodeJS examples like the one you have linked kicking about.
TLDR: unless you are testing, skip setting it up on your PC and start with setting it up on your VPS, as the VPS doesn't need a tunnel, apache/nginx are the SSL Terminator that passes to your Node script, if you use a node script link the linked exmaple in the OP
I've been testing HTTP2 Multiplexing and HTTP2/Server Push in Node.js locally and inspected it in the waterfall network graph in Chrome Dev Tools.
While using my own Node.js Server Push with res.stream.pushStream I got an "Initiator: Push (index)" in DevTools like this:
The change was noticeable as below:
I did some more research and figured I should use a reverse proxy to do the job (for example Nginx instead) and connect to my Node.js via HTTP1.1 instead as the upstream and serve HTTP2 from my reverse proxy.
After setting up the nginx.conf with http2_push_preload on; i sent some headers like these from my Node.js backend:
res.setHeader("Link","</picture.jpg>; as=image; rel=preload");
To my suprise I didn't see the "Push / (index)" indicator but "Other" like on the screenshot below and the asset listed in the link header seemed to show up on the waterfall graph quicker with a slightly lower TTFB time compared to the rest of assets.
I've been also looking for a solution to serve HTTP/2 as the reverse proxy and to download the assets via HTTP/2 from the service directly without the TLS (HTC) but it seems there's nothing like this.
Getting back to my question: How should I go about testing the HTTP/2 Server Push? Is the "Initiator: Other" a misinterpretation from the devtools? It seems to be working but it doesn't report as Server Push.
Also, are there any projects/solutions that would let for a connection to a backend upstream via HTTP2 directly?
I'm pretty sure the latter image shows slower access because of the overhead of using a reverse proxy instead of connecting to the server directly and it's magnified by the HTTP1.1 usage on the target server.
Thanks to #Barry I figured out it's actually Chrome itself as the "Other" initiator and it indeed speeds up the process a bit by using the Resource Hints Link header and not HTTP/2 Server Push itself.
The problem was actually a bug I can't seem to reproduce and it works well after a restart of the OS. Afterall Nginx was the culprit of not using the headers correctly, which kicked in when parsed in Chrome.
I have a problem with an Express.js service running on production that I'm not able to replicate on my localhost. I have already tried requesting all the urls to production again to my local machine, but on my machine everything works fine. So I suspect that the problem comes with the data on the http headers (cookies, user agents, languages...).
So, is there a way, (some express module, or sniffer that runs on ubuntu) that allows me to easily create a dump on the server with the whole header so I can later repeat those exact requests to my localhost?
You can capture network packages with https://www.wireshark.org/, analyze them and maybe find the difference between your local environment and the production one.
You can try to use a Proxy-Tool like Charles (https://www.charlesproxy.com/) or Fiddler (http://www.telerik.com/fiddler) to log your Browser Requests.
I am building a development tool, and would like to monitor requests to specific domains on a user's system.
I have already written a MITM proxy server. I would like to observe all requests to, say, api.twitter.com, without requiring users to change their code to point to my proxy server. This might be called an HTTP sniffer, I'm not sure.
I have considered:
Browser plugin (eg: chrome dev tools plugin)
/etc/hosts (but this can't map domain to domain I think, and if you did you wouldn't be able to get to the original one)
Native OSX app (learning curve)
Is there a way to observe system HTTP requests using node? I don't know where to start.