I am building a development tool, and would like to monitor requests to specific domains on a user's system.
I have already written a MITM proxy server. I would like to observe all requests to, say, api.twitter.com, without requiring users to change their code to point to my proxy server. This might be called an HTTP sniffer, I'm not sure.
I have considered:
Browser plugin (eg: chrome dev tools plugin)
/etc/hosts (but this can't map domain to domain I think, and if you did you wouldn't be able to get to the original one)
Native OSX app (learning curve)
Is there a way to observe system HTTP requests using node? I don't know where to start.
Related
I've build a Web-App that is displayed in an Electron-App with Nativefier. That already works great, but now i need to send requests from the website to the local network to talk with some local devices which are (with it's ip-address) configured in the Web-App.
I had the idea to use the Electron-App as a "proxy" to the local network by using a javascript callback from the Website to the Electron-App (don't know if this is possible, just an idea), which then makes the local request because it's running on a computer in the same network.
The reason for this post is that i need ideas/tips to secure this and prevent allowing to talk to other than the desired Web-App (available under a certain domain) by developing something protective like checking or validating the Server, validating the request by sending it's hash back to the server or other methods.
So my questions are: is it generally a good idea to do something like this or is this a huge security problem and have anyone any tips to secure the communication and only allowing the communication to in the web-app configured devices in the local network?
I'm using node js trying to send my web-page to my network, I successfully call localhost:port in my computer using express as server, the webpage loads fine trigger my webcam which I used to streaming in the webpage, and then im working to make a simple app in my phone to directly access my server, so my questions:
1.How do I able to access my server from different devices in the same wireless-network? by calling ip + port ?192.168.1.104:9001 ? cause i've tried and it didnt work.
2.I've found https with .pem something like that, is that the answer ? is there also any other way ?
3.maybe any advice before i work to make my web-app to devices? using koa? i don't even really know what is that, but i'm happily take any advices.
EDIT: i've read How could others, on a local network, access my NodeJS app while it's running on my machine?
let's say I simply using random router, so i can't configure my router-port, my server in my pc and my phone join in the same network, trying to access the server in my phone
1.How do I able to access my server from different devices in the same wireless-network?
All you need to do is find your server's IP address in this same wireless-network, and find the Node.js application's port. Then access the following URL in other devices:
http://{server_IP}:{port}
However, there are some points need to check:
Need to check firewall and confirm the port is not blocked, server IP is not blocked by test device, and test device IP is not blocked by server.
Need to check whether there is any Proxy setting in server and test device. If there is any, disable the proxy.
A computer may have many IP addresses at the same time, you need to find the correct one in the same wireless-network. For example, If you install a virtual machine software such as VMware and run a virtual system inside, your real computer will get IP address as 192.168.*.* -- this IP address looks like an intranet IP in wireless-network, but it is not, and can never be accessed by test device.
2.I've found https with .pem something like that, is that the answer?
No, HTTPS has nothing to do with this problem. HTTPS just add security (based on HTTP layer), it does not impact any HTTP connectivity. Actually, to minify the problem, it is better to only use HTTP in your scenario.
There is only one very special case that may bring your problem by HTTPS -- the test machine is configured and will block any non-HTTPS connection for security.
3.maybe any advice before i work to make my web-app to devices? using koa?
My suggestion is: As there is an HTTP connectivity issue, the first step is trying to find the root cause of that issue. Thus, it is better to make a simplest HTTP server using native Node.js, no Koa, no Express. In this way, the complexity of server will be reduced, which makes root cause investigation easier.
After the HTTP connectivity issue is fixed, you can pick up Koa or Express or any other mature Node.js web framework to help the web-app work.
4.let's say I simply using random router, so i can't...
Do you mean your server get dynamic IP address by DHCP? As long as the IP is not blocked by test device, it does not matter.
I have a windows server 2012 VPS running a web app behind Cloudflare. The app needs to initiate outbound connections based on user actions (eg upload image from URL). The problem is that this 'leaks' my server's IP address and increases risk of DDOS attacks.
So I would like to prevent my server's IP from being discovered by setting up a forward proxy. So far my research has shown that this is no simple task, and would involve setting up another VPS to act as a proxy.
Does this extra forward proxy VPS have to be running windows ? Are their any paid services that could act as a forward proxy for my server (like cloudflare's reverse proxy system)?
Also, it seems that the suggested IIS forward proxy plugin, Application Request Routing, does not work for HTTPS.
Is there a solution for both types of outgoing (HTTPS + HTTP) requests?
I'm really lost here, so any help or suggestions would be appreciated.
You are correct in needing a "Forward Proxy". A good analogy for this is the proxy settings your browser has for outbound requests. In your case, the web application behaves like a desktop browser and can be configured to make the resource request through a proxy.
Often you can control this for individual requests at the application layer. An example of doing so with C#: C# Connecting Through Proxy
As far as the actual proxy server: No, it does not need to run Windows or IIS. Yes, you can use a proxy service. The vast majority of proxy services are targeted towards consumers and are used for personal privacy or to get around network restrictions. As such, I have no direct recommendations.
Cloudflare actually has recommendations regarding this: https://blog.cloudflare.com/ddos-prevention-protecting-the-origin/.
Features like "upload from URL" that allow the user to upload a photo from a given URL should be configured so that the server doing the download is not the website origin server.
This may be a more comfortable risk mitigator, as it wouldn't depend on a third party proxy service. A request for upload could be handled as a web service call to a dedicated "file downloader" server. Keep in mind that if you have a queued process for another server to do the work, and that server is hosted in the same infrastructure, both might be impacted by a DDoS, depending on the type of DDoS.
Your question implies that you may be comfortable using a non-windows server. Many softwares exist that can operate as a proxy(most web servers), but suffer from the same problem as ARR - lack of support for the HTTP "CONNECT" verb, which is used by modern browsers to start an HTTPS connection before issuing a "GET". SQUID is very popular, open source, and supports everything to connect to.. anything. It's not trivial to set up. Apache also has support for this in "mod_proxy_connect", but I have no experience in that and the online documentation isn't very robust. It's Apache, though, so it may be worth the extra investigation.
I'm having fits accomplishing something and after scouring google & SO, throwing my hands up after a few days. Trying to do something that I think is pretty common: debug / examine all HTTP traffic while developing a node.js app.
In Windows it is as simple as firing up Fiddler and I can see all HTTP & HTTPS traffic from all processes. But I've switched platforms over to OSX and trying to make the same work.
I've tried using Charles & MITMPROXY, but all I'm seeing is the traffic to, with the response, my node.js app. My node.js app is calling external services, some using the popular request package (which I have seen how to set that up) but also using other packages, like azure-storage. What's troubling me is I can't get any of the debugging proxies to show me at the azure-storage package is sending / receiving to the endpoints they are calling.
Conceptually I think I get it... I have to tell these different things (like node.js, request & azure-storage) to go through the proxy each of these tools uses... but how can you do that without modifying their source? Can't, like how Fiddler works on Windows, you do something to "all traffic goes through this proxy"?
I'd use Fiddler on OSX but it is currently not working with no ETA in sight after talking to Telerik.
So the problem I was having is what I thought... in my specific instance the module that I was using to access Azure storage was not using the default proxy. I found a package (**global-tunnel that hijacked everything that used the request package to control it going through a proxy. Now I saw stuff show up in the HTTP debuggers I was using.
The problem now is when I am trying to reach an HTTPS endpoint... using something like Charles, it used it's own SSL cert which wasn't trusted by Azure so the connections were refused. Back to the drawing board...
We need to extend pretty old application, which make a lot of integration testing connected with remote datasources, but this cases wasn't properly writtten, so it's not an easy way to change it to some network independent stubs. Is it possible to create some kind of a script for unix, who will be listen for specific requests, let's say to google.com/api/123 and if such request found, not allow it to go forward, but return some value, which we previously mapped on that url?
You most likely need to set up a proxy server like Squid, set up redirection and route all traffic through that proxy.