My tor is connected to 127.0.0.1:9051, Its said that it not an http proxy so how can I really connect to websites programmatically using it (preferably node.js)?
(Trying to connect using http GET)
Like is there a specific way of sending requests?
Thanks in advance 🙏
You can use Axios for request and set proxy to TOR SOCKS proxy.
Like below
const axios = require('axios');
const SocksProxyAgent = require('socks-proxy-agent');
const proxyOptions = `socks5://$127.0.0.1:9050`;
const httpsAgent = new SocksProxyAgent(proxyOptions);
const baseUrl = 'https://example.com'
const client = axios.create({baseUrl, httpsAgent});
client.get('/something').then(res => res.data);
To the people who are looking for an updated answer here what's working for me. Make sure your tor instance running on another terminal tab or in the background.
It's just a regular axios setup but we need to pass the httpsAgent parameter created by socks-proxy-agent library.
const axios = require('axios')
const { SocksProxyAgent } = require('socks-proxy-agent')
const run = async () => {
try {
const httpsAgent = new SocksProxyAgent('socks://127.0.0.1:9050')
const result = await axios({
httpsAgent,
method: 'get',
url: 'https://api.ipify.org'
})
console.log(result.data)
// 185.220.101.68
} catch (err) {
console.log(err.message)
}
}
run()
Related
I am using the new (as of version 18) Node.js "fetch" API to perform HTTP requests e.g.
const response = await fetch(SOMEURL)
const json = await response.json()
This works, but I want to "mock" those HTTP requests so that I can do some automated testing and be able to simulate some HTTP responses to see if my code works correctly.
Normally I have used the excellent nock package alongside Axios to mock HTTP requests, but it doesn't appear to work with fetch in Node 18.
So how can I mock HTTP requests and responses when using fetch in Node.js?
Node 18's fetch function isn't built on the Node.js http module like most HTTP libraries (including Axios, request etc) - it's a total rewrite of an HTTP client built on the lower-level "net" library called undici. As such, "nock" cannot intercept requests made from the fetch function (I believe the team are looking to fix this, but at the time of writing, Nock 13.2.9, Nock does not work for fetch requests).
The solution is to use a MockAgent that is built into the undici package.
Let's say your code looks like this:
// constants
const U = 'http://127.0.0.1:5984'
const DB = 'users'
const main = async () => {
// perform request
const r = await fetch(U + '/' + DB)
// parse response as JSON
const j = await r.json()
console.log(j)
}
main()
This code makes a real HTTP request to a CouchDB server running on localhost.
To mock this request, we need to add undici into our project:
npm install --save-dev undici
and add some code to intercept the request:
// constants
const U = 'http://127.0.0.1:5984'
const DB = 'users'
// set up mocking of HTTP requests
const { MockAgent, setGlobalDispatcher } = require('undici')
const mockAgent = new MockAgent()
const mockPool = mockAgent.get(U)
setGlobalDispatcher(mockAgent)
const main = async () => {
// intercept GET /users requests
mockPool.intercept({ path: '/' + DB }).reply(200, { ok: true })
// perform request
const r = await fetch(U + '/' + DB,)
// parse response as JSON
const j = await r.json()
console.log(j)
}
main()
The above code now has its HTTP "fetch" request intercepted with mocked response.
I am unable to get the mocking to work. I am not sure the sample above is doing what it is intended to do. I think it's actually making a call to localhost. I haven't been able to use MockAgent to mock the fetch call. I keep getting the result of the request (which is a google error page in this example). It should be equivalent as the code above, but I changed the url. When I had localhost I got fetch errors (since it wasn't being intercepted).
const { MockAgent, setGlobalDispatcher, } = require('undici');
test('test', async ()=>{
// constants
const U = 'http://www.google.com'
const DB = 'users'
const mockAgent = new MockAgent()
const mockPool = mockAgent.get(U)
setGlobalDispatcher(mockAgent)
// intercept GET /users requests
mockPool.intercept({ path: '/' + DB }).reply(200, { ok: true })
const opts = { method: 'GET' };
// perform request
const r = await fetch(U + '/' + DB,opts)
// parse response as JSON
const j = await r.text()
console.log(j)
});
I've upgraded my Node.js to the latest version which supports fetch natively. I'm trying to use a free public proxy to test, however when I'm making a request, it still comes from me directly. I know that because I'm calling an "Get my IP" service, and it returns my hope IP.
Here's the code example:
const HttpsProxyAgent = require('https-proxy-agent');
fetch(`https://api.ipify.org?format=json`, {agent: new HttpsProxyAgent("36.32.21.45:8085")})
.then(res => res.json()).then(res => console.log(res));
Tried putting proxy settings in the header as well this way:
const proxyAgent = require("proxy-agent");
(async () => {
const res = await fetch('https://api.ipify.org?format=json', {
headers: {
agent: new proxyAgent('http://80.48.119.28:8080'),
}
});
const data = await res.json();
console.log(data); //it shows my ip
})();
What am I missing?
I am running a node express server on Amazon ec2 instance. I can connect to the website on my local machine from within the browser, and can view pages with only local files, but when accessing a page that makes an external HTTP request, it just hangs. I figure it has something to do with my inbound or outbound rules are prohibiting it somehow, but don't know enough about networking to solve it on my own.
These are the functions that are failing behind the scenes:
const axios = require('axios').default;
const freelancer = axios.create({
baseURL: 'https://www.freelancer.com/api/',
headers: {
'freelancer-oauth-v1': process.env.FREELANCER_TOKEN
}
});
/* Get User By Id */
async function getUserById(user_id) {
const result = await freelancer.get(`/users/0.1/users/${user_id}/`)
return result.data.result;
}
const GitHub = require('github-api');
const gh = new GitHub({
username: process.env.GHUSER,
password: process.env.GHPASS
});
const getRepos = async function () {
const user = await gh.getUser();
return new Promise(async (resolve, reject) => {
await user.listStarredRepos(function (err, repos) {
if (err) reject(err);
resolve(repos);
});
});
}
My routers look like this:
var express = require('express');
var router = express.Router();
const freelancer = require('../service/Freelancer');
/* GET home page. */
router.get('/', async (req, res, next) => {
const reviews = await freelancer.getMyReviews();
const self = await freelancer.getSelfData();
res.render('contact', {
header: 'Check out all my reviews!',
lead: '',
paragraphtext: 'Your review could be next on this list!',
reviews,
self
});
});
Based on the comments.
The issue was caused by using HTTP for www.github.com. The solution was to use HTTPS.
I'm having a very hard time sending commands to an interactive bash shell using a node.js websockets client. I'm running a cluster with kops and I'm able to establish the connection with a pod and get its prompt but I am unable to send commands and receive a response.
So my connection looks like:
const WebSocket = require('ws');
const fs = require('fs');
const readline = require('readline');
const unescape = require('querystring').unescape;
const escape = require('querystring').escape;
const channel = '0';
const access_token = "[the_api_token]";
const pod_name = 'nginx-726417742-s1nv2';
const host_address = 'wss://[the_api_url]';
const cmd = `${escape('/bin/bash')}&command=-i`;
const params = `stdout=1&stdin=1&stderr=1&tty=1&container=nginx&command=${cmd}`;
const url = `${host_address}/api/v1/namespaces/default/pods/${pod_name}/exec?${params}`;
const options = {
headers: {
'Authorization': `Bearer ${access_token}`
},
ca: fs.readFileSync('ca.crt'),
}
const ws = new WebSocket(url, options);
ws.on('open', () => {
console.log('connected');
});
ws.on('message', (data, flags) => {
process.stdout.write(data);
});
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout
});
rl.on('line', function (cmd) {
const data = channel + cmd;
if (ws && ws.readyState === 1) {
ws.send(data);
}
});
The ws.on('message', (data, flags) part prints the shell prompt perfectly root#nginx-726417742-s1nv2:/# but then I can type whatever and it goes trough ws.send(data) but no message is ever received and no error is generated. I have already tried to convert the data to base64 and send char by char instead of the whole line, but the result and behaviour is always the same.
If I proxy the API through kubectl proxy --disable-filter=true --port=8080 and use https://github.com/kubernetes-ui/container-terminal pointing to ws://localhost:8080 and using /api/v1/namespaces/default/pods/${pod_name}/exec I can get a working terminal, but local proxying is not an option for me.
Any help will be much appreciated.
I know that my answer comes late. But maybe I can help someone else with this. I have noticed that often the Kubernetes API requires to set an origin when using websockets. Otherwise the API returns strange errors sometimes. In your case you could try to complete your options as follows:
const options = {
headers: {
'Authorization': `Bearer ${access_token}`
},
ca: fs.readFileSync('ca.crt'),
origin: 'https://<the_api_url>:<port>'
}
Furthermore I think that it is not required to set wss as protocol (in your host_address variable). Usually the protocol should be automatically upgraded from https to wss during the connection process.
I am trying to use axios with a proxy server to make an https call:
const url = "https://walmart.com/ip/50676589"
var config = { proxy: { host: proxy.ip, port: proxy.port } }
axios.get(url, config)
.then(result => {})
.catch(error => {console.log(error)})
The proxy servers I am using are all in the United States, highly anonymous, with support for HTTP and HTTPS.
I am receiving this error:
{ Error: write EPROTO 140736580649920:error:140770FC:SSL
routines:SSL23_GET_SERVER_HELLO:unknown
protocol:../deps/openssl/openssl/ssl/s23_clnt.c:794:
In order to ensure that the problem is with axios and NOT the proxy, I tried this:
curl -x 52.8.172.72:4444 -L 'https://www.walmart.com/ip/50676589'
This totally works just fine.
How do I configure axios to work with proxies and https URL's?
Axios https proxy support is borked if using https proxies. Try passing the proxy through httpsProxyAgent using http.
const axios = require('axios');
const httpsProxyAgent = require('https-proxy-agent');
const httpsAgent = new httpsProxyAgent('http://username:pass#myproxy:port');
// or const httpsAgent = new httpsProxyAgent({ host: 'myproxy', port: 9999 });
const config = {
url: 'https://google.com',
httpsAgent
}
axios.request(config).then((res) => console.log(res)).catch(err => console.log(err))
Alternatively there is a fork of Axios that incorporates this: axios-https-proxy-fix but I'd recommend the first method to ensure latest Axios changes.
Try this. That work for me.
First
npm install axios-https-proxy-fix
Then
import axios from 'axios-https-proxy-fix';
const proxy = {
host: 'some_ip',
port: some_port_number,
auth: {
username: 'some_login',
password: 'some_pass'
}
};
async someMethod() {
const result = await axios.get('some_https_link', {proxy});
}
You can solve this problem looking this issue
At this solution instead use the proxy interface, use the http(s)Agent.
For it the solution use the native node module https-proxy-agent.
var ProxyAgent = require('https-proxy-agent');
var axios = require('axios');
const agent = ProxyAgent('http://username:pass#myproxy:port')
var config = {
url: 'https://google.com',
proxy: false,
httpsAgent: agent
};
For it works the proxy property must be equal to false.
The https-proxy-agent and node-tunnel solutions did work for me, but both of them doesn't support conditional proxying using NO_PROXY.
I found global-agent as the best solution in my case as it modifies the core http and https objects and will be applied automatically to any library that makes use of them, including axios, got, request, etc.
The usage is very simple.
npm i global-agent
npm i -D #types/global-agent
Add import 'global-agent/bootstrap'; to the entrypoint (index.ts) of the server.
Run with these env vars and make sure HTTP_PROXY, HTTPS_PROXY are NOT in the env.
export GLOBAL_AGENT_NO_PROXY='*.foo.com,baz.com'
export GLOBAL_AGENT_HTTP_PROXY=http://127.0.0.1:8080
This is how I finally ended up using it.
import { bootstrap } from 'global-agent';
const proxy = process.env.EXTERNAL_PROXY;
if (proxy) {
process.env.GLOBAL_AGENT_HTTP_PROXY = proxy;
process.env.GLOBAL_AGENT_NO_PROXY = process.env.NO_PROXY;
process.env.GLOBAL_AGENT_FORCE_GLOBAL_AGENT = 'false';
bootstrap();
logger.info(`External proxy ${proxy} set`);
}
I know this is an old post, but I hope this solution saves time for anyone facing an SSL issue with Axios.
You can use an HTTP agent, I suggest using hpagent
const axios = require("axios");
const { HttpProxyAgent, HttpsProxyAgent } = require("hpagent");
async function testProxy() {
try {
const proxy = "http://username:password#myproxy:port";
// hpagent configuration
let agentConfig = {
proxy: proxy,
// keepAlive: true,
// keepAliveMsecs: 2000,
// maxSockets: 256,
// maxFreeSockets: 256,
};
axios.defaults.httpAgent = new HttpProxyAgent(agentConfig);
axios.defaults.httpsAgent = new HttpsProxyAgent(agentConfig);
// Make a simple request to check for the IP address.
let res = await axios.get("https://api.ipify.org/?format=json");
console.log(res.data);
} catch (error) {
console.error(error);
}
}
testProxy();
Try to explicitly specify the port in the URL:
const url = "https://walmart.com:443/ip/50676589"
If you also need an HTTPS-over-HTTP tunnel, you'll find a solution in this article.
Hope this helps,
Jan
This error is because axios is trying to proxy your request via https (it takes it from your url), there is this ticket tracking it: https://github.com/axios/axios/issues/925
I lost a day of work when I updated my dependencies last week (Feb. 2020) trying to figure out why services were stalling. axios-https-proxy-fix will cause Axios to hang indefinitely without timeout or throwing an error conflicting with other libraries in npm. Using node-tunnel (https://github.com/koichik/node-tunnel) to create an agent also works.
const tunnel = require('tunnel');
class ApiService {
get proxyRequest() {
const agent = tunnel.httpsOverHttp({
proxy: {
host: 'http://proxy.example.com',
port: 22225,
proxyAuth: `username:password`,
},
});
return axios.create({
agent,
})
}
}