I'm trying to use the request-module to scrape websites and get the response-HTML. To prevent getting blocked by the target sites I need to use some proxies. But for some reason I can not accomplish to use request with a proxy.
I've also tried many different proxies and none of them will do the job.
My code:
const request = require('request');
const cheerio = require('cheerio');
var servers = [
'http://173.249.48.240:8080',
'http://85.214.250.48:3128'
]
var options = {
'url': 'http://www.google.de',
'proxy': servers[1]
};
request(options, (err,res,body) => {
console.log(res);
console.log(body);
console.log(err);
try {
var $ = cheerio.load(body);
console.log($('#ipv4').html());
} catch(err) {
//do nothing
}
})
I've tried many different proxies and have also used nightmare before but everytime I try to connect with a Proxy I get the following exception:
{ Error: connect ECONNREFUSED 85.214.250.48:3128
at Object._errnoException (util.js:1024:11)
at _exceptionWithHostPort (util.js:1046:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1182:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '85.214.250.48',
port: 3128 }
Related
Hi am trying to execute the KSQL query using npm package ksqlDb-client package, it throws an timeout error. I have attached the code as well, please let me know any issues over there.
when am hit this GET URL below method will execute https://localhost:5000/testKsql
exports.getKSQLStream = async () => {
const options = {
authorization: {
username: "admin",
password: "pw",
ssl: {
ca: CAs,
crt: myClientCert,
key: myClientKey,
}
},
host: 'https://ixxxx.xcxx.net',
timeout: 20000
}
const client = new KsqldbClient(options);
await client.connect();
const streamRes = await client.query('list streams;');
console.log("streams",streamRes);
await client.disconnect();
}
when I hit that URL from postman am getting below response in node console.
Error on Http2 client. Error: connect ETIMEDOUT 16.2XX.X4.xxx:8088
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16) {
errno: -4039,
code: 'ETIMEDOUT',
syscall: 'connect',
address: '16.2XX.X4.xxx',
port: 8088
}
I faced recently with the same issue and found that library has bug with authorization. I pushed PR in repo, but you can patch your local copy in the same way.
https://github.com/Streaminy/ksqldb-client/pull/4
I am using oneSignal for push notifications node.js. I am using the create notification api to send notification to the users, but i dont know why it works some times and sometimes gives timeout error
sendNotificationToUser(data) {
try {
var notificationData = {}
notificationData.app_id = oneSignalAppId
notificationData.headings = {
en: "Heading"
}
notificationData.contents = {
en: data.message
}
notificationData.include_player_ids = [data.deviceId]
var headers = {
"Content-Type": "application/json; charset=utf-8"
}
var options = {
host: "onesignal.com",
port: 443,
path: "/api/v1/notifications",
method: "POST",
headers: headers
}
var https = require("https")
var req = https.request(options, function (res) {
res.on("data", function (data1) {
console.log("Response:")
console.log(JSON.parse(data1))
})
})
req.on("error", function (e) {
console.log("ERROR:")
console.log(e)
})
req.write(JSON.stringify(notificationData))
req.end()
} catch (err) {
console.log("err in notification", err)
}
}
this api works 50% of times and 50% of times it responds with time out error, even all the inputs are correct
ERROR:
{
Error: connect ETIMEDOUT 104.18.225.52:443
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
errno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
syscall: 'connect',
address: '104.18.225.52',
port: 443
}
one simple solution is to directly hit the working ip, you can do this by including
host: 'onesignal.con' in headers and
host: '104.18.226.52' in options
this resolved my issue
to know more about how you can specify ip with host in https request
go here HTTPS request, specifying hostname and specific IP address
I'm trying to use NodeJS get requets using the 'request' module, so here is my code:
var request = require('request');
request({
'url':'http://whatismyip.host/',
'method': "GET",
'proxy': 'http://181.112.225.78:35482'
},function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body);
}
else{
console.log(error);
}
})
When I remove the "'proxy': 'http://181.112.225.78:35482'" line, it works perfectly, but when I let it, I have this error:
{ Error: connect ECONNREFUSED 181.112.225.78:35482
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1117:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '181.112.225.78',
port: 35482 }
I already tried to change the proxy, and I got the same error, so I have no idea where is it coming from...
Thanks you !
I am trying to get couchbase document revision identifier via sync gatetway API GET /{db}/{doc} within Node server:
function _getRev(docIdUrl, gateway, callback) {
let options = {
host: gateway.host,
path: gateway.path + docIdUrl,
port: gateway.port,
method: 'GET',
headers: {
'Content-Type': 'application/json'
}
};
// Node HTTP requests
let syncGatewayRequest = http.request(options, (response) => {
// ...unrelevant codes
});
syncGatewayRequest.on('error', (error) => {
logger.error('syncGateway connection error for ' + docIdUrl);
callback(error, null); // here is the error happening!!!!!
});
syncGatewayRequest.write(JSON.stringify({}));
syncGatewayRequest.end();
}
Then I got error:
[2017-11-03 11:07:51.961] { [Error: connect EMFILE 10.0.1.53:4985 - Local (undefined:undefined)]
code: 'EMFILE',
errno: 'EMFILE',
syscall: 'connect',
address: '10.0.1.53',
port: 4985 }
Error: connect EMFILE 10.0.1.53:4985 - Local (undefined:undefined)
at Object.exports._errnoException (util.js:870:11)
at exports._exceptionWithHostPort (util.js:893:20)
at connect (net.js:849:16)
at net.js:937:9
at nextTickCallbackWith0Args (node.js:420:9)
at process._tickCallback (node.js:349:13)
There is a context that the above function are executed asynchronously by a significant number of services, say 10,000+
I noticed the post here Nodejs connect EMFILE - How to reuse connections?
But I tried to unlimit the default connections by doing:
var http = require('http')
http.globalAgent.maxSockets = Infinity
But does not seem to work, error still ...
Anyone can let me know what's wrong here?
I find it hard to use superagent-proxy, just with the simple code:
const superagent = require('superagent')
require('superagent-proxy')(superagent)
let proxy = 'http://221.237.122.22:8118' // 设置代理
superagent
.get('http://sf.gg')
.proxy(proxy)
.timeout(3600*1000)
.end((err, res) => {
console.log(res)
console.log(res.status, res.headers);
console.log(res.body);
})
but when run, it cannot get an reply, why?
you should:
const superagent = require('superagent')
require('superagent-proxy')(superagent)
let proxy = 'http://221.237.122.22:8118' // 设置代理
superagent
.get('http://sf.gg')
.proxy(proxy)
.timeout(3600*1000)
.end((err, res) => {
if(err) {
console.error(err);
return;
}
console.log(res)
console.log(res.status, res.headers);
console.log(res.body);
})
then, you will get error such as
{ Error: connect ECONNREFUSED 221.237.122.22:8118
at Object.exports._errnoException (util.js:1018:11)
at exports._exceptionWithHostPort (util.js:1041:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '221.237.122.22',
port: 8118,
response: undefined }
You code is correct, proxy URL is not - if its exactly
'http://221.237.122.22:8118'
it means proxy do not require any login, anybody can use it with just URL, its not the case for most of the proxies, normally proxy URL is like 'http://username:password#IPADDRESS_OR_HOST:PORT'