I am using the ws module for node.js and I am having a issue, for some unknown reason it isn't connecting to the right server. I am not sure what is causing it to happen. If you need any bits of code just ask. Here is a video of what is happening. https://youtu.be/_Ji50RzyGh4
Here is the connect code:
connect() {
if (this.server == 'none' || this.server == '' || this.server == null || this.server == undefined) return;
this.ws = new WebSocket(this.server, {
headers: {
'Accept-Encoding': 'gzip, deflate, sdch',
'Accept-Language': 'pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4',
'Cache-Control': 'no-cache',
Cookie: '__cfduid=df2db54a70c595c3c2b5a0d067d1650481506608598',
Origin: origin,
Referer: origin,
Pragma: 'no-cache',
'Sec-WebSocket-Extensions': 'permessage-deflate; client_max_window_bits',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'
},
agent: proxyAgents[this.id]
});
this.binaryType = 'nodebuffer';
this.ws.onopen = this.onopen.bind(this);
this.ws.onclose = this.onclose.bind(this);
this.ws.onerror = this.onerror.bind(this);
}
Related
I am trying to access the json response on a link. It is working in my normal browser but not working when I try to access it with python requests and is sending a Cloudflare page as a response rather than json.
So far, I have tried:
Copying the headers from my browser and passing them to my request
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36',
'Accept': 'application/json, text/plain, */*',
'Accept-Language': 'en-US,en;q=0.9',
'Cache-Control': 'max-age=0',
'Referer': 'https://9gag.com/',
'Cookie': '____ri=5012; ____lo=US;'
}
req = requests.get("https://9gag.com/v1/group-posts/group/default/type/hot", headers=headers)
print(req)
Sending cookies seperately
cookies = {
'____ri': '5012', '____lo': 'US',
}
req = requests.get("https://9gag.com/v1/group-posts/group/default/type/hot", headers=headers, cookies=cookies)
print(req)
Both of these approaches are returning <Response [403]>
I am trying to make a request to "https://www.walmart.com/search/?page=1&query=" using request function or using scrapy module but getting the response code 444.
See below my snippet:
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36',
'Accept': 'application/json', 'Content-Type': 'application/json'
}
res = requests.get('https://www.walmart.com/', headers=headers)
cookie = res.cookies
res1 = requests.get('https://www.walmart.com/search/?page=1&query=',headers=res.headers,cookies=cookie)
But I m getting the res1.status_code as 444. Would appreciate any help here.
This is how you should reuse request elements:
import requests
with requests.Session() as connection:
connection.headers.update(
{
"Accept": "application/json",
"Content-Type": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/89.0.4389.86 YaBrowser/21.3.0.740 "
"Yowser/2.5 Safari/537.36",
}
)
_ = connection.get("https://www.walmart.com/")
response = connection.get('https://www.walmart.com/search/?page=1&query=')
print(response.status_code)
Output (status code):
200
I get a response like
config: {
url: '',
method: 'post',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
'accept-language': 'de-DE,de;q=0.8',
cookie: '**WANT THIS AS A STRING**',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36'
},
},
now I would like to get the cookie as a string to post it in the next request.
let cookies = response.config.headers.cookie;
I send a request through a proxy and always receive such a response
tunneling socket could not be established, cause=read ECONNRESET
or
tunneling socket could not be established, cause= socket hang up
My code
let settings = {
url: `url`,
headers: {
'Connection': 'keep-alive',
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
},
method: 'POST',
proxy: `http://${ip}:${port}`,
strictSSL: false
}
request.request(settings, (err, response, body) => {
// err here
})
what am I doing wrong ?
Now this error : Error: Tunnel creation failed. Socket error: Error: read ECONNRESET
My code:
const request = require('request'),
proxyingAgent = require('proxying-agent');
;
let settings = {
url: url,
headers: {
'Connection': 'keep-alive',
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
},
method: 'POST',
// proxy: `http://${obj.proxy[obj.proxyIdx]}`,
agent: proxyingAgent.create(`http://${obj.proxy[obj.proxyIdx]}`, url),
}
About your code, problem possibly lies in your settings object.
You need to use syntax like this:
let settings = {
url,
headers: {
'Connection': 'keep-alive',
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36"
},
method: 'POST',
proxy: `http://${ip}:${port}`,
strictSSL: false
}
Here we use ES6 to make object short.
But also, you can establish proxy connection with npm package proxying agent.
Your code should look something like this:
const proxyingAgent = require('proxying-agent');
const fetch = require('node-fetch');
const host = <your host>;
const port = <port>;
const creds = {
login: 'username',
password: 'pass'
};
const port = <proxy port>;
const buildProxy = (url) => {
return {
agent: proxyingAgent.create(`http://${creds.login}:${creds.password}#${host}:${port}`, url)
};
};
//If you don't have credentials for proxy, you can rewrite function
const buildProxyWithoutCreds = (url) => {
return {
agent: proxyingAgent.create(`http://${host}:${port}`, url)
};
};
And than you can use it with your url and credentials. We'll use fetch package.
const proxyGetData = async (url, type) => {
try {
const proxyData = buildProxyWithoutCreds(url);
// Make request with proxy. Here we use promise based library node-fetch
let req = await fetch(url, proxyData);
if (req.status === 200) {
return await req[type]();
}
return false;
} catch (e) {
throw new Error(`Error during request: ${e.message}`);
}
};
I am using the request npm module.I want to retrieve an image from a url. The request.get(url) function is returning me a '400 Bad Request', whereas the image is accessible from the browser.
The url i am hitting is : http://indiatribune.com/wp-content/uploads/2017/09/health.jpg
You could try to add some headers:
const request = require('request');
request.get({
url: 'http://indiatribune.com/wp-content/uploads/2017/09/health.jpg',
headers: {
Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-GB,en;q=0.8,en-US;q=0.6,hu;q=0.4',
'Cache-Control': 'max-age=0',
Connection: 'keep-alive',
Host: 'indiatribune.com',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
},
}, (err, response, data) => {
console.log(response, data);
});
The User-Agent seems to be enough.
Use download module . It's pretty simple.
const fs = require('fs');
const download = require('download');
download('http://indiatribune.com/wp-content/uploads/2017/09/health.jpg').pipe(fs.createWriteStream('foo.jpg'));