How can I send both query string parameters and form data parameters in nodejs. I tried the following code. but it's not working. it is just hanging and no response is comming.
var url_req = require("request");
url_req({
pool: separateReqPool,
url: "https://somthing.com/v1/message/userF",
method: 'POST',
qs: {
action: "send-message",
c: "somecode_c",
cache: new Date().getTime(),
sid: "somecode_sid"
},
form: {
convId: "userF~userT~esp",
message: "<font face='Arial' size='10'>.</font>",
sendAs: "userF",
txnId: "userF~" + new Date().getTime() + "~number",
},
headers: {
'Content-Type': 'application/json;charset=utf-8',
'Cookie': 'some cookies',
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:33.0) Gecko/20100101 Firefox/33.0',
'Accept-Language': "en-US,en;q=0.5",
'Connection': "keep-alive",
'Pragma': "no-cache",
'Cache-Control': "no-cache",
'Referer': "https://something.com/xframe.html?bc",
'Accept': ' text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Content-Length': 100,
'Host': "something.com"
}
}, function (error, response, body) {
if (error)
console.log(error);
else
console.log(body);
});
Related
I am new in Node/Jest world. I am trying to test the following function:
export async function visitPage(url: string) {
const cookie = ''
let headers = {
'authority': 'www.example.com',
'pragma': 'no-cache',
'cache-control': 'no-cache',
'upgrade-insecure-requests': '1',
'dnt': '1',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'navigate',
'sec-fetch-user': '?1',
'sec-fetch-dest': 'document',
'rtt': '350',
'downlink': '1.45',
'ect': '3g',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"macOS"',
'referer': 'https://google.com',
'accept-language': 'en-US,en;q=0.9,ur;q=0.8,zh-CN;q=0.7,zh;q=0.6',
'Accept-Encoding': 'gzip',
'cookie': cookie,
}
try {
const response = await axios({
method: 'get',
url: url,
headers: headers
});
if (response.status == 200) {
let html = response.data
if(html.includes('Hello, Sign in')) {
// console.log('Not logged in')
return false
}
}
} catch(error) {
if (error.response) {
console.log(error.response.data);
}
}
return true;
}
In test I am doing like the below:
describe('Product', () => {
it('visitPage', async () => {
const pageData = await visitPage('https://example.com');
// console.log(pageData)
expect(pageData).toBe(false);
});
});
The test gets passed but then it prints the following:
an all test suites.
Jest has detected the following 1 open handle potentially keeping Jest from exiting:
● TLSWRAP
31 |
32 | try {
> 33 | const response = await axios({
| ^
34 | method: 'get',
35 | url: url,
36 | headers: headers
Use nextTick before axios call as stated in here
await process.nextTick(() => {})
That solves my issue.
Don't understand why the response body of the following request is empty:
request:
req.headers['User-Agent'] = 'request';
req.headers['Accept'] = 'text/html';
const options = {
url: config.prodURL,
headers: req.headers,
qs: req.query,
method: req.method
};
request(options, (err, resp, body) => {
console.log(resp);
});
I'm expecting HTML in this response, but for some reason, when I look at resp it displays body: ''
Part of the resp contains the href url and if I use this in any client, it displays the HTML code.
What is wrong with this config?
update
Here's the data with the exception of hrefas I cannot share this URL. err is null, here are the rest of the values:
{ url: 'http://my.url/',
headers:
{ 'user-agent': 'PostmanRuntime/7.15.2',
accept: '*/*',
'cache-control': 'no-cache',
'postman-token': '8c208831-580b-4460-b3e0-046d85789658',
host: 'localhost:8001',
'accept-encoding': 'gzip, deflate',
connection: 'keep-alive',
'User-Agent': 'request',
Accept: 'text/html' },
qs: { query-params... },
method: 'GET'
}
I believe it might be related to the response headers, the URL requested doesn't return any, so does requestJs require a content-type to be able to parse the body?
request get no body content with header "content-length: 0".
resource: https://www.doe.gov.taipei/OpenData.aspx?SN=8A3B3293C269E096
It is a rss link, work well in browser and curl.
curl test on linux, and wget works too.
here is simple code:
const req = require('request');
req.get({
url: 'https://www.doe.gov.taipei/OpenData.aspx?SN=8A3B3293C269E096',
rejectUnauthorized: false
}, (err, res, body) => {
console.log(res)
});
test it online simulator get same result
https://repl.it/repls/ScalyAquamarineObjectpool
i think it should be a text/xml content, not empty.
response:
{
statusCode: 200,
body: '',
headers:
{ 'cache-control': 'private,No-cache',
'set-cookie':
[ 'ASP.NET_SessionId=kzdhyxq2grttp5awwpqzxyen; path=/; secure; HttpOnly' ],
'x-frame-options': 'SAMEORIGIN',
'strict-transport-security': 'max-age=0',
'x-xss-protection': '1; mode=block',
'x-content-type-options': 'nosniff',
'content-security-policy':
'frame-ancestors \'self\' https://www-mgr.gov.taipei http://www-mgr.gov.taipei',
date: 'Wed, 03 Apr 2019 06:44:47 GMT',
connection: 'close',
'content-length': '0' },
request:
{ uri:
Url {
protocol: 'https:',
slashes: true,
auth: null,
host: 'www.doe.gov.taipei',
port: 443,
hostname: 'www.doe.gov.taipei',
hash: null,
search: '?SN=8A3B3293C269E096',
query: 'SN=8A3B3293C269E096',
pathname: '/OpenData.aspx',
path: '/OpenData.aspx?SN=8A3B3293C269E096',
href:
'https://www.doe.gov.taipei/OpenData.aspx?SN=8A3B3293C269E096' },
method: 'GET',
headers: {} }
}
The signature of your callback function is wrong. In Node.JS, usually the first parameter to the callback is the (optional) error object.
Try this:
const req = require('request');
req.get({
url: 'https://www.doe.gov.taipei/OpenData.aspx?SN=8A3B3293C269E096',
rejectUnauthorized: false
}, (err, res, body) => {
console.log(res)
});
To your updated question: It seems like adding a user-agent does the trick:
const req = require('request');
req.get({
url: 'https://www.doe.gov.taipei/OpenData.aspx?SN=8A3B3293C269E096',
rejectUnauthorized: false,
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3750.0 Iron Safari/537.36'
}
}, (err, res, body) => {
console.log(body)
});
I'm using request.
I want to login a website only once, then use the same connection until it expires, so I won't have to login every time I send a request.
How can I do that?
I have tried the code below, but it doesn't retain the session. I'm using Redis to save cookie values, this is probably the wrong way, though. Two cookies are generated after you log in.
var request = require("request").defaults({jar: true, followAllRedirects: true});
var options1 = {
method: 'GET',
url: 'homepageURL',
headers:
{
'cache-control': 'no-cache',
'accept-language': 'tr,en-US;q=0.9,en;q=0.8,ru;q=0.7,it;q=0.6,fr;q=0.5,de;q=0.4,es;q=0.3,sv;q=0.2,nl;q=0.1,pl;q=0.1,pt;q=0.1,nb;q=0.1',
'accept-encoding': 'gzip, deflate, br',
referer: 'URL',
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
'upgrade-insecure-requests': '1'
}
};
request(options1, function (homeError, homeResponse, homeBody) {
if (homeResponse.statusCode == 200) {
var form = {"username": "myusername", "password": "mypassword", "secret": "mysecret", "action": "login"};
var options2 = {
method: 'POST',
url: 'siteurl + "/" + login.asp',
headers:
{
'cache-control': 'no-cache',
'accept-language': 'tr,en-US;q=0.9,en;q=0.8,ru;q=0.7,it;q=0.6,fr;q=0.5,de;q=0.4,es;q=0.3,sv;q=0.2,nl;q=0.1,pl;q=0.1,pt;q=0.1,nb;q=0.1',
'accept-encoding': 'gzip, deflate, br',
referer: 'URL',
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
'content-type': 'application/x-www-form-urlencoded',
'upgrade-insecure-requests': '1',
origin: url + "/" + login.asp
},
form: form
};
request(options2, function (loginError, loginResponse, loginBody) {
if (loginResponse.statusCode == 200) {
console.log("Successfully logged in.");
client.set("cookie1", homeResponse.request.headers.cookie, function () {
client.set("cookie2", loginResponse.request.headers.cookie, function () {
nrp.emit('profile', {});
});
});
}
});
}
});
Profile test
nrp.on('profile', function () {
client.get("cookie1", function (cookie1Err, cookie1) {
client.get("cookie2", function (cookie2Err, cookie2) {
var cookie = cookie1 + ";" + cookie2;
var options = {
method: 'GET',
url: 'site/profile',
headers:
{
'cookie': cookie,
'accept-language': 'tr,en-US;q=0.9,en;q=0.8,ru;q=0.7,it;q=0.6,fr;q=0.5,de;q=0.4,es;q=0.3,sv;q=0.2,nl;q=0.1,pl;q=0.1,pt;q=0.1,nb;q=0.1',
'accept-encoding': 'gzip, deflate, br',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
'upgrade-insecure-requests': '1'
}
};
request(options, function (error, response, body) {
console.log(body);
// it says you are not logged in, but if you use this inside the block above, it works.
});
})
});
});
I am using ajax to send post to my app. In postman everything works as expected, however in browser, data comes in as x-www-form-urlencoded and data is missing.
Below is my ajax and postman configurations and their corresponding headers in req...
$.ajax({
url: '/',
headers: {
'Accept': 'application/json',
'Content-Type': "application/json"
}, // send as JSON
data: JSON.stringify(formData),
method: 'POST'
});
headers:
{ host: 'localhost:8080',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0',
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'accept-language': 'en-US,en;q=0.5',
'accept-encoding': 'gzip, deflate',
referer: 'http://localhost:8080/',
'content-type': 'application/x-www-form-urlencoded',
'content-length': '0',
connection: 'keep-alive',
'upgrade-insecure-requests': '1',
pragma: 'no-cache',
'cache-control': 'no-cache' }
headers:
{ data: '{"name":"tester", "email": "t#t.com", "message":"This is a test"}',
contenttype: 'application/json',
datatype: 'xml/html/script/json',
'cache-control': 'no-cache',
'postman-token': 'cbaa5cbe-0ab8-4328-abfd-f5dc6a6acd90',
'user-agent': 'PostmanRuntime/7.1.1',
accept: '*/*',
host: 'localhost:8080',
'accept-encoding': 'gzip, deflate',
'content-length': '0',
connection: 'keep-alive' }
also my relevant server code:
app.use(bodyParser.json());
//handle urlencoded data
app.use(bodyParser.urlencoded({extended:false}));
app.post('/', function (req, res) {
console.log(res);
console.log('----------------------------------------------------------');
console.log(req);
//var jsonData = JSON.parse(req.headers.data);
console.log(req.body);
console.log(req.data);
let transporter = courier.createTransport({...});
let mailOptions = {...};
transporter.sendMail(mailOptions, function (error, info) {
if (error)
console.log(error);
else
console.log('email sent:' + info.response);
});
res.render('index', {
title: 'Projects',
projects: projects
});
});
app.get('/', function (req, res) {
res.render('index', {
title: 'Projects',
projects: projects
});
});
plz halp!
on you can just simply:
JSON.parse(data)
inside your
success: function(data){
var object =JSON.parse(data);
}
once the json parses the data you can do what ever you like.