I have a secured CouchDB instance running. For Replication, I was following the instructions on the "Getting Started" guide. However PouchDB doens't seem to send my credentials to the server so I'm getting an "Authentication required" error from CouchDB.
Reproduce
this.db = new PouchDB('nfcs');
this.remote = 'https://USER:PASSWORD#couchdb.pixelarbeit.de/nfcs';
let options = {
live: true,
retry: true,
continuous: true
};
this.db.sync(this.remote, options)
.on('error', err => console.log('PouchDB Error NFCs', err))
.on('active', err => console.log('PouchDB Active', err))
.on('complete', err => console.log('PouchDB Complete', err));
Headers sent to server
Request URL:https://couchdb.pixelarbeit.de/nfcs/
Request Method:OPTIONS
Status Code:401 Unauthorized
Remote Address:185.26.156.40:443
Referrer Policy:no-referrer-when-downgrade
Response Headers
Access-Control-Allow-Origin:http://localhost:8100
Access-Control-Expose-Headers:Cache-Control, Content-Type, Server
Cache-Control:must-revalidate
Connection:close
Content-Length:61
Content-Type:text/plain; charset=utf-8
Date:Mon, 16 Oct 2017 06:12:45 GMT
Server:CouchDB/1.6.1 (Erlang OTP/17)
WWW-Authenticate:Basic realm="server"
Request Headers
Accept:*/*
Accept-Encoding:gzip, deflate, br
Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4,ru;q=0.2,es;q=0.2
Access-Control-Request-Headers:authorization
Access-Control-Request-Method:GET
Cache-Control:no-cache
Connection:keep-alive
Host:couchdb.pixelarbeit.de
Origin:http://localhost:8100
Pragma:no-cache
Referer:http://localhost:8100/
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
I just fixed the issue. It seems like a PouchDB plugin broke the functionality.
Related
I am trying to scape this url https://www.myntra.com/laptop-bag/chumbak/chumbak-unisex-brown-geo-bird--printed-laptop-bag/6795882/buy using puppeteer.
It's working when i use { headless: false }, but failing in headless mode.
Then i have compared response in both cases using this.
const resp = await page.goto(url);
console.log(resp);
Then i figured out that we need to add userAgent when using headless mode. so i have added this.
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
Now it is working in both cases locally. But when i deploy to cloud function, it is still failing.
This is the screenshot taken using puppeteer.
this is some part of the response log.
_headers:
{ status: '403',
server: 'AkamaiGHost',
'mime-version': '1.0',
'content-type': 'text/html',
'content-length': '395',
expires: 'Thu, 09 Jul 2020 12:16:30 GMT',
date: 'Thu, 09 Jul 2020 12:16:30 GMT',
'set-cookie': 'AKA_A2=A; expires=Thu, 09-Jul-2020 13:16:30 GMT........
Am i missing anything?
Thanks.
update:
I have used puppeteer stealth plugin along with IP rotation. here is the code
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker')
puppeteer.use(AdblockerPlugin({ blockTrackers: true }))
And for IP rotation:
var browser = await puppeteer.launch({
headless: true,
args: ['--proxy-server=abcd-efg.proxymesh.com:12345']
});
var page = await browser.newPage();
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36');
await page.authenticate({
username: 'myusername',
password: 'mypassword'
});
IP rotation working locally but still blocked on cloud function.
Using residential proxies fixed the issue.
Initially I have deployed in cloud function and AWS lambda with IP rotation. I have used proxymesh service for IP rotation. but it provides data center proxies only. It was failed. Then i tried with residential proxies from another service. It worked.
I'm brand new to Node JS (v.10.9.0) and wanted to make a simple web scraping tool that gets statistics and ranks for players on this page. No matter what I can't make it work with this website, I tried multiple request methods including http.request and https.request and have gotten every method working with 'http://www.google.com'. However every attempt for this specific website either gives me a 301 error or a socket hang up error. The location the 301 error gives me is the same link but with a '/' on the end and requesting it results in a socket hang up. I know the site runs on port 443. Do some sites just block node js, why are browsers able to connect but not stuff like this?
Please don't link me to any other threads I've seen every single one and none of them have helped
var request = require('request');
var options = {
method: "GET",
uri: 'https://www.smashboards.com',
rejectUnauthorized: false,
port: '443'
};
request(options, function (error, response, body) {
console.log('error:', error); // Print the error if one occurred
console.log('statusCode:'); // Print the response status code if a response was received
console.log('body:', body); // Print the HTML for the Google homepage.
});
Error:
error: { Error: socket hang up
at createHangUpError (_http_client.js:322:15)
at TLSSocket.socketOnEnd (_http_client.js:425:23)
at TLSSocket.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1085:12)
at process._tickCallback (internal/process/next_tick.js:63:19) code: 'ECONNRESET' }
EDIT:
Adding this to my options object fixed my problem
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
OP Here
All I did was add:
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
To my options Object and it's working perfectly.
New code:
var request = require('request');
var options = {
method: 'GET',
uri: 'https://www.smashboards.com',
rejectUnauthorized: false,
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
};
request(options, function (error, response, body) {
console.log('error:', error); // Print the error if one occurred
console.log('statusCode:'); // Print the response status code if a response was received
console.log('body:', body); // Print the HTML for the Google homepage.
});
Thats 12+ hours I'm never getting back
I am struggling to successfully make a request using request-promise npm on a site that requires a cookie to view or for the request to be successful.
Henceforth, I have looked into cookieJars in order to store all those that are given in the repsonse after the request has been done.
const rp = require("request-promise")
var cookieJar = rp.jar()
function grabcfToken(){
let token = ""
let options = {
url : 'https://www.off---white.com/en/GB',
method: "GET",
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
resolveWithFullResponse : true
}
rp(options)
.then((response)=>{
console.log(response)
})
.catch((error)=>{
console.log(error)
})
}
Can someone tell me why the request isn't successfully going through? How do I apply the cookies that I initially get before being timed out.
const rp = require("request-promise")
var cookieJar = rp.jar()
function grabcfToken(){
let token = ""
let options = {
url : 'https://www.off---white.com/en/GB',
method: "GET",
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
resolveWithFullResponse : true,
jar: cookieJar
}
rp(options)
.then((response)=>{
console.log(response)
})
.catch((error)=>{
console.log(error)
})
}
If you're asking about including your jar which you filled with the cookies from the request to be sent to across you have to add jar: cookiejar as pasrt of your options object before sending it.
I am trying to call a soap web service from node js. But the server does not show me any output either the success or the error. I am using http server for this purpose
My output is always :
[Thu Dec 15 2016 15:12:16 GMT+0530 (India Standard Time)] "GET /views/bookNGo.html?empId=EMP2&empName=ABCD&mobile=3312&alternateMobile=678&address=hjhjh&landMark=hjgjhgj&deptId=hjhj&projectId=hjgh&app
rovingManager=kjhkj&pickupPoint=jkhjkh&dropPoint=jkhjk" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
var soap = require('soap');
console.log('One');
var url = 'https://ap2.salesforce.com/services/wsdl/class/InsertEmployee?WSDL';
var args = {"employeeId": "EMP2","employeeName": "Chandra","phoneNumber":"12345","alternatePhoneNumber": "98765","address": "Boduppal", "landMark": "Temple","departmentID": "001","projectID": "001","approvingManager": "Prashanth","pikupPoint": "Uppal","dropPoint":"Hitec"};
soap.createClient(url, function(err, client) {
client.createEmployee(args, function(err, result) {
console.log("Webservice called");
console.log(result);
});
});
I have tired running the code in https://runkit.com/npm/soap and it gives me an error saying "TypeError: Cannot read property 'InsertEmployeeService' of undefined".
Using mongoose to connect to mongolab and hosted in heroku. Method get, post , put works perfectly but deleting is the "problem".
when i tried to delete. i got it first.
Request URL:https://---------.herokuapp.com/------/54b413c2647bec02001efdd0
Request Headers
Provisional headers are shown
Accept:application/json, text/plain, */*
Origin:null
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/39.0.2171.95 Safari/537.36
After 30 seconds more or less i got this.
Remote Address:150.100.2.200:8080
Request URL:https://---------.herokuapp.com/------/54b413c2647bec02001efdd0
Request Method:DELETE
Status Code:503 Service Unavailable
Request Headersview source
Accept:application/json, text/plain, */*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8,es-419;q=0.6,es;q=0.4,fr;q=0.2
Host:-----.herokuapp.com
Origin:null
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/39.0.2171.95 Safari/537.36
Response Headersview source
Cache-Control:no-cache, no-store
Connection:keep-alive
Content-Length:484
Content-Type:text/html; charset=utf-8
Date:Mon, 12 Jan 2015 20:36:33 GMT
Server:Cowboy
I have this. Showing me an error 503 but.. if i chek on mongolab webpage the item is already gone. When i tested it in my local machine everything works fine including delete method but using heroku i got this problem.
-CORS avaliable.
-These are 3 different ways that i tried to do obtaining the same result.
exports.deleteNotificacion = function(req, res) {
var id = req.params.id;
console.log(id);
Todo.findById(id,function(err,notificacion){
notificacion.remove();
notificacion.save();
}); }
exports.deleteNotificacion = function(req, res) {
var id = req.params.id;
console.log(id);
Todo.findById(id,function(err,notificacion){
notificacion.remove (function(err){
if (!err) {
res.send('');
console.log('Removed');
}else {
console.log('ERROR: ' + err);
};
})
}); }
exports.deleteNotificacion = function(req, res) {
var id = req.params.id;
console.log(id);
Todo.findByIdAndRemove(id,function(err){
if(err){console.log("ERROR " + err);}
// res.send("eliminado");
});}
Code samples that you have posted are difficult to read, but it seems that you just don't send anything to the client if the request was successful. After 30s of waiting Heroku router timeouts and your client get a 503 page, as described in this blogpost.
One of the correct responses to a DELETE request is to send HTTP 204 code with no body:
res.status(204).end();