Problem: Need to download a private repository that is within an organization hosted on GitHub Enterprise.
I created a personal access token for my account with scope repo and stored it as an environment variable, GITHUB_ACCESS_TOKEN.
I'm using NodeJS with the request library to make the GET request. However, with the following code, I get a 401 response when I run it.
(Note: I replaced <repo-name> with the actual name of the repository).
Can someone explain why this doesn't work and point me in the right direction?
My Function :
function downloadRepository(owner, repository, branch, accessToken) {
let options = {
method: "GET",
url: `https://api.github.com/orgs/${owner.toLowerCase()}/repos/${repository.toLowerCase()}/tarball/${branch}?access_token=${accessToken}`,
headers: {
'Accept': 'application/vnd.github.v3.raw',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'
}
};
request(options, (error, response, body) => {
if(error || response.statusCode != 200) {
console.log("Could not download repository: %s", response.statusCode);
return;
}
return body;
});
}
My Main :
const request = require('request');
let wiki = downloadRepository("deep-learning-platform", "<repo-name>", "wiki",
process.env.GITHUB_ACCESS_TOKEN);
Related
I am trying to create a simple tool which will hit a specific Facebook page and extract some data. I am using NodeJS for this. I am using inbuilt https module to the GET request. The thing in my local machine it is working fine but inside the VM it is not able to get the response, it is failing every time.
I tried with axios also but it is not working either.
Previously I thought that could be firewall issue but I enabled every thing inside the firewall but then also it is not working. Very strange issue I am facing with this.
What can be the possible reason for this?
Is it Facebook blocking my request? No right? I am sending User-Agent header also but it is not working.
My code looks like below :
const options = {
hostname: 'www.facebook.com',
// port: 443,
path: `/${username}`,
method: 'GET',
headers: {
'user-agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36',
'Accept-Language': 'en-GB,en-US;q=0.8,en;q=0.6',
},
};
const request = https.request(options, (response) => {
let body = '';
console.log('response : \n' + response);
response.setEncoding('utf8');
response.on('data', (chunk) => {
body += chunk;
console.log('response on data chunk : \n' + chunk);
});
response.on('end', () => {
console.log('body : \n' + body);
const arrMatches = body.match(rePattern);
if (arrMatches && arrMatches.length > 1) {
resolve(arrMatches[1]);
} else {
reject(new Error('Facebook user not found'));
}
});
});
request.on('error', (err) => {
console.log('request error : \n' + err);
reject(err);
});
request.end();
I'm trying to scrape the html using library request on node.js. The response code is 200 and the data I get is unreadable. Here my code:
var request = require("request");
const options = {
uri: 'https://www.wikipedia.org',
encoding: 'utf-8',
headers: {
"Accept": "text/html,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
"charset": "utf-8",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/78.0.3904.108 Chrome/78.0.3904.108 Safari/537.36"
}
};
request(options, function(error, response, body) {
console.log(body);
});
As you can see, I sent the request for html and utf-8 but got a large string like f��j���+���x��,�G�Y�l
My node version is v8.10.0 and the request version is 2.88.0.
Is something wrong with the code or I'am missing something??
Any hint to overtake this problem would be appreciate.
Updated Answer:
In response to your latest post:
The reason it is not working for Amazon is because the response is gzipped.. In order to decompress the gzip response, you simply need to add gzip: true to the options object you are using. This will work for both Amazon and Wikipedia:
const request = require('request');
const options = {
uri: "https://www.amazon.com",
gzip: true
}
request(options, function(error, response, body) {
if (error) throw error;
console.log(body);
});
Lastly, if you are wanting to scrape webpages like this, it is probably best to use a web scraping framework, like Puppeteer, since it is built for web scraping.
See here for Puppeteer GitHub.
Original Answer:
Since you are just grabbing the HTML from the main page, you do not have to specify charset, encoding, or Accept-Encoding..
const request = require('request');
const options = {
uri: 'https://www.wikipedia.org',
//encoding: 'utf-8',
headers: {
"Accept": "text/html,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
//"charset": "utf-8",
//"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/78.0.3904.108 Chrome/78.0.3904.108 Safari/537.36"
}
};
request(options, function (error, response, body) {
if (error) throw error
console.log(body);
});
To take it a bit further... in this scenario, you don't need to specify headers at all...
const request = require('request');
request('https://www.wikipedia.org', function (error, response, body) {
if (error) throw error
console.log(body);
});
Thanks you the reply, when I used that to the Wikipedia page works properly, but when I use it to scrape another website like the amazon, got the same bad result
const request = require('request');
request('https://www.amazon.com', function (error, response, body) {
if (error) throw error
console.log(body);
});
I'm creating an https request, to get some hidden variables on a sign in page. I'm using the node.js package request for this. After calling the request, I'm using a callback to go back to my parse function.
class h {
constructor(username, password){
this.username = username;
this.password = password;
this.secret12 = '';
}
init() {
//Loading H without cookie
request({
uri: "http://example.com",
method: "GET",
jar: jar,
followRedirect: true,
maxRedirects: 10,
timeout: 10000,
//Need to fake useragent, otherwise server will close connection without response.
headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}
},
this.getHiddenInputs());
}
getHiddenInputs(error, response, body) {
if (!error && response.statusCode === 200) {
//Parsing body of request, to get hidden inputs required to mock legit authentication.
const dom = new JSDOM(body);
this.secret12 = (dom.window.document.querySelector('input[value][type="hidden"][name="secret12"]').value);
}
else {
console.log(error);
console.log(response.statusCode)
}
};
}
const helper = new h("Username", "Password");
helper.init();
console.log(helper);
So after calling request inside init(). I'm using the callback function to run the code that finds the Hidden Input after it has completed the request. I'm following the example from here.
Am I missing something?
You are executing this.getHiddenInputs() instead of passing it to request as a callback, so there is no actual callback given to the request call.
You could pass it like this this.getHiddenInputs.bind(this) or I'd prefer something like this (error, response, body) => this.getHiddenInputs(error, response, body)
I deployed a web app using google apps script and when I try to make requests to it using Node.js I get the following error:
Google Drive Page Not Found - Sorry, unable to open the file at this time
But everything works fine when I'm sending the requests using Postman.
This is my node.js code:
const options = {
'method': 'post',
'gzip': true,
'accept': '*/*',
'content-length': 0,
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36',
'url': 'https://script.google.com/macros/s/script-id/exec'
}
request(options, (error, response, body) => {
if(response.statusCode == 200) {
res.sendStatus(200);
}
else {
res.sendStatus(500);
}
});
This is the content of the google apps script web app:
function doPost(e) {
return ContentService.createTextOutput(JSON.stringify({success: true}));
}
There are a couple more questions related to this topic, but the solution they provide is to remove /u/# from the URL ... which does not apply in my case.
Could this happen because I'm making requests from a http server to a web app deployed on https? That's the only thing that I'm thinking of...
Update:
I've also tried to send the request from a HTTPS node server and it still doesn't work.
Making a GET request works fine.
Making POST requests from a Node.js server to a web app deployed using Google Apps Script requires the 'Content-Length' header to be specified, and also the followAllRedirects option set to true, which is needed because the initial request is a redirect to another request that returns the response.
The working code looks like this:
const options = {
'method': 'post',
'gzip': true,
'body': [],
'followAllRedirects': true,
'headers': {
'Content-Length': 0 // length of the specified `body`
},
'url': 'https://script.google.com/macros/s/scriptID/exec'
}
request(options, (error, response, body) => {
if(response.statusCode == 200) {
res.sendStatus(200);
}
else {
res.sendStatus(500);
}
});
Hopefully this will help others that are facing this issue.
When publishing the web app inside Google Apps Script, make sure you have set the "Who has access" option to "Anyone, including anonymous".
Also, it may not work if your organization does not allow sharing of Drive files outside the organization. This setting can only be changed by the GSuite admin.
I'm using NodeJS "request" module to access this particular page
http://www.actapress.com/PaperInfo.aspx?PaperID=28602
via
r = request(i, (err, resp, body) ->
if err
console.log err
else
console.log body
)
The content of "body" is different compared to when I actually access the URL via the browser. Are there some extra settings that I need to configure for request module?
try to set User-Agent header:
request({
uri: 'http://www.actapress.com/PaperInfo.aspx?PaperID=28602',
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36'
}
}, function(err, res, body) {
console.log(body);
});
You can simply use JSON.parse.
body = JSON.parse(body);