This my first time hooking up to an API. I'm attempting to query NIH project data and can seem to hook up correctly to it as I get a status code of 200. The issue I have is when I try to print the output. I get a URL rejection despite having access. The documentation suggests that if any issues come up, it may be due to my IP being blocked due to a variety of reasons. I reached out to the API support team there and I don't have any issues with a blocked IP. They had me run some curl requests via command prompt and I was able to execute those correctly. This leads me to believe I have a code issue. What am I doing wrong here?
NIH API Info
import requests
params = {
"criteria":
{
"fiscal_years":[2019,2018]
},
"include_fields": [
"ApplId","SubprojectId","FiscalYear","Organization", "ProjectNum","OrgCountry",
"ProjectNumSplit","ContactPiName","AllText","FullStudySection",
"ProjectStartDate","ProjectEndDate"
],
"offset":0,
"limit":25,
"sort_field":"project_start_date",
"sort_order":"desc"
}
response = requests.post("https://api.reporter.nih.gov/v2/projects/Search", data = params)
#print(response.status_code)
print(response.text)
Sample curl script and output:
curl -X POST "https://api.reporter.nih.gov/v2/projects/search" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"criteria\":{\"covid_response\":[\"Reg-CV\",\"CV\"]},\"include_fields\":[\"ApplId\",\"SubprojectId\",\"FiscalYear\",\"Organization\",\"ProjectNum\",\"OrgCountry\",\"ProjectNumSplit\",\"ContactPiName\",\"AllText\",\"FullStudySection\",\"ProjectStartDate\",\"ProjectEndDate\"],\"offset\":0,\"limit\":10}"
The solution is simple, but easily missed. You need json=params See this question: How to POST JSON data with Python Requests?
response = requests.post("https://api.reporter.nih.gov/v2/projects/search", json=params)
Below is the entire code with the small change:
import requests
params = {
"criteria":
{
"fiscal_years":[2019,2018]
},
"include_fields": [
"ApplId","SubprojectId","FiscalYear","Organization", "ProjectNum","OrgCountry",
"ProjectNumSplit","ContactPiName","AllText","FullStudySection",
"ProjectStartDate","ProjectEndDate"
],
"offset":0,
"limit":25,
"sort_field":"project_start_date",
"sort_order":"desc"
}
response = requests.post("https://api.reporter.nih.gov/v2/projects/search", json=params)
print(response.status_code)
print(response.text)
The start of result looks like:
{"meta":{"search_id":null,"total":160216,"offset":0,"limit":25,"sort_field":"project_start_date","sort_order":"desc","sorted_by_relevance":false,"properties":{}},"results":[{"appl_id":10396858,"subproject_id":null,"fiscal_year":2018,"project_num":"7K01AG046366-06","organization":{"org_name":"UNIVERSITY OF CONNECTICUT SCH OF MED/DNT","city":null,"country":null,"org_city":"FARMINGTON","org_country":"UNITED STATES","org_state":"CT","org_state_name":null,"dept_type":"NEUROSCIENCES","fips_country_code":null,"org_duns":["022254226"],...
Related
I am using cURL to fetch a website (which returns the error I want).
$ curl 'https://matchmaker.krunker.io/seek-game?hostname=krunker.io®ion=us-ca-sv&game=SV%3A4jve9&autoChangeGame=false&validationToken=QR6beUGVKUKkzwIsKhbKXyaJaZtKmPN8Rwgykea5l5FkES04b6h1RHuBkaUMFnu%2B&dataQuery=%7B%7D
However, when I use the node-libcurl package in Node.JS, I get an error 1020. Why does this happen?
const { curly } = require('node-libcurl')
const url = 'https://matchmaker.krunker.io/seek-game?hostname=krunker.io®ion=us-ca-sv&game=SV%3A4jve9&autoChangeGame=false&validationToken=QR6beUGVKUKkzwIsKhbKXyaJaZtKmPN8Rwgykea5l5FkES04b6h1RHuBkaUMFnu%2B&dataQuery=%7B%7D'
curly.get(url)
.then(({ statusCode, data }) => console.log(statusCode, data))
1020 is not a curl error code, and a quick google search suggest 1020 is a common-ish firewall error for Cloudflare, ref
https://community.cloudflare.com/t/error-1020-cannot-log-in-to-the-site-how-to-fix/71784
the most obvious difference for a server between a LIBcurl request and a curl request is that libcurl doesn't have an user-agent, but curl does, and it's not too uncommon to block requests lacking an user-agent (wikipedia.org is 1 example blocking requests lacking useragent),add an user-agent and try again.
curly.get(url,{userAgent:"imaginate's awesome curl script"})
.then(({ statusCode, data }) => console.log(statusCode, data))
This answer: "How to perform status checks in github repository?" ..
References the following articles:
https://help.github.com/en/github/administering-a-repository/enabling-required-status-checks
https://developer.github.com/v3/guides/building-a-ci-server/#working-with-statuses
So, to update a status check, I need to create one (and update it with a status at the same time); as per: https://developer.github.com/v3/repos/statuses/#create-a-status ... Issue a POST to:
/repos/:owner/:repo/statuses/:sha
However, I am getting a 404 (response.status_code: 404), whenever I try to post a payload to the endpoint url.
NOTE: I am building the url, using the information from the GitHub web-hook itself i.e.
content['repository']['owner']['login']
content['repository']['name']
content['pull_request']['head']['sha']
JSON Payload:
payload = { "state": "pending", "description": "The build succeeded!", "context": "continuous-integration/audit-compliance"}
Request:
headers = {'Content-Type': 'application/json',
'Authorization': "token "+gittokn}
url = https://github.com/api/v3/repos/<OWNER>/<REPO>/statuses/<PR-HEAD-SHA>
response = requests.post(url, headers=headers, json=payload, verify=False)
Response:
{'message': 'Not Found', 'documentation_url': 'https://developer.github.com/enterprise/2.18/v3/repos/statuses/#create-a-status'}
With Curl:
curl -Is --user USER:PASS https://github.com/api/v3/repos/<OWNER>/<REPO>/statuses/<PR-HEAD-SHA> | head -1
HTTP/1.1 200 OK
curl -Is --user USER:PASS https://api.github.com/repos/<OWNER>/<REPO>/statuses/<PR-HEAD-SHA> | head -1
# no result
Where am I going wrong?
Can provide additional info on request; TIA.
See that in the example the url has no v3 prefix, the template below shall work for you:
https://api.github.com/repos/ORGNAME/REPONAME/statuses/<SHA>
UPDATED ANSWER
https://github.com/api/v3/repos///statuses/
.. works fine.
The permission settings for GitHub token were the problem; Github really need to fix their exception handling; 404 it is not!
I have written a shell script code which is successfully adding a new OS Connection IP in Imperva system/instance.
## Create a new IP OS connection in a given site, server group.
create_ip()
{
JSESSIONID="${1}"
addThisIP="${2}"
siteName="${3}"
serverGroupName="${4}"
echo -e "\n- Trying to create a new OS connection IP now.\n";
##Make sure while initiating a REST API call, any parameter which has ' ' (space) in Imperva should be substituted with '%20'.
##JSESSIONID will be generated first and will be available to this function
create_new_ip_output="$(curl -ik -X POST -H "Cookie: JSESSIONID=$JSESSIONID" -H "Content-Type: application/json" -H "Accept: application/json" "https://${MX}:8083/SecureSphere/api/v1/conf/serverGroups/${siteName// /%20}/${serverGroupName// /%20}/servers/${addThisIP}" -d '{"connection-mode":"SSH","host-name":"thisLinuxServer.fdqn","OS-type":"linux","user-name":"enter_your_userID"}')";
return_code="$?";
if [[ "${return_code}" == "0" && ! `echo "${create_new_ip_output}" | grep "do not have permission"` ]]; then
echo -e "\n\n- OS connection IP (${addThisIP}) created successfully in site: ${siteName}, servergroup: ${serverGroupName} and stdout:\n${create_new_ip_output}\n";
return 0;
else
echo -e "\n\n- Failed to create a new OS connection IP (${addThisIP}) in site: ${siteName} and servergroup: ${serverGroupName}\n- using session ID: ${JSESSIONID}, error log:\n${create_new_ip_output}\n";
return 1;
fi
}
OK, the above code works fine and once run, I see valid output showing whether an IP got added successfully or what was the failure message.
Now, I'm trying to implement the same functionality in NodeJS.
To do that, I have successfully created functions to generate JSESSIONID (so that I can use it to perform multiple REST/API call operations, rather than creating a new session each time for any operation), successfully deleting a JSESSIONID (i.e. to logoff/out from Imperva session so that I don't hit the limit) and to successfully search an IP in Imperva system if it has been previously added.
Using the above shell script logic, I have written the following NodeJS code to add a new OS Connection IP in Imperva using Rest/API, but I'm getting an error.
//Imperva create IP
//qData is a hash array that has valid index/value pair values.
var impervaCreateIP = function(qData){
var deferred = Q.defer();
var data = ''
var options = {
hostname: 'myImpervaServerInstance.mycompanydomain.com',
port: 8083,
method: 'POST',
path: '/SecureSphere/api/v1/conf/serverGroups/'+ qData['siteName'] + '/' + qData['serverGroupName'] + '/servers/' + qData['ip'],
headers: {
'Content-Type': 'application/xml',
'X-Requested-With': 'Nodejs',
'Cookie': 'JSESSIONID='+qData['sid']
}
}
//svtLog() is a logger that I'm using to log messages to a file. Out of scope of this post.
svtLog('info','Imperva','Inside impervaCreateIP function')
svtLog('info','Imperva',qData['ip'])
svtLog('info','Imperva',qData['siteName'])
svtLog('info','Imperva',qData['serverGroupName'])
svtLog('info','Imperva','')
console.log('Inside impervaCreateIP')
console.log(options);
httpHelp(options, data)
.then(function(fullResponse){
var result = JSON.parse(fullResponse[1])
console.log("11 -----")
console.log(result)
console.log("22 -----")
deferred.resolve(qData['sid'])
console.log(result)
})
.fail(function(e){
svtLog('error','Imperva','Failed to add IP in Imperva')
svtLog('info','Imperva',qData['ip'])
svtLog('error','Imperva',e)
svtLog('info','Imperva','')
deferred.reject(e)
})
return deferred.promise;
}
Error message:
Inside impervaCreateIP
{ hostname: 'myImpervaServerInstance.mycompanydomain.com',
port: 8083,
method: 'POST',
path: '/SecureSphere/api/v1/conf/serverGroups/some%20Site%20NameInImperva/someServerGroupNameInImperva_01/servers/10.20.30.40',
headers:
{ 'Content-Type': 'application/xml',
'X-Requested-With': 'Nodejs',
Cookie: 'JSESSIONID=7B3C378D365B673F6C749847DEDC7D8F } }
[SyntaxError: Unexpected token <]
I'm looking for 2 things:
1. How to resolve the error (as shown above).
2. How to pass CURL's -d option {...} body parameters in the NodeJS code above, like I used it in the shell script.
PS: Changing POST to GET (method), assures that the code is fine syntax wise as it successfully shows the IP (with GET operation) with the above NodeJS code.
So, it's failing only when I use POST (i.e. when I'm trying to create an IP) -vs- GET (which I use to find an IP exist or not). I checked as per the Imperva API doc, the response comes in JSON format. Not sure if due to missing bullet# 2 question, I'm getting this Syntax error with POST.
OK.
Solution to both bullet# 1 and #2 was to set the CURL "-d option values" in NodeJS like shown below (I was setting it blank earlier):
var data = '{"connection-mode":"SSH","host-name":"thisLinuxServer.fdqn","OS-type":"linux","user-name":"enter_your_userID"}'
Now, both options and data will be sent to httpHelp(options,data) and it'll work.
The above is just an dummy data value. Usually I'd pass valid host-name in the JSON object.
Never thought that not setting the data variable would result in this syntax error.
Other thing, I was missing BIG time was the value for 'Content-Type' inside headers variable. That was the reason, the error message was having "<" bracket in it (as by default it looks for a <body>...</body> section containing some XML format but in our case, the HTML response contained or was getting returned in JSON { ... } format way for the body section. That's why it was showing an error for unexpected token <.
As per the API doc, it should have been "JSON" type, thus changed that line to
'Content-Type': 'application/json',
I used Parse's CLI with the new Heroku integration to create the scaffold NodeJS project (parse new).
The example cloud function it gives you is:
// Hello
Parse.Cloud.define('hello', function(request, response) {
response.success('Hello world! ' + (request.params.a + request.params.b));
});
I can hit this route with the following CURL command and everything works fine:
curl -X POST \
-H "X-Parse-Application-Id: b8qPYS4SLSz0WoSWXlWeQosmF2jJPUPydetg3esR" \
-H "X-Parse-REST-API-Key: TOJLbfbNXSQcBdDVnU0MnKVu7SyamQvZmorHL5iD" \
-H "Content-Type: application/json" \
-d '{"a": "Adventurous ", "b": "Parser"}' \
https://api.parse.com/1/functions/hello
But then I added a new Class to my Parse Data, inserted a row, and tried to query & return the results. I keep getting {"code":143,"error":"Invalid webhook response status: 500 Internal Server Error"} as the response.
I'm fairly certain it is not my code that is the problem and am guessing there is some configuration step or something I'm missing.
Here is my modified Parse function:
// Hello
Parse.Cloud.define('hello', function(request, response) {
var query = Parse.Query("Favorites");
query.find({ useMasterKey: true }).then(
function(results) {
response.success('win');
}, function() {
response.error('fail');
});
});
And a picture of my Parse Class with the inserted row:
I have Googled the error and can't find any good answers only poorly worded questions. I'm completely at a loss here. Thanks in advance for your help.
Looks like Parse is wrong initialised on register-webhooks.js post deploy script:
Parse.initialize(process.env.PARSE_APP_ID, "unused", process.env.PARSE_MASTER_KEY);
And without second parameter (JavaScript Key) you can't execute any Parse.Query from cloud functions.
So my solution is:
Add new PARSE_JS_KEY to Heroku Config Variables (value is JavaScript Key from Parse->Settings->Keys)
In server.js file add line:
Parse.initialize(process.env.PARSE_APP_ID, process.env.PARSE_JS_KEY, process.env.PARSE_MASTER_KEY);
before require('./cloud/main.js');
PS: Place process.env.PARSE_JS_KEY directly in register-webhooks.js initializer does not work.
I have been trying to make this post request to the github api for the last couple of days, but unfortunately the response is coming back as "bad message"
here is the piece of code we are sending in the post request using https request in node -
This is the post data
var issueData = JSON.stringify({
"title":title,
"body":comment
});
This is the options file
var options = {
host: 'api.github.com',
path: '/repos/sohilpandya/katasohil/issues?access_token='+sessions.token,
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Firefox/24.0',
},
method: 'POST'
};
This is the https request
var requestaddIssue = https.request(options, function(responseFromIssues){
responseFromIssues.setEncoding('utf8');
responseFromIssues.on('data', function(chunk){
console.log('>>>>chunk>>>>>',chunk);
issueBody += chunk;
});
responseFromIssues.on('end',function(issueBody){
console.log(issueBody);
});
});
requestaddIssue.write(issueData);
requestaddIssue.end();
I have tried another approach where the authentication token for the user is in the header as
'Authentication': 'OAuth '+ sessions.token (where we are storing token inside sessions)
But the chunk response always seems to come back with the following in the console log.
{
"message": "Not Found",
"documentation_url": "https://developer.github.com/v3/issues/#create-an-issue"
}
I have tried the same in apigee and it seems to work ok and returns to correct response. Hoping someone can find the minor error in the code above that is causing this bad message error.
Except the issueBody variable is not defined in the snippets you posted, the code is correct. I tried it using a personal access token.
The error you get appears because you need to add a scope with power to open issues.
I tried the repo and public_repo scopes and they are both working. Note that repo has access to private repositories. Here you can see the list of scopes.
If you're using OAuth, then you you should have an url looking like this:
https://github.com/login/oauth/authorize?client_id=<client-id>&scope=public_repo&redirect_uri=<redirect-uri>