I have come into a trouble that when using execSync in node.js, it's not working as directly type the command in the shell.
Here is my issue:
I use a curl to request for some data from a server, and I need to do that with a cookie because there is a login requirement.
It's easy to handle the login process and get the cookie, but it's weird that using the cookie with a curl in node.js would cause the server an "internal error". And since I don't have the permission to change the server-code, I'm looking for help about the difference of calling curl in Node.js and directly use curl.
Here is the code:
var command = 'curl --cookie cookie.txt ' + getURL();
console.log(command);
// output: curl --cookie cookie.txt http://example.com/getdata
var result = child_process.execSync(command).toString();
// will cause an internal error and the "result" is an error-reporting page.
Directly calling this in the shell:
curl --cookie cookie.txt http://example.com/getdata
Everything works, I got the data I need.
I tried to find some plots, for instance, changing the code to:
var command = 'curl --cookie cookie-bad.txt ' + getURL();
I put some wrong cookie in the cookie-bad.txt, I will get a "you are not log in" result.
So there must be something wrong with:
sending a cookie to the server to request some data with curl running inside a nodejs script with execSync.
Is there any way I can improve the code or something?
What is your Node.js version? I don't have any problem with 10.16.0.
Related
I have a curl command. I want to execute in in Python and fetch the response to pass it through other code.
curl https://api.box.com/oauth2/token -d 'grant_type=refresh_token' -d 'refresh_token=Ew38UXVKS3kc0axFt6MdDklUnoHpipxTzDWBmKlXAG9ImvGafbbaUhQndv89e677' -d 'client_id=3qkm5h4wh765b3khiws0z8hdpkc56jhs' -d 'client_secret=h9AeXzZL3KATJuaHmimFFBRBDZQrp9tr' -X POST
How can I execute the script in Python and get the response and pass it through other code?
When I am executing the curl script in CMD, I am getting this response:
{"access_token":"uz843jpIiEWnu0CcuT9as2XbA3UEQTR67","expires_in":4261,"restricted_to":[],"refresh_token":"GsDaP6VyUpHN8vDHbz9ktAjLfMLN0dFL6PMIK4fmDH8eKRqR360vDhQTBhIMZxy67","token_type":"bearer"}
From the above response I need to take the access_token value.
Like avloss said - try out requests.
Another great resource is the application Postman
It will let you try out any http calls you'd like, and then can also translate that into/out of curl and/or python requests code for you (and a bunch of other languages).
I find it really useful when trying to figure how to use requests for anything more than simple http calls.
you should use requests library (pip install requests)
import requests
url = 'https://api.box.com/oauth2/token'
data = {
'grant_type':'refresh_token',
'refresh_token':'***',
'client_id':'***',
'client_secret':'***'
}
response = requests.post(url, data).json()
print(response)
this might not be an issue per say but here is my problem : I'm using childprocess.execSync to execute curl request on pages I need the html content of. It works perfectly fine for valid pages but if the curl request to a page encounters pretty much any error code such as 404 or 403, then the result of the execSync is empty and I have no way to know what error code the curl encountered.
Is there any way to know the curl error code that happens during the childprocess.execSync ?
node.js version : 8.16.2
Well, I found a way, look like I just had to pass "-i" with my curl request to return the response headers too and not just the body, now I can just parse that and see the error code I got.
I have a python script which does a GET request to GitLab and stores the data from the response in an excel file using tablib library.
This script works fine in Windows when I execute it using python3.
I have tried to execute the same script in the Windows Subsystem for Linux (WSL) I have enabled and the script fails.
The output when I execute with python3 script.py in WSL is the following:
RESPONSE {"message":"404 Project Not Found"}
When I execute from Windows using python .\gitlab.py where python is python3:
RESPONSE [{"id":567,"iid":22}, {"id":10,"iid":3}]
I think the problem could be related to the GET api call I am doing because in WSL it returns Project Not Found.
I executed that request using curl in WSL to see if the unix in general has this issue, but I get back the expected response instead of the not found response. This was the request:
curl -X GET 'https://URL/api/v4/projects/server%2Fproducts%2FPROJECT/issues?per_page=100' -H 'Content-Type: application/json' -H 'PRIVATE-TOKEN: TOKEN' --insecure
Why is python failing in unix using Python if unix is able to execute the get request using curl? Should I enable/disable something in the request perhaps?
This is the request I am doing in my python script:
def get_items():
url = "https://URL/api/v4/projects/server%2Fproducts%2FPROJECT/issues"
payload = {}
querystring = {"state": "closed", "per_page": "100"}
headers = {
'Content-Type': "application/json",
'PRIVATE-TOKEN': os.environ.get("GITLAB_KEY") # enviromental variable added in windows
}
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
response = requests.request(
"GET", url, headers=headers, data=payload, params=querystring, verify=False)
print("RESPONSE " + response.text)
return json.loads(response.text)
UPDATE:
I have tried using the project id as well instead of the path but it didn't work
REF: https://docs.gitlab.com/ee/api/projects.html#get-single-project
GET /projects/:id
Change this:
url = "https://URL/api/v4/projects/server%2Fproducts%2FPROJECT/issues"
To
projectId = 1234 # or whatever your project id is ... Project Page, Settings -> General
url = "https://URL/api/v4/projects/" + projectId + "/issues"
Based on an answer I got in the post I did in Reddit, I found the problem.
In the python script, I am using an environmental variable which is not accessible in that way ( os.environ.get("GITLAB_KEY") ) from the WSL.
For now, I have replaced it with the hard-coded value just to check that this was really the issue. The script now works as expected.
I will find a way to access the env var again now that I know what the problem was.
I am trying to convert speech to text using Nuance so i am trying to send this request
curl "https://dictation.nuancemobility.net:443/NMDPAsrCmdServlet/dictation?appId=[INSERT YOUR APP ID]&appKey=[INSERT YOUR 128-BYTE STRING APP KEY]&id=C4461956B60B" -H "Content-Type: audio/x-wav;codec=pcm;bit=16;rate=16000" -H "Accept-Language: ENUS" -H "Transfer-Encoding: chunked" -H "Accept: application/xml" -H "Accept-Topic: Dictation" -k --data-binary #audio_16k16bit.pcm
need to upload audio file (.pcm) format.
I am using okhttp3 library following is the builder
RequestBody requestBody = new MultipartBody.Builder().setType(MultipartBody.FORM)
.addFormDataPart("file", "audio_16k16bit.pcm", RequestBody.create(MEDIA_TYPE_PNG, "audio_16k16bit"))
.build();
httpBuider.addQueryParameter("appId", "NMDPTRIAL_XXXXXXX_XXX_com20161122071457").addQueryParameter("appKey", "fadaed7b801e10d84272c0a75317d8cee13ab86ae902ab322cd6e1219fcbe79aa5d41526f225fe3497bfdbead6b4b9b7ee7122d773cd0a9fa3ebc042b7a7dc5c");
Request request = new Request.Builder().addHeader("Content-Type","audio/x-wav;codec=pcm;bit=16;rate=16000").addHeader("Accept-Language","eng-GBR").addHeader("Transfer-Encoding","chunked").addHeader("Accept","application/xml").addHeader("Accept-Topic","Dictation").post(requestBody).url(httpBuider.build()).build();
I am getting following log
HTTP ERROR 500
Problem accessing /NMDPAsrCmdServlet/dictation. Reason:
Server Error
Missing anything?
I have experienced the same just now using Jersey. Problem for me was, that Jersey was overwriting the content type header (I tried with Application_Octet_Stream).
Here is the request with which I finally got it to work:
Response myResponse = target.
request().
accept(MediaType.TEXT_PLAIN_TYPE).
header("Accept-Language","DEDE").
header("Accept-Topic", "Dictation").
header("Transfer-Encoding","chunked").
post(Entity.entity(speechStream, "audio/x-wav;codec=pcm;bit=16;rate=16000"));
I suggest using something like fiddler to find out what is really posted and comparing that to the curl post. This is how I finally found out what was wrong with my request.
Remove the header "Transfer-Encoding","chunked". My error was fixed by removing it.
My piston application works correctly when I run it locally with python manage.py runserver command but returns
urllib2.HTTPError: HTTP Error 403:
FORBIDDEN
under apache. How can I debug django-piston application?
I usually debug Piston apps by:
Setting my handlers to use Basic Authentication, even if I'm normally using something else.
Use curl to make requests
Use pdb (or ipdb) to set a breakpoint in my handler if desired.
You can conditionally change to BasicAuthentication like this:
auth = {'authentication': WhateverYouAreUsingForAuthentication(realm="YourSite")}
if getattr(settings, "API_DEBUG", None):
from piston.authentication import HttpBasicAuthentication
auth = {'authentication': HttpBasicAuthentication(realm="Spling")}
some_handler = Resource(SomeHandler, **auth)
To pass a username and password using curl, use the -u option:
curl -u username:password http://localhost:8000/api/some/endpoint/
So in your local settings module, just set API_DEBUG=True whenever you want to use basic auth.