I am using a GET request to fetch what I expect to be an XML document from an endpoint. The response has the following structure:
' <itunes:explicit>clean</itunes:explicit>\n' +
' <itunes:episode>11</itunes:episode>\n' +
' <itunes:episodeType>full</itunes:episodeType>\n' +
(This is from a console log in a Node.js function).
I haven't encountered a response like this before and am having trouble doing anything useful with it. I've tried:
Changing the response type and encoding of my GET function
Parsing the response with an XML parser - this throws an error
Removing the newline and + characters manually with regex (I'd like to avoid this, but it doesn't seem to work anyway)
It's worth saying that the response looks as you'd expect in a browser window:
Am I missing something fundamental about how this data is encoded / structured and what is the best way to turn it into something I can work with?
Rookie error. In case anyone else stumbles across this; I expected the response from my Axios GET request to be the xml. The response was actually in a data property in the response:
const response = await axios.get(url);
const myXML = response.data;
Related
I am using mockoon for API simulation. I created 2 routes there with method GET and its body contains(responds with) JSON object. I noticed that my express app is not able to parse one of the routes. But the route that has JSON object in body which contains ARRAY is getting parsed. I tested both routes with Express(by console.log) and in chrome browser(I have JSON formatter extension) and it is behaving the same meaning response that does not contain ARRAY is not getting parsed but the response with array is getting parsed(behaving normally). Let me show the screenshots:
Express(by console.log):
With array:
Without array:
Chrome(JSON Formatter extension):
With array(extension is able to parse):
Without array(extension is not able to parse):
I tried adding Header(Content-Type: application/json) to the route in mockoon. But still, I am not aware of what is going on here. Someone please explain
The express code:
const iabs_client = await axios.get(
"http://localhost:3001/iabs-client
);
Here is the route created in Mockoon(without array inside JSON):
P.S mockoon is a program that creates endpoints in localhost, useful for API simulation when developing front-end without having backend yet
The trailing comma after "something" is not valid JSON. Edit your Mockoon body to remove the comma and it should work.
i searched the internet (and stackoverflow :D) to find an answer for the following question - and found none that i understood.
background:
we want to use a python script to connect our companies CMDB with our AWX/Ansible infrastructure.
the CMDB has a REST API which supports a (halfway) proper export.
i'm currently stuck with the implementation of the correct API call.
i can call the API itself and authenticate, but i can't call the proper filter to get the results i need.
the filter is realized by having the following string within the URL (more in the attached code example)
Label LIKE "host*"
it seems that python has a problem with the *.
error message:
InvalidURL(f"URL can't contain control characters. {url!r} "
I found some bug reports that there is an issue within some python versions, but i'm way to new to properly understand if this affects me here :D
used python version 3.7.4
PS: let's see if i can get the markup right :D
i switched the called URL to determine where exactly the problem occurs.
it only occurs when i use the SQL like filter part.
this part is essential since i just want our "hosts" to be returned and not the whole CMDB itself.
#import the required classes and such
from http.client import HTTPConnection
import json
#create a HTTP connection client
client = HTTPConnection("cmdb.example.company")
#basic auth and some header details
headers = {'Content-Type': 'application/json',
'Authorization' : 'Basic my-auth-token'}
#working API call
client.request('GET', '/cmdb/rest/hosts?attributes=Label,Keywords,Tag,Description&limit=10', headers=headers)
#broken API call returns - InvalidURL(f"URL can't contain control characters. {url!r} "
client.request('GET', '/cmdb/rest/hosts?filter=Label LIKE "host*"&attributes=Label,Keywords,Tag,Description&limit=10', headers=headers)
#check and convert the response into a readable (JSON) format
response = client.getresponse()
data = response.read()
#debugging print - show that the returned data is bytes?!
print(data)
#convert the returned data into json
my_json = data.decode('utf8').replace("'", '"')
data = json.loads(my_json)
#only return the data part from the JSON and ignore the meta-overhead
text = json.dumps(data["data"], sort_keys=True, indent=4)
print(text)
so, i want to know how to properly call the API with the described filter and resolve the displayed error.
can you give me an example i can try or pin-point a beginners mistake i made?
am i affected by the mentioned python bug regarding the URL call with * in it?
thanks for helping me out :)
soooo i found my beginners mistake myself:
i used the URL from my browser - and my browser automaticly encodes the special characters within the URL.
i found the following piece of code within Python3 URL encoding guide and modified the string to fit my needs :)
import urllib.parse
query = ' "host*"'
urllib.parse.quote(query)
'%20%22host%2A%22'
Result: '%20%22host%2A%22'
%20 = " "
%22 = " " "
%2A = "*"
so the final code looks somewhat like this:
#broken API call returns - InvalidURL(f"URL can't contain control characters. {url!r} "
client.request('GET', '/cmdb/rest/hosts?filter=Label LIKE "host*"&attributes=Label,Keywords,Tag,Description&limit=10', headers=headers)
filter=Label LIKE "host*"
#fixed API call
client.request('GET', '/cmdb/rest/hosts?filter=Label%20LIKE%20%22host%2A%22&attributes=Label,Keywords,Tag,Description&limit=10', headers=headers)
filter=Label%20LIKE%20%22host%2A%22
I'm having an issue with a NodeJS REST api created using express.
I have two calls, a get and a post set up like this:
router.get('/:id', (request, response) => {
console.log(request.params.id);
});
router.post('/:id', (request, response) => {
console.log(request.params.id);
});
now, I want the ID to be able to contain special characters (UTF8).
The problem is, when I use postman to test the requests, it looks like they are encoded very differently:
GET http://localhost:3000/api/â outputs â
POST http://localhost:3000/api/â outputs â
Does anyone have any idea what I am missing here?
I must mention that the post call also contains a file upload so the content type will be multipart/form-data
You should encode your URL on the client and decode it on the server. See the following articles:
What is the proper way to URL encode Unicode characters?
Can urls have UTF-8 characters?
Which characters make a URL invalid?
For JavaScript, encodeURI may come in handy.
It looks like postman does UTF-8 encoding but NOT proper url encoding. Consequently, what you type in the request url box translates to something different than what would happen if you typed that url in a browser.
I'm requesting: GET localhost/ä but it encodes it on the wire as localhost/ä
(This is now an invalid URL because it contains non ascii characters)
But when I type localhost/ä in to google chrome, it correctly encodes the request as localhost/%C3%A4
So you could try manually url encoding your request to http://localhost:3000/api/%C3%A2
In my opinion this is a bug (perhaps a regression). I am using the latest version of PostMan v7.11.0 on MacOS.
Does anyone have any idea what I am missing here?
yeah, it doesn't output â, it outputs â, but whatever you're checking the result with, think you're reading something else (iso-8859-1 maybe?), not UTF-8, and renders it as â
Most likely, you're viewing the result in a web browser, and the web server is sending the wrong Content-Type header. try doing header("Content-type: text/plain;charset=utf-8"); or header("Content-type: text/html;charset=utf-8"); , then your browser should render your â correctly.
I wrote a simple API which will return request.query as a response.
The behavior is little different than what I am expecting.
redirectto -- I am getting the only name as part of response redirectto param.
id -- I am getting an array in response.
Why is this behaviour?
Query parameters that contain reserved characters should be URL encoded or they will fail to parse correctly.
The properly encoded URL should look something like this:
http://localhost:8082/redirect?requesttype=click&id=79992&redirectto=http%3A%2F%2Flocalhost%3A8081%2Fredirect%3Fname%3Djohn%26id%3D123
I have a server running on nodejs, and I have the following piece of code to manage a post request -
form.on('file', function (field, file) {
var RecordingInfo = JSON.parse(file.name);
...
when I tried to upload a file I got the following exception:
undefined:1
"}
SyntaxError: Unexpected end of input
at Object.parse (native)
at IncomingForm.<anonymous> (.../root.js:31:34)
...
searching around the web, I fond that this exception is caused because the data comes in bits, and the event is fired after the first bit arrives, and I don't have all the data. OK. The thing is, after a little testing I fond that from chrome I can upload large files (tried a 1.75gb file) without any problem, while firefox crashes the server with a 6kb file.
My question is - why are they different?
A sample capture can be downloaded form here. The first post is from chrome, the second from firefox.
The complete file.name string before uploading is:
// chrome
"{"subject":"flksajfd","lecturer":"אבישי וינר","path":"/גמרא","fileType":".png"}"
// firefox
"{"subject":"fdsa","lecturer":"אלקס ציקין","path":"/גמרא","fileType":".jpg"}"
(The data submitted is not the same, but I don't think it matters)
Chrome is encoding double-quotes in the JSON-encoded "filename" as %22 while Firefox is encoding them as \".
Your file upload parsing library, Formidable, explicitly truncates the filename from the last \ character. It expects double-quotes to be encoded as %22 although RFC 2616 allows backslash-escaped quotes like Firefox has implemented. You can consider this a bug in Formidable. The result is that the following JSON string:
'{"subject":"fdsa",...,"fileType":".jpg"}'
...is encoded as follows:
'{%22subject%22:%22fdsa",...,%22fileType%22:%22.jpg%22}' // Chrome
'{\"subject\":\"fdsa\",...\"fileType\":\".jpg\"}' // Firefox
...and then decoded by Formidable:
'{"subject":"fdsa",..."fileType":".jpg"}' // Chrome
'"}' // Firefox
To fix the issue you have a few choices:
Raise the issue with Formidable to correctly handle backslash-escaped quoted-value strings (or fix it yourself and submit a pull request).
Send the JSON payload in a separate part of the FormData object, e.g. using a Blob.
Transliterate all double-quote characters in your JSON-format filename to a 'safe' character that will not appear elsewhere in the string (I chose ^ as an example); replace the quote client-side and reinstate it server-side as follows.
Client:
var formData = new FormData();
formData.append('file', $scope.recording, JSON.stringify(RecordingInfo).replace(/"/g, '^');
Server
form.on('file', function (field, file) {
var RecordingInfo = JSON.parse(file.name.replace(/\^/g, '"');