I am trying to get data from the Bing search API, and since the existing libraries seem to be based on old discontinued APIs I though I'd try myself using the request library, which appears to be the most common library for this.
My code looks like
var SKEY = "myKey...." ,
ServiceRootURL = 'https://api.datamarket.azure.com/Bing/Search/v1/Composite';
function getBingData(query, top, skip, cb) {
var params = {
Sources: "'web'",
Query: "'"+query+"'",
'$format': "JSON",
'$top': top, '$skip': skip
},
req = request.get(ServiceRootURL).auth(SKEY, SKEY, false).qs(params);
request(req, cb)
}
getBingData("bookline.hu", 50, 0, someCallbackWhichParsesTheBody)
Bing returns some JSON and I can work with it sometimes but if the response body contains a large amount of non ASCII characters JSON.parse complains that the string is malformed. I tried switching to an ATOM content type, but there was no difference, the xml was invalid. Inspecting the response body as available in the request() callback actually shows bad code.
So I tried the same request with some python code, and that appears to work fine all the time. For reference:
r = requests.get(
'https://api.datamarket.azure.com/Bing/Search/v1/Composite?Sources=%27web%27&Query=%27sexy%20cosplay%20girls%27&$format=json',
auth=HTTPBasicAuth(SKEY,SKEY))
stuffWithResponse(r.json())
I am unable to reproduce the problem with smaller responses (e.g. limiting the number of results) and unable to identify a single result which causes the issue (by stepping up the offset).
My impression is that the response gets read in chunks, transcoded somehow and reassembled back in a bad way, which means the json/atom data becomes invalid if some multibyte character gets split, which happens on larger responses but not small ones.
Being new to node, I am not sure if there is something I should be doing (setting the encoding somewhere? Bing returns UTF-8, so this doesn't seem needed).
Anyone has any idea of what is going on?
FWIW, I'm on OSX 10.8, node is v0.8.20 installed via macports, request is v2.14.0 installed via npm.
i'm not sure about the request library but the default nodejs one works well for me. It also seems a lot easier to read than your library and does indeed come back in chunks.
http://nodejs.org/api/http.html#http_http_request_options_callback
or for https (like your req) http://nodejs.org/api/https.html#https_https_request_options_callback (the same really though)
For the options a little tip: use url parse
var url = require('url');
var params = '{}'
var dataURL = url.parse(ServiceRootURL);
var post_options = {
hostname: dataURL.hostname,
port: dataURL.port || 80,
path: dataURL.path,
method: 'GET',
headers: {
'Content-Type': 'application/json; charset=utf-8',
'Content-Length': params.length
}
};
obviously params needs to be the data you want to send
I think your request authentication is incorrect. Authentication has to be provided before request.get.
See the documentation for request HTTP authentication. qs is an object that has to be passed to request options just like url and auth.
Also you are using same req for second request. You should know that request.get returns a stream for the GET of url given. Your next request using req will go wrong.
If you only need HTTPBasicAuth, this should also work
//remove req = request.get and subsequent request
request.get('http://some.server.com/', {
'auth': {
'user': 'username',
'pass': 'password',
'sendImmediately': false
}
},function (error, response, body) {
});
The callback argument gets 3 arguments. The first is an error when applicable (usually from the http.Client option not the http.ClientRequest object). The second is an http.ClientResponse object. The third is the response body String or Buffer.
The second object is the response stream. To use it you must use events 'data', 'end', 'error' and 'close'.
Be sure to use the arguments correctly.
You have to pass the option {json:true} to enable json parsing of the response
Related
I created a NodeJS application which should get some data from an external API-Server. That server provides its data only as 'Content-Type: text/plain;charset=ISO-8859-1'. I have got that information through the Header-Data of the server.
Now the problem for me is that special characters like 'ä', 'ö' or 'ü' are shown as �.
I tried to convert them with Iconv to UTF-8, but then I got these things '�'...
My question is, what am I doing wrong?
For testing I use Postman. These are the steps I do to test everything:
Use Postman to trigger my NodeJS application
The App requests data from the API-Server
API-Server sends Data to NodeJS App
My App prints out the raw response-data of the API, which already has those strange characters �
The App then tries to convert them with Iconv to UTF-8, where it shows me now this '�' characters
Another strange thing:
When I connect Postman directly to the API-Server, the special characters get shown as they have too without problems. Therefore i guess my application causes the problem but I cannot see where or why...
// Javascript Code:
try {
const response = await axios.get(
URL
{
params: params,
headers: headers
}
);
var iconv = new Iconv('ISO-8859-1', 'UTF-8');
var converted = await iconv.convert(response.data);
return converted.toString('UTF-8');
} catch (error) {
throw new Error(error);
}
So after some deeper research I came up with the solution to my problem.
The cause of all trouble seems to lie within the post-process of axios or something similar. It is the step close after data is received and convertet to text and shortly before the response is generated for my nodejs-application.
What I did was to define the "responseType" of the GET-method of axios as an "ArrayBuffer". Therefore an adjustment in axios was necessary like so:
var resArBuffer = await axios.get(
URL,
{
responseType: 'arraybuffer',
params: params,
headers: headers
}
);
Since JavaScript is awesome, the ArrayBuffer provides a toString() method itself to convert the data from ArrayBuffer to String by own definitions:
var response = resArBuffer.data.toString("latin1");
Another thing worth mentioning is the fact that I used "latin1" instead of "ISO-8859-1". Don't ask me why, some sources even recommended to use "cp1252" instead, but "latin1" workend for me here.
Unfortunately that was not enough yet since I needed the text in UTF-8 format. Using "toString('utf-8')" itself was the wrong way too since it would still print the "�"-Symbols. The workaround was simple. I used "Buffer.from(...)" to convert the "latin1" defined text into a "utf-8" text:
var text = Buffer.from(response, 'utf-8').toString();
Now I get the desired UTF-8 converted text I needed. I hope this thread helps anyone else outhere since thse informations hwere spread in many different threads for me.
I'm still new enough with Node that HTTP requests trip me up. I have checked all the answers to similar questions but none seem to address my issue.
I have been dealt a hand in the Wild of having to go after JSON files in an API. I then parse those JSON files to separate them out into rows that populate a SQL database. The API has one JSON file with an ID of 'keys.json' that looks like this:
{
"keys":["5sM5YLnnNMN_1540338527220.json","5sM5YLnnNMN_1540389571029.json","6tN6ZMooONO_1540389269289.json"]
}
Each array element in the keys property holds the value of one of the JSON data files in the API.
I am having problems getting either type of file returned to me, but I figure if I can learn what is wrong with the way I am trying to get 'keys.json', I can leverage that knowledge to get the individual JSON data files represented in the keys array.
I am using the npm modules 'request' and 'request-promise-native' as follows:
const request = require('request');
const rp = require('request-promise-native');
My URL is constructed with the following elements, as follows (I have used the ... to keep my client anonymous, but other than that it is a direct copy:
let baseURL = 'http://localhost:3000/Users/doug5solas/sandbox/.../server/.quizzes/'; // this is the development value only
let keysID = 'keys.json';
Clearly the localhost aspect will have to go away when we deploy but I am just testing now.
Here is my HTTP call:
let options = {
method: 'GET',
uri: baseURL + keysID,
headers: {
'User-Agent': 'Request-Promise'
},
json: true // Automatically parses the JSON string in the response
};
rp(options)
.then(function (res) {
jsonKeysList = res.keys;
console.log('Fetched', jsonKeysList);
})
.catch(function (err) {
// API call failed
let errMessage = err.options.uri + ' ' + err.statusCode + ' Not Found';
console.log(errMessage);
return errMessage;
});
Here is my console output:
http://localhost:3000/Users/doug5solas/sandbox/.../server/.quizzes/keys.json 404 Not Found
It is clear to me that the .catch() clause is being taken and not the .then() clause. But I do not know why that is because the data is there at that spot. I know it is because I placed it there manually.
Thanks to #Kevin B for the tip regarding serving of static files. I revamped the logic using express.static and served the file using that capability and everything worked as expected.
I have a simple code that's supposed to download images from slack messages.
var url = message.file.private_url;
var destination_path = './tmp/uploaded';
var opts = {
method: 'GET',
url: url,
headers: {
Authorization: 'Bearer ' + process.env.botToken,
}
};
request(opts, function(err, res, body) {
console.log('FILE RETRIEVE STATUS',res.statusCode);
}).pipe(fs.createWriteStream(destination_path));
The code worked fine for a while, but now I'm getting this error:
An error occured in the receive middleware: TypeError: Cannot read property 'private_url' of undefined
Any help would be appreciated!
Are you using the events API?
Several changes have been made to the API recently (both Events and Web APIs). See here: https://api.slack.com/changelog/2018-05-file-threads-soon-tread
If you describe the API you're using, I might be able to provide more specific help but I suspect the issue (as described in the link above) is that the file attribute attached to messages has been replaced with a new files field (an array). The files in the array are also in a different format.
Check the JSON payload. It probably contains a files array.
I have a file in memory (buffer) - there is no file on the file system.
I want to send that buffer to another server that talks HTTP.
For example, some API A creates a file in memory, SignServer manipulates such files, and responds with a new buffer. My API takes the file from A and feeds it to SignServer.
I tried sending the file to SignServer in multiple ways, but it keeps responding with status 400 (missing field 'data' in request).
What I tried:
var http = require('http');
var querystring = require('querystring');
var data = querystring.stringify({
workerName: 'PDFSigner',
data: file_buffer
});
var request = new http.ClientRequest({
hostname: 'localhost',
port: 8080,
path: '/signserver/process',
method: 'GET',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
// I also tried 'multipart/form-data'
'Content-Length': Buffer.byteLength(data)
}
});
request.end(data);
I tried printing data, and it showed:
workerName=PDFSigner&data=
Which is bad because data wasn't set to file_buffer.
I tried printing file_buffer, and it does have content (not null, not undefined, actually has bytes inside).
So stringifying the buffer gave an empty string.
I tried doing the same thing with the request module and it didn't work either.
Note that SignServer isn't written in Node nor JavaScript. It's a Java application, so it probably doesn't work with json (which is why I'm trying to do it with querystring). Yes, I tried sending json.
The reason why data is set to an empty string is described in this issue and the solution is given in this issue.
escape and stringify the buffer like so:
var data = querystring.stringify({
workerName: 'PDFSigner',
data: escape(file_buffer).toString('binary')
});
As #robertklep mentioned, your other problem is that you can't send a big file using application/x-www-form-urlencoded. You'd need to do it with multipart/form-data.
In a browser, if I send a GET request, the request will send the cookie in the meanwhile. Now I want to simulate a GET request from Node, then how to write the code?
Using the marvelous request library cookies are enabled by default. You can send your own like so (taken from the Github page):
var j = request.jar()
var cookie = request.cookie('your_cookie_here')
j.add(cookie)
request({url: 'http://www.google.com', jar: j}, function () {
request('http://images.google.com')
})
If you want to do it with the native http:request() method, you need to set the appropriate Set-Cookie headers (see an HTTP reference for what they should look like) in the headers member of the options argument; there are no specific methods in the native code for dealing with cookies. Refer to the source code in Mikeal's request library and or the cookieParser code in connect if you need concrete examples.
But Femi is almost certainly right: dealing with cookies is full of rather nitpicky details and you're almost always going to be better off using code that's already been written and, more importantly, tested. If you try to reinvent this particular wheel, you're likely to come up with code that seems to work most of the time, but occasionally and unpredicatably fails mysteriously.
var jar = request.jar();
const jwtSecret = fs.readFileSync(`${__dirname}/.ssh/id_rsa`, 'utf8');
const token = jwt.sign(jwtPayload, jwtSecret, settings);
jar.setCookie(`any-name=${token}`, 'http://localhost:12345/');
const options = {
method: 'GET',
url: 'http://localhost:12345',
jar,
json: true
};
request(options, handleResponse);