I am trying to write a NodeJS app that grabs a image from private URL using Axios and passes it to the Azure Face SDK (documentation) to detect faces in the image and get attributes of those faces - in this case, emotions and head pose.
I have gotten a modified version of the quickstart example code here working, which makes a call to the detectWithUrl() method. However, the image that I have in my code is a ArrayBuffer, so I thought I would try calling detectWithStream() instead. The documentation for this method says it needs to be passed something of type msRest.HttpRequestBody - I found some documentation for this type, which looks like it wants to be a Blob, string, ArrayBuffer, or ArrayBufferView. The problem is, I don't really understand what those are or how I might get from a arraybuffer image to an HttpRequestBody of that type. I have worked with HTTP requests before, but I don't quite understand why one is being passed to this method, or how to make it.
const rqs = {
responseType: 'arraybuffer',
url: `${fileUrl}`,
method: 'GET',
headers: {
'Content-Type': `image/jpeg`,
'Cache-Control': 'no-cache'
},
httpsAgent: agent
};
axios.request(rqs)
.then((ret) => {
// ret.data is a ArrayBuffer type;
let detected_faces = client.face.detectWithStream(ret.data, {
returnFaceAttributes: ["Accessories", "Age", "Blur", "Emotion", "Exposure", "FacialHair", "Glasses", "Hair", "HeadPose", "Makeup", "Noise", "Occlusion", "Smile", "QualityForRecognition"],
detectionModel: "detection_01",
recognitionModel: "recognition_03"
})
.then(ok => {
console.log(ok);
console.log("face(s) detected from image.");
console.log("Face attributes for face(s):");
})
.catch(err => {
console.error(err.message);
});
console.log('ok');
resolve(ret.data);
})
.catch((err) => {
console.error('nok');
console.error(err.message);
reject(false)
});
And I had receive a message error Only absolute URLs are supported
Only absolute URLs are supported -
a. it may be due to passing relative URL to Axios in line 3, try checking the fileUrl.
b. check the face SDK is provided with the proper endpoint
relative URL - "./a/b"
absolute URL - "https://www.x.com/a/b"
how I might get from an array buffer image?
Array buffer is not needed. Looking at face SDK code Ref (hope this is right sdk), we just need streams.
we can get this by changing the responseType value to stream
responseType: 'arraybuffer', // current
responseType: 'stream', // updated - we can update it in rqs and reponse will be stream and you can pass res.data to detectWithStream function
-------------------------------------Hope the code works----------------------------------------
Recommendation:
You are creating a callback hell by using nested .then chains what is call back hell.
Try using async/await syntax to achieve the same functionality in a simplified way - Ref.
Regards,
Muhamed
Related
I am getting a 400 error (bad request) from this POST:
siteData.myPost = async function (url, data) {
let options = {
method: 'POST',
headers: {
'Content-Type': 'application/json;charset=utf-8'
},
body: JSON.stringify(data)
}
try {
const response = await fetch(quilkinUrlBase() + url,options);
return response.json();
}
catch (error) {
qPopup.Alert("Error with web request: " + error);
}
};
(the fetch address computes to "http://localhost:1337/UpdateSiteVisits").
This can't, of course, convert 'response' to json because the response is a DOC containing the 400 error message.
in server.js I have the following callback definition for the URL specified in the fetch:
app.post("/UpdateSiteVisits", sites.updateVisits);
in the debugger, the code never reaches updateVisits().
I have a very similar construct with a GET:
app.get("/GetSitesForType/:type", sites.findByType);
which works fine, so I assume the problem is with the 'options' object (which the GET doesn't use). In the POST case the data is just an integer ID, and if I change the data format to plain text (instead of json), the post works fine.
If leave the format as json, and send an object, rather than a simple integer, the post also works fine. So it looks like the system just doesn't like converting a single integer to json.
I needed to send the integer as json, so instead of using the method like this:
siteData.myPost("UpdateSiteVisits", siteID)
I needed to use
siteData.myPost("UpdateSiteVisits", { siteId: siteID })
[ later ]
Please ignore my question and answer, I was just getting mixed up with types. Such is the result from me converting from strongly-typed C# to untyped javascript!
I am trying to create a function which return the latest blocked domain from my pihole server.
I first created a JSON call, since the first call I needed was in JSON format, this is all working and I get the data needed.
However, the second function I need is to fetch plain text data and that one doesn't work, it simply returns an empty body [].
This is the function
socket.on("pihole_last", function() {
setInterval(function() {
let settings = {
method: "Get",
headers: {
"Accept": "text/html"
}
};
fetch('http://domain/admin/api.php?recentBlocked', settings)
.then(res => res.text())
.then((data) => {
console.log(data);
}).catch(error => {
return error;
});;
}, 1000)
});
The JSON function which works looks pretty much the same, the only real different is the header accept and the res.text() which should fetch the data in plain text?
The data returned from the URL is a plain text domain, no tags, no nothing.
According to this issue from the pi-hole GIT, you should provide some form of authentication. The question which you linked in your comment is 5 years old, at that time this was an unintended behaviour.
If I understand correctly the API description one way to authorize should be working with this url:
http://domain/admin/api.php?recentBlocked?auth=YOUR_TOKEN
The YOUR_TOKEN should be in:
Authorization & Token required (see WEBPASSWORD in /etc/pihole/setupVars.conf)
I have a client app in React, and a server in Node (with Express).
At server side, I have an endpoint like the following (is not the real endpoint, just an idea of what i'm doing):
function endpoint(req, res) {
res.writeHead(200, {
'Content-Type': 'text/plain',
'Transfer-Encoding': 'chunked'
});
for(x < 1000){
res.write(some_string + '\n');
wait(a_couple_of_seconds); // just to make process slower for testing purposes
}
res.end();
}
This is working perfect, i mean, when I call this endpoint, I receive the whole stream with all the 1.000 rows.
The thing is that I cannot manage to get this data by chunks (for each 'write' or a bunch of 'writes') in order to show that on the frontend as soon as i'm receiving them..(think of a table that shows the rows as soon as i get them from the endpoint call)
In the frontend I'm using Axios to call the API with the following code:
async function getDataFromStream(_data): Promise<any> {
const { data, headers } = await Axios({
url: `http://the.api.url/endpoint`,
method: 'GET',
responseType: 'stream',
timeout: 0,
});
// this next line doesn't work. it says that 'on' is not a function
data.on('data', chunk => console.log('chunk', chunk));
// data has actually the whole response data (all the rows)
return Promise.resolve();
}
The thing is that the Axios call returns the whole data object after the 'res.end()' on the server is called, but I need to get data as soon as the server will start sending the chunks with the rows (on each res.write or whenever the server thinks is ready to send some bunch of chunks).
I have also tried not to use an await and get the value of the promise at the 'then()' of the axios call but it is the same behavior, the 'data' value comes with all the 'writes' together once the server does the 'res.end()'
So, what I doing wrong here ? maybe this is not possible with Axios or Node and I should use something like websockets to solve it.
Any help will be very appreciate it because I read a lot but couldn't get a working solution yet.
For anyone interested in this, what I ended up doing is the following:
At the Client side, I used the Axios onDownloadProgress handler that allows handling of progress events for downloads.
So, I implemented something like this:
function getDataFromStream(_data): Promise<any> {
return Axios({
url: `http://the.api.url/endpoint`,
method: 'GET',
onDownloadProgress: progressEvent => {
const dataChunk = progressEvent.currentTarget.response;
// dataChunk contains the data that have been obtained so far (the whole data so far)..
// So here we do whatever we want with this partial data..
// In my case I'm storing that on a redux store that is used to
// render a table, so now, table rows are rendered as soon as
// they are obtained from the endpoint.
}
}).then(({ data }) => Promise.resolve(data));
}
I retrieve this URL from using some previous logic - https://bots.rocket.chat/file-upload/ZSzGtWdXyP8DZwrQ9/Audio%20record.mp3
Now I need to download it for further processing, but I tried using the download, axios.request, and other packages available. My best guess for them failing is that the actual URL of the audio is different then what I retrieved.
Something like (It changes after a while) -https://s3.amazonaws.com/uploads.use1.cloud.rocket.chat/KqCQiHfeFaKdSEGvy/uploads/GENERAL/XmYemgXfudSzBQstR/ZSzGtWdXyP8DZwrQ9?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAILPK6SHTK5RJZLHQ%2F20190518%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190518T115900Z&X-Amz-Expires=120&X-Amz-Signature=8d793d14fbb33d78b084a5ed153106ba19440feb7bf05525aeed530b5a0907f9&X-Amz-SignedHeaders=host
Also, I'm able to download the file using the same code when I provide it with the end URL (the longer one).
So need to figure out how to reach that URL in node.
Also to access the first URL you need to make an account on https://bots.rocket.chat
This is the code I have currently. The url passed to getFile function is the is a relative path which I append to the prefix url to form the above mentioned shorter URL, and call axios and this doesn't work.
But when I replace the url with a long url which is similar to the longer one posted above the code works just fine.
async function getFile(url) {
var fs = require('fs');
return await axios.request({
responseType: 'arraybuffer',
followAllRedirects: true,
url: `https://bots.rocket.chat${url}`,
method: 'get',
headers: {
'Content-Type': 'audio/mpeg',
},
}).then((result) => {
const outputFilename = 'file2.mp3';
fs.writeFileSync(outputFilename, result.data);
return outputFilename;
})
.catch(err => console.log(err));
;
}
I'm trying to send a multipart/form-data image to another server, using unirest. Their .attach() works with a fs.createReadStream(), however, I have not been able to convert the buffer to an image. The logical step is to convert the buffer to Uint8Array first, then creating a read stream from it. However, this throws up an error message saying that the array must not contain null values. Removing 0 entries from the array will almost certainly break the image.
Image is not null, has all the bytes, and even sending the image data as a giant string works.
Here's what I tried:
imageBytes = new Uint8Array(image.buffer)
unirest
.post(someURL)
.headers(headers)
.attach("image", fs.createReadStream(imageBytes))
.end(response => {
console.log(response.body);
});
The alternatives are:
1. Attaching the buffer directly, which sends the raw data as a form field. Not ideal, and might run into image size restrictions.
2. Write the file to storage instead of keeping it in memory. This would be handling some sensitive information, thus would require auto-deletion post a certain amount of time, leading to more work.
EDIT: I ended up switching to request, as that allowed inline 'files' from buffers. The code to do so is below:
request({
uri: someURL,
method: "POST",
formData: {
"image": {
value: image.buffer,
options: {
filename: image.originalname,
contentType: image.mimetype
}
}
}
}, (err, resp, body) => {
if (err) {
console.log("ERROR -> " + err)
}
if (body) {
console.log(body)
}
})
EDIT2: Please also put in encoding: null in the request options if you follow this. Don't be like me and spend a day tracking down why your returned data is of an alien format. :)