I was working on a simple code to download missing images from a site and save it in the local system, given its complete URL. I am able to get the data in binary format as response and also I am able to save it properly. But when I try to open the image it shows the format is not supported by the system. I tried to save some js and css file and they are being saved properly and I am able to view them as well. But I am having problem with all the image formats.
Here is the code I wrote:
try {
response = await axios.get(domain + pathOfFile);
console.log(response);
fs.writeFile(localBasePath + pathOfFile, response.data, "binary", (error) => {
if (error) console.log("error while writting file", error.message);
});
} catch (error) {
console.log("error in getting response", error.message);
}
domian: contains the base domain of the site
pathOfFile: contains the path of file on that domain
localBasePath: the base folder where I need to store the image
I even tried to store the response in a buffer and then tried to save the image, but still I am facing the same problem.
Any suggestions would be appreciated.
You need to define responseEncoding while calling axios.get method.
Change your line to:
response = await axios.get(domain + pathOfFile, {responseEncoding: "binary"});
Related
I have followed many solutions provided in the previous questions but mine is not working. The problem is in .json extension. Whenever I use filename.json, the app will crash with ERR_CONNECTION_RESET but successfully created an empty .json file. However, if I change the extension to filename.txt, the fs.writeFile will successfully create the filename.txt with the data inside and the app will work as expected. Did I miss any configuration here to create the JSON file?
Here is the example code I used.
var jsonData = '{"persons":[{"name":"John","city":"New York"},{"name":"Phil","city":"Ohio"}]}';
// parse json
var jsonObj = JSON.parse(jsonData);
console.log(jsonObj);
// stringify JSON Object
var jsonContent = JSON.stringify(jsonObj);
console.log(jsonContent);
fs.writeFile("./public/output.json", jsonContent, 'utf8', function(err) {
if (err) {
console.log("An error occured while writing JSON Object to File.");
return console.log(err);
}
console.log("JSON file has been saved.");
});
So, ERR_CONNECTION_RESET means that the connection was closed midway. My guess, as in the comments, would be that it's a reloading server.
Try using --ignore public/**/*.json and it should work.
I'm trying to create pages that will take a user information and save them to the database. the user information are {name, age....... picture}, when I put the information without a picture it work fine and the data saved to the database but when I try to put the picture with them it gives me the error.
I'm sorry for the picture quality.
any one can help me with this.
I'm using nodejs and react
thx :)
You would need to send the picture as Base64 in your object, as it follows:
var data = {
name: 'John',
age: 27,
picture: 'data:image/png;base64,R0lGODlhPQBEAJos...',
}
In NodeJS, if using Express, the picture will be req.body.picture. So, all you need to do is store the file, then get the temp path do do what you need.
You can store base64 file doing:
var filePath = './tmp/myPicture.png';
fs.writeFile(filePath, req.body.picture, 'base64', (err) => {
if (err) {
res.json({ err: 'Error while creating temp file from base64.' });
} else {
// Your file was uploaded, so you can read your file here.
}
});
I would like to download a .csv file from the browser, intercept it, and extract the data to convert it into a JSON object. Most responses say to use the requests.buffer(), however, my situation is unique as it always says the buffer is empty, but the file downloads.
I have tried to pull requests.buffer()
downloadPage.on('request', request => {
console.log(request.isNavigationRequest());
console.log(nextRequest);
if (request.isNavigationRequest() && !nextRequest) {
return request.abort();
}
initialRequest = false;
request.continue();
});
downloadPage.on('response', async (response) => {
console.log(response.buffer());
file_data = JSON.parse(response.buffer());
})
await Promise.all([
downloadPage.goto('https://clients.messagelabs.com/Tools/Track-And-Trace/DownloadCsv.ashx?sessionid=' + json_data.request.SessionId).catch(err => console.log(err)),
page.waitForNavigation()
])
Since I am on a corporate network, it won't let me upload my images, but...
When I proceed to the link above on downloadPage.goto(...) it automatically begins a .csv file download. Than the page closes. I think the page closing is clearing the buffer, however, I can't seem to intercept the response to grab the file data before this happens. Any ideas are appreciated.
Please do not link me to another github that tells me to use the request.buffer(), as I have tried many variations.
Error: Protocol error (Network.getResponseBody): No data found for resource with given identifier
Context
I am working on a Proof of Concept for an accounting bot. Part of the solution is the processing of receipts. User makes picture of receipt, bot asks some questions about it and stores it in the accounting solution.
Approach
I am using the BotFramework nodejs example 15.handling attachments that loads the attachment into an arraybuffer and stores it on the local filesystem. Ready to be picked up and send to the accounting software's api.
async function handleReceipts(attachments) {
const attachment = attachments[0];
const url = attachment.contentUrl;
const localFileName = path.join(__dirname, attachment.name);
try {
const response = await axios.get(url, { responseType: 'arraybuffer' });
if (response.headers['content-type'] === 'application/json') {
response.data = JSON.parse(response.data, (key, value) => {
return value && value.type === 'Buffer' ? Buffer.from(value.data) : value;
});
}
fs.writeFile(localFileName, response.data, (fsError) => {
if (fsError) {
throw fsError;
}
});
} catch (error) {
console.error(error);
return undefined;
}
return (`success`);
}
Running locally it all works like a charm (also thanks to mdrichardson - MSFT). Stored on Azure, I get
There was an error sending this message to your bot: HTTP status code InternalServerError
I narrowed the problem down to the second part of the code. The part that write to the local filesystem (fs.writefile). Small files and big files result in the same error on Azure.fs.writefile seams unable to find the file
What is happpening according to stream logs:
Attachment uploaded by user is saved on Azure
{ contentType: 'image/png',contentUrl:
'https://webchat.botframework.com/attachments//0000004/0/25753007.png?t=< a very long string>',name: 'fromClient::25753007.png' }
localFilename (the destination of the attachment) resolves into
localFileName: D:\home\site\wwwroot\dialogs\fromClient::25753007.png
Axios loads the attachment into an arraybuffer. Its response:
response.headers.content-type: image/png
This is interesting because locally it is 'application/octet-stream'
fs throws an error:
fsError: Error: ENOENT: no such file or directory, open 'D:\home\site\wwwroot\dialogs\fromClient::25753007.png
Some assistance really appreciated.
Removing ::fromClient prefix from attachment.name solved it. As #Sandeep mentioned in the comments, the special characters where probably the issue. Not sure what its purpose is. Will mention it in the Botframework sample library github repository.
[update] team will fix this. Was caused by directline service.
I am using multer-gridfs-storage and gridfs-stream to store my video in the backend (Express/Node). When I try to retrieve the file to play on my front end (React) the player refuses to recognize the source.
I am using Video-React to display the video on download. The download is successful, I get a Binary string back from the backend, which I converted to a Blob.
try{
fileBlob = new Blob([res.data], {type : res.headers['content-type']});
}catch(err){
console.log('Error converting to blob');
console.log(err);
}
This is my Video-React player being rendered
<Player
autoPlay
ref="player"
>
<source src={this.state.fileURL} />
<ControlBar autoHide={false} />
</Player>
Then I tried two techniques
readDataAsURL
let reader = new FileReader();
reader.onload = function(event){
//rThis is just a reference to the parent function this
rThis.setState({fileURL: reader.result}, () => {
rThis.refs.player.load();
});
}
try{
reader.readAsDataURL(fileBlob);
}catch(err){
console.log('Error trying readDataURL');
console.log(err);
}
src is being set correctly but the video never loads
URL.createObjectURL
let vidURL = URL.createObjectURL(fileBlob);
rThis.setState({fileURL: vidURL}, () => {
rThis.refs.player.load();
});
src is set to a blob: url but still nothing
Is this an issue with Video-react or should I be doing something else? Any pointers to references I could look at will also help. What am I doing wrong? dataURL works in the case of images, I checked, but not video.
So after some more reading, I finally figured out the problem. Since I'm using gridfs-stream I'm actually piping the response from the server. So I was never getting the whole file, and trying to convert res.data, which is just a chunk, was a mistake. Instead, in my res object, I found the source url within the config property.
res.config.url
This contained my source url to which my server was piping the chunks. Should have figured it out earlier, considering I picked GridFS storage for precisely this reason.