So I have a use case where the client uploads a small TSV file, the file is opened and parsed on the server, and results are written to a new file on the server.
Since the TSV file will be tiny (under 1 MB), I am wondering if it is even necessary to upload the file to the server (writing it to disk) before parsing it. Instead, could the file contents be captured when the user clicks "upload file"? I could then store the file contents in an array, each item representing a line in the file.
Thoughts?
You don't need to stream the file to disk, but be aware that you should set clear and concise limits so that a person could not, say, upload a 5GB file and make your service crash from memory exhaustion. You just need to be aware that you're limited to your available amount of memory(likely less) when you process something completely in memory. It's also possible to stream parse it, so that you don't need to save it to disk before parsing it. In your case it sounds easiest to just upload it into memory, and make certain that you put a limit(maybe like 5mb limit) on the upload file size.
Are you asking whether this option is feasible or whether it's a good idea?
Regarding feasibility, it is entirely possible using the FileReader API to parse the content and then a simple Meteor.call onto whatever method is appending to the file on disk. The code would like like follows:
function onSubmit(event, template) {
var file = template.$('.your-file-input-elemt').files[0];
var filereader = new FileReader();
filereader.onload = function(fileevent) {
Meteor.call('processTSV', filereader.readAsText(file));
};
}
If you're talking about whether it's a good idea, then it comes down to browser support. Are you okay with users without the FileReader API not getting support from your application? If so it's considerably easier to deal with than handling uploads with something like CollectionFS.
Related
I am writing a nodejs app which works with fonts. One action it performs is that it downloads a .ttf font from the web, converts it to a base64 string, deletes the .ttf and uses that string in other stuff. I need the .ttf file stored somewhere, so I convert it. This process takes like 1-2 seconds. I know heroku has an ephemeral file system but I need to store stuff for such a short time. Is there any way I can store my files? Using fs.writeFile currently returns this error:
Error: EROFS: read-only file system, open '/app\test.txt']
I had idea how about you make an action, That would get font, convert it and store it on a global variable before used by another task.
When you want to use it again, make sure you check that global variable already filled or not with that font buffer.
Reference
Singleton
I didn't know that you could store stuff in /tmp directory. It is working for the moment but according to the dyno/ephemeral system, it gets cleaned frequently so I don't know if it may cause other problems in the long run.
I'm using nodejs and developing file upload system on my personal project.
From one blog, (it's not english site so I won't link it here) I saw that I have to re-encoding fil. If not, high-level attacker can insert malicious shell code in normal file (he told jpeg as an example) even I limit the extension of uploaded files and change the original filename and hide the filepath.
My question is, so how to re-encoding files in nodejs??
there are several types of file and I don't know how to do it.
Thanks in advance.
While doing a file upload, there are several hardening to increase security.
Rename the file name with extension
Use whitelist to check the file's extension.
Use whitelist to check the file's mime types.
Use whitelist to check the file's magic bytes.
Set content-type and no-sniff headers to response if you buffer (accessible) your files from HTTP
Re-encoding means in here actually is the fourth step in above.
Determine file type with reading magic bytes;
const FileType = require('file-type');
const readChunk = require('read-chunk');
(async () => {
const buffer = readChunk.sync('Unicorn.png', 0, 4100);
console.log(await FileType.fromBuffer(buffer));
//=> {ext: 'png', mime: 'image/png'}
})();
There are several packages that you can decide to use:
https://github.com/sindresorhus/file-type
I would like to wrap a data file (~1MB) to golang app and then use that data in os.exec. The app runs on Linux.
How to define the data in the app, as a string or []byte, variable, or Const?
Should be defined in a global scope, or wrapped in a func?
How to pass the data from the app memory to the executed process ?
For building the data file(s) in to your program, you have a number of choices. You are correct in that you could manually copy/paste the data file(s) in to the program as types string, []byte as variables, but there are other packages/applications for go that handle this for you already that can also minimize your app's memory footprint. One example that comes to mind is go-bindata (link to repo) which does just this.
As for using the file in os/exec, I'm assuming you're trying to either pass the entire file to the program using either a file path or a raw string. For file paths, you would have to write the entire file to disk first. If you're trying to pass the entire file contents as a raw string, you can still use go-bindata or a string of the data file as arg in os/exec.Command.
P.S. - go-bindata has not seen updates in a while, so I would encourage you to seek more active forks or alternatives if you're having trouble using it.
I am using streams to download the excel file. Please find the code below
$response = new \Zend\Http\Response\Stream();
$response->setStream(fopen($myfile, 'r'));
$response->setStatusCode(200);
$headers = new \Zend\Http\Headers();
$headers->addHeaderLine("Content-Type: application/vnd.ms-excel; charset=UTF-8")
->addHeaderLine('Content-Disposition', 'attachment; filename=my.xls')
->addHeaderLine( "Content-Transfer-Encoding: binary" )
->addHeaderLine('Content-Length', filesize($myfile));
$response->setHeaders($headers);
The file generated is proper but when the same file I am enforcing the user to download there are two empty lines coming in the downloaded excel file. I did some researching and thought may it's http version and header lines but I think it's not because if I try otherwise I get those two empty line in start of excel report.
Please note there are no empty space in the start of the content.
Any idea ?? why it's so?
Mind the streamed response does in fact not really provide a stream context to the client. It just buffers the stream internally and sends out the response in one go.
That being said, I have created a controller plugin to send attachments from a file path or directly with the binary data addressed to a variable. It's in my common Soflomo\Common library. I haven't had the issues you described and I use some more headers than you do.
Tell me if that piece of code works for you. One of the differences is you use the size of the original file as the size of the response. I am not sure, as this might cause an indifference with the cached streamed response. Try to just grab the contents and do a strlen() on this content.
I hope my Q was clear ... I am curious about the typical way to code for someone clicking File|Open, and selecting a file that is inappropriate for the program--like someone using a word processing program and trying to open a binary file.
In my case, my files have multiple streams streamed together. I'm unsure how to have the code validate whether an improper file was selected before the app throws a stream read exception. (Or is the way to handle the situation to just write code to catch a stream read exception?)
Thanks, as always.
I think it's quite usual that you have code that just tries to open the file, and if it fails, an error is shown to the user. Most file formats has some kind of header with a "magic number", so that the reader can tell if it's not the right file very quickly after reading the first few bytes of the file.
Magic number at the start of the file generally helps -- if you have control of the file format.
Otherwise, yeah -- catch the exception and put up a dialog.