Is there a way to get a file object in Node with file meta data like mimetype, originalname, path, etc..? I'd like to read a file from disk and get this information, rather than a Buffer.
Here is one nodejs package that could help you work with file meta data information.
https://www.npmjs.com/package/fs-meta
Related
Is there any way to read a parquet file schema from Node.JS?
If yes, how?
I saw that there is a lib, parquetjs but as I saw it from the documentation it can only read and write the contents of the file.
After some investigation, I've found that the parquetjs-lite can do that. It does not read the whole file, just the footer and then it extracts the schema from it.
It works with a cursor and the way I saw it there is two s3.getobject calls, one for the size and one for the given data.
I am sending a gzip file from Postman to a Flask endpoint. I can take that binary file with request.data and read it, save it, upload it, etc.
My problem is that I can't take its name. How can I do that?
My gzip file is called "test_file.json.gz" and my file is called "test_file.json".
How can I take any of those names?
Edit:
I'm taking the stream data with io.BytesIO(), but this library doesn't contain a name attribute or something, although I can see the file name into the string if I just:
>>>print(request.data)
>>>b'\x1f\x8b\x08\x08\xca\xb1\xd3]\x00\x03test_file.json\x00\xab\xe6RPP\xcaN\xad4T\xb2RP*K\xcc)M5T\xe2\xaa\x05\x00\xc2\x8b\xb6;\x16\x00\x00\x00'
Further to the comment, I think the code which handles your upload is relevant here.
See this answer regarding request.data:
request.data Contains the incoming request data as string in case it came with a mimetype Flask does not handle.
The recommended way to handle file uploads in flask is to use:
file = request.files['file']
file is then of type: werkzeug.datastructures.FileStorage.
file.stream is the stream, which can be read with file.stream.read() or simply file.read()
file.filename is the filename as specified on the client.
file.save(path) a method which saves the file to disk. path should be a string like '/some/location/file.ext'
source
let dataCer = '0�\u0007\u00060�\u0006��\u0003\u0002\u0001\u0002\u0002\u0010Q��\u0000����K��Z�Q��0\n\u0006\b*�\u0003\u0007\u0001\u0001\u0003\u00020�\u0001l1\u001e0\u001c\u0006\.............'
fs.writeFile('111.cer', dataCer);
let dataPdf = '%PDF-1.4\r\n1 0 obj\r\n<< \r\n/Length 9947\r\n/Filter /FlateDecode\r\n>>\r\nstream\r\nX��]�n#9p}���\u000f���\u0005\b\u0002X��<\'X \u001f�\u001b\u0010 \u0001���H�,6�R�Z�\u0014�N`�\n�T�t�ڼT\u0015���?ԋz��_�{IN_Bz�����O.............'
fs.writeFile('111.pdf', dataPdf);
The data dataCer and dataPdf I get from the application using the GET requests. I can only get this data in this encoding.
And now I need to save them as files.
Also, I will need to then save any data to the file in the same way (zip, rar, png, jpeg, ...).
When i use fs.writeFile, I get files that do not open.
fs.writeFile, can not keep the original state data, ignoring the encoding does not give me the desired result.
Please tell me how to get around this error?
Or which library can save data to any file in node.js, while ignoring the encoding?
I am trying to write an API in python (Falcon) to accept a file from multipart-form parameter and put the file in MINIO object storage. The problem is I want to send the file to Minio without saving it in any temp location.
Minio-python client has a function using which we can send the file.
`put_object(bucket_name, object_name, data, length)`
where data is the file data and length is total length of object.
For more explanation: https://docs.min.io/docs/python-client-api-reference.html#put_object
I am facing problem accumulating the values of "data" and "length" arguments in the put_object function.
The type of file accepted in the API class is of falcon_multipart.parser.Parser which cannot be sent to Minio.
I can make it work if I write the file to any temp location and then read it from the desired location and send.
Can anyone help me finding a solution to this?
I tried reading file data from the Parser object and tried converting the file to bytes io.BytesIO. But it did not work.
def on_post(self,req, resp):
file = req.get_param('file')
file_data = file.file.read()
file_data= io.BytesIO(file_data)
bucket_name = req.get_param('bucket_name')
self.upload_file_to_minio(bucket_name, file, file_data)
def upload_file_to_minio(self, bucket_name, file, file_data):
minioClient = Minio("localhost:9000", access_key='minio', secret_key='minio', secure=False)
try:
file_stat = sys.getsizeof(file_data)
#file_stat = file_data.getbuffer().nbytes
minioClient.put_object(bucket_name, "SampleFile" , file, file_stat)
except ResponseError as err:
print(err)
Traceback (most recent call last):
File "/home/user/.local/lib/python3.6/site-packages/minio/helpers.py", line 382, in is_non_empty_string
if not input_string.strip():
AttributeError: 'NoneType' object has no attribute 'strip'
A very late answer to your question. As of Falcon 3.0, this should be possible leveraging the framework's native multipart/form-data support.
There is an example how to perform the same task to AWS S3: How can I save POSTed files (from a multipart form) directly to AWS S3?
However, as I understand, MinIO requires either the total length, which is unknown, or alternatively, it requires you to wrap the upload as a multipart form. That should be doable by reading reasonably large (e.g., 8 MiB or similar) chunks into the memory, and wrapping them as multipart upload parts without storing anything on disk.
IIRC Boto3's transfer manager does something like that under the hood for you too.
I know how to write and read from a file in S3 using boto. I'm wondering if there is a way to append to a file without having to download the file and re-upload an edited version?
There is no way to append data to an existing object in S3. You would have to grab the data locally, add the extra data, and then write it back to S3.