I currently have a node.js server running where I can grab a csv file stored in storage bucket and store that to a local file.
However, when I try to do the same thing with a xlsx file, it seems to mess up the file and cannot be read when I download it to a local directory.
Here is my code for getting the file to a stream:
async function getFileFromBucket(fileName) {
var fileTemp = await storage.bucket(bucketName).file(fileName);
return await fileTemp.download()
}
and with the data returned from above code, I store it into local directory by doing the following:
fs.promises.writeFile('local_directory', DataFromAboveCode)
It seems to work fine with .csv file but does not work with .xlsx file where I can open the csv file but xlsx file gets corrupted and cannot be opened.
I tried downloading the xlsx file directly from the storage bucket on google cloud console but it seems to work fine, meaning that somethings gone wrong in the downloading / saving process
Could someone guide me to what I am doing wrong here?
Thank you
Related
I am using multer npm library to read files from postman and I am getting the file and its details in my node js code(checked via logging req.file), but my concern is I don't want the file to get stored in my local machine, I just want to extract the data from the file and process further for my requirements.
Is this possible or anyone can suggest me some solutions to this.
Thanks in advance
As I read multer libraries, it streams the uploaded file in the disk, and then it processes the file.
If you only want to use the properties like mime/type or originalfilename of the file you can use multer and in the storage options after getting the uploaded file properties, use cb(null, false) to prevent storing the file, but if you want to process the file, you can remove it from the disk after your process is done.
When uploading csv file created by windows; an error is generated when trying to parse the file using exceljs.
Here is the error message:
message: "Can't find end of central directory : is this a zip file ?
If it is, see https://stuk.github.io/jszip/documentation/howto/read_zip.html"
It happens because of the uploading stage, you haven't uploaded the file successfully so that can not reading later. Please double check whether you uploading or having the file on cloud storage successfully or not
I am accessing a website that allows me to download CSV file. I would like to store the CSV file directly to the blob container. I know that one way is to download the file locally and then upload the file, but I would like to skip the step of downloading the file locally. Is there a way in which I could achieve this.
i tried the following:
block_blob_service.create_blob_from_path('containername','blobname','https://*****.blob.core.windows.net/containername/FlightStats',content_settings=ContentSettings(content_type='application/CSV'))
but I keep getting errors stating path is not found.
Any help is appreciated. Thanks!
The file_path in create_blob_from_path is the path of your local file, looks like "C:\xxx\xxx". This path('https://*****.blob.core.windows.net/containername/FlightStats') is Blob URL.
You could download your file to byte array or stream, then use create_blob_from_bytes or create_blob_from_stream method.
Other answer uses the so called "Azure SDK for Python legacy".
I recommend that if it's fresh implementation then use Gen2 Storage Account (instead of Gen1 or Blob storage).
For Gen2 storage account, see example here:
from azure.storage.filedatalake import DataLakeFileClient
data = b"abc"
file = DataLakeFileClient.from_connection_string("my_connection_string",
file_system_name="myfilesystem", file_path="myfile")
file.append_data(data, offset=0, length=len(data))
file.flush_data(len(data))
It's painful, if you're appending multiple times then you'll have to keep track of offset on client side.
I'm writing a small archiving script (in node.js) to move files on my Google Drive to a predetermined folder if they contain .archive.7z in the filename. The script is run periodically as a cron job, and the file movement has not caused any issues, but files still in the process of being uploaded by my desktop client are moved before they're finished. This terminates the upload and results in corrupted files in the destination folder.
Files still being uploaded from my desktop to Google Drive are returned by the following function anyway:
async function getArchivedFiles (drive) {
const res = await drive.files.list({
q: "name contains '.archive.7z'",
fields: 'files(id, name, parents)',
})
return res.data.files
}
Once the files are moved and renamed with the following code, the upload terminates from my client (Insync) and the destination files are ruined.
drive.files.update({
fileId: file.id,
addParents: folderId,
removeParents: previousParents,
fields: 'id, parents',
requestBody: {
name: renameFile(file.name)
}
})
Is there any way to check if a file is still being uploaded before moving it?
It turns out that a tiny placeholder-type file is being created on uploads. I'm not sure if this is a Google Drive API behaviour or something unique to the Insync desktop client. This file seems to upload separately and thus can be freely renamed once it's complete.
I worked around this problem by including the file's md5 hash in the filename, and updating my script to only move files when the hash in their filename matches the md5Checksum retrieved from the Google Drive API.
Uploading zip file to sharepoint using REST is not saving the file contents. The operation is sucessful. But when the file is downloaded from site to local machine, its failing to open the zip and showing 0 bytes
Did you follow:
https://learn.microsoft.com/en-us/sharepoint/dev/sp-add-ins/upload-a-file-by-using-the-rest-api-and-jquery
Creating the item (with the meta data) in the library and upload the file (binary) are two seperate actions you have to handle to complete the upload. I think you only add an item to the library and not uploading the binary data?