I'm trying to download the entire folder from Firebase Storage so I can zip those files and upload them back to Firebase Storage while generating a download link.
I have read a lot of posts and code but everything seemed getting away from my scope.
Do you have a clear example on how to download a Firebase Storage Folder to a Cloud Functions Temporary Folder or at least some hints on I could do it, keeping in mind that I am targeting a specific folder?
There is no bulk download operation provided by the SDK. The general pattern for downloading all files with some shared prefix using the node SDK for Cloud Storage will be:
list the files using getFiles at the prefix (folder) of interest
iterate them
download each one separately
Related
Right now I can copy a single file to a different location but is there a way to copy the entire contents of a folder to a different location.
In firebase-storage there is no concept of folder. There's the concept of reference e.g. "images/mountains.png" which you may have misinterpreted it as folder.
In order to copy all the content of similar references, you need to list all the files, then move the desire list of files over to new reference e.g. "archive/mountains.png"
This post may be useful
How to get a list of all files in Cloud Storage in a Firebase app?
I am using Youtube-dl to download transcript, it works fine on my machine (local server) where I provide the __Dirname into the Options params to upload files. But I want to use Google Cloud functions, so how can I substitute __dirname with Cloud storage ??
Thank you !!
Upload from Youtube-dl it's not possible. To upload files into Google Cloud storage is possible if you upload a file already in your disk.
You will need to download the file from whichever program you mention (as mentioned in the comments, you can download it to a temporal folder), upload the file to GCS and then delete it from your temporal folder.
What you can actually do? you can for example run a script inside of a Google Cloud Instance with a gsutil command to upload the files into a bucket.
I have a backend Nodejs application and I am fetching and streaming files in the background when a certain event happens in the client.
I have deployed the backend to Google App Engine.
The file downloading is working fine but I am a bit confused where the files are downloaded and stored ? In the app I am creating a folder relative to the deployed app folder and storing them there with createWriteStream. I also init a git repository where the files are (using simple-git npm module)
It seems the files are not accessible via the cloud shell since I can not find them there
Can I for example create a storage bucket and use "normal" file operations command there (and init the repo there)
-Jani
To store data downloaded you want to store it in Cloud Storage, you can find a complete guide in this Using Cloud Storage documentation.
Under almost any circumstances you want to download files into the App Engine Deployment since the instances doesn't have much memory to store data, and also when the deployment scales up and down you are prone to lost data
With AWS S3, I can upload a file test.png to any directory I like, regardless of whether or not it exists... because S3 will automatically generate the full path & directories.
For example, if I when I upload to S3, I use the path this/is/a/new/home/for/test.png, S3 will create directories this, is, a, ... and upload test.png to the correct folder.
I am migrating over to Azure, and I am looking to use their file storage. However, it seems that I must manually create EVERY directory... I could obviously do it programmatically by checking to see if the folder exists and if not, create it... but wow...why should I work so hard?
I did try:
file_service.create_file_from_path('testshare', 'some/long/path', 'test.png', 'path/to/local/location/of/test.png')
However, that complains that the directory does not exist... and will only work if I either manually create the directories or replace some/long/path with None.
Is it possible to just hand Azure a path and have it create the directories?
Azure Files closely mimics OS File System and thus in order to push a file in a directory, that directory must exist. What that means is if you need to create a file in a nested directory structure, that directory structure must exist. Azure File service will not create that structure for you.
A better option in your scenario would be to use Azure Blob Storage. It closely mimics Amazon S3 behavior that you mentioned above. You can create a Container (similar to Bucket in S3) and then upload a file with a name like this/is/a/new/home/for/test.png.
However please note that the folders are virtual in Blob Storage (same as S3) and not the real one. Essentially the name with which the blob (similar to Object in S3) will be saved is this/is/a/new/home/for/test.png.
I have a required to download a file from SFTP server and the file downloaded is stored to local folder say "D:\Data\tempData.csv"
I have to read the data from local file and consume in my application for other data manipulation.
This job is created using web hooks scheduler in Azure Web Jobs.
I am unable to download file to azure and then read from there.
Can some one help me to use a location for temp data which is equivalent to "D:\Data\tempData.csv" in local system in the azure environment.
Suggest a place in azure where can I download file and then to read from there.
Thanks in Advance.
What I tried?
Tried using SSH.NET dll to download file from SFTP to local folder
Again to read from local folder to my application
Tried looking at BLOB storage usage, which was not approved Tech Arch.
In an Azure Web App, you can create files anywhere under d:\home (for persistent files) or under d:\local (temporary files). See this page for more details on the file system. Try using Kudu Console to see those locations.
How you get the file in that location sounds mostly unrelated to your primary question about what location you can use.
In Azure Environment, the "Web-Jobs" are stored in its local folder where known as "D:\home" and "D:\local" is the local folder used by the Web-hooks.
I was in need to use a folder for temporary usage of downloading a file from SFTP server and again read the file from that local temporary location file and consume it in my application.
I have used the "D:\local\Temp" as the temporary folder which is created by the code after checking the folder existence, then after creating the folder the code will download a file from server and store to this location and then read from the same location and delete the file from that temporary folder.
Thanks all for your help, #David Ebbo Thanks.