With AWS S3, I can upload a file test.png to any directory I like, regardless of whether or not it exists... because S3 will automatically generate the full path & directories.
For example, if I when I upload to S3, I use the path this/is/a/new/home/for/test.png, S3 will create directories this, is, a, ... and upload test.png to the correct folder.
I am migrating over to Azure, and I am looking to use their file storage. However, it seems that I must manually create EVERY directory... I could obviously do it programmatically by checking to see if the folder exists and if not, create it... but wow...why should I work so hard?
I did try:
file_service.create_file_from_path('testshare', 'some/long/path', 'test.png', 'path/to/local/location/of/test.png')
However, that complains that the directory does not exist... and will only work if I either manually create the directories or replace some/long/path with None.
Is it possible to just hand Azure a path and have it create the directories?
Azure Files closely mimics OS File System and thus in order to push a file in a directory, that directory must exist. What that means is if you need to create a file in a nested directory structure, that directory structure must exist. Azure File service will not create that structure for you.
A better option in your scenario would be to use Azure Blob Storage. It closely mimics Amazon S3 behavior that you mentioned above. You can create a Container (similar to Bucket in S3) and then upload a file with a name like this/is/a/new/home/for/test.png.
However please note that the folders are virtual in Blob Storage (same as S3) and not the real one. Essentially the name with which the blob (similar to Object in S3) will be saved is this/is/a/new/home/for/test.png.
Related
Possible to rename an underlying file while Unzipping using Logic App? I am calling an HTTP activity to download a ZIP file. That Zip contains only 1 Underlying file with some value appended to the name. I want to store the Unzipped file with a better name so that it can be used further. Is it possible ?
Incoming ZIP File --> SAMPLEFile.ZIP
Underlying File --> SampleTextFile20200824121212.TXT
Desired File --> SampleTextFile.TXT
Suggestions ?
As far as I know, we can't implement this requirement directly in "Extract archive to folder" action. We can just rename the file by copy it from one folder to another folder (shown as below).
You can create a new ticket on feedback page to ask azure team for this feature.
I am working in Spark where we need to write the data to S3 bucket after performing some tranformations. I know that while writing dtaa to HDFS/S3 via Spark throws an exception if the folder path already exists. So in our case if S3://bucket_name/folder already exists while writing the data to the same S3 bucket path, it will throw an exception.
Now the possible solution is to use mode as OVERWRITE while writing through Spark. But that would delete all the files already present in it. I want to have a kind of APPEND functionality with the same folder. So if folder already has some files, then it would just add more files to it.
I am not sure if API out of the box gives any such functionality. Of course there is an option where I can create a temporary folder inside a folder and save the file. After that I can move that file to its parent folder and delete the temporary folder. But this kind of approach is not best.
So please suggest how to proceed with this.
I created a spring-boot web project and uploaded it to server already(centOS7).
currently the img upload to jar file on server is stored inside the static package in jar file
this makes the jar file very large and hard to edit.
can some one give me a idea to store the img somewhere else on server and how to find the position of picture out of jar inside html.
First of all, you have to decide, in which directory you are going to store your files and create it:
mkdir /path/to/your/dir
Then assign a newly created directory to your application user:chown <your user>:<users group> /path/to/your/dir
Then, don't forget to give read/write permissions for the user, under which you run your app - to the already created directory.
chmod 600 /path/to/your/dir - this will allow your app to only read/write to the directory and prevent the execution of files within it (for security reasons).
Then just replace the path you already have - with the new one (to the newly created directory).
Please, be aware that there is a lot of security stuff to consider when you're going to store files on your server.
By the way, please consider reading about different storage options like AWS S3, Google Cloud Storage and Ceph.
Please, take into account - that if you're going to store your files on your server - then you should take care about them (for example: keep eyes on space, make sure you have mirroring across discs and so on so forth). With AWS S3, for example, you don't need to care about all of that stuff and it's very cheap.
I am working on an automation piece where I need to download all files from a folder inside a S3 bucket irrespective of the file name. I understand that the using boto3 in python I can download a file like:
s3BucketObj = boto3.client('s3', region_name=awsRegion, aws_access_key_id=s3AccessKey, aws_secret_access_key=s3SecretKey)
s3BucketObj.download_file(bucketName, "abc.json", "/tmp/abc.json")
but I was then trying to download all files irrespective of what filename to be specified in this way:
s3BucketObj.download_file(bucketName, "test/*.json", "/test/")
I know the syntax above could be totally wrong but is there a simple way to do that?
I did find a thread which helps here but seems a bit complex: Boto3 to download all files from a S3 Bucket
There is no API call to Amazon S3 that can download multiple files.
The easiest way is to use the AWS Command-Line Interface (CLI), which has aws s3 cp --recursive and aws s3 sync commands. It will do everything for you.
If you choose to program it yourself, then Boto3 to download all files from a S3 Bucket is a good way to do it. This is because you need to do several things:
Loop through every object (there is no S3 API to copy multiple files)
Create a local directory if it doesn't exist
Download the object to the appropriate local directory
The task can be made simpler if you do not wish to reproduce the directory structure (eg if all objects are in the same path). In that case, you can simply loop through the objects and download each of them to the same directory.
I have a required to download a file from SFTP server and the file downloaded is stored to local folder say "D:\Data\tempData.csv"
I have to read the data from local file and consume in my application for other data manipulation.
This job is created using web hooks scheduler in Azure Web Jobs.
I am unable to download file to azure and then read from there.
Can some one help me to use a location for temp data which is equivalent to "D:\Data\tempData.csv" in local system in the azure environment.
Suggest a place in azure where can I download file and then to read from there.
Thanks in Advance.
What I tried?
Tried using SSH.NET dll to download file from SFTP to local folder
Again to read from local folder to my application
Tried looking at BLOB storage usage, which was not approved Tech Arch.
In an Azure Web App, you can create files anywhere under d:\home (for persistent files) or under d:\local (temporary files). See this page for more details on the file system. Try using Kudu Console to see those locations.
How you get the file in that location sounds mostly unrelated to your primary question about what location you can use.
In Azure Environment, the "Web-Jobs" are stored in its local folder where known as "D:\home" and "D:\local" is the local folder used by the Web-hooks.
I was in need to use a folder for temporary usage of downloading a file from SFTP server and again read the file from that local temporary location file and consume it in my application.
I have used the "D:\local\Temp" as the temporary folder which is created by the code after checking the folder existence, then after creating the folder the code will download a file from server and store to this location and then read from the same location and delete the file from that temporary folder.
Thanks all for your help, #David Ebbo Thanks.