I'm facing weird issue. I have one Azure Blob storage container which consists of 2 folders, let's say folder A and folder B. If I copy the files using WinSCP only to folder A, the file is copied in folder A as well as folder B and vice versa.
I see this issue only using WinSCP with Blob storage and not with Azcopy or Azure storage explorer. Also i don't see this issue with Azure file storage using WinSCP.
Any help appreciated.
Related
I have a use case where I am Transferring XML files from Windows fileshare to Azure Blob Storage and then loading data to Snowflake Tables. I am using Matillion to achieve this.
The Windows Fileshare gets Zipped XML file which contains .xml and .xml.chk files. I am using Azure Blob Storage component of Matillion to Copy the .xml files to Snowflake table and have set Purge = True to delete them afterwards.
I need help in deleting the leftover .xml.chk files from Blob Storage Container. Also, once the data loading is complete, I would like to delete the zipped files from Windows Fileshare.
Thanks,
Shivroopa
you can delete the files from blob storage using the matillion python script component (orchestration->scripting->Python script)
here is an example of python code to delete blob items and containers
Delete Blob Example
I don't see a way to delete files on the windows machine from Matillion other than creating an API endpoint on the fileshare and calling the API from Matillion.
I have a Azure VM (Win 2016) where I have a folder where we have file coming every 5 minute.
Now I want to create and Window Service which will run on Azure VM and if any file exist, it will move to Azure File storage.
Could someone guide whats need to do or any other approach?
As I see it, you have 2 options:
Mount File Storage Share as a network drive. Once you mount the share as a network drive (and get a drive letter) you can simply use System.IO namespace to perform IO operations. Please see this link for more details: https://learn.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-windows.
Use Microsoft's Azure Storage SDK which is a wrapper over Azure Storage REST API and upload files from the local folder to the share in Azure File Storage. Please note that once the file is uploaded in Azure File Storage, you would need to manually delete the file from your server to achieve move operation.
I have few csv file present on my local system. I want to upload them into Azure blobs in a particular directory structure. I need to create the directory structure as well on azure.
Please suggest the possible options to achieve that.
1 - Create Your storage in Azure
2 - get Azure storage explorer
https://azure.microsoft.com/en-us/features/storage-explorer/
3- start the app, login and Navigate to your Blob
4- Drag and drop folder and files :)
basically this MS provided Software allows you to use your storage account in a classic folders and files structure
I can not find the API for Node.js to copy everything in one fileShare to another fileShare in different storage account. Does anyone run into this problem before?
All I can find is the azCopy solution.
Thank you very much
I can not find the API for Node.js to copy everything in one fileShare
to another fileShare in different storage account.
This is because no such API exists. Node SDK for Azure Storage is a wrapper over Azure Storage REST API and the REST API only supports copying files (neither directories nor shares).
What you would need to do is list all files from a share and then copy these files individually which is what azcopy does.
Currently i have some ftp where on it i have some deep structure of folders and files within it. It could be even 10 levels down from root folder. As i migrated already with success my local database to azure database, i wonder also whether is there any azure ftp i could use to migrate this as well. I know we have something like Azure storage and i could create Container for it of type File or Blobs - are one of those could be used like particural ftp - could i create folder structure there somehow using container and either File or Blob for that purpose, how it works there? Does either container blob or file for such purposes?
Let me add to what NDJ has written. So both Azure Blobs and Files would serve your purpose.
As mentioned by NDJ, Azure Blob Storage is a 2-level hierarchy system. At the top you have a blob container and the each blob container contains 0 or more files. So it does not support a folder structure per se but as NDJ mentioned, you can create an illusion of a sub folder by using appropriate blob delimiters (usually /). If you were to compare it with local file system, a directory at the root level (C:) is a container in blob storage and then the files would go in there. So imagine you have a folder called images in C:\ of your computer, that would be a container in blob storage. Now imagine that you have 2 sub folders beneath this folder (let's call them hires and lores) and both of them contains some files (say image1.png). When you move them to Azure Blob Storage, the container name would be images but the blob names would be hires/image1.png and lores/image1.png. Some of the storage explorers would take this delimiter (/) and show you that your container contains 2 folders and inside each folder you have an image called image1.png but in reality there are only 2 blobs in that blob container.
Azure File Service is a close match to your local file system. At the top level, you've got a Share and each share will container directories and files. Each directory can again contain many directories and files.
As NDJ mentioned, there's no FTP access to Azure Storage but there are many tools that will allow you to upload files from local computer to Azure Storage and many of them will preserve the file hierarchy. You can always write code to upload the files yourself. If you decide to use Azure Files, you can simply mount a File Storage Share as a network drive on your local computer and then transfer the files from your local computer to Azure Files as if you're transferring files from one drive to another.
UPDATE
Regarding difference between Azure Blob Storage and File Storage, both are used to store files. There are a few differences that I could think of:
A Share in Azure File Storage can be mount as a network drive on your local computer/Azure VM whereas a Blob Container in Azure Blob Storage can't. So if you have an application which writes files to local file system, you can take the application as is and make use of Azure File Storage and write the file to that network drive without making many changes to your code (typical example of Lift-And-Shift kind of application.
You can set ACL on a Blob Container whereas you can't do the same on a Share. This makes Azure Blob Storage ideal for storing static content (images, css, js) for your websites. For exposing files in File Storage, you would need to resort to Shared Access Signature.
You can set the size of a Share (default is 5GB) whereas no such thing exist for a Blob Container. A blob container can go up to the size of a storage account.
To understand Azure Files, I would recommend reading this: https://azure.microsoft.com/en-in/documentation/articles/storage-dotnet-how-to-use-files/.
Azure blob supports 10 levels down (up to 254. Basically the files are stored non hierarchically, but each / separator gives the appearance of directories.
It's relatively trivial to write something to move files to azure, as far as I know there is no ftp functionality yet - but it has been requested. It looks like some people have already created some code for this
You can now use Storage Explorer across all platforms to easily work within any folder structure.