I am attempting to copy a file to Azure File Share via .Net Storage client (v12) for an integration.
The plan is copy file from Azure Storage to Azure File Share in the same account. That works fine.
I am using the StartCopyAsync method and it works as expected during testing.
However, for the live integration, the consumer is reading the file during the copy operation with zero bytes. The consumer then deletes the file. However, they have accessed and copied deleted the file before it was complete. Is there anything I can do to prevent access to the file during copy operation?
They have the file share mounted an SMB share and monitored and BizTalk is just seeing the file instantly.
Options looked at:
Uploading file and renaming - Azure File Share doesn't seem to support rename?
Uploading Marked ReadOnly and Hidden. SMB doesn't care if the file is ReadOnly and the other part of integration can't change to ignore hidden files
Take File Lease - To prevent deletion - still has the problem of client reading the empty file as soon as it hits the file share
Something else?
Related
I've tried removing a file in an Azure File Share using
the az CLI
Azure Storage Explorer
Both yield the error:
The specified resource may be in use by an SMB client. and ErrorCode:SharingViolation
I've tried listing file handles with the Azure Powershell and az CLI commands, but no file handles are shown. Supposedly, this should reveal any file locks.
I've also tried rebooting everything (that I know of!) that is connected to this file share. Other files in the same directory can be deleted. Everything else with this file share seems normal.
Any idea how I can find the source of the lock, and how to delete it?
Can you check any other client accessing the share?
Create another test file in the same storage account(fileshare) for testing purpose and see are facing the similar issue?
Sharingviolation: The operation failed because the object is already opened and does not allow the sharing mode that the caller requested.
Based on the error message you may refer to this article: https://learn.microsoft.com/en-us/rest/api/storageservices/managing-file-locks
which provides detailed information on file locks
Try to Unlock all Azure file share locks
This article lists common problems that are related to Microsoft Azure Files when you connect from Windows clients. It also provides possible causes and resolutions for these problems. In addition to the troubleshooting steps in this article: Unable to delete files
Our application stores files uploaded from our customers to blob storage. These files are exchanged between different parties (our customers and their suppliers). Is there a way to check the uploaded files for viruses? The Antimalware service seems to just check virtual machines, but I cannot get any information about using it to check files as a service.
A great solution would be if we could store such a file in Azure Storage as an "on hold" file till it is checked. Then we would need a service to check this file and returns the result. If the file is virus-free we could then move it to the final destination.
Azure Storage is just... storage. There are no utilities built in, such as antivirus. You'd need to do your antivirus check on your own. Since antivirus tools typically only work with local OS storage, you'd need to place your "on hold" content (as you referred to it) on a local disk somewhere that you have antivirus installed and then copy to blob storage once your antivirus check is done.
How you accomplish managing this, and which software you use, is up to you. But VMs, App Services, and Cloud Services (web/worker roles) all have local disks available.
As the other answer states Azure Storage is just storage. There are a couple of ways you could do this though,
The first solution would be to run your own anti-virus and use this either as a gateway or programatically download the file from the Blob storage, check the file and then take the appropriate action. It's possible to run something like ClamAV to do this yourself.
Alternatively you could use a third party service like AttachmentScanner (which is exactly what you mention in your comment) which will accept a URL or a direct file upload. With Azure you can generate a temporary url pointing to the file with an expiration of a few minutes, pass the URL to AttachmentScanner and then take the appropriate action depending on the result.
I read an article about virus scanning for blob storage. Might be useful for you.
This guy is using an azure function trigger for the blob to catch the changes and sending the blob file to a virus scanner. The virus scanner is running in a docker container.
Full implementation details are available in the link below
https://peterrombouts.nl/2019/04/15/scanning-blob-storage-for-viruses-with-azure-functions-and-docker/
You can use Azure Defender for Storage to detect following:
Suspicious access patterns - such as successful access from a Tor exit node or from an IP considered suspicious by Microsoft Threat Intelligence
Suspicious activities - such as anomalous data extraction or unusual change of access permissions
Upload of malicious content - such as potential malware files (based on hash reputation analysis) or hosting of phishing content
And to enable it you need to go to Advanced security:
I setup an "azinbox" folder on an azure file storage container. I setup a console application (job) on a VM to check every 30 seconds for a file in that folder. If the job finds it, it moves the file from azinbox to a vminbox folder on the VM. As soon as the files shows up on the VM, if it has a virus, it gets quarantined and the file is deleted from the vminbox. The job on the vm then checks 30 seconds later to see if the file is still in the vminbox. If it is, it must be OK. The job moves the validated file to an azoutbox folder on the azure file storage container. From the Web Site perspective, 1) upload the file to azinbox 2) wait a minute and check the azoutbox. If the file is found, the website moves the file from the azoutbox to its final destination.
I admit it is a crappy solution because it takes a LONG time to complete a file upload. A minute or two can seem like a long time to upload a simple PDF to the user especially if they have more than one to upload.
Also, this requires you setup an entire VM server JUST to validate a file for a virus.
If anyone has a better option, please let me know.
I have an Azure Web App, which will generate pdf files at runtime and write them to disk. Can I trust that these files will be persisted?
I am concerned that if my image is spun down and brought back up again then the files might have disappeared.
Or perhaps Azure decides to move the website to a different machine or different datacentre, where these files would not exist.
I know there are cloud based options such as blob storage, but I would prefer the simplicity of writing to disk and having access over FTP.
Anything that you write under the d:\home folder is guaranteed to be persisted. See the File System section in this for more details on this topic.
I have an old legacy application built on .NET remoting, and transferring data via XML via with FTP.
Esentially, a CRM system is sending XML files to a directory on the web server, which has a windows service that uses a filewatcher to process the incoming XML file, updating the database.
Similarly, changes on the web application serialize down into an XML file into an out folder, that the CRM polls via FTP every 5 minutes.
Trying to map the best services to convert this to for Azure.
You could use Azure Blobs or Azure Files for this.
Azure Blobs: This is the lowest cost option, while still providing high throughput. However, note that Azure Blobs do not have File Watcher functionality, so you would have to poll the directory every few minutes to check for a new file. If you delete files after processing them, then this is really easy - all you have to do is list and see if there are any files. If you want to retain the files, then you might have to do more, since the file list will get big over time. Let me know if this is the case and I can suggest some options.
Azure Files: This is an SMB share that you can mount from a VM in the same region. This will map pretty closely to your exising filesystem based code, including FileWatcher. However, note that Azure Files can only be mounted by a VM in the same region.
I have a WCF service hosted on Windows Azure as a "cloud service." When the service starts, it needs to populate data from files/disk to its memory so it is accessed fast (cached in other words). Right now I'm using like C:\Documents\Filestoprocess folder so that the WCF calls the folder and populates data data in that folder in its memory. I have like 5,000 small files. How do I do this in Azure? Is there a folder path that I can call within the WCF so that the WCF calls these files and opens each files and saves each data in the files? I'm not really looking for complicated Blob access through network using bandwidth. I'm looking for simple disk I/O access to these files from the WCF "cloud service" that is running on its own public web address.
You should try to use a cloud storage service to store data, as if you write to the local file system it can get destroyed on a restart of the service or recycling of the service.
You can look into using the azure drive service, which is like creating a disk dive. It is on top of blob storage.
But if you really want to write and read data on the local file system check out this blog post http://blog.codingoutloud.com/2011/06/12/azure-faq-can-i-write-to-the-file-system-on-windows-azure/
It talks about setting up your service definition to allow writing to the local file system.
Depending on the size of your instances you'll get a non-presistent disk where you can store this kind of temporary data. The minimum is 20GB for an extra small instance. You shouldn't access the disk directly, but you need to use a local resource instead which you can configure in your service definition file or in Visual Studio (double click your Web / Worker Role).
This storage is non-persistent, this means if you delete your deployment, if you decrease the number of instances, in case of hardware problems, ... you loose all data saved here. If you want to persist your files you should use blob storage instead. But in your case, where you need the files as some kind of caching mechanism, local resources are perfect.
And if your goal is to cache data you might want to take a look at the caching features included in Windows Azure: Caching in Windows Azure
Blob access is not complex. In fact, you could do a single download of a zip file from blob storage to local disk, unzip it, then prime your wcf service from those 5,000 small files.
Check out this msdn page documenting DownloadBlobToFile(). The essential parts:
CloudBlobClient blobClient =
new CloudBlobClient(blobEndpoint, new StorageCredentialsAccountAndKey(accountName, accountKey));
// Return a reference to the blob.
CloudBlob blob = blobClient.GetBlobReference("mycontainer/myblob.txt");
// Download the blob to a local file.
blob.DownloadToFile("c:\\mylocalblob.txt");
Now: I don't agree with saving to the root folder on C:. Rather, you should grab some local storage (easily configurable). Once you configure local storage in your role configuration, just ask the role environment for it, and ask for root path:
var localResource = RoleEnvironment.GetLocalResource("mylocalstorage");
var rootPath = localResource.RootPath;
Note: As #KingPancake mentioned, you could use an Azure drive. However: remember that an Azure drive can only be writeable by one instance. You'd need to make additional snapshots for your other instances. I think it's much simpler for you to go with a simple blob, copy your files down (either as single zip or individual files), and go from there.
You mentioned concern with network+bandwidth. You don't pay for bandwidth within the same data center. Also: It's extremely fast: 100Mbps per core. So even with a Small instance, you'll have your files copied down very quickly, moreso when you go to larger instance sizes.
One last thought: The only other ways to gain access to your 5,000 files, without using blob storage or Azure Drives (which are mounted as vhd's in blob storage) would be to either download the files from an external source or bundle them with your Windows Azure package (and then they'd show up in your app's folder, under whatever subfolder you stuck them in). Bundling has two downsides:
Longer time to upload your deployment package due to added size
Inability to change any of the individual files without redeploying the package.
By storing in a blob, you can easily change one (or all) of your small files without redeploying your code - you'd just need to signal it to either re-read from blob storage or restart the instances so they automatically download the new files.