I'm trying to resize the image from blob storage using the Azure function - the easy task, lots of samples, works great, but. works only when resized image is saved to a different file. My problem is that I would like to replace the original image with resized one - with the sane location and name.
when I set output blob to be the same as input blob, it is triggered over and over again without the finish.
is there any way I could change blob using azure function and store result in the same file?
The easiest option is to accept two invocations for the same file, but add a check of the size of the incoming file. If the size is already OK, do nothing and quit without changing the file again. This should break you out of the loop.
Blob trigger uses Storage Logs to watch for new or changed blobs. It then compares the changed blob against Blob Receipts in a container named azure-webjobs-hosts in the Azure storage account. Each receipt has ETag associated with it, so when you change a blob, the ETag changes and the Blob is submitted to the function again.
Unless you want to go fancy and update ETag's in receipts from within a function (not sure if it's feasible), your changed files will go for re-processing.
Related
I have a logic app that is triggered when there is a change to the blob, this works fine but what if I want this process to run and overwrite these files is this possible. I seem to lose all dynamic options as soon as I change the bloc modified trigger
I am not sure if I understand your question. But if you just want a scheduled job kind of process to pick or put files from/to azure blob storage, you can use blob actions rather than trigger. You can use a 'Recurrence' trigger to start the logic app and use one of the appropriate blob actions to do the required operation. Let me know if you are looking for something else.
Using Azure ADF and currently my setup is as follows:
Event based triggerd on a input BLOB on file upload. File upload triggers a copy activity to output BLOB, and this action is followed by a delete operation on the input BLOB. The input BLOB can take 1 or many files at once (not sure how often the file is scanned/how quickly the event triggers the pipeline). Reading up on the delete function documentation it says:
Make sure you are not deleting files that are being written at the same time.
Would my current setup delete files that are being written?
Event based trigger on file upload >> Write from input Blob to Output Blob >> Delete input Blob
I've made an alternative solution which does a get metadata activity based on event in the beginning of the pipeline, and then does a for loop which deletes the files at the end, not sure if this is necessary though. Would my original solution suffice in an unlikely event where I'm receiving files every 15seconds or so?
Also while I'm at it, in a get metadata activity how can I get the actual path to the file, not just the file name?
Thank you for the help.
Delete Active says:
Make sure you are not deleting files that are being written at the
same time.
Your settings are:
Event based trigger on file upload >> Write from input Blob to Output
Blob >> Delete input Blob
Only after the active Write from input Blob to Output Blob finished(the deleting files are not being written), then the Delete input Blob can works.
Your questions: Would my current setup delete files that are being written?
So did you test these steps? You must test by yourself and you will get the answer.
Please notice:
Delete activity does not support deleting list of folders described by wildcard.
Any other suggestions:
You don't need to using delete actives to delete the input blob after Write from input Blob to Output Blob finished.
You can learn from Data flow, it's Source settings support delete the source file(input blob) after the copy active completed.
Hope this helps.
I could not use Leon Yue solution because my source dataset was a sftp one, which is not supported by Azure dataflows.
To deal with this problem, I used the Filter by last modified of the dataset. I set the End Time to the time the pipeline has started.
With this solution, only the files added to the source before the pipeline started will be consumed by both the copy and delete activities.
In Azure blob upload, a file is overwritten if you upload a new file with the same file name (in the same container).
I would like to rename the new file before saving it, to avoid overwriting any files - Is this possible?
Scenario:
Upload file "Image.jpg" to container "mycontainer"
Upload file "Image.jpg" to container "mycontainer" (with different content)
Rename second "Image.png" to "Image_{guid}.jpg" before saving it to "mycontainer".
You cannot rename a blob (there's no API for it). Your options:
check if blob name exists, prior to uploading, and choosing a different name for your about-to-be-uploaded blob if the name is already in use
simulate rename by copying existing blob to new blob of different name, then deleting original blob
As #juunas pointed out in comments: You'd have to manage your workflow to avoid potential race condition regarding checking for existence, renaming, etc.
I recommend using an "If-None-Match: *" conditional header (sometimes known as "If-Not-Exists" in the client libraries). If you include this header on either your PutBlob or PutBlockList operations, the call will fail and data will not be overwritten. You can catch this client-side and retry the upload operation (with a different blob name.)
This has two advantages over checking to see if the blob exists before uploading. First, you no longer have the potential race condition. Second, calling Exists() adds a lot of additional overhead - an additional HTTP call for every upload, which is significant unless your blobs are quite large or latency doesn't matter. With the access condition, you only need multiple calls when the name collides, which should hopefully be a rare case.
Of course, it might be easier / cleaner to just always use the GUID, then you don't have to worry about it.
Needing to rename may be indicative of an anti-pattern. If your ultimate goal is to change the name of the file when downloaded, you can do so and keep the blob name abstract and unique.
You can set the http download filename by assigning ContentDisposition property with
attachment;filename="yourfile.txt"
This will ensure that the header is set when the blob is accessed either as public or a SAS url.
I am using Azure Storage Nodejs and what i need to do is to copy image from one blob to another.
First i tried to getBlobToFile to get the image on temp location in disk and then just createBlockBlobFromFile from that temp location. That method did the task, but for some reason it didn't copied completely in 10% of cases.
The i was trying to use getBlobToText and the result of that put into createBlockBlobFromText, also tried to put options which is need blob to be image container. That method failed completely, image not even opened after copy.
Perhaps there is a way to copy blob file and paste it in other blobl but i didn't find that method.
What else can i do?
I'm not sure what your particular copy-error is, but... with getLocalBlobToFile(), you're actually physically moving blob content from blob storage to your VM (or local machine), and then with createBlockBlobFromLocalFile() you're pushing the entire contents back to blob storage, which is resulting in two physical network moves.
The Azure Storage system supports blob-copy as a 1st-class operation. While it's available via REST API call, it's also wrapped in the same SDK you're using, in the method BlobService.startCopyBlob() (source code here). This will instruct the storage to initiate an async copy operation, completely within the storage system (meaning no download+upload on your side). You'll be able to set source and destination, set timeouts, etc. (all parameters are fully documented in the source code).
The link in the accepted answer is broken, although the method is correct: the method startCopyBlob is documented here
(Updated: Jan 3, 2020) https://learn.microsoft.com/en-us/javascript/api/azure-storage/BlobService?view=azure-node-latest#azure_storage_BlobService_createBlockBlobFromLocalFile
(The old link) https://learn.microsoft.com/en-us/javascript/api/azure-storage/BlobService?view=azure-node-latest#azure_storage_BlobService_createBlockBlobFromLocalFile
if i upload a file on azure blob in the same container where the file is existing already, it is over-writing the file, how to avoid overwriting the same? below i am mentioning the scenario...
step1 - upload file "abc.jpg" on azure in container called say "filecontainer"
step2 - once it gets uploaded, try uploading some different file with the same name to the same container
Output - it will overwrite existing file with the latest uploaded
My Requirement - i want to avoid this overwrite, as different people may upload files having same name to my container.
Please help
P.S.
-i do not want to create different containers for different users
-i am using REST API with Java
Windows Azure Blob Storage supports conditional headers using which you can prevent overwriting of blobs. You can read more about conditional headers here: http://msdn.microsoft.com/en-us/library/windowsazure/dd179371.aspx.
Since you want that a blob should not be overwritten, you would need to specify If-None-Match conditional header and set it's value to *. This would cause the upload operation to fail with Precondition Failed (412) error.
Other idea would be to check for blob's existence just before uploading (by fetching it's properties) however I would not recommend this approach as it may lead to some concurrency issues.
You have no control over the name your users upload their files with. You, however, have control over the name you store those files with. The standard way is to generate a Guid and name each file accordingly. The chances of conflict is almost zero.
A simple pseudocode looks like this:
//generate a Guid and rename the file the user uploaded with the generated Guid
//store the name of the file in a dbase or what-have-you with the Guid
//upload the file to the blob storage using the name you generated above
Hope that helps.
Let me put it that way:
step one - user X uploads file "abc1.jpg" and you save it io a local folder XYZ
step two - user Y uploads another file with same name "abc1.jpg", and now you save it again in a local folder XYZ
What do you do now?
With this I am illustrating that your question does not relate to Azure in any way!
Just do not rely on original file names when saving files. Where-ever you are saving them. Generate random names (GUIDs for example) and "attach" the original name as meta-data.