I'm uploading files from a Raspberry Pi to Azure Blob storage using an Azure IoT hub, using this microsoft tutorial as the basis for my C# code, and it's working fine.
Looking at the Microsoft documentation for the method UploadToBlobAsync(), "If the blob already exists, it will be overwritten."
I'm wondering if there's any way to restrict the device's permissions to create-only in the Azure portal or via PowerShell. My concern is that should someone access the device's storage and get the device id and key they would have the means to delete or overwrite files previously uploaded by that device in the storage container.
As a work-around I could have a server-side process pick up files once they've been received and move them elsewhere, but if the device id/key was restricted to create-only then I wouldn't need this overhead.
The method UploadToBlobAsync (assembly Microsoft.Azure.Devices.Client.UWP) is a wrapper of the REST API sequence calls for uploading a blob to the Azure Storage container.
The following sequence is processed:
REST API call to the Azure IoT Hub to obtain a reference for uploading blob, see the following screen snippet:
As you can see in the above picture, the sasToken for this operation has been generated for read/write.
Once the device received the above response, the REST API PUT the blob can be called.
Here is my suggestion. The device can call REST API Get the metadata of the blob, see the following screen snippet:
Based on the above result, this sequence can be either skipped or continue for actually uploading blob using the REST API PUT.
This is a last step of the sequence (very important). The device need to send a notification to the Azure IoT Hub with the status of the uploading sequence. The following screen snippet shows this REST API call:
Well, as you can see the above step #2 can decide about the skipping or overwriting the upload blob process.
Related
I have video files arriving into Azure blob storage.
When the file arrives, I want to invoke a REST api and pass the file in the api's POST method.
What is the Azure blob storage setting where-in I can configure the api call trigger (when new file arrives) and specify that the file needs to be POST'ed?
What is the Azure blob storage setting where-in I can configure the
api call trigger (when new file arrives) and specify that the file
needs to be POST'ed?
There are many ways to accomplish this.
One possible solution is to make use of Blob Triggered Azure Function. The Function will be triggered when a blob is created. In your Function code, you can invoke your API and post the blob data.
Other option would be to use Azure Logic Apps where you can define a workflow that gets invoked when a blob is created.
I have a NodeJS app that can successfully send data to the Azure IoT Hub, which in turn sends the data to the Azure Blob Storage.
That all works fine.
I can also manually send a message with the Message to device tool of the Azure Devices portal and I can receive this message in my NodeJS app.
What I want to do now, is to send a confirmation message back to the NodeJS app when the blob has been created or deleted.
Can someone please guide me on how to do that? There is little information out there to send messages back to the Azure Client.
I see that I can create a subscription to the Azure Blob Storage but I don't know how to hook it up to the Azure IoT Hub.
Cheers
This is possible by using an Azure Function with a Blob Storage trigger. You could write an Azure Function that subscribes to blob storage changes and use the IoT Hub Service SDK to send a message back to the device. I'm assuming you're using IoT Hub's message routing feature to store the telemetry in blob storage. This comes with a challenge, as there is no way to tell from the name of the blob what device it originated from. You would need to combine it with a blob input to read the file input.
Do you absolutely need this confirmation on the device side? Another note, if you're not interested in persisting telemetry, but instead want to upload a file from your NodeJS app, you could consider the IoT Hub File Upload feature instead.
You can use a device twins to represent a state between the device and service sides.
Updating a state (device twin) at the service side, the notification message is delivered to the device side.
I see log messages in azure iot device client source code like this:
log.debug("Connection already opened by TransportClient."); or
log.info("Device client opened successfully");
My question is where these log messages going? how to get that messages for debug purpose?
Thanks
In general, Blob Storage is added as a 'logging endpoint' which shall encompass a storage account, container in the account and blob in the container. The blobs of type 'Block blobs' shall be utilized for storing text and binary data.
All logs get stored in 'Block blobs' in a container named $logs, which is automatically created when Storage Analytics is enabled for a storage account. The $logs container is located in the blob namespace of the storage account, for example: http://.blob.core.windows.net/$logs
To view and analyze your log data, you should download the blobs that contain the log data you are interested in to a local machine. There are many tools like AzCopy, Azure Storage Data Movement library, Azure Import/Export service to import or export data to and from your storage account. To view the logs, you can also use any tool that can access Azure blob storage, such as Visual Studio or Cerebrata Azure Management Studio.
In case of azure-iot-sdk, each IoT hub exposes a set of endpoints(service endpoints) for the solution's back end to communicate with the devices. An IoT hub has a default built-in-endpoint (messages/events). By default, messages are routed to the built-in service-facing endpoint (messages/events) that is compatible with Event Hubs. You can refer to below link to various methods to read from built-in endpoint https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-read-builtin
You can also create custom endpoints to route messages to by linking other services in your subscription to the IoT Hub. In case if the custom endpoint is created, a message is routed to multiple endpoints whose routing queries it matches. There are two storage services IoT Hub can route messages to Azure Blob Storage and ADLS (Azure Data Lake Storage) Gen2 accounts. You can refer to the below link to various methods to read from custom endpoint https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-read-custom
In the scenario of logs from IoT SDK itself, the logs can get logged in stdout or stderr as per type of log and based on the deployment environment and also can be redirected accordingly as per the requirement. The SDK uses debug library for detailed logs. The below link can be helpful in giving additional details https://github.com/Azure/azure-iot-sdk-node/wiki/Troubleshooting-Guide-Devices
Is it possible to use some ready made construct in azure cloud environment to ingest the events (in json format) that are currently stored in azure blob storage and have it submit those events directly to azure event hub without writing any (however small) custom code? In other words, I would like to use configuration driven approach only.
Sure. You can try to use Azure Logic Apps to realize your needs without any code or just with some function expressions, please refer to the offical documents of Azure Logic Apps to know more details.
The logic flow is as the figure below.
You can refer to my sample below to make it works.
Here is my sample to receive an event from my EventHub and transfer to Azure Blob Storage to create a new blob for storing the event data.
Create an Azure Logic App instance on Azure portal, it should be easy for you.
Move to the tab Logic app designer to configure the logic flow.
Click Save and Run buttons. Then, use ServiceBusExplorer (downloaded from https://github.com/paolosalvatori/ServiceBusExplorer/releases) to send event message and check whether new blob created using AzureStorageExplorer. It works fine after a few minutes.
I have successfully used this feature to upload files to a storage container in Azure blob storage. I wanted to ask if this form of file upload has the same integrity checks using MD5 hash as there is with normal blob storage uploading. This link describes that feature for blob storage. Also, it seems this check is optional, if so, is there a way for me to ensure that this happens when I upload from my iot device using the azure-iot-sdk's.
I have tested this issue with Azure IoT SDK for CSharp. When uploading the file to azure iot hub, I uses fiddler to catch the http request. Actually, the headers in the request contains Content-MD5. So if you use Azure IoT SDK for CSharp, you need not to check the option for MD5. And you can refer to the method UploadFromStreamAsync in Microsoft Azure Storage SDK for .NET, this method is called when upload file via UploadFromStreamAsync method.
Update:
MD5 cannot be calculated for an existing blob because it would require reading the existing data. Please disable storeBlobContentMD5.