Azure : How to write path to get a file from a time series partitioned folder using the Azure logic apps - azure

I am trying to retrieve a csv file from the Azure blob storage using the logic apps.
I set the azure storage explorer path in the parameters and in the get blob content action I am using that parameter.
In the Parameters I have set the value as:
concat('Directory1/','Year=',string(int(substring(utcNow(),0,4))),'/Month=',string(int(substring(utcnow(),5,2))),'/Day=',string(int(substring(utcnow(),8,2))),'/myfile.csv')
So during the run time this path should form as:
Directory1/Year=2019/Month=12/Day=30/myfile.csv
but during the execution action is getting failed with the following error message
{
"status": 400,
"message": "The specifed resource name contains invalid characters.\r\nclientRequestId: 1e2791be-8efd-413d-831e-7e2cd89278ba",
"error": {
"message": "The specifed resource name contains invalid characters."
},
"source": "azureblob-we.azconn-we-01.p.azurewebsites.net"
}
So my question is: How to write path to get data from the time series partitioned path.

The response of the Joy Wang was partially correct.
The Parameters in logic apps will treat values as a String only and will not be able to identify any functions such as concat().
The correct way to use the concat function is to use the expressions.
And my solution to the problem is:
concat('container1/','Directory1/','Year=',string(int(substring(utcNow(),0,4))),'/Month=',string(int(substring(utcnow(),5,2))),'/Day=',string(int(substring(utcnow(),8,2))),'/myfile.csv')

You should not use that in the parameters, when you use this line concat('Directory1/','Year=',string(int(substring(utcNow(),0,4))),'/Month=',string(int(substring(utcnow(),5,2))),'/Day=',string(int(substring(utcnow(),8,2))),'/myfile.csv') in the parameters, its type is String, it will be recognized as String by logic app, then the function will not take effect.
And you need to include the container name in the concat(), also, no need to use string(int()), because utcNow() and substring() both return the String.
To fix the issue, use the line below directly in the Blob option, my container name is container1.
concat('container1/','Directory1/','Year=',substring(utcNow(),0,4),'/Month=',substring(utcnow(),5,2),'/Day=',substring(utcnow(),8,2),'/myfile.csv')
Update:
As mentioned in #Stark's answer, if you want to drop the leading 0 from the left.
You can convert it from string to int, then convert it back to string.
concat('container1/','Directory1/','Year=',string(int(substring(utcNow(),0,4))),'/Month=',string(int(substring(utcnow(),5,2))),'/Day=',string(int(substring(utcnow(),8,2))),'/myfile.csv')

Related

ADF: can't build simple Json file transformation (one field flattening)

I need a help in transforming simple json file inside Azure Data Flow. I need to flatten just one field date_sk in example here:
{
"date_sk": {"string":"2021-09-03"}
"is_influencer": 0,
"is_premium": -1,
"doc_id": "234"
}
Desired transformation:
"date_sk": {"string":"2021-09-03"}
to become
"dateToGroupBy" : "2021-09-03"
I create source stream, note the strange projection Azure picks, there is no "string" field anymore, but this is how automatic Azure transformation works for some reason:
Data preview of the same source stream node:
And here's how it suggest me to transform it in a separate "Derived Column" modifier. I played with the right part, but this is the only format (date_sk.{}) that does not display any error I was able to pick:
But then output dateToGroupBy field happens to be empty:
Any ideas on what could got wrong and how can I build the expected transformation? Thank you
Alright, it happened to be a Microsoft bug in ADF.
ADF stumbles upon "string" field name as JSON field, can't handle it, though schema and data validation passes through Ok, and showing no errors.
When I replace date_sk": {"string":"2021-09-03"} by date_sk": {"s1":"2021-09-03"} or anything other than string everything starts working just fine
and dateToGroupBy is filled with date values taken from date_sk.s1
When I return string back, it shows NULL in output values.
It supposed to either show error on verification stage or handle this field naming properly.

Azure Resource Manager template chained functions

I am trying to remove / from the URL using azure function before assigning to output variable value
"webappStorageUri":{
"type": "string",
"value": "[take(reference(resourceId('Microsoft.Storage/storageAccounts', variables('webappStorageName'))).primaryEndpoints.web, length(reference(resourceId('Microsoft.Storage/storageAccounts', variables('webappStorageName'))).primaryEndpoints.web)-1]"
}
Returned value from length function should be the value for take function. This is not working. I get following error on deployment. I don't get anything out of this error message. Does Azure support chained function execution? Is this is right approach to remove / from the URL?
Error message
[error]Deployment template language expression evaluation failed: 'Unable to parse language expression 'take(reference(resourceId('Microsoft.Storage/storageAccounts', variables('webappStorageName'))).primaryEndpoints.web, length(reference(resourceId('Microsoft.Storage/storageAccounts', variables('webappStorageName'))).primaryEndpoints.web)-1': expected token 'RightParenthesis' and actual 'Integer'.'. Please see https://aka.ms/arm-template-expressions for usage details.
I'm not sure what you are trying to achieve, but your function has issues with brackets, and you cannot really substract by appending -1 in a random place.
"[take(reference(variables('webappStorageName')).primaryEndpoints.web,
sub(length(reference(variables('webappStorageName')).primaryEndpoints.web), 1))]"
line breaks for readability only

Azure Logic App - How to upload file to Azure Blob Storage from byte array

I am trying to create a Logic App that is triggered by an HttpRequest that contains as payload a JSON request. Inside of this JSON, one field contains the file:
{
"EntityId": "45643",
"SharedGuid": "00000000-0000-0000-0000-000000000000",
"Document": {
"DocumentName": "MyFileName.pdf",
"File": "JVBERi0xLjMKJfv8/f4KMS.....lJUVPRg=="
}}
This "file" content is being generated by the customer application by using the following C# function: File.ReadAllBytes("local path here").
I managed to upload the byte array to blob storage. But the file is not valid once it is uploaded in the Blob Storage.
I tried different file contents for the file in the JSON schema definition as: string, binary, application/octet-stream.
Any help will be appreciated.
Did you do the operation to convert the byte to Base64String in your httprequest code, just like the code below:
byte[] b = File.ReadAllBytes(#"filepath");
string s = Convert.ToBase64String(b);
According to the file content you provided, it seems you have convert it to base64string as above, so I provide the solution below:
For this requirement, you can just parse the response data as string(do not need to use "binary" in schema) in your "Parse JSON" action and then use base64ToBinary() method in the "Create blob" action, please refer to the screenshot shown as below:
The expression in "Blob content" is:
base64ToBinary(body('Parse_JSON')?['Document']?['File'])
Hope it helps~
If still have any problem, please feel free to let me know.

UsageAggregates API of azure is giving a blank response with continuation token

I called below AZURE API from postman.
https://management.azure.com/subscriptions/{SubscriptionID}/providers/Microsoft.Commerce/UsageAggregates?api-version=2015-06-01-preview&reportedstartTime=2019-12-29T00%3a00%3a00%2b00%3a00&reportedEndTime=2019-12-30T00%3a00%3a00%2b00%3a00&$top=1
I got a response with empty value field along with some nextlink. when I again called the API with Nextlink URL response was having blank value field.
{
"value": [],
"nextLink": somelink
}
I am able to get proper response using same API for some different subscription.
Empty Array shows that it doesn;t have any usage details available for requested time frame.
Firstly as suggested by Tony, check if you have the usage resource for the requested date.
**Important**
Please note that the dateTime value format must be URL encoded as ISO 8601 format, and non-numeric characters must use escape codes (i.e. colon is escaped to %3a, plus sign is escaped to %2b) so that it is URI friendly. These refer to the start and end time ranges of your query. This dateTime parameter must also be specified in Universal Time Coordinated (UTC).
Secondly , Set {aggregationGranularity} to ‘Hourly’. This is an optional parameter with two discrete potential values: Daily and Hourly. As the values suggest, the former one returns the data in daily granularity whereas the latter one is hourly resolution. Daily is the default.
Sample URI:
**https://management.azure.com/subscriptions/{subscription-Id}/providers/Microsoft.Commerce/UsageAggregates?api-version=2015-06-01-preview\&reportedStartTime=2014-05-01T00%3a00%3a00%2b00%3a00\&reportedEndTime=2015-06-01T00%3a00%3a00%2b00%3a00\&aggregationGranularity=Hourly\&showDetails=f**
Hope it helps.
Make sure that you have enough permissions for the subscription in question.
Empty result [] can be returned when the permissions are missing. There is no error shown in such case.
The easiest way to ensure the permissions are available is to use Contributor or Owner role.
Additionally, Get-UsageAggregates -Debug (from Az PowerShell module) can be used to check how the request URL should look like.
Get-UsageAggregates -ReportedStartTime "2022-05-24 10:00" -ReportedEndTime "2022-05-24 15:00" -Debug -AggregationGranularity Hourly

Azure Resource Manager - Convert value to 'lower'

I was recently using ARM templates to deploy multiple resources into Azure. While deploying Storage accounts, I ran into an issue which was due to some constraints put up by Azure like
Name of Storage Account should not contain upper case letters
Its max length should be 24.
I want this name from the user and can handle the 2nd issue using the "maxLength" property on 'parameters'. But for lower case, there is no such property in 'parameters' also I'm unable to find any function which will convert the user entered value to lower case.
What I expect:
Method to convert the user entered value in lower case.
Any other method to suit my use case.
Thanks in advance.
You should look at the string function reference of the ARM templates.
you need to create a variable (or just add those functions to the name input, like so:
"name": "[toLower(parameters('Name'))]"
or add a substring method, something like this:
"variables": {
"storageAccountName": "[tolower(concat('sawithsse', substring(parameters('storageAccountType'), 0, 2), uniqueString(subscription().id, resourceGroup().id)))]"
},

Resources