Stream Analytics Job -> DataLake ouput - azure

Question

I want to set up CI/CD (ARM template) with StreamAnalytics Job with output set to DataLake Store.
https://learn.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs/outputs#microsoftdatalakeaccounts
The issue comes with refreshToken:
"It is recommended to put a dummy string value here when creating the data source
and then going to the Azure Portal to authenticate the data source
which will update this property with a valid refresh token"
Furthermore after 90-days refresh token is outdated and you need to do "Renvew Authorization"
https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-data-lake-output#renew-data-lake-store-authorization
I tried to authorize ServicePrincipal.
How to do automatic deployment for ASA with DataLake?
How to handle issue with this 90-days token validitiy?
Maybe you wiped the trail :)

Answer 1

at this time it is not yet possible. Sorry for the inconvenience. However we know this is very important and we will add Service Principal auth in the near future (we are looking at the exact ETA).
In the meantime you need to renew manually the token. This can be done without losing any data by (1) stopping the job, (2) changing the token, then (3) restarting the job last time it was stopped.
Let me know if you have any further question.

Answer 2

As far as I know quite soon MSI-based authentication will be in preview.
But if you need an immediate solution (to e.g. be able to have a VSTS pipeline running through without error) you can do the following:
Create template (e.g. with the CICD NuGet Package [1])
Manipulate the ARM Template <jobName>.JobTemplate.json
Add the output datasource object for the ADLS output object
If you work with Visual Studio you can get the values quite easily from the ADLS output JSON
It is important to set refreshToken to some fake value
Like the following:
"outputs": [
{
"name": "xxx",
"properties": {
"serialization": {
"type": "Json",
"properties": {
"encoding": "UTF8",
"format": "LineSeparated"
}
},
"datasource": {
"type": "Microsoft.DataLake/Accounts",
"properties": {
"accountName": "xxx",
"tenantId": "xxx-xxx-xxx-xxx-xxx",
"tokenUserPrincipalName": "xxx#xxx.com",
"tokenUserDisplayName": "xxx, xxx",
"filePathPrefix": "xxx/{date}/{time}",
"dateFormat": "yyyy/MM/dd",
"timeFormat": "HH",
"refreshToken": "faketoken"
}
}
}
},
...
Deploy the ARM Template
The job will start successfully but it is necessary to renew the token, therefore
Stop the job
Renew the authentication of the ADLS output
Start the job
Resources
[1] CICD NuGet Package