Azure Cognitive Services Read Request with Powershell - azure

I'm trying to use the Azure Cognitive Services API with Powershell to 'read' the contents of a .jpg in a blob store. Everything I am trying to do works perfectly using the Azure API demo/test page, so I am fairly sure that this, to a degree, proves that some elements I'm using in my code are valid. Well, at least they are when using the API testing tool.
Here is my Powershell:
Clear-Host
$myUri = "<ENDPOINT value from "keys and endpoint blade">/vision/v3.0/read/analyze?language=en"
$imagePath = "<path to image in blob. accessible online and anonymously>"
$subKey = "<KEY #1 from "keys and endpoint" blade>"
$headersHash = #{}
$headersHash.Add( "Host", "westeurope.api.cognitive.microsoft.com" )
$headersHash.Add( "Ocp-Apim-Subscription-Key", $subKey )
$headersHash.Add( "Content-Type","application/json" )
$bodyHash = #{ "url" = $imagePath }
out-host -InputObject "Sending request:"
$response = Invoke-WebRequest -uri $myUri `
-Method Post `
-Headers $headersHash `
-Body $bodyHash `
-verbose
"Response: $response"
When I send that, all I ever get is a:
Invoke-WebRequest : The remote server returned an error: (400) Bad Request.
At C:\scratch\testy.ps1:15 char:13
+ $response = Invoke-WebRequest -uri $myUri `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-W
ebRequest], WebException
+ FullyQualifiedErrorId :
WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
I must be missing something basic but I cannot see what. There are no published examples of using Powershell to access the CS APIs that I can find but there is an example with Python (requests) and I'm pretty sure I'm emulating and representing what goes into the Python example correctly. But then again, its not working so something isn't right.
Strangely, when I try to recreate this in Postman, I get a 202 but no response body, so its not possible for me to view or extract the apim-request-id in order to manufacture the next request to retrieve the results.

Found the issue. The problem was made clear when I wrapped the call in a try/catch block and put this in the catch block:
$streamReader =
[System.IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
$ErrResp = $streamReader.ReadToEnd() | ConvertFrom-Json
$streamReader.Close()
I was then able to look at the contents of the $ErrResp variable and there was a fragment of a string which said "unable to download target image.." or something to that effect. Odd because I could use the URL I was supplying to instantly connect to and get the image.. so it had to be the way the URL was being injecting into the body.
It was.
When using a hashtable as the body, where your Content-Type is 'application/json' all you need to do, it seems, is use convertto-json with your hash first. This worked and I instantly got my 202 and the pointer to where to collect my results.
Hopefully this will help someone, somewhere, sometime.

Related

How to correctly pass data between two Azure PowerShell Functions

I have two PowerShell Azure functions. The first one collects data and the second one pushes data to an Azure Storage Table. I can see my first function is calling the second function but the second function isn't getting the data. Here is my first function:
$body = #{"partitionKey"="01";"rowKey"="02";"userId"="00001"}
$dataJson = $body | ConvertTo-Json
$functionUri = 'https://functionNumber2'
Invoke-WebRequest -Uri $functionUri -Method POST -Body $dataJson
And my second function looks like this:
using namespace System.Net
# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)
$tableData = $Request.Body.Address
# Write to the Azure Functions log stream.
Write-Host "PowerShell HTTP trigger function processed a request."
# Interact with query parameters or the body of the request.
$name = $Request.Query.Name
if (-not $name) {
$name = $Request.Body.Name
}
Write-Host "Name $name"
$body = "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response."
if ($name) {
$body = "Hello, $name. This HTTP triggered function executed successfully."
}
# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]#{
StatusCode = [HttpStatusCode]::OK
Body = $body
})
Push-OutputBinding -Name tableName -Value $tableData -Clobber
The error I'm getting is ERROR: Cannot bind argument to parameter 'Value' because it is null. Exception : Type : System.Management.Automation.ParameterBindingValidationException Message : Cannot bind argument to parameter 'Value' because it is null. And if I print out $tableData there is nothing there. I am doing something wrong when passing the data.
Can anyone help me with this?
After reproducing from my end I observed that you are receiving this error because of the below line.
$tableData = $Request.Body.Address
I could able to make this work by using the below line.
$tableData = $Request.Body | ConvertFrom-Json
RESULTS:

Azure Devops Run Result Steps and Run Summary Details Attachments API

I am trying to fetch the attachments of the Test Run Result steps and Test Run Result Summary Details but unable to find any API related to that
I am attaching the Image below Rectangle 1 is the attchments for Test Run Result Steps and Rectangele 2 is the attachments for Test Run Result Summary Detais
If anyone have any knowledge about these perticular apis please let me know.
I have checked the AZURE API Documentation but couldn't find the specific API if I have missied something please let me know.
Thanks
By calling the Get Test Result Attachments REST API, we can get all the IDs of the attachments:
GET https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/Results/{testCaseResultId}/attachments?api-version=6.0-preview.1
After that, if you want to get the attachments you can call Attachments - Get Test Result Attachment Zip REST API with the specific Attachment ID.
GET https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/Results/{testCaseResultId}/attachments/{attachmentId}?api-version=6.0-preview.1
Please note that the REST API Attachments - Get Test Result Attachment Zip will display the context of the attachments instead of download the attachments directly. If you want to download the attachments, you can write a script to save them to a local directory. The following PowerShell script for your reference:
$AttachmentsOutfile = "D:\Test\HellWorld.java"
$connectionToken="You PAT Here"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::
ASCII.GetBytes(":$($connectionToken)"))
$AuditLogURL = "https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/Results/{testCaseResultId}/attachments/{attachmentId}?api-version=6.0-preview.1"
$AuditInfo = Invoke-RestMethod -Uri $AuditLogURL -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get –OutFile $AttachmentsOutfile
UPDATE:
However the Get Test Result Attachments REST API can only get the attachments attached from the test run UI (attached by clicking the Add attachment button).
To get the attachments of the Test Run Result steps and Test Run Result Summary, we can call Results - Get REST API with parameter detailsToInclude=iterations added:
GET https://dev.azure.com/{organization}/{project}/_apis/test/Runs/{runId}/results/{testCaseResultId}?detailsToInclude=iterations&api-version=6.0
After that we can download the attachments by their ID. The following PowerShell script for your reference to download them in a loop:
Param(
[string]$orgurl = "https://dev.azure.com/{org}",
[string]$project = "Test0924",
[string]$downloadlocation = "C:\temp\1025\",
[string]$TestRunId = "1000294",
[string]$ResultId = "100000",
[string]$user = "",
[string]$token = "PAT"
)
# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
#List test result and test step attachments:
$testresultUrl = "$orgurl/$project/_apis/test/Runs/$TestRunId/Results/$($ResultId)?detailsToInclude=iterations&api-version=6.0"
$attachments = (Invoke-RestMethod -Uri $testresultUrl -Method Get -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}).iterationDetails.attachments
ForEach ($attachment in $attachments) {
#Get test result and step attachments:
$attachmentid = $attachment.id
$attachmentname = $attachment.name
$attachmenturl = "$orgurl/$project/_apis/test/Runs/$TestRunId/Results/$ResultId/attachments/$($attachmentid)?api-version=6.0"
Invoke-RestMethod -Uri $attachmenturl -Method Get -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -OutFile $downloadlocation\$attachmentname
}

How to use Azure App Configuration REST API

Trying to figure out how to use Azure AppConfiguration REST API (mostly to retrieve and create key-values). So far I found two sources of information: Configuration Stores REST API docs and this GitHub repo Azure App Configuration.
How these two sources are corresponding with each other? They apparently describe some different AppConfig REST API.
I managed to retrieve values from my AppConfig store using this type of URI and AAD authorization
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.AppConfiguration/configurationStores/{configStoreName}/listKeyValue?api-version=2019-10-01 But it allows to get only one value of one particular key.
The other approach uses URI based on AppConfig endpoint {StoreName}.azconfig.io/kv/... and must have more flexible ways to retrieve data. But I can't make it work. I tried to follow instructions. And I tried to make a request to this URI using AAD token as I did for the first type of API. In both cases I get 401 auth error.
Could anyone share some detailed working examples (Powershell, Postman)? Any help would be appreciated.
https://management.azure.com/ is the Azure Resource Management API, while the azconfig.io one is App Configuration's own API.
I think you should use App Configuration's own API. The same Azure AD token will not work for this API however. You need to request another access token with resource=https://yourstorename.azconfig.io or scope=https://yourstorename.azconfig.io/.default, depending if you use v1 or v2 token endpoint of Azure AD.
Use the $headers in the script to authenticate your api calls:
function Sign-Request(
[string] $hostname,
[string] $method, # GET, PUT, POST, DELETE
[string] $url, # path+query
[string] $body, # request body
[string] $credential, # access key id
[string] $secret # access key value (base64 encoded)
)
{
$verb = $method.ToUpperInvariant()
$utcNow = (Get-Date).ToUniversalTime().ToString("R", [Globalization.DateTimeFormatInfo]::InvariantInfo)
$contentHash = Compute-SHA256Hash $body
$signedHeaders = "x-ms-date;host;x-ms-content-sha256"; # Semicolon separated header names
$stringToSign = $verb + "`n" +
$url + "`n" +
$utcNow + ";" + $hostname + ";" + $contentHash # Semicolon separated signedHeaders values
$signature = Compute-HMACSHA256Hash $secret $stringToSign
# Return request headers
return #{
"x-ms-date" = $utcNow;
"x-ms-content-sha256" = $contentHash;
"Authorization" = "HMAC-SHA256 Credential=" + $credential + "&SignedHeaders=" + $signedHeaders + "&Signature=" + $signature
}
}
function Compute-SHA256Hash(
[string] $content
)
{
$sha256 = [System.Security.Cryptography.SHA256]::Create()
try {
return [Convert]::ToBase64String($sha256.ComputeHash([Text.Encoding]::ASCII.GetBytes($content)))
}
finally {
$sha256.Dispose()
}
}
function Compute-HMACSHA256Hash(
[string] $secret, # base64 encoded
[string] $content
)
{
$hmac = [System.Security.Cryptography.HMACSHA256]::new([Convert]::FromBase64String($secret))
try {
return [Convert]::ToBase64String($hmac.ComputeHash([Text.Encoding]::ASCII.GetBytes($content)))
}
finally {
$hmac.Dispose()
}
}
# Stop if any error occurs
$ErrorActionPreference = "Stop"
$uri = [System.Uri]::new("https://{myconfig}.azconfig.io/kv?api-version=1.0")
$method = "GET"
$body = $null
$credential = "<Credential>"
$secret = "<Secret>"
$headers = Sign-Request $uri.Authority $method $uri.PathAndQuery $body $credential $secret
Sauce: https://github.com/Azure/AppConfiguration/blob/master/docs/REST/authentication/hmac.md#JavaScript

getting an error on running a powershell command

I am getting this following error:
exception calling "UploadString" with "2" argument(s): "The remote server returned an error: (500)
Internal Server Error." At C:\Cinegy_Type\Helpers-Type.ps1:146 char:5 + $web.UploadString($url,
$xmlDoc.OuterXml) + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [],
MethodInvocationException + FullyQualifiedErrorId : WebException
The method is this:
function Type-UpdateVariable([string]$name, [string]$value, [string]$type = "Text", [string]$server="localhost", [int]$instance=0)
{
#make a Type PostRequest XML document using .Net XML document object
$xmlDoc = New-Object System.Xml.XmlDocument;
#add root element for request - which is a 'PostRequest' element
$xmlRootElem = $xmlDoc.AppendChild($xmlDoc.CreateElement('PostRequest'));
#create SetValue element and define variable Name, Type and Value
$xmlSetValueElem = $xmlRootElem.AppendChild($xmlDoc.CreateElement('SetValue'));
$xmlSetValueElem.SetAttribute("Name", $name);
$xmlSetValueElem.SetAttribute("Type",$type);
$xmlSetValueElem.SetAttribute("Value", $value);
#create a .Net webclient which will be used to perform the HTTP POST
$web = new-object net.webclient
#Air requires that the data is in XML format and declared properly - so add the HTTP Header to state this
$web.Headers.add("Content-Type", "text/xml; charset=utf-8")
#perform the actual HTTP post to the IP and port (which is 5521 + instance number) of the XML data
$url = "http://" + $server + ":" + (5521 + $instance) + "/postbox"
$web.UploadString($url, $xmlDoc.OuterXml)
}
Line 146 is:
$web.UploadString($url, $xmlDoc.OuterXml)
what I do not understand is that this was working yesterday.
The full code can be found here: (Do not use line number here for reference, this has 3 scripts.)
https://pastebin.com/RtisgmiB
The web server you're posting to is having a problem; see meaning of a HTTP status code 500. Unless the data passed to your function are problematic, your code looks fine. There is nothing you're missing in that Powershell error message, which would give you a clue as to the issue.
Check the error log on the web server (best option)
Output the parameters to the console with the purpose of submitting the request manually through Postman, cURL, etc.
Good luck!

How to configure Webhook activity for runbooks execution in Azure Data Factory v2?

I am working on running runbooks (powershell and graphical) from ADF. One of the ways I found to accomplish this task is to use webhooks. I will have runbooks running in parallel and in series (if dependency exists on previous runbook).
Overall,
If a flat file is dropped in Azure Blob storage then it triggers the pipeline that contains respective runbook(s). This part is working.
The webhook of runbook(s) are used in ADF webhook activity. This is where I am facing the problem. I am unsure about what should be in the body of webhook activity?
After some research I was able to find something about Callback uri that needs to be added (or somehow generated) in the body of the webhook. How can I get this Callback uri? If I don't add proper callback uri then the activity runs till timeout. I believe the functioning should be webhook activity completes when the runbook it's running is executed successfully so we can move on to next webhook activity in a pipeline. I have tried web activity as well but it's the same issue.
The body I am using right now is just below json.
{"body":{"myMessage":"Sample"}}
I have referenced:
https://vanishedgradient.com/2019/04/25/webhooks-with-azure-data-factory/
https://mrpaulandrew.com/2019/06/18/azure-data-factory-web-hook-vs-web-activity/
https://social.msdn.microsoft.com/Forums/en-US/2effcefb-e65b-4d5c-8b01-138c95126b79/in-azure-data-factory-v2-how-to-process-azure-analysis-service-cube?forum=AzureDataFactory
Thanks for the links, they are useful sources. I've managed to get this working for a pipeline that calls a runbook to resize azure analysis services. Having the runbook return failure and success information was not well documented.
Here's some code to assist a little, which i've taken from several places but a lot from the open issue (https://github.com/MicrosoftDocs/azure-docs/issues/43897) on this Microsoft page: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-webhook-activity
The datafactory Webhook activity passes in some "Headers", SourceHost which is #pipeline().DataFactory and SourceProcess which is #pipeline().Pipeline. This was so we can do some checking to confirm that the runbook is being run by acceptable processes.
The Body of the call is then other variables we required:
#json(concat('{"AnalysisServer":"', pipeline().parameters.AASName, '", "MinimumSKU":"', pipeline().parameters.SKU,'"}') )
Your runbook needs the WebhookData parameter
param
(
[Parameter (Mandatory=$false)]
[object] $WebhookData
)
You can then grab all the bits you need, including checking if a callbackuri was provided:
if ($WebhookData)
{
# Split apart the WebhookData
$WebhookName = $WebhookData.WebhookName
$WebhookHeaders = $WebhookData.RequestHeader
$WebhookBody = $WebhookData.RequestBody | Convertfrom-Json
$WebhookADF = $WebhookHeaders.SourceHost
$WebhookPipeline = $WebhookHeaders.SourceProcess
Write-Output -InputObject ('Runbook started through webhook {0} called by {1} on {2}.' -f $WebhookName, $WebhookPipeline, $WebhookADF)
# if there's a callBackURI then we've been called by something that is waiting for a response
If ($WebhookBody.callBackUri)
{
$WebhookCallbackURI = $WebhookBody.callBackUri
}
...
}
The variable $WebHookHeaders: #{Connection=Keep-Alive; Expect=100-continue; Host=sXXevents.azure-automation.net; SourceHost=**MYDATAFACTORYNAME**; SourceProcess=**MYPIPELINENAME**; x-ms-request-id=**UNIQUEIDENTIFIER**}
You can then grab information out of your json body: $AzureAnalysisServerName = $WebHookBody.AnalysisServer
Passing an error/failure back to your runbook is relatively easy, note that I put my success/update message in to $Message and only have content in $ErrorMessage if there's been an error:
$ErrorMessage = "Failed to do stuff I wanted"
if ($ErrorMessage)
{
$Output = [ordered]#{ output= #{
AzureAnalysisServerResize = "Failed" }
error = #{
ErrorCode = "ResizeError"
Message = $ErrorMessage
}
statusCode = "500"
}
} else {
$Output = [ordered]#{
output= #{
"AzureAnalysisServerResize" = "Success"
"message" = $Outputmessage
}
statusCode = "200"
}
}
$OutputJson = $Output | ConvertTo-Json -Depth 10
# if we have a callbackuri let the ADF Webhook activity know that the script is complete
# Otherwise it waits until its timeout
If ($WebhookCallBackURI)
{
$WebhookCallbackHeaders = #{
"Content-Type"="application/json"
}
Invoke-WebRequest -UseBasicParsing -Uri $WebhookCallBackURI -Method Post -Body $OutputJson -Header $WebhookCallbackHeaders
}
I then end the if ($WebhookData) { call with an else to say the runbook shouldn't be running if not called from webhook:
} else {
Write-Error -Message 'Runbook was not started from Webhook' -ErrorAction stop
}
Passing back an error message was quiet easy, passing back a success message has been traumatic, but the above seems to work, and in my datafactory pipeline i can access the results.
Output
{
"message": "Analysis Server MYSERVERNAME which is SKU XX is already at or above required SKU XX.",
"AzureAnalysisServerResize": "Success"
}
Note that with the Invoke-WebRequest, some examples online don't specify -UseBasicParsing but we had to as the runbook complained: Invoke-WebRequest : The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer's first-launch configuration is not complete.
I'm not sure if this is best practice but I have something that is working in a Powershell Workflow Runbook.
If the runbook has a webhook defined then you use the webhookdata parameter. Your request body needs to be in JSON format and the $WebhookData param picks it up. For example supposed the Body in your webhook activity looks like this:
{"MyParam":1, "MyOtherParam":"Hello"}
In your runbook you pick up the parameters this way:
Param([object]$WebhookData)
if($WebhookData){
$parameters=(ConvertFrom-Json -InputObject $WebhookData.RequestBody)
if($parameters.MyParam) {$ParamOne = $parameters.MyParam}
if($parameters.MyOtherParam) {$ParamTwo = $parameters.MyOtherParam}
}
The variables in your runbook $ParamOne and $ParamTwo are populated from the parsed JSON Body string. The data factory automatically appends the callBackUri to the Body string. You don't need to create it.
You have to use the $WebhookData name. It's a defined property.
I hope this helps.
Apologies for delay. I had found the complete solution few months back. Thanks to Nick and Sara for adding the pieces. I used similar code as return code. We were using graphical runbooks with limited changes allowed so I just added return code (Powershell) at the end of the runbook to have little to no impact. I plugged in below code:
if ($WebhookData)
{
Write-Output $WebhookData
$parameters = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)
if ($parameters.callBackUri)
{
$callbackuri = $parameters.callBackUri
}
}
if ($callbackuri)
{
Invoke-WebRequest -Uri $callbackuri -UseBasicParsing -Method POST
}
Write-Output $callbackuri
After this I added an input parameter using "Input and Output" button available in the runbook. I named the input parameter as "WebhookData" and type as "Object". The name of input parameter is case-sensitive and should match the parameter used in Powershell code.
This resolved my issue. The runbook started when called from ADF pipeline and moved to next pipeline only when the underlying runbook called by the webhook was completed.

Resources