I'm in the process of creating an Azure Logic App to work with Abbyy's OCR REST API.
I use the Create SAS URI by path action which returns Web URL. Web URL returns the FQDN, incuding the SAS token, to my blob.
Web URL is passed to an Http action as a query parameter. In code view, the relevant part of the JSON looks like:
"method": "GET",
"uri": "https://cloud.ocrsdk.com/processRemoteImage?source=#{body('Create_SAS_Uri_by_path')?['WebUrl']}&language=English&exportformat=pdfSearchable&description=blah&imageSource=scanner"
The uri resolves thus:
https://cloud.ocrsdk.com/processRemoteImage?source=https://mysaaccount.blob.core.windows.net/inbox/180730110047_0001.pdf?sv=2017-04-17&sr=b&sig=2IGMt1qDZthaBSyvD3WJ6T1zc36Wr%2FNoiB4Wki5Lf28%3D&se=2018-08-16T11%3A16%3A48Z&sp=r&language=English&exportformat=pdfSearchable&description=blah&imageSource=scanner"
This results in the error (450):
<?xml version="1.0" encoding="utf-8"?><error><message language="english">Invalid parameter: sr</message></error>
Which is basically picking up sr= query parameter from the SAS token and of course, the API doesn't have an sr argument, and even if it did, its value would be wrong.
I did find this question and I attempted "percent-escape" the ampersands (&) by adjusting my code to use the replace function, thus:
"method": "GET",
"uri": "https://cloud.ocrsdk.com/processRemoteImage?source=#{replace(body('Create_SAS_Uri_by_path')?['WebUrl'],'%26','%2526' )}&language=English&exportformat=pdfSearchable&description=blah&imageSource=scanner"
However, this has no effect. I.e the resulting URI is the same as above. Interestingly, it appears the SAS token itself has made use of "percent-escape".
If anyone has any suggestion on how to resolve or work around this problem, I'd be most greatful if you would share your thoughts.
Does anyone know if the LogicApp actions are opensource and if so what the GitHub link is. I can then raise an issue.
Resolved it.
Basically, I was on the right track, but I used %26 when I should have used &, so using the code above, it should read:
"method": "GET",
"uri": "https://cloud.ocrsdk.com/processRemoteImage?source=#{replace(body('Create_SAS_Uri_by_path')?['WebUrl'],'&','%26' )}&language=English&exportformat=pdfSearchable&description=blah&imageSource=scanner"
And therefore, the URI reads:
https://cloud.ocrsdk.com/processRemoteImage?source=https://mysaaccount.blob.core.windows.net/intray/180730110047_0001.pdf?st=2018-08-17T10%3A55%3A38Z%26se=2018-08-18T10%3A55%3A38Z%26sp=r%26sv=2018-03-28%26sr=b%26sig=FTRoVgV7MRz5d5gTgrEs6D0QSy3268BqscZX1LHbJYQ%3D&language=English&exportformat=pdfSearchable&description=blah&imageSource=scanner
Next step: convert XML body into JSON...
Update 1
I have updated (17/08/2018) the replace with value from %2526 to %26. I was clearly getting my knickers in a twist. Must have been trying to double-escape, which isn't needed.
Although Microsoft seem to partially percent-encode the SAS token, I note they percent-encode = with %3D, however, the Abbyy API doesn't seem to care about =. (Testing from Postman).
Update 2
Not sure why Update 1 did work. It could be because I set the access policy to blob (anonymous read access for blobs only), and setting it back to private hadn't taken effect, or it might be the wrong content-type. Anyway, it worked and then it didn't. Well, it was replacing the & with %26 without issue, and I tried escaping the = too, but the endpoint didn't like it, testing via Postman and Logic App.
My trigger is actually a C# BlobTrigger* Azure Function App, and I have written code to generate the SAS token, so I used dotnet's Uri.EscapeDataString() method:
string fullUrl = cloudBlobContainer.Uri + "/" + blobName + sasToken;
string escapedUri = Uri.EscapeDataString(fullUrl);
log.LogInformation($"Full URL: {fullUrl}");
log.LogInformation($"Escaped URL: {escapedUri}");
return escapedUri;
Update 3
Just seen this line, created by the Logic App designer:
"path": "/datasets/#{encodeURIComponent(encodeURIComponent('https://someaccount.sharepoint.com/sites/eps'))}/tables//items//attachments",
Looks like encodeURIComponent() might be a function to do the same thing as Uri.EscapedDataString and would replace replace() function. I haven't tested it yet.
*the reason I am using a function app for the trigger is down to cost. Although there is a Logic App trigger that runs when a new blob is detected, it has to run to a schedule. My plan is consumption, and I have read that MS charge when a Logic App runs, regardless of whether it does anything or not. IMHO, it is inefficient to have a task triggering every 5 seconds, when 90% of the time there isn't anything for it to do. The function app is better suited to my requirements, although there is apparently a ~10 minute "warm-up" period, if the app has gone to sleep. I can live with that.
Related
I would like to use the Microsoft Azure Cognitive Service Speech-to-text. It offers a REST API, which I succesfully have used. I can point to an Azure blob storage using a SAS URI, and the files in the container are transcribed.
My problem is, that when I try to retrieve the transcription results from the API, they are published to a public url. Since voice data can be sensitive, I would like to keep the results stored privately. Is there any way to do this?
I does not seem like it is an option in the API schema, although you can set a destinationContainerUrl. I have tried to set the destinationContainerUrl, but the result does not appear in the container.
I have only used the API reference, which is why I am not posting any code.
You've found the correct option. Using destinationContainerUrl will write the results into this container. Make sure you provide a container SAS which allows listing and writing.
When the job succeeds, the results should be there. Please check that status of your job, maybe it was set to failed.
Documentation about transcriptions:
https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/batch-transcription
If the job succeeds and the files are not on this container, please let us know the $.self link of the job and the creation time, to help us gathering the logs.
Ok. So the solution was super simple. I just did the post request json wrong. destinationContainerUrl needs to be under properties, as shown below:
{"contentUrls": ["LINK-TO-BLOB-SAS-URI-TOKEN"],
"properties": {
"diarizationEnabled": false,
"wordLevelTimestampsEnabled": false,
"punctuationMode": "DictatedAndAutomatic",
"profanityFilterMode": "Masked",
"destinationContainerUrl": "LINK-TO-BLOB-SAS-URI-TOKEN"
},
"locale": "en-US",
"displayName": "Transcription from blob to blob"
}
We have many dozens of build pipelines and we want to pause and resume (re-enable) build pipelines from a simple webapp interface as we are making config changes frequently. Here is the MS doc explaining this API:
https://learn.microsoft.com/en-us/rest/api/azure/devops/build/builds/update%20build?view=azure-devops-rest-5.0#definitionqueuestatus
From this documentation, it appears I need to hit the REST API and change/toggle the DefinitionQueueStatus -- however, this documentation only shows a sample for a build specific operation, whereas I want to pause then re-enable the entire build pipeline. What is the proper way to make this call?
I'm using fetch - and I've tried many dozen formats in the call - the 'ourorg' and 'ourproject' are correct (we use this call structure for many other calls), but all fails for this call below. I grabbed the 'definitionID' from the URL I can visibly see when in the Azure devops portal on the specific build pipeline page, and I'm using it for the {buildID} as I don't know what else to put there. Any guidance to help here is appreciated - I don't need to use fetch btw - any working sample will help here:
fetch(https://dev.azure.com/our_org/our_projectname/_apis/build/builds/definitionId=1593?retry=true&api-version=5.0 {
method: 'PATCH ',
credentials: 'same-origin',
body: 'DefinitionQueueStatus: "Enabled"'
}).then(function(response) {
console.log(response);
})
It seems that the body is incorrect in your post. Here is sample about how to use POSTMAN to access Azure DevOps Services REST APIs.
Generate the PAT, and then record the token, it is important to use to authorization, please see this document.
Create a new request in POSTMAN, it is recommended to put the request in a collection for Azure DevOps Services REST API;
Select the authorization as Basic Auth, you can input the username as any value, and the password as the token which is generated in step1.
Basic Auth
Set the REST API which you want to use,and select the request method type(GET,POST,FETCH ....), here you use https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}?api-version=5.0.
In the Body tab, you can set the request body as raw in json format, and input the value as following:
{
"buildNumber":"#20190607.2",
"buildNumberRevision":1,
"definition":
{
"id":1,
"createdDate":null,
"queueStatus":"paused"
}
}
Everthing is ready now, you can send the request, if sccuess, you will get the response from the REST API.
In your post, the body content is incorrect, the Request Body should meet the format in the REST API document. The DefinitionQueueStatus is a type in definitions. In addition, if you send the request with parameter retry, you will get the message The request body must be empty when the retry parameter is specified..
I am creating an ASP.NET MVC5 action method that implements a password reset endpoint and accepts a click-through from an email message containing a token. My implementation uses OWIN middleware and closely resembles the ASP.NET Identity 2.1 samples application.
As per the samples application, the token is generated by UserManager and embedded into a URL that is sent to the user by email:
var token = await UserManager.GeneratePasswordResetTokenAsync(user.Id);
var encoded = HttpServerUtility.UrlTokenEncode(Encoding.UTF8.GetBytes(token));
var uri = new Uri(Url.Link("ResetPasswordRoute", new { id = user.Id, token = encoded }));
The link in the email message targets an MVC endpoint that accepts the token parameter as one of its route segments:
[Route("reset-password/{id}/{token}"]
public async Task<ActionResult> PasswordResetAsync(int id, string token)
{
token = Encoding.UTF8.GetString(HttpServerUtility.UrlTokenDecode(token));
// Implementation here
}
However, requests to this endpoint (using a URL generated in the above manner) fail with Bad Request - Invalid URL.
It appears that this failure occurs because the URL is too long. Specifically, if I truncate the token segment, it connects correctly to the MVC endpoint (although, of course, the token parameter is no longer valid). Specifically, the following truncated URL works ...
http://localhost:53717/account/reset-password/5/QVFBQUFOQ01uZDhCRmRFUmpIb0F3RS9DbCtzQkFBQUFzcko5MEJnYWlrR1RydnVoY2ZwNEpnQUFBQUFDQUFBQUFBQVFaZ0FBQUFFQUFDQUFBQUNVeGZZMzd4OTQ3cE03WWxCakIwRTl4NkVSem1Za2ZUc1JxR2pwYnJSbmJ3QUFBQUFPZ0FBQUFBSUFBQ0FBQUFEcEpnVXFXS0dyM2ZPL2dQcWR1K2x6SkgxN25UVjdMYlE2UCtVRG4rcXBjU0FBQUFE
... but it will fail if one additional character is added ...
http://localhost:53717/account/reset-password/5/QVFBQUFOQ01uZDhCRmRFUmpIb0F3RS9DbCtzQkFBQUFzcko5MEJnYWlrR1RydnVoY2ZwNEpnQUFBQUFDQUFBQUFBQVFaZ0FBQUFFQUFDQUFBQUNVeGZZMzd4OTQ3cE03WWxCakIwRTl4NkVSem1Za2ZUc1JxR2pwYnJSbmJ3QUFBQUFPZ0FBQUFBSUFBQ0FBQUFEcEpnVXFXS0dyM2ZPL2dQcWR1K2x6SkgxN25UVjdMYlE2UCtVRG4rcXBjU0FBQUFEf
I believe that the default IIS configuration setting for maxUrlLength should be compatible with what I am trying to do, but I have also tried explicitly setting it to a larger value, which did not solve the problem.
However, using Fiddler to examine the server response, I can see that the working URL generates a server response with the following header ...
Server: Microsoft-IIS/8.0
... whereas the longer URL is rejected with a response containing the following header ...
Server: Microsoft-HTTPAPI/2.0
This seems to imply that the URL is not being being rejected by IIS, but by a middleware component.
So, I am wondering what that component might be and how I might work around its effect.
Any suggestions please?
Many thanks,
Tim
Note: Although my implementation above Base64 encodes the token before using it in the URL, I have also experimented with the simpler approach used in the sample code, which relies on the URL encoding provided by UrlHelper.RouteUrl. Both techniques suffer from the same issue.
You should not be passing such long values in the application path of the URL as they are limited in length to something like 255 characters.
A slightly better alternative is to use a query string parameter instead:
http://localhost:53717/account/reset-password/5?token=QVFBQUFOQ01uZDhCRmRFUmpIb0F3RS9DbCtzQkFBQUFzcko5MEJnYWlrR1RydnVoY2ZwNEpnQUFBQUFDQUFBQUFBQVFaZ0FBQUFFQUFDQUFBQUNVeGZZMzd4OTQ3cE03WWxCakIwRTl4NkVSem1Za2ZUc1JxR2pwYnJSbmJ3QUFBQUFPZ0FBQUFBSUFBQ0FBQUFEcEpnVXFXS0dyM2ZPL2dQcWR1K2x6SkgxN25UVjdMYlE2UCtVRG4rcXBjU0FBQUFEf
That should be safe for at least 2000 characters (full URL) depending on the browser and IIS settings.
A more secure and scalable approach is to pass a token inside an HTTP header.
I'm just in the process of experimenting with the Azure rest api's in preparation for some work I'm heading into...
Within node.js I'm trying to obtain a list of webspaces hosted by my subscription using the following code...
var websiteOptions = {
hostname: 'management.core.windows.net',
path: '/<<Subscription-ID>>/services/webspaces',
method: 'GET',
pfx: new Buffer(managementCert, 'base64'),
strictSSL: true,
headers: {
'x-ms-version': '2013-06-01'
}
};
websiteOptions.agent = new https.Agent(websiteOptions);
var request = https.request(websiteOptions,function(res){
console.log('Status Code:', res.statusCode);
console.log('Headers:', res.headers);
When executed it returns back with..
<Error xmlns="http://schemas.microsoft.com/windowsazure"
xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<Code>AuthenticationFailed</Code>
<Message>The server failed to authenticate the request. Verify that the certificate is valid and is associated with this subscription.
</Message>
</Error>
The strange thing is that if I replace ..
/<<subscription-ID>>/services/webspaces
with...
/<<subscription-ID>>/services/hostedservices
That specific call works like a charm, returning a list of hosted services - so this leads me to believe that I'm encoding/attaching my management certificate correctly.
I have tried various api versions in the headers without much luck, I have also read various sections of this article without much luck.
So why are my credentials working in one section of the API and not the websties section
What am I missing, I'm sure it's something really simple :|
OK so now I feel daft, and it just goes to show that even big companies can get it wrong..
I have been playing a bit more and decided to drop a forward slash onto the end of the API call for the web spaces endpoint.. and...
Bingo... it worked like a treat!!!
Would have been nice if Microsoft was consistent across it's API endpoints, the hosted services end point for instance didn't need this trailing slash.
Also, accurate error messages would have been nice too, instead of just reporting an invalid credentials exception!!!
As always.. the simple things in life :(
I am trying to make a simple REST call to the Set Blob Properties API (http://msdn.microsoft.com/en-us/library/windowsazure/hh452235) to just turn off/on logging. I have gotten the REST API call to successfully work for retrieving Blob Properties, so I know my hashing algorithms, headers-setting, and Authentication signature creation works, but I can't seem to get it working on the Set Properties side of things. I keep getting an error on the Authentication Header, so I know I'm not doing something right there.
I have copied below what is being created and eventually hashed and put into the auth header string. The online documentation (http://msdn.microsoft.com/en-us/library/windowsazure/dd179428) does not really help in determining which of these fields are absolutely required for this particular type of Blob request, so I've tried filling most of them in, but I don't seem to get a difference response regardless of what I fill in. I've also tried the Shared Key Lite authentication, which would be preferred since it's much more lightweight, but that doesn't seem to work either when I fill in all 5 of those fields.
Shared Key Authentication for Blob Services:
PUT\n
\n
\n
130\n
(MD5_CONTENT_HASH)
\n
\n
\n
\n
\n
\n
\n
x-ms-date:Tue, 19 Jun 2012 19:53:58 GMT\n
x-ms-version:2009-09-19\n
/(MY_ACCOUNT)/\n
comp:properties\n
restype:service
Is there anything obvious I'm missing here? The values (MD5_CONTENT_HASH) and (MY_ACCOUNT) are of course filled in when I make the request call, and the similar request call to "GET" the properties works fine when I send it. The only difference between that one and this is that I'm sending the MD5_content, along with the content-length. I may be missing something obvious here, though.
Any advice would be greatly appreciated! Thanks in advance.
-Vincent
EDIT MORE INFO:
Programming Language I'm using: Objective-C (iOS iPhone)
I'm also using ASIHTTPRequest to make the request. I simply define the request, setRequestMethod:#"PUT", then I create the request body and convert it to NSData to calculate the length. I attach the request-body data via the appendPostData method to the request. I then build the auth string above, hash the whole thing, and attach it to the request as a header called "Authorization".
Request Body String I'm using:
<?xml version=\"1.0\" encoding=\"utf-8\"?><StorageServiceProperties><Logging><Version>1</Version></Logging></StorageServiceProperties>
I know this is an incomplete request body, but I was planning on waiting for it to give a failure on "missing request body element" or something similar, until I proceeded on creating the full XML there. (could that be my issue?)
Error I get from the server:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:accc4fac-2701-409c-b1a7-b3a528ce7e8a
Time:2012-06-20T14:36:50.5313236Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request '(MY_HASH)' is not the same as any computed signature. Server used following string to sign: 'POST
130
x-ms-date:Wed, 20 Jun 2012 14:36:50 GMT
x-ms-version:2009-09-19
/(MY_ACCOUNT)/
comp:properties
restype:service'.</AuthenticationErrorDetail></Error>
What's odd is that the error I get back from the server seems to look like that, no matter how many parameters I pass into the Authentication signature.
Thanks for any help you can offer!
Comparing your signed string and the error message indicates that you're sending a POST request but signing as though you're sending a PUT.