We've an Azure based solution that tracks user actions on a SharePoint web site. A SharePoint extension produces a JSON payload that is sent to an API Management resource with the task to log it to Event Hub.
Useless I describe next parts of the architecture since will go beyond the purpose of the question.
We're used to deploy the whole solution automating DevOps with a custom software.
In the last deploy the APIM seems unable to work properly. The request took about 3 minutes and ends with 502 - Web server received an invalid response while acting as a gateway or proxy server.
I get this error using postman where it should normally return a mocked 200 after logging to Event Hub.
This is the XML policy used:
<set-body>#{
var body = context.Request.Body.As<JObject>();
body.Add(new JProperty("event_id", Guid.NewGuid().ToString()));
body.Add(new JProperty("ip_address", context.Request.IpAddress));
body.Add(new JProperty("ingest_status", "unknown"));
return body.ToString();
}</set-body>
<log-to-eventhub logger-id="pagesLogger">#(
context.Request.Body.As<string>(preserveContent: true)
)</log-to-eventhub>
<mock-response status-code="200" content-type="application/json" />
This is the top-level ALL APIs policy:
<cors>
<allowed-origins>
<origin>*</origin>
</allowed-origins>
<allowed-methods>
<method>GET</method>
<method>POST</method>
<method>OPTIONS</method>
</allowed-methods>
<allowed-headers>
<header>*</header>
</allowed-headers>
</cors>
The weird thing is that also commenting log-to-eventhub I get an error response: 500 - The request timed out.
I tried to create another API in the same APIM resource but I'm still unable to get a 200 from a mocked response. Also creating a brand new APIM resource it's not able to solve the problem.
Enabling application insights on APIM did not help to get further informations. The same also with Event Hub diagnostics.
I don't expect a solution but I would really appreciate hints that help me diagnose the problem.
It was an Azure issue. Without changing anything after two days everything
is back to normality
Related
Using the Azure portal, I’m unable to send test requests to the Echo API (and all other backend APIs).
When sending a request, I’m getting the following error:
HTTP/1.1 401 Access Denied
cache-control: private, s-maxage=0
content-length: 152
content-type: application/json
date: Tue, 12 Apr 2022 05:13:28 GMT
vary: Origin
www-authenticate: AzureApiManagementKey realm="https://AAAA.azure-api.net/echo",name="Ocp-Apim-Subscription-Key",type="header"
{
"statusCode": 401,
"message": "Access denied due to missing subscription key. Make sure to include subscription key when making requests to an API."
}
The request works fine when I tick the “Bypass CORS proxy” checkbox and through Postman.
I have the following global inbound CORS policy:
<policies>
<inbound>
<cors allow-credentials="true">
<allowed-origins>
<origin>https://AAAA.developer.azure-api.net</origin>
<origin>https://AAAA.azure-api.net</origin>
</allowed-origins>
<allowed-methods preflight-result-max-age="300">
<method>*</method>
</allowed-methods>
<allowed-headers>
<header>*</header>
</allowed-headers>
<expose-headers>
<header>*</header>
</expose-headers>
</cors>
</inbound>
<backend>
<forward-request />
</backend>
<outbound />
<on-error />
</policies>
and the inbound base policy set on the Echo API.
I haven't expereinced this problem previously. Any ideas how I can bupass the CORS error while submitting test request in the APIM portal?
HTTP/1.1 401 Access Denied
www-authenticate: AzureApiManagementKey realm="https://AAAA.azure-api.net/echo",name="Ocp-Apim-Subscription-Key",type="header"
"message": "Access denied due to missing subscription key. Make sure to include subscription key when making requests to an API."
In this Troubleshooting Steps of Unauthorized errors (401) while invoking APIs in Azure, it is mentioned clearly:
Due to Wrong Ocp-Apim-Subscription-Key, this error occurs.
When you create the APIM, the Echo API is subscribed to built-in subscriptions by default.
Each subscription has two subscription keys that can be used.
Scenario 1:
By default, Echo API is registered to the Built-in all-access subscription so it will work perfectly until the subscription key is matched:
Scenario 2:
There are 2 more product subscriptions that come by default when an APIM instance is created which are Starter and Unlimited.
When the API is subscribed to that product subscriptions, then the subscription key passing in the header should match with the Original Product Subscription Keys available in the Subscriptions Menu.
Here, the Echo API is subscribed to both the products Starter and Unlimited as shown in 1st Image.
That Product Subscriptions has given with some permissions called Administrators, Developers and Guests. Any one among these should have on the user to access the APIs subscribed these products.
In the 3rd Image, you can see what APIs are subscribed to Starter Product like Echo API.
If any of the above workaround did not solve the issue, please refer the troubleshooting steps doc provided that shows all of the causes that produces this specific error 401 Unauthorized and Missing the Subscription Key along with the resolution.
ByPass CORS option allows the requests originating from any domain. Sometime, allowing the cross-domain access also fixes the issue. Try Azure Cross-domain policy as given below to allow access from 'any' domain (you can specify your domain too).
<cross-domain>
<cross-domain-policy>
<allow-http-request-headers-from domain='*' headers='*' />
</cross-domain-policy>
For details refer MS documentation on managing cross domain access : https://learn.microsoft.com/en-us/azure/api-management/api-management-cross-domain-policies
I have engaged with Microsoft on this and they are investigating the issue. As per MS initial investigation “Microsoft Defender for Cloud Apps" creates a Proxy that intercepts all requests going out of Azure portal and it seem like MCAS proxy is either removing or modifying headers from the outgoing request thus causing this behaviour. Microsoft has pointed to the following document for reference: Troubleshooting - What is cas.ms, mcas.ms, or mcas-gov.us?. MS has advised that they don’t have any ETA for the fix and that they are investigating further. Their recommendation is to check the Bypass CORS proxy option as workaround for the time being.
We're trying to use a client certificate to authenticate when calling an OData service in SAP S/4HANA. And we're calling from an azure APIM instance. As certificate we've created a self-signed certificate and configured SAP S/4HANA according to this blog (https://blogs.sap.com/2020/05/03/x.509-certificate-based-logon-to-odata-services/)
Then we test this from the browser it works like a charm.
But calling from azure APIM we get the following response from SAP S/4HANA:
<?xml version="1.0" encoding="utf-8"?> <error xmlns:xsi="http://www.w3.org/2001/XMLSchema-Instance">
<code>HTTP/404/E/Not Found</code>
<message> Service /sap/opu/odata/sap/xxxxyyyy/xxyyzz call was terminated because the corresponding service is not available.The termination occurred in system UFI with error code 404 and for the reason Not found. Please select a valid URL. If it is a valid URL, check whether service /sap/opu/odata/sap/xxxxyyyy/xxyyzz is active in transaction SICF. If you do not yet have a user ID, contact your system administrator. </message>
SAP S/4HANA support says that then calling from browser they can 'see' certificate in payload but then calling from APIM, the payload is 'empty'.
I've got the trace logs from the SAP S/4HANA gateway server and I've noticed this subtly difference calling from browser vs calling from APIM:
Browser call (successfull):
[Thr 140708195055360] HttpModGetDefRules: determined the defactions: COPY_CERT_TO_MPI (1)
APIM call (failed):
[Thr 140708197697280] HttpModGetDefRules: determined the defactions: NOTHING (0)
So the certificate is obviously reaching SAP S/4HANA gateway server but not the SAP S/4HANA Odata server. So somehow, for some reason it's lost on the SAP S/4HANA gateway server only then it comes from azure APIM.
I've tried to make the calls 100% identical (same headers same values) but I can't control the way the certificate is added in azure apim or can one ?
I read that one can set the certificate body using policy below:
<authentication-certificate body="#(context.Variables.GetValueOrDefault<byte[]>("byteCertificate"))" password="optional-certificate-password" />
but I can't figure out how to get a proper value for "byteCertificate".
Has anyone done this? Or has anyone had a similar issue?
We finally found the solution!
Thanks to microsoft APIM support team, thanks a lot :)
APIM acts like a reverse proxy and adds headers related to this role. The header "X-Forwarded-For" causes SAP to deny the request with the above misleading error message. We found a solution that SAP could configure:
ICM parameter "icm/HTTPS/accept_ccert_for_x_forwarded_for_requests" has to be set to "true" - per default it's set to "false".
(The header can't be deleted with a policy on APIM side.)
I am having difficulty adding text into my url path between my frontend and backend when using Azure Api Management.
I have a Azure functions api with a url like this:
e.g. https://pXXXXX-myapi.azurewebsites.net/api/pXXXXX/alm/{name}
The default api created by Azure API Management is this:
e.g. https://myapi.azure-api.net/pXXXXX/pXXXXX/alm/{name}
Ideally I'd like to Frontend to remove the redundant pXXXXX:
e.g. https://myapi.azure-api.net/pXXXXX/alm/{name}
If I remove it from the front end e.g. GET /pXXXXX/alm/{name} becomes GET /alm/{name}.
Then I receive a 500 error.
I assume this is because removing "/pXXXXX" removes it from the backend path as well.
So I have tried to use the rewrite-uri policy like so:
<rewrite-uri template="/pXXXXX/alm/{name}" copy-unmatched-params="false" />
I've tried multiple variations of this with no success.
Assuming
you've changed the frontend for your operation to "GET /alm/{name}"
and your backend function app service url is "https://pXXXXX-myapi.azurewebsites.net/api"
and your API Base URL is "https://myapi.azure-api.net/pXXXXX"
then rewrite-uri exactly like you pasted is correct. You want to add leading "pXXXXX" to the url.
I've reproduced exact same scenario on my side and it works.
500 error does not mean there is something wrong with the path. If you would wrongly configure your url you would get 404. Most probably there is an internal error inside your function app or other policies in the APIM. Just call this API with Ocp-Apim-Trace header set to true and subscription key that allows tracing and you will see what is wrong. Or you can paste trace here if you want so we can help.
Posting my own answer but thanks to those who tried to help.
I believe the issue came from an error from APIM itself.
I assume it was not setting up the policies correctly, because now it is automatically including set-backend-service in the inbound policy whereas it did not over the past 2 days when I have been repeatably re-creating my api.
Now my inbound policy looks like this:
<inbound>
<base />
<set-backend-service id="apim-generated-policy" backend-id="pXXXXX-myapi" />
<rewrite-uri template="/pXXXXX/alm/{name}" copy-unmatched-params="false" />
</inbound>
And my front end api looks as before:
/pXXXXX/alm/{name}
I have an Azure App Service hosting an OData endpoint that is behind an Azure API Management (APIM) instance. To prevent calling the App Service directly it is protected by a certificate that only the APIM has.
When I call the APIM URL through Chrome or Postman, it behaves as expected. Just one request with no redirects or funny business, and it returns the OData root.
Here is a Fiddler log of a request to the APIM using Postman
However, when using the same URL as an OData source in Power Query using OData.Feed(), it returns a 301 which forwards to the backend URL, which obviously fails because that URL is protected by a certificate. Here is a Fiddler log of a request to the APIM using Power Query in Excel
I've configured the subscription key to be passed in the headers, but I've also tried it as a query param and it doesn't work in Power Query either way. I've also tried using an OData entity endpoint directly (to avoid the $metadata call) with no luck.
The user agent Power Query uses is Microsoft.Data.Mashup, but I haven't found any documentation about its compatibility with APIM, but that shouldn't matter, right?
In typical fashion after working on this for two days, I discovered the answer right after posting on StackOverflow. I'll leave this question up in case anyone has the same issue.
The problem was that the Power Query connector automatically follows #odata.context links for metadata, and #odata.nextLink links for paging. These links still had the app service site as the host instead of the APIM host.
So a quick edit of the outbound rules in APIM was able to fix the issue
<outbound>
<base />
<set-variable name="backendBaseUrl" value="#(context.Request.OriginalUrl.Scheme + "://" + context.Request.OriginalUrl.Host.ToString() + context.Api.Path)" />
<find-and-replace from="#("http://" + context.Request.Url.Host.ToString())" to="#((string)context.Variables["backendBaseUrl"])" />
<find-and-replace from="#("https://" + context.Request.Url.Host.ToString())" to="#((string)context.Variables["backendBaseUrl"])" />
</outbound>
Here I have to rules to replace http and https URLs just in case some configuration changes.
I am using Azure API Management as front end for my Logic App. The "subscription required" setting needs to be enabled as we do need the protection. However, we must send the key via query parameter because our calling application only supports GET, not POST.
So my API call was sent to Azure using the format of https://my.azure-api.net/myapi/manual/paths/invoke?subscription-key=mykey
Now in Azure API setting I did create a policy set to delete action on the "subscription-key" query parameter, but here's the problem:
Even though the parameter is removed from the request body into Logic App, upon digging into the "RAW" outputs in Logic App where it shows various headers, we can see the subscription-key in these two headers:
"X-WAWS-Unencoded-URL": "/myapi/manual/paths/invoke?subscription-key=xxx
"X-Original-URL": /myapi/manual/paths/invoke?subscription-key=xxx
In other words, the full original query URL was made available to Logic App before the parameter was removed. This exposes the API subscription key to the Logic App.
Is there any workaround for this?
Ah I see now that those headers were actually sent automatically by Azure API Management to the backend Logic App API, so all I had to do was to set header policies to remove them in addition to the query parameter policy.
<set-query-parameter name="subscription-key" exists-action="delete" />
<set-header name="X-WAWS-Unencoded-URL" exists-action="delete" />
<set-header name="X-Original-URL" exists-action="delete" />
This takes care of it.