We configured a function proxy for our website approximately 2 months ago and got everything working as expected. Last night around 8:00-8:30pm EST the proxy stopped working, receiving "Internal server error" 500 messages when accessing it's endpoints. We haven't changed anything on our end so I don't know why this all of a sudden started.
We proxy our domain to various endpoints. The endpoint that stopped working are proxies to pages we are hosting on GitHub pages. Proxies to other services such as other Azure App service instances are still working.
I ran a Proxy-Trace-Enabled: true request to the proxy and find the following error in the trace log:
"backend": [
{
"source": "forward-request",
"timestamp": "2018-01-31T01:45:36.4810022Z",
"elapsed": "00:00:00.0037370",
"data": {
"message": "Request is being forwarded to the backend service.",
"request": {
"method": "GET",
"url": "https://xxxxxxxxxx.github.io/xxxxxxxxxx/",
"headers": [
{
"name": "Cache-Control",
"value": "no-cache"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip"
},
{
"name": "Cookie",
"value": "__cfduid=xxxxxxxxxx"
},
{
"name": "Max-Forwards",
"value": "10"
},
{
"name": "User-Agent",
"value": "PostmanRuntime/7.1.1"
},
{
"name": "CF-IPCountry",
"value": "US"
},
{
"name": "X-Forwarded-For",
"value": "xxxxxxxxxx, xxxxxxxxxx, xxxxxxxxxx"
},
{
"name": "CF-RAY",
"value": "xxxxxxxxxx-MIA"
},
{
"name": "X-Forwarded-Proto",
"value": "https"
},
{
"name": "CF-Visitor",
"value": "{\"scheme\":\"https\"}"
},
{
"name": "Postman-Token",
"value": "xxxxxxxxxx"
},
{
"name": "CF-Connecting-IP",
"value": "xxxxxxxxxx"
},
{
"name": "X-WAWS-Unencoded-URL",
"value": "/"
},
{
"name": "X-Original-URL",
"value": "/"
},
{
"name": "X-ARR-LOG-ID",
"value": "xxxxxxxxxx"
},
{
"name": "DISGUISED-HOST",
"value": "xxxxxxxxxx.com"
},
{
"name": "X-SITE-DEPLOYMENT-ID",
"value": "xxxxxxxxxx"
},
{
"name": "WAS-DEFAULT-HOSTNAME",
"value": "xxxxxxxxxx.azurewebsites.net"
},
{
"name": "Content-Length",
"value": "0"
}
]
}
}
},
{
"source": "forward-request",
"timestamp": "2018-01-31T01:45:36.5122512Z",
"elapsed": "00:00:00.0363283",
"data": {
"messages": [
"Error occured while calling backend service.",
"The request was aborted: Could not create SSL/TLS secure channel."
]
}
}
],
I am not sure why there are "The request was aborted: Could not create SSL/TLS secure channel." errors since I can access the GitHub pages version of the website that it is proxying to without issue (no SSL issues). We've had to disable use of the proxy for now and change our DNS to point directly to the GitHub page until we can resolve this.
What's going on here?
Why did this break all of a sudden with no changes on our end?
Maybe GitHub suddenly switched to TLS 1.2-only and walked away?
$ curl --tlsv1.0 -vki https://microsoft.github.io
...
* gnutls_handshake() failed: Error in protocol version
$ curl --tlsv1.1 -vki https://microsoft.github.io
...
* gnutls_handshake() failed: Error in protocol version
$ curl --tlsv1.2 -vki https://microsoft.github.io
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
...
* subject: C=US,ST=California,L=San Francisco,O=GitHub, Inc.,
CN=www.github.com
* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,
CN=DigiCert SHA2 High Assurance Server CA
...
HTTP/1.1 200 OK
I don't know if you can tell Functions Proxy to use a particular TLS version for outbound connections, you know, the equivalent of
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
Related
I'm trying to verify Client Certificates in Azure API Management. I created a new instance and I'm using the default Echo API.
I followed this documentation https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-mutual-certificates-for-clients
and this one for testing with Postman https://medium.com/#jkewley/testing-client-certificate-authentication-to-azure-api-management-with-postman-e1cfae52fc35
I'm using the following Policy in the Echo API All operations Inbound just checking if any certificate is present:
<policies>
<inbound>
<choose>
<when condition="#(context.Request.Certificate == null)">
<return-response>
<set-status code="403" reason="Missing client certificate" />
</return-response>
</when>
</choose>
<base />
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>
In the Custom Domains tab, I have my Endpoint Gateway with Negotiate client certificate and Default SSL binding enabled.
When testing without the policy it works fine. With the policy, I get "403 - Missing client certificate".
My PostMan logs show my local pfx file being sent. I have used that same CA certificate successfully with an Apigee setup that I'm trying to replicate.
Postman Console
The APIM Trace shows no sign of that certificate
{
"traceId": "1e2950a4-7ae9-4489-9175-dd6b7a8e6872",
"traceEntries": {
"inbound": [
{
"source": "api-inspector",
"timestamp": "2021-03-08T16:45:36.1300291Z",
"elapsed": "00:00:00.0002376",
"data": {
"request": {
"method": "POST",
"url": "https://xxxxxx-poc-apim.azure-api.net/echo/resource",
"headers": [
{
"name": "Ocp-Apim-Subscription-Key",
"value": "20c7x7x22xa5xdxc8a1x857bb651000a"
},
{
"name": "X-Forwarded-For",
"value": "76.98.XX.XXX"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Content-Length",
"value": "102"
},
{
"name": "Content-Type",
"value": "text/plain"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip,deflate,br"
},
{
"name": "Host",
"value": "xxxxxxx-poc-apim.azure-api.net"
},
{
"name": "User-Agent",
"value": "PostmanRuntime/7.26.10"
}
]
}
}
},
{
"source": "api-inspector",
"timestamp": "2021-03-08T16:45:36.1300291Z",
"elapsed": "00:00:00.0002401",
"data": {
"configuration": {
"api": {
"from": "/echo",
"to": {
"scheme": "http",
"host": "echoapi.cloudapp.net",
"port": 80,
"path": "/api",
"queryString": "",
"query": {
},
"isDefaultPort": true
},
"version": null,
"revision": "1"
},
"operation": {
"method": "POST",
"uriTemplate": "/resource"
},
"user": "-",
"product": "-"
}
}
},
{
"source": "cors",
"timestamp": "2021-03-08T16:45:36.1300291Z",
"elapsed": "00:00:00.0002602",
"data": "Origin header was missing or empty and the request was classified as not cross-domain. CORS policy was not applied."
},
{
"source": "choose",
"timestamp": "2021-03-08T16:45:36.1300291Z",
"elapsed": "00:00:00.0002753",
"data": {
"message": "Expression was successfully evaluated.",
"expression": "context.Request.Certificate == null",
"value": true
}
},
{
"source": "set-status",
"timestamp": "2021-03-08T16:45:36.1300291Z",
"elapsed": "00:00:00.0002817",
"data": {
"message": [
"Response status code was set to 403",
"Response status reason was set to 'Missing client certificate'"
]
}
},
{
"source": "return-response",
"timestamp": "2021-03-08T16:45:36.1300291Z",
"elapsed": "00:00:00.0002863",
"data": {
"message": "Return response was applied",
"response": {
"status": {
"code": "Forbidden",
"reason": "Missing client certificate"
},
"headers": [
]
}
}
}
],
"outbound": [
{
"source": "transfer-response",
"timestamp": "2021-03-08T16:45:36.1300291Z",
"elapsed": "00:00:00.0003120",
"data": {
"message": "Response headers have been sent to the caller."
}
}
]
}
}
I have tried a lot of things. I tried using SoapUI instead of Postman I try with another CA certificate.
I tried on another APIM that has a CA certificate but is behind an App Gateway. Always the same result.
I'm out of ideas.
I found the issue. My company is using Netskope for web traffic control and it was messing with the Certificates.
I discovered it by testing it from my home computer which was working fine.
When connecting to the APIM URL from my work laptop, my Web Browser was not showing the default .azure-api.net certificate but instead a certificate .goskope.com.
We added *.azure-api.net domain to bypass Netskope checks and it solved the issue.
Problem: I need to determine the nature of this failure so I can know how to further troubleshoot. I have come up with a few hypothesis:
- could be a firewall/proxy configuration within Azure
- could be a misconfiguration with Kroger's API
- could be a public certificate rejection from Azure app
- could be something totally unrelated to any of above
Details: I’m attempting to connect to Kroger's developer API. The following code has been simplified for this post. (I have been using IHttpClientFactory previously to generate my HTTPClient). This works locally, but once it’s deployed to an Azure web service, I am presented with a 403 message (which appears to be coming from Azure and not the external API):
You don't have permission to access http://api.kroger.com on this server.
This same code works within Azure for other 3rd party APIs over HTTPS, so the suspicion I have had is that this error comes from not using correct client certificates, so Azure attempts to call over http?
I have tried many things to resolve this, including uploading a public certificate that I downloaded from https://api.kroger.com to my azure app service. It appears to pull the certificate correctly, but the request still fails with the same message.
Relevant code is below (Without using client certificate):
using(var client = _requestFactory.CreateClient()))
{
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("basic", _clientId);
client.DefaultRequestHeaders.Add("Accept", "application/json");
var newRequest = new Dictionary<string, string>
{
{ "grant_type", "client_credentials" },
{ "scope", "product.compact" }
};
var response = await client.PostAsync($"https://api.kroger.com/v1/connect/oauth2/token", new FormUrlEncodedContent(newRequest));
return Ok(await response.Content.ReadAsStringAsync());
}
Below is the full response from the server.
{
"version": "1.1",
"content": {
"headers": [{
"key": "Content-Type",
"value": ["text/html"]
}, {
"key": "Content-Length",
"value": ["299"]
}, {
"key": "Expires",
"value": ["Sat, 08 Feb 2020 19:18:55 GMT"]
}]
},
"statusCode": 403,
"reasonPhrase": "Forbidden",
"headers": [{
"key": "Server",
"value": ["AkamaiGHost"]
}, {
"key": "Mime-Version",
"value": ["1.0"]
}, {
"key": "Date",
"value": ["Sat, 08 Feb 2020 19:18:55 GMT"]
}, {
"key": "Connection",
"value": ["close"]
}, {
"key": "Set-Cookie",
"value": ["akaalb_Digital_ALB_API=~op=KT_Digital_API_KCVG_F5:api-kcvg|~rv=47~m=api-kcvg:0|~os=75b4a9ec926d2a9e67035451773cec6c~id=63ba4b3e2a027e4d53b693e2fded5ac3; path=/; HttpOnly; Secure; SameSite=None"]
}],
"trailingHeaders": [],
"requestMessage": {
"version": "1.1",
"content": {
"headers": [{
"key": "Content-Type",
"value": ["application/x-www-form-urlencoded"]
}, {
"key": "Content-Length",
"value": ["51"]
}]
},
"method": {
"method": "POST"
},
"requestUri": "https://api.kroger.com/v1/connect/oauth2/token",
"headers": [{
"key": "Authorization",
"value": ["basic {removed}"]
}, {
"key": "Accept",
"value": ["application/json"]
}, {
"key": "Request-Context",
"value": ["appId={removed}"]
}, {
"key": "Request-Id",
"value": ["|{removed}"]
}, {
"key": "traceparent",
"value": ["{removed}"]
}],
"properties": {}
},
"isSuccessStatusCode": false
}
"headers": [{
"key": "Server",
"value": ["AkamaiGHost"]
}
Almost certainly Akamai issue. I have seen 403 errors due to header order or even user-Agent header that Akamai does not like.
I am trying to test sending an event to the Amazon's Event Gateway for my Alexa Smart Home skill using Postman but I keep receiving an 'invalid access token exception.' I have read the Amazon's documentation on this but apparently I am missing something.
When I enable my skill, my Smart Home Lambda receives the AcceptGrant.
{
"directive": {
"header": {
"namespace": "Alexa.Authorization",
"name": "AcceptGrant",
"messageId": "b2862179-bc56-4bb2-ac05-ce55c7a3e977",
"payloadVersion": "3"
},
"payload": {
"grant": {
"type": "OAuth2.AuthorizationCode",
"code": "ANSVjPzpTDBsdfoRSyrs"
},
"grantee": {
"type": "BearerToken",
"token": "Atza|IwEB..."
}
}
}
}
My lambda sends a POST to 'https://api.amazon.com/auth/o2/token' to receive the Access and Refresh tokens. It then stores those tokens. Next, my Lamdba responds with the following:
{
"event": {
"header": {
"namespace": "Alexa.Authorization",
"name": "AcceptGrant.Response",
"messageId": "b2862179-bc56-4bb2-ac05-ce55c7a3e977",
"payloadVersion": "3"
},
"payload": {}
}
}
I then get a message web page that I have successfully linked my skill - all is good.
Next, I try to send an event to Amazon's Alexa event gateway using the Postman app. I put the Access token (I also tried the Refresh token) in the header as a 'BearerToken' type and the in the 'scope' of the 'endpoint' object.
POST https://api.amazonalexa.com/v3/events?Content-Type=application/json&charset=UTF-8
with a header that specifies a Bearer Token (Access token received earlier) and a body that contains the following:
{
"event": {
"header": {
"messageId": "abc-123-def-456",
"namespace": "Alexa",
"name": "ChangeReport",
"payloadVersion": "3"
},
"endpoint": {
"scope": {
"type": "BearerToken",
"token": "<access token>"
},
"endpointId": "MySmartSwitch-001"
},
"payload": {
"change": {
"cause": {
"type": "RULE_TRIGGER"
},
"properties": [
{
"namespace": "Alexa.ModeController",
"name": "mode",
"value": "Backup",
"timeOfSample": "2020-01-02T09:30:00ZZ",
"uncertaintyInMilliseconds": 50
}
]
}
}
},
"context": {
"properties": [
{
"namespace": "Alexa.PowerController",
"name": "powerState",
"value": "ON",
"timeOfSample": "2020-01-02T09:30:00Z",
"uncertaintyInMilliseconds": 60000
},
{
"namespace": "Alexa.EndpointHealth",
"name": "connectivity",
"value": {
"value": "OK"
},
"timeOfSample": "2020-01-02T09:30:00Z",
"uncertaintyInMilliseconds": 0
}
]
}
}
The response received is '401 Unauthorized'
{
"header": {
"namespace": "System",
"name": "Exception",
"messageId": "95bd23c3-76e6-472b-9c6d-74d436e1eb61"
},
"payload": {
"code": "INVALID_ACCESS_TOKEN_EXCEPTION",
"description": "Access token is not valid."
}
}
I figured out the issue. I was mistakenly sending parameters: Content-Type=application/json and charset=UTF-8 as well including them in the header - my bad. You just need to include them in the header.
I followed the tutorial found at
https://learn.microsoft.com/en-us/azure/devops/extend/develop/service-endpoints?view=azure-devops
The api call is right because if I don't authenticate it shows the correct path in the error message. Entering the url that it shows in the message in a browser gives the correct json.
The picklist object will act like it is trying to load but will end up showing no results found.
Can you use a personal company api to load a picklist using server endpoint?
How do you debug this. All I see in the network tab on debugging tools is the 200 from azure making the post.
This is the JSON
{
"meta": {
"record_count": 5,
"source": "mongodb",
"searchParams": {
"versions.start_date": {
"$gte": "2019-08-30T00:11:55.329Z"
}
},
"paging": {
"num": 10,
"start": 0
}
},
"screenshots": [
{
"screenshot_test_id": 3946619,
"url": "https://www.google.com",
"test_name": null,
"created_date": "2019-10-27T21:26:32.0 ```
This is the datasources from the vss.extension
"dataSources": [
{
"name": "CBTAPIReturn",
"endpointUrl": "{{endpoint.url}}",
"resultSelector": "jsonpath:$[*].screenshots"
}
]
inputs and bindings from the task
"inputs": [
{
"name": "CBTService",
"type": "connectedService:CBTServiceEndpoint",
"label": "CBT service/server end point",
"defaultValue": "",
"required": true,
"helpMarkDown": "Select the CBT end point to use. If needed,selecton 'manage', and add a new service endpoint of type 'CBT server connection'"
},
{
"name": "CBT API Response",
"type": "pickList",
"label": "CBT API Response",
"required": true,
"helpMarkDown": "Select the name combination that you want to use.",
"properties": {
"EditableOptions": "True"
}
}
],
"dataSourceBindings": [
{
"target": "CBT API Response",
"endpointId": "$(CBTService)",
"dataSourceName": "CBTAPIReturn"
```
I am trying to send some images to power app using a custom connector.
My OpenAPI file looks this -
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "Upload To Function"
},
"host": "XXXXXXXXXXXXX/.azurewebsites.net",
"paths": {
"/api/HttpTriggerCSharp/name/{name}": {
"post": {
"tags": [
"HttpTriggerCSharp"
],
"operationId": "UploadImage",
"consumes": [
"multipart/form-data"
],
"produces": [
"application/json",
"text/json",
"application/xml",
"text/xml"
],
"parameters": [
{
"name": "file",
"in": "formData",
"required": true,
"type": "file",
"x-ms-media-kind": "image"
},
{
"name": "name",
"in": "path",
"required": true,
"type": "string"
}
],
"responses": {
"200": {
"description": "Saved successfully"
},
"400": {
"description": "Could not find file to upload"
}
},
"summary": "Image Upload",
"description": "Image Upload"
}
}
},
"definitions": {}
}
My function works as I have tested it out using POSTMAN, and also within the Azure function portal. But when I test it out on Powerapps using this swagger file, I get Status code 404, Resource not found error.
I call my function like this in Powerapps -
Uploadtofunction.uploadimage(MainimageView.image,{filename:"test")
Any idea, on why it's not working?
Edit: Here is how I tested it out on POSTMAN. I think its also related to the filename in swagger. Because when I add name/filename in the body, I get 500 internal server issue.