Add authentication to Azure Device Streams - azure

We want to use IOT hub and use device streams to proxy our internal Web API for maintenance purposes. This works quite fine, but we also need some kind of authorization. The shared access policy is not fine-grained enough for this purpose.
I would rather use some kind of authentication token (JWT) to pass in the request that can be checked by the device itself. If the token can be validated and the use has the proper rights, then the connection is accepted and otherwise it's reject. The only value that can be configured is the name, so I need to encode the token in the name. The client code could look something like this:
Client code
var deviceStreamRequest = new DeviceStreamRequest(streamName: "WebAPI?token=<JWT-token here>");
var result = await serviceClient.CreateStreamAsync(deviceId, deviceStreamRequest, cancellationToken).ConfigureAwait(false);
Device code
var streamRequest = await deviceClient.WaitForDeviceStreamRequestAsync(cancellationToken).ConfigureAwait(false);
if (streamRequest != null)
{
var token = GetTokenFromName(streamRequest.Name);
if (!CheckClaim(token, "WebAPI"))
{
await deviceClient.RejectDeviceStreamRequestAsync(streamRequest, cancellationToken).ConfigureAwait(false);
return;
}
await deviceClient.AcceptDeviceStreamRequestAsync(streamRequest, cancellationToken).ConfigureAwait(false);
// ...
}
The DeviceStreamRequest class does contain an AuthenticationToken, but it seems to be the authentication token that is used to connect back to IOT using the websocket and cannot be used for other purposes.
I there a better way to pass the token then using the name?

The current version of the Microsoft.Azure.Devices.DeviceStreamRequest didn't allow to populate a request payload to the device like we have in the device direct method. The request payload is the best place for sending an additional (or business) data to the device related to the streaming preprocessing such as the B2B decisions, etc.
Note, that using a streamName for passing your token is not the correct way, see the following code snippet from the .Net Reflector:
public override Task<DeviceStreamResponse> CreateStreamAsync(string deviceId, DeviceStreamRequest deviceStreamRequest, CancellationToken cancellationToken)
{
return this.CreateStreamAsync(GetDeviceStreamUri(deviceId, deviceStreamRequest.StreamName), deviceStreamRequest, cancellationToken);
}
where, the GetDeviceStreamUri has the following implementation:
private static Uri GetDeviceStreamUri(string deviceId, string streamName)
{
deviceId = WebUtility.UrlEncode(deviceId);
object[] args = new object[] { deviceId, streamName };
return new Uri("/twins/{0}/streams/{1}?api-version=2018-08-30-preview".FormatInvariant(args), UriKind.Relative);
}
As you can see the above Uri, there is already hardcoded query parameter such as ?api-version=2018-08-30-preview
However, there is a workaround for your solution based on the using an underlying communication (no using the SDK package), for instance, the REST APIs. Note, that this feature is still in the preview.
For demonstration of the iot device streaming, I am using my Azure IoT Hub Tester.
The following screen snippet shows the invoker POST request to the IoT Hub for device streaming:
As you can see, the Microsoft.Azure.Devices.DeviceStreamRequest can handled only the stream name and two headers such as iothub-streaming-response-timeout-in-seconds and iothub-streaming-connect-timeout-in-seconds.
Posting this reqest, the IoT Hub will send the message to the device. The following screen snippet shows a received message in my virtual MQTT device1:
Now, the device1 can evaluate a message payload to make a decision for either the accepting (code 202) or rejecting (code 4xx, etc.) the streaming process. Note, that there is a response time limit (in my example is 15 seconds) from the invoker call.
Once, the device accepted this streaming process, the invoker will receive the response from the IoT Hub with details in the headers for creating a websocket communication and payload from the device. The following is an example of the headers:
iothub-streaming-is-accepted: True
iothub-streaming-url: wss://centralus.centralus-001.streams.azure-devices.net:443/bridges/ih/rk2019-iot/d/device1/sid/****
iothub-streaming-auth-token: ***
iothub-streaming-ip-address: 0.0.0.0
Based on that, the invoker can evaluate a device response payload before creating a websocket communication.

Related

Spring Integration Tcp project

I have a project that part of it is using Tcp connection, the case is as per below , I will also include a screen shot.
We have two clients, client 1 and client 2 those are conveyor belts so if we receive data on client one input we should send the reply to client 2 output and vise vers, I'm sure we can do it using Spring integration Tcp and probably getways. Am I approaching correctly Tcp integration at this case?
Yet I do not have code implementation but started to put something on it.
Sounds like you implementing a chat (or similar user-to-user) communication.
No, gateways won't help you here.
You need to have a TcpReceivingChannelAdapter and TcpSendingMessageHandler connected to the same AbstractServerConnectionFactory. The TcpSendingMessageHandler is registered as a TcpSender with that connection and all the sending connections are stored in the Map<String, TcpConnection> connections. When we produce a message to this MessageHandler, it tries to consult that registry like this:
private void handleMessageAsServer(Message<?> message) {
// We don't own the connection, we are asynchronously replying
String connectionId = message.getHeaders().get(IpHeaders.CONNECTION_ID, String.class);
TcpConnection connection = null;
if (connectionId != null) {
connection = this.connections.get(connectionId);
}
if (connection != null) {
So, on the receiving side (TcpReceivingChannelAdapter and its sub-flow) you need to ensure somehow that you really set a proper IpHeaders.CONNECTION_ID header for producing so-called reply in the end to a desired client.
You probably can react for the TcpConnectionOpenEvent via #EventListener and register some business key with the connectionId for the future correlation. When you send a message, you supply that target user business key, in the TcpReceivingChannelAdapter sub-flow you take that business key and obtain a desired connectionId from you registry. And enrich it into the IpHeaders.CONNECTION_ID header for automatic logic in the TcpSendingMessageHandler.
When TcpConnectionCloseEvent happens you have to remove its respective entry from your custom registry.
Since TCP/IP comes without headers support there is no any out-of-the-box mechanism to implement such a correlation feature.
Although TcpConnectionOpenEvent might not be enough for you since there is no any business info when connection is established. Perhaps you would need to implement some hand-shake logic in the TcpReceivingChannelAdapter flow to distinguish a real message and connection metadata for registering in the custom registry.
See more info in the docs: https://docs.spring.io/spring-integration/docs/current/reference/html/ip.html#ip-correlation
It might be also better for your use-case to look into a WebSocket support: https://docs.spring.io/spring-integration/docs/current/reference/html/web-sockets.html#web-sockets

Intermittent 501 response from InvokeDeviceMethodAsync - Azure IoT

InvokeDeviceMethodAsync is intermittently (and only recently) returning a status code of 501 within the responses (the response body is null).
I understand this means Not Implemented. However, the method IS implemented - in fact, it's the only method that is. The device is using Microsoft.Azure.Devices.Client (1.32.0-preview-001 since we're also previewing the Device Streams feature).
Setup, device side
This is all called at startup. After this, some invocations succeed, some fail.
var deviceClient = DeviceClient.CreateFromConnectionString(connectionDetails.ConnectionString, TransportType.Mqtt);
await deviceClient.SetMethodHandlerAsync("RedactedMethodName", RedactedMethodHandler, myObj, cancel).ConfigureAwait(true);
Call, server side
var methodInvocation = new CloudToDeviceMethod("RedactedMethodName")
{
ResponseTimeout = TimeSpan.FromSeconds(60),
ConnectionTimeout = TimeSpan.FromSeconds(60)
};
var invokeResponse = await _serviceClient.InvokeDeviceMethodAsync(iotHubDeviceId, methodInvocation, CancellationToken.None);
What have I tried?
Check code, method registration
Looking for documentation about 501: can't find any
Looking through the source for the libraries (https://github.com/Azure/azure-iot-sdk-csharp/search?q=501). Just looks like "not implemented", i.e. nothing registered
Turning on Distributed Tracing from the Azure portal, with sampling rate 100%. Waited a long time, but still says "Device is not synchronised with desired settings"
Exploring intellisense on the DeviceClient object. Not much there!
What next?
Well, I'd like to diagnose.
What possible reasons are there for the 501 response?
Are there and diagnostic tools, e.g. logging, I have access to?
It looks like, there is no response from the method within the responseTimeoutInSeconds value, so for test purpose (and the real response error) try to use a REST API to invoke the device method.
You should received a http status code described here.

Spring Integration - Manage 401 Error in http outbound adapter call

I am new to spring integration.
I have a flow on which I need to perform an http or a tcp call depending on some conditions.
The problem I am focused on is related to the http call.
The rest endpoint called needs an accessToken as header parameter for authentication that is provided by a spring service that has 2 methods getCurrentAccessToken() and refreshAccessToken(). I want to call the method refresh accessToken only when the currentAccessToken is expired.
What I would like to do is to add the following logic when performing the call to the rest api:
If the token is expired the rest endpoint returns a 401 and I would like to intercept in the flow this error and retry the request by adding a refreshed access token.
#Bean
public IntegrationFlow clientIn(AbstractServerConnectionFactory server,
AbstractClientConnectionFactory client, LogService logService) {
return IntegrationFlows.from(Tcp.inboundAdapter(client)
.enrichHeaders(t -> t.headerFunction(IpHeaders.CONNECTION_ID, message -> this.client, true))
.log(msg -> "client: " + logService.log(msg))
.<byte[], Boolean>route(this::shouldForwardToHttp,
mapping -> mapping.subFlowMapping(true, sf -> sf
.enrichHeaders(t -> t.header("Content-Type", MimeTypeUtils.APPLICATION_JSON_VALUE))
.<byte[], RequestMessage>transform(this::buildRequestFromMessage)
.<RequestMessage, HttpEntity>transform(this::getHttpEntity)
.handle(Http.outboundGateway(restUrl).httpMethod(HttpMethod.POST)
.expectedResponseType(ResponseMessage.class))
.<ResponseMessage, byte[]>transform(p -> this.transformResponse(p))
.handle(Tcp.outboundAdapter(client))).subFlowMapping(false,
t -> t.handle(Tcp.outboundAdapter(server).retryInterval(1000))))
.get();
}
HttpEntity getHttpEntity(RequestMessage request) {
MultiValueMap<String, String> mv = new HttpHeaders();
mv.add("accessToken", tokenProvider.getCurrentAccessToken());
HttpEntity entity = new HttpEntity(request, mv);
return entity;
}
I have tried by adding a requestHandlerRetry advice and redirecting it to a recoveryChannel, but I was not able to return something to the caller flow in order to get the response with the status code and retry the call with the new accessToken.
Any idea on how I can implement this?
I don't think you need a retry advice since you definitely are simulating it via catching that 401 exception and calling a service back with refreshed token. Sounds more like recursion. To achieve it properly I would suggest to take a look into an ExpressionEvaluatingRequestHandlerAdvice: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#message-handler-advice-chain. Yes, it is similar to the retry one and it also has that failureChannel, but there is no built-in retry since we are going to simulate it calling the same endpoint again and again when necessary.
To simplify a recursion logic, I would extract that .handle(Http.outboundGateway(restUrl).httpMethod(HttpMethod.POST) .expectedResponseType(ResponseMessage.class)) into a separate flow and use a gateway() with an input channel for that flow in the main flow instead.
A failureChannel sub-flow should re-route its message back to the input of the gateway flow.
What is the most important part in this logic is to carry on all the original request message headers which includes a required for the gateway logic replyChannel.
See more docs about gateways: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#gateway.
When an ExpressionEvaluatingRequestHandlerAdvice sends a message to the failureChannel, it comes as an ErrorMessage with a MessageHandlingExpressionEvaluatingAdviceException as a payload. The message which causes a failure and has all the required headers is there in the getFailedMessage() property. So, you take that message, request for fresh token, add it into headers of a new message based on that original. In the end you send this new message to the input channel of the IntegrationFlow for an HTTP request. When all is good, the result of the HTTP call is going to be forwarded to the mentioned replyChannel from headers and in therefore to the main flow for next steps.

Outbound TCP Connection issue cause be sending data to event hub and data lake from azure function

I'm working on a Azure function with http POST trigger, once client call it and post a json data, I will send it to event hub and save to data lake.
once it got hitted by the high traffic, 20k/hour, azure functino will generate high outbound TCP connection, which will exceed the limitation (1920) of the plan.
does high outbound TCP connection cause by writing to event hub, data lake, or both?
is there a way to reduce it so I don't have to pay more to upgrade our plan?
how to debug it to trouble shooting the problem?
here is the code of send data to event hub:
EventHubClient ehc = EventHubClient.CreateFromConnectionString(cn);
try
{
log.LogInformation($"{CogniPointListener.LogPrefix}Sending {batch.Count} Events: {DateTime.UtcNow}");
await ehc.SendAsync(batch);
await ehc.CloseAsync();
}
catch (Exception exception)
{
log.LogError($"{CogniPointListener.LogPrefix}SendingMessages: {DateTime.UtcNow} > Exception: {exception.Message}");
throw;
}
here is the send data to data lake:
var creds = new ClientCredential(clientId, clientSecret);
var clientCreds = ApplicationTokenProvider.LoginSilentAsync(tenantId, creds).GetAwaiter().GetResult();
// Create ADLS client object
AdlsClient client = AdlsClient.CreateClient(adlsAccountFQDN, clientCreds);
try
{
using (var stream = client.CreateFile(fileName, IfExists.Overwrite))
{
byte[] textByteArray = Encoding.UTF8.GetBytes(str);
stream.Write(textByteArray, 0, textByteArray.Length);
}
// Debug
log.LogInformation($"{CogniPointListener.LogPrefix}SaveDataLake saved ");
}
catch (System.Exception caught)
{
string err = $"{caught.Message}Environment.NewLine{caught.StackTrace}Environment.NewLine";
log.LogError(err, $"{CogniPointListener.LogPrefix}SaveDataLake");
throw;
}
Thanks,
I just raised an issue with Azure SDK https://github.com/Azure/azure-sdk-for-net/issues/26884 reporting the problem of socket exhaustion when using ApplicationTokenProvider.LoginSilentAsync.
The current version 2.4.1 of Microsoft.Rest.ClientRuntime.Azure.Authentication uses the old version 4.3.0 of Microsoft.IdentityModel.Clients.ActiveDirectory that creates a new HttpClientHandler on every call.
Creating HttpClientHandler on every is bad. After HttpClientHandler is disposed, the underlaying socket connections are still active for significant time (in my experience 30+ seconds).
There's a thing called HttpClientFactory that ensures HttpClientHandler is not created frequently. Here's a guide from Microsoft explaining how to use HttpClient and HttpClientHandler properly - Use IHttpClientFactory to implement resilient HTTP requests.
I wish they reviewed their SDKs to ensure they follow their own guidelines.
Possible workaround
Microsoft.IdentityModel.Clients.ActiveDirectory since version 5.0.1-preview supports passing a custom HttpClientFactory.
IHttpClientFactory myHttpClientFactory = new MyHttpClientFactory();
AuthenticationContext authenticationContext = new AuthenticationContext(
authority: "https://login.microsoftonline.com/common",
validateAuthority: true,
tokenCache: <some token cache>,
httpClientFactory: myHttpClientFactory);
So it should be possible to replicate what ApplicationTokenProvider.LoginSilentAsync does in your codebase to create AuthenticationContext passing your own instance of HttpClientFactory.
The things you might need to do:
Ensure Microsoft.IdentityModel.Clients.ActiveDirectory with version of after 5.0.1-preview is added to the project
Since the code is used in Azure functions, HttpClientFactory needs to be set up and injected. More info can be found in another StackOverflow answer
Replace calls ApplicationTokenProvider.LoginSilentAsync(tenantId, creds) with something like that (this code is an inlined version of LoginSilentAsync that passes httpClientFactory to AuthenticationContext
var settings = ActiveDirectoryServiceSettings.Azure;
var audience = settings.TokenAudience.OriginalString;
var context = new AuthenticationContext(settings.AuthenticationEndpoint + domain,
settings.ValidateAuthority,
TokenCache.DefaultShared,
httpClientFactory);
var authenticationProvider = new MemoryApplicationAuthenticationProvider(clientCredential);
var authResult = await authenticationProvider.AuthenticateAsync(clientCredential.ClientId, audience, context).ConfigureAwait(false);
var credentials = new TokenCredentials(
new ApplicationTokenProvider(context, audience, clientCredential.ClientId, authenticationProvider, authResult),
authResult.TenantId,
authResult.UserInfo == null ? null : authResult.UserInfo.DisplayableId);
I really don't replicating the logic in the workaround, but I don't think there's any other option until it's fixed properly in Microsoft.Rest.ClientRuntime.Azure.Authentication
Good luck!
TCP connections are limited in specific numbers depending on the plan you have your functions on (Consumption or a static plan in any level B/S/P).
For high workloads I prefer to either
A: Use a queue with a separate function and limiting the concurrency by the function batch size and other settings
or
B: Use a SemaphoreSlim in order to control concurrency of outgoing traffic. (https://learn.microsoft.com/de-de/dotnet/api/system.threading.semaphoreslim?redirectedfrom=MSDN&view=netframework-4.7.2)

Spring Integration - TCP - Response Correlation

I'm new to Spring Integration. The situation is that I've to connect to Tcp server dynamically(i.e. the DNS will be dynamically generated at runtime based on some params). Because of this I'm using Service Activator to manually create Tcp Connections and send messages. I've overridden CachingClientConnectionFactory to make use of shared connections concept(with single-use='false'). I was listening to messages using TcpReceivingChannelAdaptor by overriding "onMessage" method. The problem is that the server either responds with a Success or failure(with Generic messages) with no CorrelationID. Is there any way to correlate the request with the response ?
I tried using TcpOutboundGateway, but with this approach also I get the same problem. I used TcpConnectionSupport to send messages :
//Sample Code.
final String correlationId = "" // Dynamic unique number
TcpOutboundGateway outboundGateway = new TcpOutboundGateway(){
public synchronized boolean onMessage(Message<?> message) {
ByteArrayToStringConverter converter = new ByteArrayToStringConverter();
String response = converter.convert((byte[]) message
.getPayload());
logger.info(correlationId);
return false;
}
};
DefaultCachingClientConnectionFactory connFactory = new DefaultCachingClientConnectionFactory();
TcpConnectionSupport con = connFactory.obtainConnection();
GenericMessage<String> msg = new GenericMessage<String>("Sample Message" + correlationId);
con.registerListener(outboundGateway);
con.send(msg);
// DefaultCachingClientConnectionFactory is the subclass of CachingClientConnectionFactory.
When I send multiple messages, every time I get the same correlation printed in the "onMessage" method.
I read here that Outbound Gateway will correlate messages. Please help me. Maybe I'm doing something wrong.
Thanks
Unless you include correlation data in the message you can't correlate a response to a request.
The gateway achieves this by only allowing one outstanding request on a socket at a time; hence the reply has to be for the request. This is not very useful at high volume with a shared connection; hence the caching client cf was introduced. The gateway keeps a map of outstanding requests based on the connection id.
The gateway, in conjunction with the caching client connection factory should do what you need. However, overriding onMessage is not a good idea, because that's where the reply correlation is done.

Resources