We are needing to write some software, that receives events one at a time, and we need to ingress them into ADX. We are struggling to understand how the Kusto Client is meant to be utilized.
public void SaveEvent(Object event)
{
var _kcsb = new KustoConnectionStringBuilder(Uri).WithAadApplicationKeyAuthentication(
applicationClientId: "{}",
applicationKey: "{}",
authority: TenantId);
using var ingestClient = KustoIngestFactory.CreateQueuedIngestClient(_kcsb);
//// Create your custom implementation of IRetryPolicy, which will affect how the ingest client handles retrying on transient failures
IRetryPolicy retryPolicy = new NoRetry();
//// This line sets the retry policy on the ingest client that will be enforced on every ingest call from here on
((IKustoQueuedIngestClient)ingestClient).QueueOptions.QueueRequestOptions.RetryPolicy = retryPolicy;
var ingestProperties = new KustoIngestionProperties(DatabaseName, TableName)
{
Format = DataSourceFormat.json,
IngestionMapping = new IngestionMapping { IngestionMappingKind = Kusto.Data.Ingestion.IngestionMappingKind.Json, IngestionMappingReference = MappingName }
};
// Build the stream
var stream = new MemoryStream();
using var streamWriter = new StreamWriter(stream: stream, encoding: Encoding.UTF8, bufferSize: 4096, leaveOpen: true);
using var jsonWriter = new JsonTextWriter(streamWriter);
packet.Id = DateTime.UtcNow.Ticks;
var serializer = new JsonSerializer();
serializer.Serialize(jsonWriter, event);
streamWriter.Flush();
stream.Seek(0, SeekOrigin.Begin);
// Tell the client to ingest this
await ingestClient.IngestFromStreamAsync(data, ingestProperties);
}
Now I have several concerns with this. We are calling this function 300 to 500 times a second. I believe the custom client has built in batching, but do we not then need to use a singleton instance of the custom client?
Next thing is that I am creating a steam per event and then calling ingerss. This feels wrong? is there no way I can setup the custom client etc, and then just enqueue each event into the custom client as we receiver them?
Related
At the moment Im simulating device where every 30 seconds I send telemetry data to IoT hub.
Here is simple code:
s_deviceClient = DeviceClient.Create(s_iotHubUri, new DeviceAuthenticationWithRegistrySymmetricKey(s_myDeviceId, s_deviceKey), TransportType.Mqtt);
using var cts = new CancellationTokenSource();
var messages = SendDeviceToCloudMessagesAsync(cts.Token);
await s_deviceClient.CloseAsync(cts.Token);
await messages;
cts.Cancel();
And function to send message:
string combinedString = fileStrings[0] + fileStrings[1];
var telemetryDataString = converter.SerializeObject(combinedString);
using var message = new Message(Encoding.UTF8.GetBytes(telemetryDataString))
{
ContentEncoding = "utf-8",
ContentType = "application/json",
};
await s_deviceClient.SendEventAsync(message);
await Task.Delay(interval);
Everything works fine and I created .exe file that was running without problems. But computer where code is running tends to shut-off from time to time which is problematic. So I tried to move this to Azure Time Trigger Function. While in logs everything looks ok, messages aren't actually posted to IoT hub.
I tried to find solution but have not been able to find anything. Is it possible to send messages as device with azure function?
You seem to be closing your DeviceClient before you start using it to send messages. Try the following:
public async Task Do()
{
// Using statement will dispose your client after you're done with it.
// No need to close it manually.
using(var client = DeviceClient.Create(s_iotHubUri, new DeviceAuthenticationWithRegistrySymmetricKey(s_myDeviceId, s_deviceKey), TransportType.Mqtt))
{
// Send messages, await for completion.
await SendDeviceToCloudMessagesAsync(client);
}
}
private async Task SendDeviceToCloudMessagesAsync(DeviceClient client)
{
string combinedString = fileStrings[0] + fileStrings[1];
var telemetryDataString = converter.SerializeObject(combinedString);
using var message = new Message(Encoding.UTF8.GetBytes(telemetryDataString))
{
ContentEncoding = "utf-8",
ContentType = "application/json",
};
await client.SendEventAsync(message);
await Task.Delay(interval);
}
I want to show message to end user when blob is taking so much time for uploading and downloading. I found useful blog here.
Simply linear retry policy
public static RetryPolicy LinearRetry(int retryCount, TimeSpan intervalBetweenRetries)
{
return () =>
{
return (int currentRetryCount, Exception lastException, out TimeSpan retryInterval) =>
{
// Do custom work here
// Set backoff
retryInterval = intervalBetweenRetries;
// Decide if we should retry, return bool
return currentRetryCount < retryCount;
};
};
}
But here I didn't get how to send response to user back while retrying. Is this right way or anything else. Please suggest
OperationContext class in Storage Client Library has an event called Retrying that you can consume and send message back to the client.
For example, I created a simple console application which tries to create a blob container. When I ran this application, I deliberately turned off Internet access so that I can simulate a situation where operation would be retried. Then in this event consumer, I simply write something back to console. You could simply raise another event from there that would send a message back to your client.
var requestOptions = new BlobRequestOptions()
{
RetryPolicy = new ExponentialRetry(),
};
var operationContext = new OperationContext();
operationContext.Retrying += (sender, args) =>
{
Console.WriteLine("I'm retrying ....");
};
var cloudStorageAccount = new CloudStorageAccount(new StorageCredentials(accountName, accountKey), true);
var blobClient = cloudStorageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference("test");
container.CreateIfNotExists(requestOptions, operationContext);
Is there any way to add the type of the response dto to the rabbitmq response message's headers collection?
(My consumer is using spring's rabbitmq handler which seems to depend on explicit type information inside the mq header when deserializing.)
Currently servicestack's mq producer already returns serveral headers, such as "content_type='application/json".
I am in need of an additional header, e.g. "typeId"="HelloResponse", so that the consuming web app knows how to deserialize the message, even in RPC cases where the response queue name is some kind of GUID.
Is there some kind of configuration which would enable me to archieve such an behaviour? Or some hook before the message gets published so that I can add the header myself?
I've added support for automatically populating the Message Body Type in RabbitMQ's IBasicProperties.Type as well as adding support for both Publish and GetMessage Filters in this commit.
Here's an example of configuring a RabbitMqServer with custom handlers where you can modify the message and its metadata properties when its published and received:
string receivedMsgApp = null;
string receivedMsgType = null;
var mqServer = new RabbitMqServer("localhost")
{
PublishMessageFilter = (queueName, properties, msg) => {
properties.AppId = "app:{0}".Fmt(queueName);
},
GetMessageFilter = (queueName, basicMsg) => {
var props = basicMsg.BasicProperties;
receivedMsgType = props.Type; //automatically added by RabbitMqProducer
receivedMsgApp = props.AppId;
}
};
mqServer.RegisterHandler<Hello>(m =>
new HelloResponse { Result = "Hello, {0}!".Fmt(m.GetBody().Name) });
mqServer.Start();
Once Configured any message published or received will go through the above handlers, e.g:
using (var mqClient = mqServer.CreateMessageQueueClient())
{
mqClient.Publish(new Hello { Name = "Bugs Bunny" });
}
receivedMsgApp.Print(); // app:mq:Hello.In
receivedMsgType.Print(); // Hello
using (IConnection connection = mqServer.ConnectionFactory.CreateConnection())
using (IModel channel = connection.CreateModel())
{
var queueName = QueueNames<HelloResponse>.In;
channel.RegisterQueue(queueName);
var basicMsg = channel.BasicGet(queueName, noAck: true);
var props = basicMsg.BasicProperties;
props.Type.Print(); // HelloResponse
props.AppId.Print(); // app:mq:HelloResponse.Inq
var msg = basicMsg.ToMessage<HelloResponse>();
msg.GetBody().Result.Print(); // Hello, Bugs Bunny!
}
This change is available from ServiceStack v4.0.33+ that's now available on MyGet.
We are getting the below exception while reading data using JsonTextReader
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
JsonTextReader jsonReader - parameter
while (hasRecords(jsonReader, JsonToken.StartObject, null, null)) //Row
{
...
//it's ok to read this all into memory - it's just one row's worth of data
JArray values = (JArray)JToken.ReadFrom(jsonReader);
Also, including the code for HttpPost implementation for better clarity
HttpClientHandler handler = new HttpClientHandler() { Credentials = taskProfileInfo.Credential };
HttpClient httpClient = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(taskProfileInfo.CommandTimeout) };
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage response;
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, url);
request.Content = new StringContent(postBody, Encoding.UTF8, "application/json");
response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
//using (var responseStream = await response.Content.ReadAsStreamAsync())
//{
// using (var reader = new StreamReader(responseStream))
// {
// responseFromAPI = reader.ReadToEnd();
// }
//}
return new JsonTextReader(new StreamReader(await response.Content.ReadAsStreamAsync()));
Appreciate if any one can help us ..
Edit: Please note that we are able to debug it locally and works fine. Only problem when we run this as Worker Role in Azure Cloud service.
I finally addressed this issue. Just to close this (might help someone) -
After doing remote debugging we found the below inner exception :
{"The request was aborted: The request was canceled."}
And, the root cause for this issue is that we did set the timeout to less than what the actual read (JsonTextReader) operation would take. The below line of code which sets the time out :
HttpClient httpClient = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(taskProfileInfo.CommandTimeout) };
So, the FIX is to increase time out value so that request will not be cancelled while reading data.
I fixed increasing the seconds in the web config, it's an alternative:
sessionState timeout="50"
The web socket is written in javascript by my colleague. I managed to connect. First of all I have to log in on the application using a test account. I have to send the email and password through a json. I have installed the Json.Net packet using NuGet.
Some code that I found on my reaserch is this, but I do not understand how to send my data using that segment.
var buffer = new byte[1024];
var segment = new ArraySegment<byte>(buffer);
webSocket.SendAsync(segment, WebSocketMessageType.Text, true, CancellationToken.None);
Of course, I can use an object
User user=new User();
user.Email="bla#bla.com";
user.Password="pass";
string json = JsonConvert.SerializeObject(user);
But it will not be of any use because the method SendAsync accepts only byte type on segment.
All I want is to send that data, and if log in succeeds, I should receive other data (in Json format) about the user.
As a side note, I am quite new to web sockets, I used http protocols from ASP.NET WEB API 2.
I have no idea about Windows Phone 8, but by the code you pasted it seems similar to the regular .NET ClientWebSocket, so here you have some examples:
public static Task SendString(ClientWebSocket ws, String data, CancellationToken cancellation)
{
var encoded = Encoding.UTF8.GetBytes(data);
var buffer = new ArraySegment<Byte>(encoded, 0, encoded.Length);
return ws.SendAsync(buffer, WebSocketMessageType.Text, true, cancellation);
}
public static async Task<String> ReadString(ClientWebSocket ws)
{
ArraySegment<Byte> buffer = new ArraySegment<byte>(new Byte[8192]);
WebSocketReceiveResult result = null;
using (var ms = new MemoryStream())
{
do
{
result = await ws.ReceiveAsync(buffer, CancellationToken.None);
ms.Write(buffer.Array, buffer.Offset, result.Count);
}
while (!result.EndOfMessage);
ms.Seek(0, SeekOrigin.Begin);
using (var reader = new StreamReader(ms, Encoding.UTF8))
return reader.ReadToEnd();
}
}
If something does not compile or exists in WP8, just find an equivalent.
#vtortola is a working example in case your data comes in multiple segmented messages, but if all data comes in a single message you don't need all those streams to read the message, you just need to do this:
public static async Task<String> ReadString(ClientWebSocket socket)
{
var reciveBuffer = new byte[32000];
var result = await socket.ReceiveAsync(new ArraySegment<byte>(reciveBuffer), CancellationToken.None);
if (result.MessageType == WebSocketMessageType.Close)
{
await socket.CloseAsync(WebSocketCloseStatus.NormalClosure, string.Empty, CancellationToken.None);
}
return Encoding.ASCII.GetString(reciveBuffer , 0, result.Count);
}
If your message is splited in multiple segments or you don't know how your message is comming then you have to do like #vtortola
Also if you want to keep receiving messages you can do a while and call ReadString inside, like this:
while (socket.State == WebSocketState.Open)
{
var msg = ReadString(socket)
//do something with your message...
}