I am making a small App that should list the number of items in my Azure queues.
When I use FetchAttributesAsync and ApproximateMessageCount in a Console App, I get the expected result in ApproximateMessageCount after a call to FetchAttributesAsync (or FetchAttributes).
When I use the same in a Universal Windows app, ApproximateMessageCount remains stuck at null after a call to FetchAttributesAsync (FetchAttributes is not available there).
Console code:
CloudStorageAccount _account;
if (CloudStorageAccount.TryParse(_connectionstring, out _account))
{
var queueClient = _account.CreateCloudQueueClient();
Console.WriteLine(" {0}", _account.QueueEndpoint);
Console.WriteLine(" ----------------------------------------------");
var queues = (await queueClient.ListQueuesSegmentedAsync(null)).Results;
foreach (CloudQueue q in queues)
{
await q.FetchAttributesAsync();
Console.WriteLine($" {q.Name,-40} {q.ApproximateMessageCount,5}");
}
}
Universal App code:
IEnumerable<CloudQueue> queues;
CloudStorageAccount _account;
CloudQueueClient queueClient;
CloudStorageAccount.TryParse(connectionstring, out _account);
queueClient = _account.CreateCloudQueueClient();
queues = (await queueClient.ListQueuesSegmentedAsync(null)).Results;
foreach (CloudQueue q in queues)
{
await q.FetchAttributesAsync();
var count = q.ApproximateMessageCount;
// count is always null here!!!
}
I have tried all kinds of alternatives, like Wait()'s and such on the awaitables. Whatever I try, the ApproximateMessageCount stays a null with dertermination :-(.
Am I missing something?
I think you have discovered a bug in the storage client library. I looked up the code on Github and essentially instead of reading the value for Approximate Message Count header, the code is reading the value for Lease Status header.
In QueueHttpResponseParsers.cs class:
public static string GetApproximateMessageCount(HttpResponseMessage response)
{
return response.Headers.GetHeaderSingleValueOrDefault(Constants.HeaderConstants.LeaseStatus);
}
This method should have been:
public static string GetApproximateMessageCount(HttpResponseMessage response)
{
return response.Headers.GetHeaderSingleValueOrDefault(Constants.HeaderConstants.ApproximateMessagesCount);
}
I have submitted a bug for this: https://github.com/Azure/azure-storage-net/issues/155.
Related
I am trying to use StartCopyFromUri or StartCopyFromUriAsync to copy a blob from one storage account to another. Even though status.HasCompleted when I try to get the ETag either through
1. var etag = await _siteStorageClient.GetBlobETag(containerPath, asset.BlobName);
//this is the response from WaitForCompletionAsync
2. var etag = complete.GetRawResponse().Headers.Where(x => x.Name == "ETag").FirstOrDefault().Value;
I've tried both methods and both return an Etag that doesn't match what is shown in the properties of the blob when I log in through Azure Portal. It is almost as if the file wasn't done copying(or race condition) when the Etag check was executed. I couldn't find any usage samples on github for the SDK.
Any ideas what could be going awry?
This a similar question but using the older SDK. How to copy a blob from one container to another container using Azure Blob storage SDK
//Storage class
public async Task<CopyFromUriOperation> CopyFile(string containerName, string blobName, Uri sourceUri)
{
var container = _blobServiceClient.GetBlobContainerClient(containerName);
var blockBlobClient = container.GetBlockBlobClient(blobName);
//Made this the synchronous version try and block
//this is the target client
var status = await blockBlobClient.StartCopyFromUriAsync(sourceUri);
while(!status.HasCompleted)
{
//Per Documentation this calls UpdateStatusAsync() periodically
//until status.HasCompleted is true
await status.WaitForCompletionAsync();
}
return status;
}
//Calling Code
var status = await _siteStorageClient.CopyFile(container,BlobName, sasUri);
var etag = await _siteStorageClient.GetBlobETag(container, BlobName);
I was able to get this working after a few tries and troubleshooting. It would only happen in the Azure environment and not when running the web app locally.
Initially the status.WaitForCompletionAsync() was inside the loop and I started getting a socket error. I believe it was getting called too many times and was causing port exhaustion(just speculation at this point).
But this is what is working now.
public async Task<CopyFromUriOperation> CopyFile(string containerName, string blobName,Uri sourceUri)
{
var container = _blobServiceClient.GetBlobContainerClient(containerName);
var blockBlobClient = container.GetBlockBlobClient(blobName);
var status = await blockBlobClient.StartCopyFromUriAsync(sourceUri);
await status.WaitForCompletionAsync();
while(status.HasCompleted == false)
{
await Task.Delay(100);
}
return status;
}
I'm looking to have our distributed event logging have proper correlation. For our Web Applications this seems to be automatic. Example of correlated logs from one of our App Services API:
However, for our other (non ASP, non WebApp) services were we use Log4Net and the App Insights appender our logs are not correlated. I tried following instructions here: https://learn.microsoft.com/en-us/azure/azure-monitor/app/correlation
Even after adding unique operation_Id attributes to each operation, we're not seeing log correlation (I also tried "Operation Id"). Example of none correlated log entry:
Any help on how to achieve this using log4net would be appreciated.
Cheers!
Across services correlation IDs are mostly propagated through headers. When AI is enabled for Web Application, it reads IDs/Context from the incoming headers and then updates outgoing headers with the appropriate IDs/Context. Within service, operation is tracked with Activity object and every telemetry emitted will be associated with this Activity, thus sharing necessary correlation IDs.
In case of Service Bus / Event Hub communication, the propagation is also supported in the recent versions (IDs/Context propagate as metadata).
If service is not web-based and AI automated correlation propagation is not working, you may need to manually get incoming ID information from some metadata if any exists, restore/initiate Activity, start AI operation with this Activity. When telemetry item is generated in scope of that Activity, it will get proper IDs and will be part of the overarching trace. With that, if telemetry is generated from Log4net trace that was executed in the scope of AI operation context then that telemetry should get right IDs.
Code sample to access correlation from headers:
public class ApplicationInsightsMiddleware : OwinMiddleware
{
// you may create a new TelemetryConfiguration instance, reuse one you already have
// or fetch the instance created by Application Insights SDK.
private readonly TelemetryConfiguration telemetryConfiguration = TelemetryConfiguration.CreateDefault();
private readonly TelemetryClient telemetryClient = new TelemetryClient(telemetryConfiguration);
public ApplicationInsightsMiddleware(OwinMiddleware next) : base(next) {}
public override async Task Invoke(IOwinContext context)
{
// Let's create and start RequestTelemetry.
var requestTelemetry = new RequestTelemetry
{
Name = $"{context.Request.Method} {context.Request.Uri.GetLeftPart(UriPartial.Path)}"
};
// If there is a Request-Id received from the upstream service, set the telemetry context accordingly.
if (context.Request.Headers.ContainsKey("Request-Id"))
{
var requestId = context.Request.Headers.Get("Request-Id");
// Get the operation ID from the Request-Id (if you follow the HTTP Protocol for Correlation).
requestTelemetry.Context.Operation.Id = GetOperationId(requestId);
requestTelemetry.Context.Operation.ParentId = requestId;
}
// StartOperation is a helper method that allows correlation of
// current operations with nested operations/telemetry
// and initializes start time and duration on telemetry items.
var operation = telemetryClient.StartOperation(requestTelemetry);
// Process the request.
try
{
await Next.Invoke(context);
}
catch (Exception e)
{
requestTelemetry.Success = false;
telemetryClient.TrackException(e);
throw;
}
finally
{
// Update status code and success as appropriate.
if (context.Response != null)
{
requestTelemetry.ResponseCode = context.Response.StatusCode.ToString();
requestTelemetry.Success = context.Response.StatusCode >= 200 && context.Response.StatusCode <= 299;
}
else
{
requestTelemetry.Success = false;
}
// Now it's time to stop the operation (and track telemetry).
telemetryClient.StopOperation(operation);
}
}
public static string GetOperationId(string id)
{
// Returns the root ID from the '|' to the first '.' if any.
int rootEnd = id.IndexOf('.');
if (rootEnd < 0)
rootEnd = id.Length;
int rootStart = id[0] == '|' ? 1 : 0;
return id.Substring(rootStart, rootEnd - rootStart);
}
}
Code sample for manual correlated operation tracking in isolation:
async Task BackgroundTask()
{
var operation = telemetryClient.StartOperation<DependencyTelemetry>(taskName);
operation.Telemetry.Type = "Background";
try
{
int progress = 0;
while (progress < 100)
{
// Process the task.
telemetryClient.TrackTrace($"done {progress++}%");
}
// Update status code and success as appropriate.
}
catch (Exception e)
{
telemetryClient.TrackException(e);
// Update status code and success as appropriate.
throw;
}
finally
{
telemetryClient.StopOperation(operation);
}
}
Please note that the most recent version of Application Insights SDK is switching to W3C correlation standard, so header names and expected format would be different as per W3C specification.
The last two days I have started getting MessagingEntityNotFoundException when my code calls Microsoft.Azure.ServiceBus.Core.MessageReceiver.PeekAsync on dead letter queue message receiver. The code have not been changed before the exception started to occur, and have been running without problems for the last two months.
The full exception message is
Microsoft.Azure.ServiceBus.MessagingEntityNotFoundException: A request handler with id '56216' was not found in the entity management node $management.
The code runs in a time triggered Azure Function.
The code for setting up the message receivers:
using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using Microsoft.Azure.ServiceBus.Management;
public class DeadLetterQueueMonitor {
private static List<MessageReceiver> _messageReceivers;
private async Task<List<MessageReceiver>> GetMessageReceivers(ILogger logger)
{
if (_messageReceivers == null)
{
_messageReceivers = new List<MessageReceiver>();
var managementClient = new ManagementClient(ConnectionStrings.ServiceBusConnectionString);
var topics = await managementClient.GetTopicsAsync();
foreach (var topic in topics)
{
var subscriptions = await managementClient.GetSubscriptionsAsync(topic.Path);
_messageReceivers.AddRange(subscriptions.Select(subscription =>
{
string entityPath = EntityNameHelper.FormatDeadLetterPath(EntityNameHelper.FormatSubscriptionPath(topic.Path, subscription.SubscriptionName));
return new MessageReceiver(ConnectionStrings.ServiceBusConnectionString, entityPath);
}));
}
}
_messageReceivers = _messageReceivers
.Select(r => r.IsClosedOrClosing ? new MessageReceiver(ConnectionStrings.ServiceBusConnectionString, r.Path) : r)
.ToList();
return _messageReceivers;
}
}
Any suggestions for what might be wrong?
EDIT 25/2-2019 (as response to comments):
I am using the following Nuget packages:
Microsoft.Azure.Amqp 2.3.0
Microsoft.Azure.ServiceBus 3.1.0
Microsoft.Azure.Services.AppAuthentication 1.0.3
I have added a little more context for the method that creates the MessageReceivers.
I developed a couple of microservice using Azure functions, every service has independent use case and different programming language.
Now I have a use case to use all service in below order, So I developed one more Azure function to use all service in given order. below code running well.
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)]HttpRequestMessage req, TraceWriter log)
{
string returnValue = string.Empty;
dynamic data = await req.Content.ReadAsStringAsync();
if (data == null)
{
return req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a value in the request body");
}
else
{
string body = data.ToString();
var transformResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.TransformServiceEndPoint, body, HttpMethod.POST);
var validationResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.ValidationServiceEndPoint, transformResult.Result.ToString(), HttpMethod.POST);
if (validationResult.Result != null && Convert.ToBoolean(validationResult.Result))
{
var encryptionResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.EncryptionServiceEndPoint, transformResult.Result.ToString(), HttpMethod.POST);
var storageResult = await HttpRestHelper.CreateRequestAsync(AppConfiguration.StorageServiceEndPoint, encryptionResult.Result.ToString(), HttpMethod.POST);
returnValue = storageResult.Result.ToString();
}
else
{
returnValue = "Validation Failed";
}
return req.CreateResponse(HttpStatusCode.OK, returnValue, "text/plain");
}
}
Question
If every microservice takes 1 min to execution, I have to wait ~4min in my Super Service and billed for 4+ min. (We don't need to pay for waiting time :) https://www.youtube.com/watch?v=lVwWlZ-4Nfs)
I want to use Azure Durable functions here but didn't get any method to call external url.
Please help me or suggest a better solution.
Thanks In Advance
Durable Orchestration Functions don't work with arbitrary HTTP endpoints. Instead, you need to create individual functions as Activity-triggered.
Orchestration will use message queues behind the scenes rather than HTTP. HTTP is request-response in nature, so you have to keep the connection and thus pay for it.
Queue-based orchestrator can also give you some extra resilience in face of intermittent failures.
The web socket is written in javascript by my colleague. I managed to connect. First of all I have to log in on the application using a test account. I have to send the email and password through a json. I have installed the Json.Net packet using NuGet.
Some code that I found on my reaserch is this, but I do not understand how to send my data using that segment.
var buffer = new byte[1024];
var segment = new ArraySegment<byte>(buffer);
webSocket.SendAsync(segment, WebSocketMessageType.Text, true, CancellationToken.None);
Of course, I can use an object
User user=new User();
user.Email="bla#bla.com";
user.Password="pass";
string json = JsonConvert.SerializeObject(user);
But it will not be of any use because the method SendAsync accepts only byte type on segment.
All I want is to send that data, and if log in succeeds, I should receive other data (in Json format) about the user.
As a side note, I am quite new to web sockets, I used http protocols from ASP.NET WEB API 2.
I have no idea about Windows Phone 8, but by the code you pasted it seems similar to the regular .NET ClientWebSocket, so here you have some examples:
public static Task SendString(ClientWebSocket ws, String data, CancellationToken cancellation)
{
var encoded = Encoding.UTF8.GetBytes(data);
var buffer = new ArraySegment<Byte>(encoded, 0, encoded.Length);
return ws.SendAsync(buffer, WebSocketMessageType.Text, true, cancellation);
}
public static async Task<String> ReadString(ClientWebSocket ws)
{
ArraySegment<Byte> buffer = new ArraySegment<byte>(new Byte[8192]);
WebSocketReceiveResult result = null;
using (var ms = new MemoryStream())
{
do
{
result = await ws.ReceiveAsync(buffer, CancellationToken.None);
ms.Write(buffer.Array, buffer.Offset, result.Count);
}
while (!result.EndOfMessage);
ms.Seek(0, SeekOrigin.Begin);
using (var reader = new StreamReader(ms, Encoding.UTF8))
return reader.ReadToEnd();
}
}
If something does not compile or exists in WP8, just find an equivalent.
#vtortola is a working example in case your data comes in multiple segmented messages, but if all data comes in a single message you don't need all those streams to read the message, you just need to do this:
public static async Task<String> ReadString(ClientWebSocket socket)
{
var reciveBuffer = new byte[32000];
var result = await socket.ReceiveAsync(new ArraySegment<byte>(reciveBuffer), CancellationToken.None);
if (result.MessageType == WebSocketMessageType.Close)
{
await socket.CloseAsync(WebSocketCloseStatus.NormalClosure, string.Empty, CancellationToken.None);
}
return Encoding.ASCII.GetString(reciveBuffer , 0, result.Count);
}
If your message is splited in multiple segments or you don't know how your message is comming then you have to do like #vtortola
Also if you want to keep receiving messages you can do a while and call ReadString inside, like this:
while (socket.State == WebSocketState.Open)
{
var msg = ReadString(socket)
//do something with your message...
}