I am having difficulties working with azure pageable.
I have these dificulties in several places.......
What happens is, if i interact with an AzurePageable and something goes wrong, the thread just doesnt return ........
For example yesterday I had too many requests to Azure Appconfiguration, the following piece of code would just hang.......
// Get all the settings from Azure app configuration
private static Dictionary<string, string> GetAllXYZSettings(ConfigurationClient client)
{
var settingsSelector = new SettingSelector() { KeyFilter = "xyz:*" };
var settings = client.GetConfigurationSettings(settingsSelector);
Dictionary<string, string> config = new();
foreach (dynamic setting in settings)
{
string settingValue = (string)setting.Value;
if (!string.IsNullOrEmpty(settingValue))
{
config.Add(setting.Key, settingValue);
}
}
return config;
}
I tried several things, wrapped everything in a try catch but my thread would just not return.
How should i read the config so i can do some correct error handeling .....
Same behaviour is also observed when reading the servicebus queues.......
wrapped in try catch, didn't work
Related
I have a ASP.NET Web API application with supporting Azure Web Job with functions that are triggered by messages added to a storage queue by the API's controllers. Testing the Web API is simple enough using OWIN but how do I test the web jobs?
Do I run a console app in memory in the test runner? Execute the function directly (that wouldn't be a proper integration test though)? It is a continious job so the app doesn't exit. To make matters worse Azure Web Job-functions are void so there's no output to assert.
There is no need to run console app in memory. You can run JobHost in the memory of your integration test.
var host = new JobHost();
You could use host.Call() or host.RunAndBlock(). You would need to point to Azure storage account as webjobs are not supported in localhost.
It depends on what your function is doing, but you could manually add a message to a queue, add a blob or whatever. You could assert by querying the storage where your webjob executed result, etc.
While #boris-lipschitz is correct, when your job is continious (as op says it is), you can't do anything after calling host.RunAndBlock().
However, if you run the host in a separate thread, you can continue with the test as desired. Although, you have to do some kind of polling in the end of the test to know when the job has run.
Example
Function to be tested (A simple copy from one blob to another, triggered by created blob):
public void CopyBlob(
[BlobTrigger("input/{name}")] TextReader input,
[Blob("output/{name}")] out string output)
{
output = input.ReadToEnd();
}
Test function:
[Test]
public void CopyBlobTest()
{
var blobClient = GetBlobClient("UseDevelopmentStorage=true;");
//Start host in separate thread
var thread = new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
var host = new JobHost();
host.RunAndBlock();
});
thread.Start();
//Trigger job by writing some content to a blob
using (var stream = new MemoryStream())
using (var stringWriter = new StreamWriter(stream))
{
stringWriter.Write("TestContent");
stringWriter.Flush();
stream.Seek(0, SeekOrigin.Begin);
blobClient.UploadStream("input", "blobName", stream);
}
//Check every second for up to 20 seconds, to see if blob have been created in output and assert content if it has
var maxTries = 20;
while (maxTries-- > 0)
{
if (!blobClient.Exists("output", "blobName"))
{
Thread.Sleep(1000);
continue;
}
using (var stream = blobClient.OpenRead("output", "blobName"))
using (var streamReader = new StreamReader(stream))
{
Assert.AreEqual("TestContent", streamReader.ReadToEnd());
}
break;
}
}
I've been able to simulate this really easily by simply doing the following, and it seems to work fine for me:
private JobHost _webJob;
[OneTimeSetUp]
public void StartupFixture()
{
_webJob = Program.GetHost();
_webJob.Start();
}
[OneTimeTearDown]
public void TearDownFixture()
{
_webJob?.Stop();
}
Where the WebJob Code looks like:
public class Program
{
public static void Main()
{
var host = GetHost();
host.RunAndBlock();
}
public static JobHost GetHost()
{
...
}
}
I've tried to come up with something from the example in the WebJobsSDK gitHub
var eventHubConfig = new EventHubConfiguration();
string eventHubName = "MyHubName";
eventHubConfig.AddSender(eventHubName,"Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=SendRule;SharedAccessKey=xxxxxxxx");
eventHubConfig.AddReceiver(eventHubName, "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=ReceiveRule;SharedAccessKey=yyyyyyy");
config.UseEventHub(eventHubConfig);
JobHost host = new JobHost(config);
But I'm afraid that's not far enough for someone of my limited "skillset"!
I can find no instance of JobHostConfiguration that has a UseEventHub property (using the v1.2.0-alpha-10291 version of the Microsoft.AzureWebJobs package), so I can't pass the EventHubConfiguration to the JobHost.
I've used EventHub before, not within the WebJob context. I don't see if the EventHostProcessor is still required if using the WebJob triggering...or does the WebJob trigger essentially act as the EventHostProcessor?
Anyway, if anyone has a more complete example for a simpleton like me that would be really sweet! Thanks
From the documentation here, you should have all the information you need.
What you are missing is a reference of the Microsoft.Azure.WebJobs.ServiceBus.1.2.0-alpha-10291 nuget package.
The UseEventHub is an extension method that is declared in this package.
Otherwise your configuration seems ok.
Here is an example on how to receive or send messages from/to an EventHub:
public class BasicTest
{
public class Payload
{
public int Counter { get; set; }
}
public static void SendEvents([EventHub("MyHubName")] out Payload x)
{
x = new Payload { Counter = 100 };
}
public static void Trigger(
[EventHubTrigger("MyHubName")] Payload x,
[EventHub("MyHubName")] out Payload y)
{
x.Counter++;
y = x;
}
}
EventProcessorHost is still required, as the WebJob just provides the hosting environment for running it. As far as I know, EventProcessorHost is not integrated so deeply into WebJob, so its triggering mechanism cannot be used for processing EventHub messages. I use WebJob for running EventProcessorHost continuously:
public static void Main()
{
RunAsync().Wait();
}
private static async Task RunAsync()
{
try
{
using (var shutdownWatcher = new WebJobsShutdownWatcher())
{
await Console.Out.WriteLineAsync("Initializing...");
var eventProcessorHostName = "eventProcessorHostName";
var eventHubName = ConfigurationManager.AppSettings["eventHubName"];
var consumerGroupName = ConfigurationManager.AppSettings["eventHubConsumerGroupName"];
var eventHubConnectionString = ConfigurationManager.ConnectionStrings["EventHub"].ConnectionString;
var storageConnectionString = ConfigurationManager.ConnectionStrings["EventHubStorage"].ConnectionString;
var eventProcessorHost = new EventProcessorHost(eventProcessorHostName, eventHubName, consumerGroupName, eventHubConnectionString, storageConnectionString);
await Console.Out.WriteLineAsync("Registering event processors...");
var processorOptions = new EventProcessorOptions();
processorOptions.ExceptionReceived += ProcessorOptions_ExceptionReceived;
await eventProcessorHost.RegisterEventProcessorAsync<CustomEventProcessor>(processorOptions);
await Console.Out.WriteLineAsync("Processing...");
await Task.Delay(Timeout.Infinite, shutdownWatcher.Token);
await Console.Out.WriteLineAsync("Unregistering event processors...");
await eventProcessorHost.UnregisterEventProcessorAsync();
await Console.Out.WriteLineAsync("Finished.");
}
catch (Exception ex)
{
await HandleErrorAsync(ex);
}
}
}
private static async void ProcessorOptions_ExceptionReceived(object sender, ExceptionReceivedEventArgs e)
{
await HandleErrorAsync(e.Exception);
}
private static async Task HandleErrorAsync(Exception ex)
{
await Console.Error.WriteLineAsync($"Critical error occured: {ex.Message}{ex.StackTrace}");
}
I have a following test code that is placed inside a WebJob project. It is triggered after any blob is created (or changed) inside "cBinary/test1/" storage account.
The code works.
public class Triggers
{
public void OnBlobCreated(
[BlobTrigger("cBinary/test1/{name}")] Stream blob,
[Blob("cData/test3/{name}.txt")] out string output)
{
output = DateTime.Now.ToString();
}
}
The question is: how to get rid of ugly hard-coded const string "cBinary/test1/" and ""cData/test3/"?
Hard-coding is one problem, but I need to create and maintain couple of such strings (blob directories) that are created dynamically - depend of supported types. What's more - I need this string value in couple of places, I don't want to duplicate it.
I would like them to be placed in some kind of configuration provider that builds the blob path string depending on some enum, for instance.
How to do it?
You can implement INameResolver to resolve QueueNames and BlobNames dynamically. You can add the logic to resolve the name there. Below is some sample code.
public class BlobNameResolver : INameResolver
{
public string Resolve(string name)
{
if (name == "blobNameKey")
{
//Do whatever you want to do to get the dynamic name
return "the name of the blob container";
}
}
}
And then you need to hook it up in Program.cs
class Program
{
// Please set the following connection strings in app.config for this WebJob to run:
// AzureWebJobsDashboard and AzureWebJobsStorage
static void Main()
{
//Configure JobHost
var storageConnectionString = "your connection string";
//Hook up the NameResolver
var config = new JobHostConfiguration(storageConnectionString) { NameResolver = new BlobNameResolver() };
config.Queues.BatchSize = 32;
//Pass configuration to JobJost
var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
}
Finally in Functions.cs
public class Functions
{
public async Task ProcessBlob([BlobTrigger("%blobNameKey%")] Stream blob)
{
//Do work here
}
}
There's some more information here.
Hope this helps.
I know that the services get wired-up by instantiating the BasicAppHost, and the IoC by using the ConfigureContainer property, but where is the right place to add the filters? The test in question never fire the global filter:
[TestFixture]
public class IntegrationTests
{
private readonly ServiceStackHost _appHost;
public IntegrationTests()
{
_appHost = new BasicAppHost(typeof(MyServices).Assembly)
{
ConfigureContainer = container =>
{
//
}
};
_appHost.Plugins.Add(new ValidationFeature());
_appHost.Config = new HostConfig { DebugMode = true };
_appHost.GlobalRequestFilters.Add(ITenantRequestFilter);
_appHost.Init();
}
private void ITenantRequestFilter(IRequest req, IResponse res, object dto)
{
var forTennant = dto as IForTenant;
if (forTennant != null)
RequestContext.Instance.Items.Add("TenantId", forTennant.TenantId);
}
[TestFixtureTearDown]
public void TestFixtureTearDown()
{
_appHost.Dispose();
}
[Test]
public void CanInvokeHelloServiceRequest()
{
var service = _appHost.Container.Resolve<MyServices>();
var response = (HelloResponse)service.Any(new Hello { Name = "World" });
Assert.That(response.Result, Is.EqualTo("Hello, World!"));
}
[Test]
public void CanInvokeFooServiceRequest()
{
var service = _appHost.Container.Resolve<MyServices>();
var lead = new Lead
{
TenantId = "200"
};
var response = service.Post(lead); //Does not fire filter.
}
}
ServiceStack is set at 4.0.40
Updated
After perusing the ServiceStack tests (which I highly recommend BTW) I came across a few example of the AppHost being used AND tested. It looks like the "ConfigureAppHost" property is the right place to configure the filters, e.g.
ConfigureAppHost = host =>
{
host.Plugins.Add(new ValidationFeature());
host.GlobalRequestFilters.Add(ITenantRequestFilter);
},
ConfigureContainer = container =>
{
}
Updated1
And they still don't fire.
Updated2
After a bit of trial and error I think it's safe to say that NO, the filters are not hooked up while using the BasicAppHost. What I have done to solve my problem was to switch these tests to use a class that inherits from AppSelfHostBase, and use the c# servicestack clients to invoke the methods on my service. THIS does cause the global filters to be executed.
Thank you,
Stephen
No the Request and Response filters only fire for Integration Tests where the HTTP Request is executed through the HTTP Request Pipeline. If you need to test the full request pipeline you'd need to use a Self-Hosting Integration test.
Calling a method on a Service just does that, i.e. it's literally just making a C# method call on a autowired Service - there's no intermediate proxy magic intercepting the call in between.
I am experiencing intermittent SaveChangesException when creating CRM entities using the early binding APIs (generated context) via a Threading.Task.
Using the code below I am able to recreate the exception - note that it is intermittent and I can have several runs without experiencing it.
As you can see there is no shared state in my code, I am creating the CRM Connection, service and context per operation.
The exception thrown is:
Microsoft.Xrm.Sdk.SaveChangesException: An error occured while processing this request. ---> System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index
Am I missing something here or are the early bound SDK classes unsafe to use in a multi-threaded environment?
class Program
{
static void Main(string[] args)
{
foreach (var index in Enumerable.Range(1, 400))
{
var capturedIndex = index;
Task.Factory.StartNew(() => CreateContact(capturedIndex));
}
Console.Read();
}
private static void CreateContact(int i)
{
var contact = new Contact {FirstName = "Test", LastName = string.Concat("Contact_", i)};
var context = new XrmServiceContext(new OrganizationService(new CrmConnection("Crm"))); // The generated context is taken from the SDK samples
context.AddObject(contact);
try
{
context.SaveChanges();
Console.WriteLine(string.Concat("Contact created - ", i));
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
}
}
The exception occurs at a lower level within the WCF configuration. For those using late-bound entities, a workaround is to disable the ProxyTypesEnabled property on the CrmConnection. Another workaround is to set the ServiceConfigurationMode to PerInstance but unfortunately, this comes with a performance hit. Note: both of these can be set through the connection string as well.
var connection1 = new CrmConnection("Crm")
{
ProxyTypesEnabled = false
}; // late-bound only
var connection2 = new CrmConnection("Crm")
{
ServiceConfigurationInstanceMode = ServiceConfigurationInstanceMode.PerInstance
};