Background: I've deployed an MVC3 application to 2 Azure Web Role instances, but I'm confused as to how I can test out the possibility of one of these instances failing.
Is there a way that I can test to ensure that my Web Role code works seamlessly when one of my instances is taken offline?
Can I manually stop one of them? Or somehow configure the load balancer to force all traffic to one of the servers?
Thanks!
If you have RDP access enabled to your instances you can very easily remove one or more instance out from LoadBalancer even when the instance is running healthy without writing any line of code. You just need to RDP to your instance and then use PowerShell scripts to take the instance off from loadbalancer. In my following blog I have described the exact procedure:
http://blogs.msdn.com/b/avkashchauhan/archive/2012/01/27/windows-azure-troubleshooting-taking-specific-windows-azure-instance-offline.aspx
The above details also help to run load testing by removing N instances from total M instances .
If you use the StatusCheck event to set the status to Busy, your instance won't receive any more requests from the loadbalancer. You might want to write some code to send messages to your instances that will set them as busy for some time (using queues for example).
public override bool OnStart()
{
RoleEnvironment.StatusCheck += RoleEnvironmentStatusCheck;
return base.OnStart();
}
// Use the busy object to indicate that the status of the role instance must be Busy
private volatile bool busy = true;
private void RoleEnvironmentStatusCheck(object sender, RoleInstanceStatusCheckEventArgs e)
{
If (this.busy)
{
// Sets the status of the role instance to Busy for a short interval.
// If you want the role instance to remain busy, add code to
// continue to call the SetBusy method
e.SetBusy();
}
}
Reference: http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.serviceruntime.roleenvironment.statuscheck.aspx
Besides that you can also simply reboot your instance (using the Windows Azure portal or Remote Desktop).
Steve Marx (aka smarx) developed a tool called WazMonkey. It is the azure counterpart of the tool Chaos Monkey developed by the netflix team to simulate broken instances in amazon AWS. WazMonkey terminates instances of a windows azure cloud service randomly to test the resilience of a cloud applications.
Related
I have an Azure logic app that correctly creates an Azure Container Instance. The container starts, does its job and terminates. I need to collect its logs with the appropriate connector and write them to an azure blob.
I have all the pieces in place but I do not know how to wait for the container to terminate before using the "get logs of container" connector to collect logs.
If the container job would last a predictable amount of time, I could use the Delay connector before getting the logs and it would suffice (I've tried with short jobs and it works well).
But my jobs may last several hours, depending on some external factors, so the Delay technique does not work.
I've tried with the "Until" connector, together with delay and the "get properties of a container group" container to wait until the state of the container is not "terminated", but without success (maybe I did it wrong). Anyway this can be quite expensive, since every "check" is billed.
How can I wait for the container to terminate before asking for its logs?
thanks.
starting from Charles Xu's answer, the correct sequence when setting the variable is
this uses the "state" container instance variable instead of "provisioning state". The latter is about the creation of the container group, the first is about the state of the container instance, which is what I need.
I added a delay to decrease the number of (paid) runs of the connector.
If you want to get the logs of a container group and nobody knows when it can terminate. In the logic app, you can use a variable to store the state of the container group and then use an until control the loop which will get the terminate of the container group until it comes true.
Here are the steps:
Create a container group;
Get the provision state of the container group;
Initialize a variable to store the provision state of the container group;
In the untile, get the provision state of the container group until the state is equal to terminate;
Get the logs of the container group.
The whole structure:
The Initialize variable and Until steps:
Since you mentioned using "Until" in logic app maybe expensive, here I provide another solution for your reference.
We can create a time trigger azure function and set the cron expression every 1 minutes, create a service plan(free tier) for the function, so we don't need to pay for the running cost of the function(but may pay for the storage of the function).
The function will run every 1 minute. In function we need to get the properties of the container by using this rest api and then if the state is "terminated", call the logic app request to trigger the logic app(the logic app should be created as "When a HTTP request is received").
The function code(in local, before deployed to azure) should be as below:
using System;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
namespace hurytimeFun
{
public static class Function1
{
[FunctionName("Function1")]
public static void Run([TimerTrigger("0 */1 * * * *")]TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
//1. using rest api to get the properties of your container
//2. if the state is "terminated"
// call the logic app request to trigger the logic app
}
}
}
Hope it would be helpful to your problem~
Once a container terminates, everything is lost. You could install the log-analytics-containers from the Azure marketplace for logging and log before the container terminates
I want to migrate typical windows service which is written in .net to Azure using Service fabric.
To implement this , I am creating one service fabric application containing one micro service as guest executable which uses .exe of windows service and deploying application package to service fabric cluster.
After deploying service fabric application on cluster I want windows service should install & start automatically on all nodes however at any time application is running on any single node. I want windows service should run on only one node at a time.
Please kindly help to implement this.
You can certainly run your service as a guest executable. Making sure it only runs on one node can be done by setting the instance count to 1 in the manifest, like so:
<Parameters>
<Parameter Name="GuestService_InstanceCount" DefaultValue="-1" />
</Parameters>
...
<DefaultServices>
<Service Name="GuestService">
<StatelessService ServiceTypeName="GuestServiceType"
InstanceCount="[GuestService_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
Or, you could actually migrate it, not just re-host it in the SF environment...
If your Windows Service is written in .NET and the you wan't to benefit from Service Fabric then the job of migrating the code from a Windows Service to a Reliable Service in Service Fabric should not be to big.
Example for a basic service:
If you start by creating a Stateless Service in a Service Fabric application you end up with a service implementation that looks like (comments removed):
internal sealed class MigratedService : StatelessService
{
public MigratedService(StatelessServiceContext context)
: base(context)
{ }
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
return new ServiceInstanceListener[0];
}
protected override async Task RunAsync(CancellationToken cancellationToken)
{
// TODO: Replace the following sample code with your own logic
// or remove this RunAsync override if it's not needed in your service.
long iterations = 0;
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
ServiceEventSource.Current.ServiceMessage(this.Context, "Working-{0}", ++iterations);
await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
}
}
The RunAsync method starts running as soon as the Service is up and running on a node in the cluster. It will continue to run until the cluster, for some reason, decides to stop the service, or move it to another node.
In your Windows Service code you should have a method that is run on start. This is usually where you set up a Timer or similar to start doing something on a continuous basis:
protected override void OnStart(string[] args)
{
System.Timers.Timer timer = new System.Timers.Timer();
timer.Interval = 60000; // 60 seconds
timer.Elapsed += new System.Timers.ElapsedEventHandler(this.OnTimer);
timer.Start();
}
public void OnTimer(object sender, System.Timers.ElapsedEventArgs args)
{
...
DoServiceStuff();
Console.WriteLine("Windows Service says hello");
}
Now grab that code in OnTimer and put it in your RunAsync method (and any other code you need):
protected override async Task RunAsync(CancellationToken cancellationToken)
{
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
DoServiceStuff();
ServiceEventSource.Current.ServiceMessage(this.Context,
"Reliable Service says hello");
await Task.Delay(TimeSpan.FromSeconds(60), cancellationToken);
}
}
Note the Task.Delay(...), it should be set to the same interval as your Windows Service had for it's Timer.
Now, if you have logging in your Windows Service and you use ETW, then that should work out of the box for you. You simply need to set up some way of looking at those logs from Azure now, for instance using Log Analytics (https://learn.microsoft.com/en-us/azure/log-analytics/log-analytics-service-fabric).
Other things you might have to migrate is if you run specific code on shut down, on continue, and if you have any parameters sent to the service on startup (for instance connection strings to databases). Those need to be converted to configuration settings for the service, look at SO 33928204 for a starting point for that.
The idea behind service fabric is so that it manages your services, from deployment and running. Once you've deployed your service/application to the service fabric instance it will be just like running a windows service (kinda) so you wont need to install your windows service. If you're using something like TopShelf you can just run the exe and everything will run totally fine within service fabric.
My cloud service is deployed with two instances by default and has auto-scaling feature on which makes Azure to deploy additional instances as needed. I need to signal all instances currently running to invalidate their internal caches. Ideally by POSTing to a public facing url. But Azure load balancer will route the request randomly to one of the instances only.
I probably can utilize Service Bus topics/subscriptions to solve the problem but that looks like overkill. Is there an easier way?
From https://msdn.microsoft.com/library/azure/microsoft.windowsazure.serviceruntime.roleenvironment.changing.aspx:
RoleEnvironment.Changing Event
Occurs before a change to the service configuration is applied to the running instances of a role.
The Changing event and the Changed event are used together to identify and manage configuration changes to the service model. By using the Changing event, an instance can respond to a configuration change [...]
To be honest i can't remember if this is how you do it, but i believe i'm pretty close:
private void RoleEnvironmentChanging(object sender, RoleEnvironmentChangingEventArgs e)
{
if ((e.Changes.Any(change => change is RoleEnvironmentConfigurationSettingChange)))
{
// Deal with e.Cancel
// Invalidate cache
}
}
Here's more on what events are triggered on configuration change:
https://azure.microsoft.com/en-us/blog/responding-to-role-topology-changes/
The following diagram shows which events fire in an example scenario containing a single role. 2 instances are deployed initially, the deployment is then scaled to 4 instances, then back down to 3, and finally the deployment is stopped.
I'm trying to find a solution that I can use to perform virus scanning on files that have been uploaded to Azure blob storage. I wanted to know if it is possible to copy the file to local storage on a Worker Role instance, call Antimalware for Azure Cloud Services to perform the scan on that specific file, and then depending on whether the file is clean, process the file accordingly.
If the Worker Role cannot call the scan programmatically, is there a definitive way to check if a file has been scanned and whether it is clean or not once it has been copied to local storage (I don't know if the service does a real-time scan when new files are added, or only runs on a schedule)?
There isn't a direct API that we've found, but the anti-malware services conform to the standards used by Windows desktop virus checkers in that they implement the IAttachmentExecute COM API.
So we ended up implementing a file upload service that writes the uploaded file to a Quarantine local resource, then calling the IAttachmentExecute API. If the file is infected then, depending on the anti-malware service in use, it will either throw an exception, silently delete the file or mark it as inaccessible. So by attempting to read the first byte of the file, we can test if the file remains accessible.
var type = Type.GetTypeFromCLSID(new Guid("4125DD96-E03A-4103-8F70-E0597D803B9C"));
var svc = (IAttachmentExecute)Activator.CreateInstance(type);
try {
svc.SetClientGuid(ref clientGuid);
svc.SetLocalPath(path);
svc.Save();
}
finally
{
svc.ClearClientState();
}
using (var fileStream = File.OpenRead(path))
{
fileStream.ReadByte();
}
[Guid("73DB1241-1E85-4581-8E4F-A81E1D0F8C57")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IAttachmentExecute
{
void SetClientGuid(ref Guid guid);
void SetLocalPath(string pszLocalPath);
void Save();
void ClearClientState();
}
I think the best way for you to know is simply take an Azure VM (IaaS) and activate Microsoft Antimalware extension. Then you may log into it and do all the necessary check and tests against the service.
Later, you will apply all this into the Worker Role (there is a similar PaaS extension available for that, calles PaaSAntimalware).
See the next excerpt from https://msdn.microsoft.com/en-us/library/azure/dn832621.aspx:
"In PaaS, the VM agent is called GuestAgent, and is always available on Web and Worker Role VMs. (For more information, see Azure Role Architecture.) The VM agent for Role VMs can now add extensions to the cloud service VMs in the same way that it does for persistent Virtual Machines.
The biggest difference between VM Extensions on role VMs and persistent VMs is that with role VMs, extensions are added to the cloud service first and then to the deployments within that cloud service.
Use the Get-AzureServiceAvailableExtension cmdlet to list all available role VM extensions."
I'm doing my first project but large one on developing Azure Application with Intergration Component.
Currently most of the integration are done using SSIS Packages and would like to transform them on to Worker Role in Azure.
Could someone please help me to understand the following queries regarding Worker Role please?
Is there way to start or stop the Worker role (just like SSIS or Windows Schedulers) via GUI? If not how to achieve this?
How do I know my worker role has been running or not running (including why it's not running ie. logs)
How do I spin multiple worker role based on time (i.e. (9:00AM to 11:00AM spin 4 roles and scale down on quiet period)
Does the following code creates any poison message or dead lock (if multiple there are 10,000 messages to process and every 5 seconds the new thread (Processsing.run) is started?
while(true)
{
var thread = new Thread(Run);
thread.start();
Thread.Sleep(5000);
Trace.WriteLine("Working", "Information");
}
public class PhotoProcessing
{
public static void Run()
{
// Read from queue
CloudQueueMessage msg =
Storage.Queue.GetNextMessage();
while(msg != null)
{
string[] message = msg.AsString.Split('$');
if(message.Length == 2)
{
AddWatermark(message[0], message[1]);
}
// Message has been read so remove it
Storage.Queue.DeleteMessage(msg);
// Get next message if any
msg = Storage.Queue.GetNextMessage();
}
}
Is there way to start or stop the Worker role (just like SSIS or Windows Schedulers) via GUI? If not how to achieve this?
There are actually many ways to achieve this. You can use Windows Azure Portal to do or you could use 3rd party tools (like our Cloud Storage Studio) or you could write your own application using Windows Azure Service Management API (http://msdn.microsoft.com/en-us/library/ee460799.aspx)
How do I know my worker role has been running or not running (including why it's not running ie. logs)
Again you could use one of the GUI based tools to see the status of your roles. As far as why the roles are not running, you would need to enable Windows Azure Diagnostics in your worker role (http://msdn.microsoft.com/en-us/library/gg433048.aspx)
How do I spin multiple worker role based on time (i.e. (9:00AM to 11:00AM spin 4 roles and scale down on quiet period)
You can write your own application using Windows Azure Service Management API to do so or you could make use of 3rd party tools like AzureWatch from Paraleap or Azure Management Cmdlets (both from Microsoft and our company). While the cmdlets will get the job done, I believe Azure Watch is much more sophisticated solution. We wrote a blog post for autoscaling some days back which you can find here: http://www.cerebrata.com/Blog/post/Scale-your-Windows-Azure-instances-with-Azure-Management-Cmdlets.aspx.