I want to add tracing to my startup tasks. I was trying to enable DiagnosticsMonitorTraceListener in a powershell script that runs before all the actual startup tasks. Code:
Add-Type -Path ./Microsoft.WindowsAzure.Diagnostics.dll
$listener = new-object -type Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener
[System.Diagnostics.Trace]::Listeners.Add($listener)
$credentials = new-object -type Microsoft.WindowsAzure.StorageCredentialsAccountAndKey -argumentlist "ACCOUNT_NAME", "ACCOUNT_KEY"
$dmConfig = Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitor]::GetDefaultInitialConfiguration();
$dmConfig.Logs.ScheduledTransferPeriod = [System.TimeSpan]::FromMinutes(1)
[Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitor]::Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", $dmConfig)
For some reason my traces like this:
[System.Diagnostics.Trace]::TraceInformation("Starting startup tasks")
never end up transferred to Azure Storage. The script runs without any problem.
I know that there is a way to make Azure Diagnostic Infrastructure to copy logs generated by startup tasks but as far as I understand, the DiagnosticMonitor gets started only when a role starts. What if a startup task fail and the role never gets to run - the logs are not persisted.
Please let me know if I'm going the wrong way to solve this. Thanks.
One alternative if you are using Visual Studio Ultimate is to use IntelliTrace. That can look at a lot of things during startup. For external tasks, and things like that, check out:
http://blog.smarx.com/posts/windows-azure-startup-tasks-tips-tricks-and-gotchas
And
http://leastprivilege.com/2011/03/04/logging-output-of-azure-startup-tasks-to-the-event-log/
Related
I am working on Devops project to run QA powershell codes as custom script extensions . I need to run it on multiple Virtual machines (minimum 10). I figured out how to install custom script extension parallelly in VMs. But I did not find a solution uninstall custom script extensions parallelly in Vms . Please help. I am ok with ARM template or using Azure CLI .
one way would be to use jobs, something like this (rough sketch):
"vm1","vm2","vm3" | Foreach-Object {
Start-Job -ScriptBlock {
Remove-AzureRMVMCustomScriptExtension -ResourceGroupName xxx -VmName $using:PSItem -Name extensionname -Force
}
}
the above will work if you have azurermcontextautosave enabled. ARM Templates are not capable of removing custom script extension, you might experiment with Complete mode, but its a bit dangerous.
But honestly you just need to use forceUpdateTag to just force extension to rerun, without removing it
Is there any way through code to notify users about IIS application recycle or IIS reset?
better if I could check using APIs so that i can dispatch the message to central server which could sent out email or send instant messages?
There are many ways to do this. One of the easiest, I have realized is to use powershell. You could use something like:
Get-EventLog -log system -Source IISCTLS,WAS,W3SVC | Format-Table
Source,Message -Wrap -auto | Out-File some1.txt
to get a good txt file containing all the app pools restarts and IIS restarts.
If you would like to do it through IIS API then check http://www.muqeetkhan.com/how-to-use-microsoft-web-administration-from-powershell
You could easily translate that powershell script to a C# program as well.
Hope this helps.
You may use windows scheduled tasks with event trigger for this purpose. They are relatively simple to implement: find some recycle event in eventvwr (in System logs locate an event with ID 5074), right click on it, choose "Attach a Task to this event". Then choose "Start a program" and specify your batch/powershell script or an exe, etc. You can pass any data from the event as an argument. You can also define multiple events as triggers for this task.
I've created a start-up task for an Azure website that does the following:
Creates an AppPool
Converts a Virtual Directory into a Virtual Application
I've created a powershell script that carries out these tasks.
I've set up the startup element in the Service Definition
<Startup>
<Task commandLine="MyStartup.cmd" executionContext="elevated" taskType="simple">
</Task>
</Startup>
All good so far.
However, I've found out, rather late, that:
IIS may not be fully configured during the startup task stage in the
startup process, so role-specific data may not be available.
I take this to mean that the website may not exist on IIS when the powershell script is run. I've tested the script and sure enough it fails because it can't find the virtual directory on IIS.
My question is: Is there a way to ensure the powershell script is run after the website is created on IIS?
Please note, I don't really want to use Microsoft.WindowsAzure.ServiceRuntime.RoleEntryPoint.OnStart if possible.
Thanks in advance.
I followed on from #kwill's suggestion and created the following:
Powershell:
while(!($myWeb = Get-Website -name "*MyWeb*")){
Write-Host "Website not installed. Waiting 30 seconds..."
Start-Sleep 30
}
# Proceed with installation
and configuration
<Task commandLine="MyBackground.cmd" executionContext="elevated" taskType="background">
</Task>
This works.
Check out the role architecture diagram at http://blogs.msdn.com/b/kwill/archive/2011/05/05/windows-azure-role-architecture.aspx. You can see that the startup tasks are executed before IISConfigurator creates the application pool for your website. This means that the only place you can make modifications to the apppool is in OnStart.
I haven't tried this, but you could create a background type startup task which will let the rest of the startup process (ie. running IISConfigurator) proceed while your startup task is still running, and then within that startup just loop until the virtual directory is detected.
I'm currently working on an Azure project that works 100% locally with emulator resources. I'm now trying to deploy a worker role, but I'm running into an issue that I'm not sure how to troubleshoot.
Upon deploying the worker role in my Azure portal, the two instances continually loop through "recycling".
I can try to RDP into the role, but I only have about a minute to look around before the connection closes, I'm assuming due to the recycling.
After some searching it doesn't seem like this is a super common problem. Is there something trivial I'm overlooking that could be causing this issue? How would you go about troubleshooting this? Thank you for your time :)
In case of missing Reference you can troubleshoot this issue by:
Unzip your CSPKG file and then again unzip .CSSX file (just rename CSSX to zip) and match that everything references and static content is all there.. This way you can match what is on VM. Also in 2 minute windows when you RDP, try to look for Application event log for exception and get it because that would be the key to find the root cause.
IF you could see the exception in event log and look for the exception, you sure can find where it was generated. You can also use Intellitrace which might require you to redeploy the app.
Also there are ways were copying WinDBG and locking to the specific process you can debug it. I am not sure how much you would want to try but just copy the WinDBG to VM and use it would be enough (not sure how much experience you have with WinDBG though and how much time you would want to spent.)
Also been pestered by this role recycle issue numerous times. Here is the sequence of steps to debug persistent role recycles:
Debugging Azure Role Recycles
Enable Remote Access to your role - RDP login
Check eventvwr.msc (Windows Logs -> Application, App & Service Logs->Windows Azure)
Review the Azure text file logs across both C:\logs and c:\resources
Review custom logs in the Volume E: or F: for any custom role startup logging
Run AzureTools and attach to startup processes (download WinDBG, use Utils->Attach Debugger, select process - WaWorkerHost/WaIISHost, etc), use G to continue and watch debugger output for assemblies failing to load.
Installing Azure Debugging Tools via Powershell
PS> md c:\tools; Import-Module bitstransfer; Start-BitsTransfer http://dsazure.blob.core.windows.net/azuretools/AzureTools.exe c:\tools\AzureTools.exe; c:\tools\AzureTools.exe
If all items above fail - try using other tools in the AzureTools treasure trove - such as fusion logging, etc, this approach above will work!
WinDBG Sample Output - Failing to Locate Assembly (WaIISHost)
The most likely cause is that you have a missing assembly. One tactic to catch this is to wrap any startup processing in a master try/catch that manual logs the error to Azure storage.
If you added any referrences, check to make sure they're set to copylocal=true and that any external assets that were included in your service package were also set to be included.
From Avkash above:
Yes. this mean some issue in your Worker Role code is causing your Worker Role Host Process to crash.. If you look your fault stack you must see the function or the link from your code which generate this fault. IF you need help open a free Azure Support incident to Windows Azure Support team and they will help you.
Just a suggestion: Also Check the installable(if any)and any other references you use are 64bit.Azure VMs have 64bit OS. Once i was stuck up with this kind of problem due to 32/64 bit issues.
Are your worker roles exiting their work loop? A local recycle is very fast and you might not notice it, but spin-up time in the cloud can be long.
If the issue is caused by a startup batch file, I have stopped the loop by editing the batch file on the instance to include "exit /b 0" at the beginning. This will tell Azure that the startup was successful and you then have all the time you need to diagnose issues without the VM getting killed.
I created "Timer Job" in two ways,
1.Created a SharePoint Project through the Visual Studio 2010 and added several Features and webparts that's successfully deployed later i added the "Timer job" through the feature with the scope "Site" the timerjob not available in "Job Definitions".
2.When created a project with timer job feature only it's available in "Job Definitions".
I need to deploy the timer job along with other features as mention in first scenario
Regards,
Pavan.
You should deploy timer jobs in features scoped "Web Application". Please post the code you register your job with if this does not resolve your issue.
I've successfully deployed timer jobs to features scoped as tightly as 'Web' with no issues.
Doing it this way made it easier for me get the job settings to associate with specific list instances and I don't have to hard code any site collection URLs or whatever into the job code.
You still have to make sure you get it registered with the Web Application, and here's how I designed my feature activation for my web-scoped feature:
Public Overrides Sub FeatureActivated(ByVal properties As SPFeatureReceiverProperties)
Dim web As SPWeb = DirectCast(properties.Feature.Parent, SPWeb)
' Remove job if it exists.
DeleteJobAndSettings(web.Site.WebApplication)
' Create the job.
Dim job As New TimerJob_myClass(TimerJob_myClass.JobName, web.Site.WebApplication)
Also, even though it's a Web feature and shows up under 'Manage Features', it's better to mark it hidden. This type of feature can only be activated by running an stsadm command line from PowerShell. For example:
stsadm -o activatefeature -url http://intranet.contoso.local/ -id 01234567-AAAA-BBBB-CCCC-DDDDEEEEFFFF
You should run PowerShell as a user who has rights on the content database associated with the site your working with. The app pool that would have normally run the feature receiver code during a GUI activation does not have access to the config database and that's why it fails via GUI.
my problem solved in production by activating feature with the following powershell
Enable-SPFeature -identity "sample_Job Definition" -URL http://portaluat
my job definition scope was Site