An automated Windows update this morning left my Windows Server 2012 R2 Classic Virtual Machine on Azure in a semi-crashed state. The VM is a web server, and all the files and applications in it are still accessible via the browser. In other words, IIS and a number of other services are still running. Unfortunately, however, the VM is not accessible via Remote Desktop and is unresponsive to commands from the Azure management interface on the portal.azure.com website.
This type of error is quite common and can be found reported on many other websites. The error has been happening to Windows users (not just Windows Server) for many years already, and none of the solutions online will work for Azure users, because they involve restarting from a CD, pressing shift-f8 during boot, issuing DOS commands, restoring from backup, or unchecking certain properties in VMWare or other software.
Does anybody have a real solution for this problem on Microsoft Azure?
After struggling with this for weeks, I think I was able to fix this with the help of Microsoft support! I decide to post the solution here in case it can help someone in the future. Here are the three things that you need to do to fix this:
1-Restore the VM from a backup prior to the crash. The VM with the "Undoing Changes" crash is pretty much toast at this point. Now, proceed to steps 2 and 3 to ensure that the next batch of Windows Updates won't crash it again!
2-On your new VM, ensure that the Environment Variables for TEMP and TMP both point to C:\Windows\TEMP. In my case, they were both pointing to a temporary folder in the logged in user's profile.
3-Ensure that C:\Windows\TEMP is always empty. I achieved this by setting up a scheduled task that runs a simple BAT file that deletes all files and folders inside of the C:\Windows\TEMP once a day. I spoke with a Microsoft representative who said that even though you may have plenty of hard drive space in your C:\ drive, the Windows TEMP folder is really not supposed to get much bigger than 500MB. When it gets very large you may have some issues with Windows Updates (mine was just under 500MB when the updates were failing).
I would recommend contacting Azure support as something may have to be done by an engineer to fix the issue and unfortunately classic VMs don't have the redeploy feature.
I've added only InboundPort 3389 RPD, and works well now.
Related
Help! The folder C:\ProgramData\Microsoft\Crypto\SystemKeys is growing out of control. It is doing this on some of our servers and some desktops. We are a medium to small business and use Active Directory (not Azure AD). I've heard that this folder is used by IIS, SQL Server, Remote Desktop Licence Server (maybe other things too?). My guess is it has something to do with Remote Desktop as I've not seen this problem with any computer that hasn't been using Remote Desktop. I've heard that some of the keys in this folder are important so you can't just delete it. Anyone have any idea what to do to get it to stop, what causes it, or how to clean it up?
Here my file properties:
Thank you for any help.
I found an answer to this question. Turns out in my case it was because of an application writing to this directory. Contacted them and they are implementing a fix in the next release (October 2021). In the mean time, I've deleted the extra keys in the folder. So far no issues.
After upgrading system to Windows 10 - os 1803 we are getting below issues while working with ClearCase 8.0.1.x/9.0.1.x
Unable to checkin/checkout.
Not able to create views.
Not able to add any file to source control.
The system hangs & crashes while performing any ClearCase operation.
There is no error message, but I have attached screenshot for reference.
Please let us know if there is any issue with the Windows 10 ver(1803), any security system enabled?
Or has ClearCase provided any fix?
We have tried 9.0.1.5 and issue still persists.
This is what we got from windows event log.
The computer has rebooted from a bugcheck.
The bugcheck was:
0x000000c2 (0x0000000000000004, 0x00000000535be990, 0x000000000004efd3, 0xfffff803e01848b1)
for most of them whoever has upgraded to windows 1803 ver :( for people who are still using ver1709 it is working perfectly fine
Then I would recommand contacting IBM support: only them can update their ClearCase 9/Windows 10 compatibility matrix and confirm if MVFS is supported on a more recent (1803) Windows 10 edition.
We also facing same problem and I have raised the case with IBM. Still not yet resolved. As IBM said there are some limitations to work ClearCase with windows 10 and windows 2016.
We tried all the options except Secure boot disable. If possible please do disable secure boot option in Windows 10 and try to checkin/checkout code from CleraCase
Note : It works for Snapshot views. That means the issue related to MVFS
I'm seconding #VonC's recommendation to open a ticket with IBM. When you do that, save a step and collect a clearbug2 and a kernel memory dump to send in as soon as the case is opened. It will save the turn-around time of us asking you for it. If the installed programs list doesn't list installed security software (DLP, Privilege management sw like Avecto, other endpoint security tools), please list those separately as well.
I would also love to know who # IBM told you there are "limitations" with Win10-1803.
There are a few issues with Windows 10 "version upgrades" breaking things, but they generally don't cause system crashes. Windows 10 upgrades are actually full OS installs that then (imperfectly) migrate application settings. Anything that uses custom network providers (ClearCase is one example) will find that the network providers will be broken or partially broken. Reinstalling is usually required. Again, that has not yet been reported as a cause of a BSOD.
If the upgrade/reinstall didn't fix view creation, please post a separate question on the view creation issue. There may be things we can do to the SMB 2 caches to allow view creation to work in cases where the view storage is not on the client host.
I noticed that the screen shot you posted is a Terminal Services disconnect screenshot. Does the issue only occur over a Terminal Services client connection or does it also happen on a local connection?
My site is hosted on Azure. I need to programmatically turn off Internet Explorer's default enhanced security configuration settings whenever I repave or redeploy a new box on Azure.
How do I do this?
I found this article on another site http://jetlounge.net/blogs/teched/archive/2009/10/25/fix-ie-esc-won-t-turn-off-internet-explorer-enhanced-security.aspx. It included the following command line syntax, but on my local box I couldn't find the IEHARDEN.INF file it referred to. I also don't think this solution is Azure-specific.
rundll32.exe setupapi.dll,InstallHinfSection IESoftenAdmin 128 %windir%\inf\IEHARDEN.INF
I need to turn off these default hardening settings under Azure because I have a 3rd party IE screen capture DLL that needs to execute Javascript on webpages.
I think that this approach, shaped in a Windows Azure StartupTask running in Elevated execution context will help you.
Just remember that the .bat or .cmd file you create needs to be UTF8 encoded. There used to be some issues with the batch files if they are not UTF8.
UPDATE
I decided to update the answer, because it would have been too long for a second comment. I want to first make clear that I do not intend to offend anyone and the next is just mine personal view and thoughts.
Well, I mine vision might be (is) distorted through mine prism. But, I think that these specifics has nothing to do with Windows Azure itself.
These are OS related configuration specifics and the approach would be one and the same (with some variations) regardless of a (hosting/cloud) provider. If you had to deploy your solution to a dedicated (or virtual) server, you would had to create some kind of scheduled task, or startup task to make these configuration changes. Or even interactively login to make these changes.
Since Windows Azure offers the StartUp Task, it is up to us (developers) to decide what to do and how to shape the OS according to our needs.
The OS configuration changes that one can possibly need are only limited by the total ammount of all available Windows Server 2008/R2 configuration options. I personally do not believe that these needs to be reflected in Windows Azure documentation by any means. They have their place in Windows Server documentation. It is arguable which are "commonly used", because what might be common for one, might also be "never needed" for others ...
I now have my Windows Azure environment set up so that I can access my Worker Role with Remote Desktop. However, I'm not sure how to proceed at the moment. After much digging I found a web site that was offline but in Google's cache there was mention of attaching to the Worker Role running in the Azure Cloud from the Visual Studio debugger. But I only have Visual Developer (not studio) 2010 and I have searched all over and as far as I can see there is no such option to attach to a remote server. I am able to publish my project to the Azure Cloud without error and I have a "healthy" instance of my Worker Role showing as active and running.
I did connect with RDP through the Azure Management portal. The login worked fine and up came the remote desktop window. I searched through much of what I could find and was unable to find my Worker Role. I must have the wrong impression of RDP, because I had hoped to see the Worker Role's main display form when I logged in, just like I do when I debug it locally in the Cloud Emulator. But instead all I saw was a blank desktop with some base level server inspection and management routines. I even checked the Event Viewer for Application related messages and saw none.
So now I'm stuck wondering if my Worker Role is actually running or not, despite the seemingly positive status messages from the Management Portal, and I still want to attach to my Worker Role for debugging through Visual Developer, if it's possible, but I am unable to figure out how.
Anyone with experience in this area that can give me some solid tips on what to do next, please respond.
UPDATE: I believe my worker role may be running because I opened a command window and did a Netstat and saw it listening on the correct port. However, that may just be my Worker Role shell class that starts the custom EXE I have it launch as a spawned proces. I still haven't confirmed if my custom EXE is running yet.
UPDATE-2: Just ran TaskList from a command window and the custom EXE is listed.
UPDATE-3: Everything is working as I just ran a remote test of the service so that's not a problem. Still want to know how to attach to the Worker Role from Visual Developer 2010 for remote debugging, and if it's possible to see the custom EXE's display form like I do when doing local debugging in the Cloud Emulator.
-- roschler
There is a set of articles here which goes in length on how to set up for remote debugging in Azure:
http://blogs.u2u.be/peter/post/2011/06/21/Remote-debugging-an-Azure-Worker-role-using-Azure-Connect-Remote-desktop-and-the-remote-debugger.aspx
http://blogs.u2u.be/peter/post/2011/06/24/Remote-debugging-an-Azure-worker-role-using-Azure-Connect-remote-desktop-and-remote-debugger-part-2.aspx
http://blogs.u2u.be/peter/post/2011/06/26/Remote-debugging-a-Windows-Azure-Worker-Role-using-Azure-Connect-Remote-desktop-and-the-remote-debugger-part-3.aspx
The key takeaway is that you don't need to actually install Visual Studio on Azure, you only need to copy the Remote Debugger bits and then use Azure Connect to add your developer machine to the Virtual Network.
You can setup Remote Debugging with Visual Studio 2012
http://code.msdn.microsoft.com/Remote-Debugging-Windows-dedaaec9
When you say:
But instead all I saw was a blank desktop with some base level server inspection and management routines.
this is exactly what you get with an Azure VM. It's a basic OS install, plus the bare minimum of Azure stuff it needs to run and the code you've uploaded. There's no fancy monitoring or health checks available on the machine by default, you're expected to have provided those yourself to have them available without having to RDP into the machine to check on it.
RDP is very good for tracking down certain problems, like checking that a startup task will run, checking which directories items are installed in and just generally being nosey. If you need extra tools to track down a problem, you can just install them while you're connected to the server. For example I have RDPed into a server and installed the Microsoft Debugging Tools, to track down a memory issue.
I suppose you could remote into your VM, install Visual Studio there, and debug the process...
I also suppose it might be possible to enable remote debugging (not sure what's involved there, but such a thing exists, and it works over TCP) and debug from a local instance of Visual Studio.
To my knowledge, neither is commonly done.
Based on other answers, you would be better off writing a log file to a local storage. You can read the file from RDP if you reallyhace to. Keep in mind, debugging on Azure isn't really simple, and rightly so.
What I was thinking though was, maybe you could run the process using the user's credentials. I can't verify at the moment, but you have a better shot of seeing the ui when you rdp.
We are starting with Sharepoint development with a team of three and are currently setting up our development environments. We would like to avoid installing a Server 2008 for each developer, thus a single terminal server has been setup, using Remote Windows to start a VS2008 instance on each developer's machine. Now we would like to separate developers' testing environments (i.e. a different site colletion per developer), but have realized that the assemblies would need to be installed into GAC to show properly on the site. But since there is AFAIK only one GAC, developers wouldn't be able to test their stuff independently.
Is there any way we could create separate testing environments without installing a bunch of 2008 Servers?
So you're all going to remote in an fire up Visual Studio and be compiling stuff and restarting IIS, etc?
You're going to be stamping on each other's toes.
A wiser choice nowadays is to use Hyper-V (or some other virtualisation).
We use Windows Server 2008 on our laptops, and use Hyper-V to run our dev environments. We then have a dev environment (sandbox) each, and these have VS2008, SVN, Nunit, etc.
Our code is tested against each other thanks to CruiseControl on the only shared Hyper-V.
This has been great for us... we distribute the load, we can work on the move, we don't step on each others toes and if we need to do a demo we can switch Hyper-Vs and demo from the demo Hyper-V (branched from the dev one early on so that the environments are known).
Go virtual and don't look back.
PS: I've just seen your comment about one server... just put Hyper-V on that and run 3 instances. That's also what we do ;)
I don't know about installing the server on everything, but this sounds like an ideal task for Virtual Machines rather than physical ones- where I work we using VMWare a whole lot for this kind of work and it does very well.
It's also useful to be able to roll back to a snapshot when it comes to testing installation processes and so on.
No. In addition to the GAC there are all the SharePoint files in the 12 hive, such as features and site templates. It's not worth what you save on server costs.
(Of course if you don't use the GAC, but deploy to the bin folder, and you don't touch anything in the 12 hive, you can give each developer their own web application on the same server. But this approach puts a lot of restrictions on what they can do. It's still not worth it.)
Virtual machines will work, but they can be slow to develop on. For instance, you'll need to restart the application pool for every GAC deploy - which means a pause of maybe 15-60 seconds to reload the application, (depending on the hardware). This will become annoying.
Virtual machines work better for test and production, where you don't restart the application so often.
I recommend a physical server for each developer. This will minimize the code-deploy-test cycle time, and make sure they don't have to worry about stepping on each others toes.
You are on the wrong track with Terminal Services - its just not going to give you any separation.
A lot of people do recommend developing on W003/2008 server directly, and it does simplify some things like remote debugging.
I prefer the more traditional method of using VMWare to run virtual machines. These can be running on a local or remote host. Remote debugging is a little more complex to setup but still possible.
Finally - if possible then deploy to the bin dir rather than the GAC. This will make it much easier to deploy automatically after compilation.
The contributors are right that there are lots of stumbling blocks to multi-developer single server environments.
Number one developers will be trying to attach to the same Web Application process w2ps.exe so creating separate Web Applications on different ports is a must unless you are prepared to share time debugging. How to setup a development environment for sharepoint 2013
The second problem is when you try to collaborate and use shared components/features. Having a desire to work separately is debatable, I believe that the team developers should be collaborating and sharing so combing work is desirable to ensure seamless integration into a single final solution and that no work is duplicated. The multi-developer single server environment works perfectly until you try to collaborate 'One common mistake is to have one “development server” used by all team developers. Unless team members are working on totally unrelated components and never need to do common things such as restart IIS or attach a debugger to an IIS process, this type of environment generally doesn’t work well.' http://technet.microsoft.com/en-us/magazine/dn145990.aspx We made this mistake through lack of experience and knowledge, but once you make it it's possible to work round it.
My first attempt to share features was to copy developer 1's project into developer 2's solution and add a reference to it in developer’s 2's project and add all the features to developer 2's package. Deploying this works fine for developer 2, until as I discovered if developer 1 detaches their solution from the debugger it retracts the solution based on the duplicated solution id from the farm and therefore from each developer's web application. Therefore developer 2 has the rug pulled out from underneath them. Although this is a part solution and seemed to work for a while, it took me a while to work out what was happening and what combinations of dev 1 and 2 deployments make each other’s work and not work.
So I found a better solution. Under the project properties in Visual Studio under SharePoint tab there is a combo box called 'Auto-retract after debugging'. This by default retracts the solution when the developer stops the attached debugger and pulls the features out from underneath the other developers. Unticking this box prevents the retract and leaves each developers individual solutions deployed at farm level and on reattaching to the debugger just replaces the solution with minimal fuss.
In my experience recycling the IIS application pool is so fast other developers don't even notice, but with a larger team than 2 this might become more prevalent, so perhaps someone else could add their experiences. I also guess unless the other develops try to attach at exactly the same time that the recycle is happening it'll be fine, so is a really small chance of having a cross over time, and simply detaching and reattaching will fix this if it is ever experienced.