We have Azure PAAS service implementation.In that in each Instance we have 10 customer and each customer owns a mounted cloud drive(page blob) to store the some files.
This deployment is available for last one year in azure .
For last 2-3 weeks we observe that is 1-2 cloud drive(page blob) getting un-mounted from this instance .We got some error information from the System log of event viewer which is added and this error is also not consistent. Currently as work around we are rebooting the Instance daily which we remount the vhd (pageblob) again.
Guest OS version-1.18
Azure SDK 1.7
Please let us know what is reason for this issue?
Error details
Log Name: System
Source: PlugPlayManager
Date: 4/22/2013 11:10:50 AM
Event ID: 12
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: RD00155D477FE9
Description:
The device 'Msft VHD Disk SCSI Disk Device' (SCSI\Disk&Ven_Msft&Prod_VHD_Disk\1&26c3c0c&0&000002) disappeared from the system without first being prepared for removal.
Log Name: System
Source: WaDrivePrt
Date: 4/22/2013 11:10:49 AM
Event ID: 4
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: RD00155D477FE9
Description:
'/lwe_2f44e5e3.vhd' failed to renew lease the specified XDisk.
Related
For background knowledge: Compute/Memory nova instances in Rackspace don't come with a local root volume, Rackspace has a policy to create them with an external SSD bootable volumes. Now the question:
I am trying to create a Compute flavor instance in Rackspace using pyrax api, in a way that Rackspace does in its UI(https://support.rackspace.com/how-to/boot-a-server-from-a-cloud-block-storage-volume/) as follows:
pyrax.cloudservers.servers.create(hostname,image.id,
flavor.id,block_device_mapping,
security_groups=security_groups,
nics=networks, key_name=key)
where
block_device_mapping = {"vda": "59fb72d5-0b33-46c2-b10b-33fed25c5f74:::1"},
the long 32 digit number is the volume_id of the volume I create before server creation using
pyrax.cloud_blockstorage.create(name=volume_name, size=volume_size,
volume_type=volume_type).
I get an error saying:
Policy doesn't allow memory_flavor:create:image_backed to be performed.(HTTP 403).
Also for other flavors which come with a local root volume(needless to say I don't have reference those with 'block_device_mapping' param), the pyrax api for instance creation works fine.
Here is a little thread on the topic in the pyrax/rackspace repo on github: https://github.com/rackspace/pyrax/issues/484 that discusses about the issue.
Is there something I am missing?
When a bootable volume is created, image_id(OS image id) should be specified to boot the volume:
pyrax.cloud_blockstorage.create(name=volume_name, size=volume_size,
volume_type=volume_type,image=image.id)
Also The block_device_map needs some more params:
block_device_map = [{
'boot_index': '0',
'source_type': 'image',
'destination_type': 'volume',
'delete_on_termination': True,
'uuid': image.id,
'volume_size': int(requested_size),
'device_name': 'vda'
}]
And here's the final catch in actually not getting a 403 Forbidden error:
While creating a server instance, don't specify the image id again in the pyrax call params, otherwise pyrax gets confused with what image to boot the instance. Hence just put a None to image_id in the params for pyrax.cloudservers.servers.create() as:
pyrax.cloudservers.servers.create(
hostname,
image=None,
flavor=flavor.id,
block_device_mapping_v2=block_device_map,
security_groups=security_groups,
nics=networks,
key_name=key)
As stated in the azure support page, I'm writing on Stack Overflow to find a solution to my issue, but for me this looks a little bit off topic...
When I'm trying to delete a group resource via linux terminal I get:
Delete resource group Default-Storage-WestEurope? [y/n] y
+ Deleting resource group Default-Storage-WestEurope
error: Long running operation failed with error: 'Invalid status code with response body "{"Error":{"Code":"ResourceGroupDeletionBlocked","Target":null,"Message":"Deletion of resource group 'Default-Storage-WestEurope' failed as resources with identifiers 'Microsoft.ClassicStorage/storageAccounts/bitnamiwesteuropecfuropu' could not be deleted. The provisioning state of the resource group will be rolled back. The tracking Id is 'f791a8f0-a28a-4fe3-b491-c6251b51d987'. Please check audit logs for more details.","Details":[{"Code":null,"Target":"/subscriptions/5fdcf34e-ecda-408e-b3ba-e706ac34dba6/resourceGroups/Default-Storage-WestEurope/providers/Microsoft.ClassicStorage/storageAccounts/bitnamiwesteuropecfuropu","Message":"{\"error\":{\"code\":\"StorageAccountOperationFailed\",\"message\":\"Unable to delete storage account 'bitnamiwesteuropecfuropu': 'Storage account bitnamiwesteuropecfuropu has some active image(s) and/or disk(s), e.g. bitnami-bitnami-redis-3.2.1-0-westeurope-CfuROpU. Ensure these image(s) and/or disk(s) are removed before deleting this storage account.'.\"}}","Details":null}]}}" occurred when polling for operation status.'.
info: Error information has been recorded to /home/giumbai/.azure/azure.err
error: group delete command failed
Edit: So I've made some progress, but still not enough. So i have a blob that has an image with an lease, in order to bake the lease i used this command: azure storage blob lease break -a bitnamiwesteuropecfuropu -k <my key> then i was prompted to insert the container name and blob name.
But didn't worked, i get this error, that i don't really understand:
{ ArgumentNullError: Required argument blob for function _leaseImpl is not defined
<<< async stack >>>
at throwMissingArgument (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:417:9)
at ArgumentValidator._.extend.exists (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:447:7)
at ArgumentValidator._.extend.string (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:426:10)
at /usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:4661:9
at Object.validateArgs (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:495:3)
at Object.BlobService._leaseImpl (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:4660:14)
at Object.BlobService.breakLease (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:1253:8)
at Function.Object.defineProperty.value (/usr/lib/node_modules/azure-cli/node_modules/streamline/lib/callbacks/builtins.js:367:19)
at __1 (/usr/lib/node_modules/azure-cli/lib/util/storage.util.js:423:41)
at StorageUtil_performStorageOperation__1 (/usr/lib/node_modules/azure-cli/lib/util/storage.util.js:421:5)
at StorageUtil_breakLease__10 (/usr/lib/node_modules/azure-cli/lib/util/storage.util.js:1609:31)
at breakLease (/usr/lib/node_modules/azure-cli/lib/commands/storage/storage.blob.js:817:17)
at breakBlobLease (/usr/lib/node_modules/azure-cli/lib/commands/storage/storage.blob.js:802:5)
<<< raw stack >>>
at throwMissingArgument (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:417:9)
at ArgumentValidator._.extend.exists (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:447:7)
at ArgumentValidator._.extend.string (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:426:10)
at /usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:4661:9
at Object.validateArgs (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:495:3)
at Object.BlobService._leaseImpl (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:4660:14)
at Object.BlobService.breakLease (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:1253:8)
at Function.Object.defineProperty.value (/usr/lib/node_modules/azure-cli/node_modules/streamline/lib/callbacks/builtins.js:367:19)
at __$__1 (/usr/lib/node_modules/azure-cli/lib/util/storage.util.js:423:41)
at __func (/usr/lib/node_modules/azure-cli/node_modules/streamline/lib/callbacks/runtime.js:47:5)
stack: [Getter/Setter],
name: 'ArgumentNullError',
argumentName: 'blob',
message: 'Required argument blob for function _leaseImpl is not defined',
__frame:
{ name: 'StorageUtil_performStorageOperation__1',
line: 402,
file: '/usr/lib/node_modules/azure-cli/lib/util/storage.util.js',
prev:
{ name: 'StorageUtil_breakLease__10',
line: 1598,
file: '/usr/lib/node_modules/azure-cli/lib/util/storage.util.js',
prev: [Object],
calls: 3,
active: false,
offset: 11,
col: 30 },
calls: 1,
active: false,
offset: 19,
col: 4 },
rawStack: [Getter] }
ArgumentNullError: Required argument blob for function _leaseImpl is not defined
<<< async stack >>>
at throwMissingArgument (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:417:9)
at ArgumentValidator._.extend.exists (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:447:7)
at ArgumentValidator._.extend.string (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:426:10)
at /usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:4661:9
at Object.validateArgs (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:495:3)
at Object.BlobService._leaseImpl (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:4660:14)
at Object.BlobService.breakLease (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:1253:8)
at Function.Object.defineProperty.value (/usr/lib/node_modules/azure-cli/node_modules/streamline/lib/callbacks/builtins.js:367:19)
at __1 (/usr/lib/node_modules/azure-cli/lib/util/storage.util.js:423:41)
at StorageUtil_performStorageOperation__1 (/usr/lib/node_modules/azure-cli/lib/util/storage.util.js:421:5)
at StorageUtil_breakLease__10 (/usr/lib/node_modules/azure-cli/lib/util/storage.util.js:1609:31)
at breakLease (/usr/lib/node_modules/azure-cli/lib/commands/storage/storage.blob.js:817:17)
at breakBlobLease (/usr/lib/node_modules/azure-cli/lib/commands/storage/storage.blob.js:802:5)
<<< raw stack >>>
at throwMissingArgument (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:417:9)
at ArgumentValidator._.extend.exists (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:447:7)
at ArgumentValidator._.extend.string (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:426:10)
at /usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:4661:9
at Object.validateArgs (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/common/util/validate.js:495:3)
at Object.BlobService._leaseImpl (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:4660:14)
at Object.BlobService.breakLease (/usr/lib/node_modules/azure-cli/node_modules/azure-storage/lib/services/blob/blobservice.js:1253:8)
at Function.Object.defineProperty.value (/usr/lib/node_modules/azure-cli/node_modules/streamline/lib/callbacks/builtins.js:367:19)
at __$__1 (/usr/lib/node_modules/azure-cli/lib/util/storage.util.js:423:41)
at __func (/usr/lib/node_modules/azure-cli/node_modules/streamline/lib/callbacks/runtime.js:47:5)
Edit2: Interesting i managed to delete the remaining image i had to brake the lease on the image and on the container, so both the image and container are now deleted, but when i try to delete the empty storage i get:
Failed to delete storage account 'bitnamiwesteuropecfuropu'. Unable to delete storage account 'bitnamiwesteuropecfuropu': 'Storage account bitnamiwesteuropecfuropu has some active image(s) and/or disk(s), e.g. bitnami-bitnami-redis-3.2.1-0-westeurope-CfuROpU. Ensure these image(s) and/or disk(s) are removed before deleting this storage account.'.
Proof :)
Successfully deleted blob 'bitnami-images/bitnami-bitnami-redis-3.2.1-0-westeurope-CfuROpU'.
You should not threat to leave the service for ever this will not get your answers any faster. If you read the error message it is pretty clear on that what it is happening. You are trying to delete a storage account that has a disk that it is attached to a machine. You cannot delete a storage account with a disk that is with a running machine. Go to the portal and check the storage account you will find that the storage account has a file inside it. If you click on this vcs file you are going to see that the state of this file it is locked and the lease has infinity duration. Checking on your account you até going to find a machine that has a disk file on this storage account. Delete the virtual machine and delete the storage account after the lease will be released when you delete the storage account.
Worker role instance stops daily and i get following error in WADWindowsEventLogsTable.
Instance status is unhealthy after this event is logged.
Fault bucket , type 0
Event Name: PDUWICA
Response: Not available
Cab Id: 0
Problem signature:
P1: 8
P2: 1.6
P3: 6.3.0.0
P4: 1033
P5: 61
P6:
P7:
P8:
P9:
P10:
Attached files:
These files may be available here:
Analysis symbol:
Rechecking for solution: 0
Report Id: a32d8f81-b668-11e5-80c8-000d3a316974
Report Status: 0
Hashed bucket:
Can you enable Intellitrace, re-deploy your instance (suggest to use local emulator if possible for a staging instance), get a more specific information about any unhealthy experience as posted and report the same? Review this link for a step by step guide on how to enable and use Intellitrace (http://justazure.com/microsoft-azure-cloud-services-part-5-debug-monitor-scale/)
Edit:
Once logged in to the instance with Remote assistance.
Try to access the Schedule Tasks, following is the screenshot on my PC and that might get you some specific information.
Once the scheduler runs, try to dig in and look for anything suspicious
I have an existing mvc4 web project which I wanted to deploy to a cloud service and to start using the azure data caching.
I have added the windows azure caching nuget packages to two projects in the solution, the web project and a class library project both of which will need these.
I then add a web role for the web project, and I have updated the datacache identifier reference in the web.config to point to the web role which is enabled for co located caching.
I can run this locally on the emulator without any problems while I don't have any datacache code. But the moment I put in code to access the datacache is when I have problems. Just this code caused the web project to hang:
var cache = new DataCache("default");
There are not errors that I am aware of, either in the vs output or errors generated from the web application, it just hangs.
What is the best way to start diagnosing where this problem lies?
UPDATE
I have just noticed the following errors generated in the application event log:
Application: CacheServiceEmulator.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.ApplicationServer.Caching.DataCacheException
Stack:
at Microsoft.ApplicationServer.Caching.AzureCommon.AzureUtility.ProcessException(System.Exception)
at Microsoft.ApplicationServer.Caching.CacheServiceEmulator.CacheServiceEmulator.Main(System.String[])
Faulting application name: CacheServiceEmulator.exe, version: 1.0.4797.0, time stamp: 0x506f41ec
Faulting module name: KERNELBASE.dll, version: 6.2.9200.16451, time stamp: 0x50988aa6
Exception code: 0xe0434352
Fault offset: 0x000000000003811c
Faulting process ID: 0x13cc
Faulting application start time: 0x01ce1b74c41f996d
Faulting application path: D:\Users\Tony\My Documents\Visual Studio 2012\Projects\Seqential\Didbook_ws\Didbook.net\Didbook.net v1.0\didbook.net Web.Azure\csx\Debug\roles\didbook.net Web\plugins\Caching\CacheServiceEmulator.exe
Faulting module path: C:\WINDOWS\system32\KERNELBASE.dll
Report ID: 03114030-8768-11e2-beaf-68942335e1fe
Faulting package full name:
Faulting package-relative application ID:
Fault bucket -936878625, type 5
Event Name: CLR20r3
Response: Not available
Cab Id: 0
Problem signature:
P1: cacheserviceemulator.exe
P2: 1.0.4797.0
P3: 506f41ec
P4: Microsoft.ApplicationServer.Caching.AzureServerCommon
P5: 1.0.4797.0
P6: 506f41df
P7: 3d
P8: 18
P9: SWOUM0PNYW4I1S3EYHEY4VNB5OWO0LJ1
P10:
Attached files:
C:\Users\Tony\AppData\Local\Temp\WER90C9.tmp.WERInternalMetadata.xml
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_cacheserviceemul_667e21a2e47da59aad2c601844d8dcfd3d291a_28d494fe
Analysis symbol:
Rechecking for solution: 0
Report ID: 03114030-8768-11e2-beaf-68942335e1fe
Report Status: 0
Hashed bucket: 700c7356d6308372410cf1d2baaf5d77
Does that help track down what is happening?
One other piece of info that may help is that if I create a brand new solution, add a web role and enable co located caching I can get it to work fine -- it just appears something specific to this solution.
The Azure Caching emulator starts logman.exe passing the cnf parameter as 30:00, but logman may reject it if the format is not compatible with your regional settings.
All you have to do, is to change the Long time setting to "HH:mm:ss" and it'll works.
Can you dump your cscfgs here starring out storage keys, also check the events and stack traces in application server channel (Admin) , it would have a better stack trace.
I developed a application site where one facility is FAQ in which user can post text data without any limit.
I hv two server to run the application whenever a single field (question or answer) size is huge (like one page long) one of the server is giving service unavailable. I checked in log the error detail is
-------------------
Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1002
Date: 1/23/2012
Time: 3:29:49 PM
User: N/A
Computer: BA5SWWW006
Description:
Application pool 'pool_name' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
--------------------
AND ALSO
-------------------
Event Type: Error
Event Source: VsJITDebugger
Event Category: None
Event ID: 4096
Date: 1/23/2012
Time: 3:29:44 PM
User: NT AUTHORITY\NETWORK SERVICE
Computer: BA5SWWW006
Description:
An unhandled win32 exception occurred in w3wp.exe [10896]. Just-In-Time debugging this exception failed with the following error: Debugger could not be started because no user is logged on.
Check the documentation index for 'Just-in-time debugging, errors' for more information.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 02 00 5c 80 ..\
------------------------
but the other server is working fine. i check all d basic setting of both is same.
and also there no such logged in error for any other module.
Even error wont occur for same module if the text in question or answer is less.
when this occur it ask for enter the user credentials i couldn't understand why it prompt for such?????
i use my-sql with longtext field to store question or answer.
May be best to try the IIS Debug Diagnostics Tool to further diagnose the problem.
This SO question has plenty of other suggestions: How to diagnose IIS fatal communication error problem