I want to monitor and get information regarding the different instances in an Azure Virtual Machine Scale Set (VMSS).
I used the command (Python):
vmss = compute_client.virtual_machine_scale_sets.list(resource_group, scale_set_name)
But I am not able to get the result I am expecting.
Any suggestions what to do?
You can use the following code to get the ip and powerstate.
compute_client = ComputeManagementClient(credentials, subscription_id)
vmss = compute_client.virtual_machine_scale_set_vms.list(resource_group_name="", vmss="")
for item in vmss:
print("name: ", item.name)
ni_reference = item.network_profile.network_interfaces[0].id
resource_client = ResourceManagementClient(credentials, subscription_id)
nic = resource_client.resources.get_by_id(
ni_reference,
api_version='2017-12-01')
ip_reference = nic.properties['ipConfigurations'][0]['properties']
print("ip info: ", ip_reference)
instance_view = compute_client.virtual_machine_scale_set_vms.get_instance_view(resource_group_name="", vmss="", instance_id=item.instance_id)
print(instance_view.statuses[1].code)
result:
name: yangtestvmss_1
ip info: {'provisioningState': 'Succeeded', 'privateIPAddress': '10.0.0.5', 'privateIPAllocationMethod': 'Dynamic', 'subnet': {'id': '/subscriptions/e5b0fcfa-e859-43f3-8d84-5e5fe29f4c68/resourceGroups/yangtestvmss/providers/Microsoft.Network/virtualNetworks/yangtestvmssVnet/subnets/default'}, 'primary': True, 'privateIPAddressVersion': 'IPv4', 'isInUseWithService': False}
PowerState/running
name: yangtestvmss_3
ip info: {'provisioningState': 'Succeeded', 'privateIPAddress': '10.0.0.7', 'privateIPAllocationMethod': 'Dynamic', 'subnet': {'id': '/subscriptions/e5b0fcfa-e859-43f3-8d84-5e5fe29f4c68/resourceGroups/yangtestvmss/providers/Microsoft.Network/virtualNetworks/yangtestvmssVnet/subnets/default'}, 'primary': True, 'privateIPAddressVersion': 'IPv4', 'isInUseWithService': False}
PowerState/running
If you want to get the VMs information, please use the following code.
subscription_id = 'subscription Id'
credentials = ServicePrincipalCredentials(client_id=CLIENT, secret=KEY, tenant=TENANT_ID)
client = ComputeManagementClient(credentials, subscription_id)
vmss = client.virtual_machine_scale_set_vms.list("resourcegroup Name","VMSS name")
for item in vmss:
print("id:",item.id)
print("name",item.name)
Test Result:
There is a cool tool that a guy from Microsoft has been build for monitoring VMSS
see this link VMSS Dashboard
The mentioned tool helps you to see the status of VMs in the scale set: you can see the update domain and fault domain grouping of VMs. It lets you start or deallocate a VM. The code is for more than two years ago.
Related
Hi we are trying to use NodeJS to return IP address WHOIS information before we send the requesting IP address to the rest of our app - That part is easy.
However the part that is not easy is, selecting only the Organization part of the whois information.
for example this is a whois and what it returns
whois 137.184.236.168
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object
refer: whois.arin.net
inetnum: 137.0.0.0 - 137.255.255.255
organisation: Administered by ARIN
status: LEGACY
whois: whois.arin.net
changed: 1993-05
source: IANA
# whois.arin.net
NetRange: 137.184.0.0 - 137.184.255.255
CIDR: 137.184.0.0/16
NetName: DIGITALOCEAN-137-184-0-0
NetHandle: NET-137-184-0-0-1
Parent: NET137 (NET-137-0-0-0-0)
NetType: Direct Allocation
OriginAS: AS14061
Organization: DigitalOcean, LLC (DO-13)
RegDate: 2019-11-13
Updated: 2020-04-03
Comment: Routing and Peering Policy can be found at https://www.as14061.net
Comment:
Comment: Please submit abuse reports at https://www.digitalocean.com/company/contact/#abuse
Ref: https://rdap.arin.net/registry/ip/137.184.0.0
OrgName: DigitalOcean, LLC
OrgId: DO-13
Address: 101 Ave of the Americas
Address: FL2
City: New York
StateProv: NY
PostalCode: 10013
Country: US
RegDate: 2012-05-14
Updated: 2022-05-19
Ref: https://rdap.arin.net/registry/entity/DO-13
OrgAbuseHandle: ABUSE5232-ARIN
OrgAbuseName: Abuse, DigitalOcean
OrgAbusePhone: +1-347-875-6044
OrgAbuseEmail: abuse#digitalocean.com
OrgAbuseRef: https://rdap.arin.net/registry/entity/ABUSE5232-ARIN
OrgTechHandle: NOC32014-ARIN
OrgTechName: Network Operations Center
OrgTechPhone: +1-347-875-6044
OrgTechEmail: noc#digitalocean.com
OrgTechRef: https://rdap.arin.net/registry/entity/NOC32014-ARIN
OrgNOCHandle: NOC32014-ARIN
OrgNOCName: Network Operations Center
OrgNOCPhone: +1-347-875-6044
OrgNOCEmail: noc#digitalocean.com
OrgNOCRef: https://rdap.arin.net/registry/entity/NOC32014-ARIN
The only thing we are interested in is Organization: DigitalOcean, LLC (DO-13)
As we want to drop all IP addresses from this host provider.
We noticed that we have been successful at stopping Google and AWS via using host command but Digital Ocean does not work this way and we need to do it via Whois.
I know in NodeJS I would request the information
exec("whois "+ip, (error, stdout, stderr) => {
console.log(stdout);
}
Could use a regular expression:
const organizationPattern = /^organization:\s*(.+)$/im;
const match = organizationPattern.exec(stdout);
const organization = match ? match[1] : 'unknown';
console.log(organization);
I am trying to write a code which will read values from excel file and will create VMs in Google Cloud. I am facing problem at two locations, while creating tags if I use 'items': [tag] or while creating service account scope it starts giving me error.
import os, json
import googleapiclient.discovery
from google.oauth2 import service_account
import csv
credentials = service_account.Credentials.from_service_account_file('G:/python/json/mykids-280210.json')
compute = googleapiclient.discovery.build('compute', 'v1', credentials=credentials)
def create_instance(compute, vm_name, image_project, image_family, machinetype, startupscript, zone, network,
subnet, project, scope, tag):
# Get the latest Debian Jessie image.
image_response = compute.images().getFromFamily(
project=image_project, family=image_family).execute()
source_disk_image = image_response['selfLink']
# Configure the machine
machine_type = "zones/" + zone + "/machineTypes/" + machinetype
startup_script = startupscript
config = {
'name': vm_name,
'machineType': machine_type,
'description': 'This VM was created with python code',
'tags': {
'items': ['external', 'home', 'local'] #'items': [tag] <~~~~~~~~~~~
},
'deletionProtection': False,
'labels': {'env': 'dev', 'server': 'mytower', 'purpose': 'personal'},
# Specify the boot disk and the image to use as a source.
'disks': [
{
'boot': True,
'autoDelete': True,
'initializeParams': {
'sourceImage': source_disk_image,
}
}
],
# Specify a network interface with NAT to access the public
# internet.
'networkInterfaces': [{
'network': 'global/networks/' + network,
'subnetwork': 'regions/us-central1/subnetworks/' + subnet,
'accessConfigs': [
{'type': 'ONE_TO_ONE_NAT', 'name': 'External NAT'}
]
}],
# Allow the instance to access cloud storage and logging.
'serviceAccounts': [{
'email': 'default',
'scopes': [
#'https://www.googleapis.com/auth/devstorage.read_write', 'https://www.googleapis.com/auth/logging.write'
#scope # scope <~~~~~~~~~~~~~~~~~~~~
]
}],
'scheduling': {
"preemptible": True
},
# Metadata is readable from the instance and allows you to
# pass configuration from deployment scripts to instances.
'metadata': {
'items': [{
# Startup script is automatically executed by the
# instance upon startup.
'key': 'startup-script',
'value': startup_script
}]
}
}
return compute.instances().insert(
project=project,
zone=zone,
body=config).execute()
# [END create_instance]
with open('vms.csv', newline='') as csvfile:
data = csv.DictReader(csvfile)
for row in data:
vm_name = row['vm_name']
image_project = row['image_project']
image_family = row['image_family']
machinetype = row['machinetype']
startupscript = row['startupscript']
zone = row['zone']
network = row['network']
subnet = row['subnet']
project = row['project']
scope = row['scopes']
tag = row['tags']
print(create_instance(compute, vm_name, image_project, image_family, machinetype, startupscript, zone, network,
subnet, project, scope, tag))
csvfile.close()
error when use scope variable
G:\python\pythonProject\venv\Scripts\python.exe G:/python/pythonProject/read-excel-gcp/vm/create_vm.py
Traceback (most recent call last):
File "G:\python\pythonProject\read-excel-gcp\vm\create_vm.py", line 100, in <module>
print(create_instance(compute, vm_name, image_project, image_family, machinetype, startupscript, zone, network,
File "G:\python\pythonProject\read-excel-gcp\vm\create_vm.py", line 79, in create_instance
return compute.instances().insert(
File "G:\python\pythonProject\venv\lib\site-packages\googleapiclient\_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "G:\python\pythonProject\venv\lib\site-packages\googleapiclient\http.py", line 915, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://compute.googleapis.com/compute/v1/projects/mykids-280210/zones/us-central1-a/instances?alt=json returned "One or more of the service account scopes are invalid: 'https://www.googleapis.com/auth/devstorage.read_write', 'https://www.googleapis.com/auth/logging.write''". Details: "One or more of the service account scopes are invalid: 'https://www.googleapis.com/auth/devstorage.read_write', 'https://www.googleapis.com/auth/logging.write''">
Process finished with exit code 1
I get a similar error when I use tag variable.
I have # the value the way I am passing in the above code.
Below is my csv file details
vm_name,image_project,image_family,machinetype,startupscript,zone,network,subnet,project,scopes,tags
python-vm1,debian-cloud,debian-9,e2-micro,G:/python/json/startup-script.sh,us-central1-a,myvpc,subnet-a,mykids-280210,"https://www.googleapis.com/auth/devstorage.read_write', 'https://www.googleapis.com/auth/logging.write'","external', 'home', 'local'"
python-vm2,debian-cloud,debian-9,e2-micro,G:/python/json/startup-script.sh,us-central1-a,myvpc,subnet-a,mykids-280210,"https://www.googleapis.com/auth/devstorage.read_write', 'https://www.googleapis.com/auth/logging.write'","external', 'home', 'local'"
I am not sure that when the value are passed directly it works, but when passing the value through variable, it fails.
I have marked the problem area with <~~~~~~~~~~~~
Please suggest if anyone understands the issue.
#d.s can you try changing your scope format to something like this:
'serviceAccounts': [
{
'email': 'default'
'scopes':[
'https://www.googleapis.com/auth/compute',
'https://www.googleapis.com/auth/servicecontrol',
'https://www.googleapis.com/auth/service.management.readonly',
'https://www.googleapis.com/auth/logging.write',
'https://www.googleapis.com/auth/monitoring.write',
'https://www.googleapis.com/auth/trace.append',
'https://www.googleapis.com/auth/devstorage.read_write']}]
The listed scopes are the default scopes that you will need for an instance. I think the problem you are facing is you where trying to only list two scopes which are not enough to allow you to deploy your instance.
I have created aws infrastructure with collection EC2, Redshift, VPC etc. via CLOUDFORMATION. Now I want to delete it in particular reverse order. Exa. All resources are dependent on VPC. VPC should be deleted at the end. But somehow every stack is deleting but VPC stack is not deleting via python BOTO3.It shows some subnet or network interface dependency error. But when I try to delete via console, It deletes it successfully.
Has anyone faced this issue?
I have tried to delete everyting like loadbalancer which is attached to it. But still VPC is not deleting.
AWS CloudFormation creates a dependency graph between resources based upon DependsOn references in the template and references between resources.
It then tries to deploy resources in parallel, but takes dependencies into account.
For example, a Subnet might be defined as:
Subnet1:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.0.0.0/24
VpcId: !Ref ProdVPC
In this situation, there is an explicit reference to ProdVPC, so CloudFormation will only create Subnet1 after ProdVPC has been created.
When a CloudFormation stack is deleted, the reverse logic is applied. In this case, Subnet1 will be deleted before ProdVPC is deleted.
However, CloudFormation is not aware of resources created outside of the stack. This means that if a resource (eg an Amazon EC2 instance) is created inside the Subnet, then stack deletion will fail because the Subnet cannot be deleted while there is an EC2 instance using it (or, more accurately, an ENI is attached to it).
In such situations, you will need to manually delete the resources that are causing the "delete failure" and then try the delete command again.
A good way to find such resources is to look in the Network Interfaces section of the EC2 management console. Make sure that there are no interfaces connected to the VPC.
As you specified that you are having issues with deleting VPC within stacks containing lambdas which themselves are in VPC, this most probably could be because of the network interfaces being generated by lambdas to connect to other resources in the VPC.
Technically these network interfaces should be auto-deleted when lambdas are undeployed from the stack but in my experience, I have observed orphaned ENI's which doesn't let the VPC be undeployed.
For this reason, I created a custom resource backed lambda which cleans up the ENI's after all lambdas within VPC's have been undeployed.
This is the cloud formation part where you setup the custom resource and pass the VPC ID
##############################################
# #
# Custom resource deleting net interfaces #
# #
##############################################
NetInterfacesCleanupFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: src
Handler: cleanup/network_interfaces.handler
Role: !GetAtt BasicLambdaRole.Arn
DeploymentPreference:
Type: AllAtOnce
Timeout: 900
PermissionForNewInterfacesCleanupLambda:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:invokeFunction
FunctionName:
Fn::GetAtt: [ NetInterfacesCleanupFunction, Arn ]
Principal: lambda.amazonaws.com
InvokeLambdaFunctionToCleanupNetInterfaces:
DependsOn: [PermissionForNewInterfacesCleanupLambda]
Type: Custom::CleanupNetInterfacesLambda
Properties:
ServiceToken: !GetAtt NetInterfacesCleanupFunction.Arn
StackName: !Ref AWS::StackName
VPCID:
Fn::ImportValue: !Sub '${MasterStack}-Articles-VPC-Ref'
Tags:
'owner': !Ref StackOwner
'task': !Ref Task
And this is the corresponding lambda. This lambda tries 3 times to detach and delete orphaned network interfaces and if fails if it can't which means there's still a lambda which is generating new network interfaces and you need to debug for that.
import boto3
from botocore.exceptions import ClientError
from time import sleep
# Fix this wherever your custom resource handler code is
from common import cfn_custom_resources as csr
import sys
MAX_RETRIES = 3
client = boto3.client('ec2')
def handler(event, context):
vpc_id = event['ResourceProperties']['VPCID']
if not csr.__is_valid_event(event, context):
csr.send(event, context, FAILED, validate_response_data(result))
return
elif event['RequestType'] == 'Create' or event['RequestType'] == 'Update':
result = {'result': 'Don\'t trigger the rest of the code'}
csr.send(event, context, csr.SUCCESS, csr.validate_response_data(result))
return
try:
# Get all network intefaces for given vpc which are attached to a lambda function
interfaces = client.describe_network_interfaces(
Filters=[
{
'Name': 'description',
'Values': ['AWS Lambda VPC ENI*']
},
{
'Name': 'vpc-id',
'Values': [vpc_id]
},
],
)
failed_detach = list()
failed_delete = list()
# Detach the above found network interfaces
for interface in interfaces['NetworkInterfaces']:
detach_interface(failed_detach, interface)
# Try detach a second time and delete each simultaneously
for interface in interfaces['NetworkInterfaces']:
detach_and_delete_interface(failed_detach, failed_delete, interface)
if not failed_detach or not failed_delete:
result = {'result': 'Network interfaces detached and deleted successfully'}
csr.send(event, context, csr.SUCCESS, csr.validate_response_data(result))
else:
result = {'result': 'Network interfaces couldn\'t be deleted completely'}
csr.send(event, context, csr.FAILED, csr.validate_response_data(result))
# print(response)
except Exception:
print("Unexpected error:", sys.exc_info())
result = {'result': 'Some error with the process of detaching and deleting the network interfaces'}
csr.send(event, context, csr.FAILED, csr.validate_response_data(result))
def detach_interface(failed_detach, interface):
try:
if interface['Status'] == 'in-use':
detach_response = client.detach_network_interface(
AttachmentId=interface['Attachment']['AttachmentId'],
Force=True
)
# Sleep for 1 sec after every detachment
sleep(1)
print(f"Detach response for {interface['NetworkInterfaceId']}- {detach_response}")
if 'HTTPStatusCode' not in detach_response['ResponseMetadata'] or \
detach_response['ResponseMetadata']['HTTPStatusCode'] != 200:
failed_detach.append(detach_response)
except ClientError as e:
print(f"Exception details - {sys.exc_info()}")
def detach_and_delete_interface(failed_detach, failed_delete, interface, retries=0):
detach_interface(failed_detach, interface)
sleep(retries + 1)
try:
delete_response = client.delete_network_interface(
NetworkInterfaceId=interface['NetworkInterfaceId'])
print(f"Delete response for {interface['NetworkInterfaceId']}- {delete_response}")
if 'HTTPStatusCode' not in delete_response['ResponseMetadata'] or \
delete_response['ResponseMetadata']['HTTPStatusCode'] != 200:
failed_delete.append(delete_response)
except ClientError as e:
print(f"Exception while deleting - {str(e)}")
print()
if retries <= MAX_RETRIES:
if e.response['Error']['Code'] == 'InvalidNetworkInterface.InUse' or \
e.response['Error']['Code'] == 'InvalidParameterValue':
retries = retries + 1
print(f"Retry {retries} : Interface in use, deletion failed, retrying to detach and delete")
detach_and_delete_interface(failed_detach, failed_delete, interface, retries)
else:
raise RuntimeError("Code not found in error")
else:
raise RuntimeError("Max Number of retries exhausted to remove the interface")
The link to the lambda is https://gist.github.com/revolutionisme/8ec785f8202f47da5517c295a28c7cb5
More information about configuring lambdas in a VPC - https://docs.aws.amazon.com/lambda/latest/dg/vpc.html
Does anyone have experience with the Python SDK (v: v2.0.0rc6) and cloning/duplicating running VM's into another resource group?
getting the OS disk to start. Will need the data disk as well
managed_disk = self.compute_client.disks.get(resource_group_name=source_rg_name, disk_name=vm.storage_profile.os_disk.name)
make a snapshot of the os disk.
self.compute_client.snapshots.create_or_update(
self.config.resource_group_name,
'SNAPSHOT-' + virtual_machine,
{
'location': managed_disk.location,
'creation_data': {
'create_option': 'Copy',
'source_uri': managed_disk.id
}
}
)
create the VM. Throws exception below.
result = self.compute_client.virtual_machines.create_or_update(
self.config.cybric_resource_group_name,
virtual_machine,
azure.mgmt.compute.models.VirtualMachine(
location=vm.location,
os_profile=vm.os_profile,
hardware_profile=vm.hardware_profile,
network_profile=azure.mgmt.compute.models.NetworkProfile(
network_interfaces=[
azure.mgmt.compute.models.NetworkInterfaceReference(
id=nic_obj['id'],
primary=True
),
],
),
storage_profile=azure.mgmt.compute.models.StorageProfile(
os_disk=azure.mgmt.compute.models.OSDisk(
caching=azure.mgmt.compute.models.CachingTypes.none,
create_option=azure.mgmt.compute.models.DiskCreateOptionTypes.attach,
name=dup_virtual_machine,
managed_disk=azure.mgmt.compute.models.ManagedDiskParameters(
id=managed_disk.id
),
),
image_reference = azure.mgmt.compute.models.ImageReference(
publisher=vm.storage_profile.image_reference.publisher,
offer=vm.storage_profile.image_reference.offer,
sku=vm.storage_profile.image_reference.sku,
version=vm.storage_profile.image_reference.version,
),
),
),
)
Exception:
Failed to create virtual machines: Azure Error: InvalidParameter
Message: Cannot attach an existing OS disk if the VM is created from a
platform or user image. Target: osDisk
The error log reason is that create_option is wrong, you should use FromImage not attach. You could check Azure Python SDK in this link.
Hi am looking to get Cloud Formation Stack Status I tried with below code it's printing all the properties in the stack Object but I need only StackStatus please check with code and Response and help me to overcome this.
Advance appreciations
import boto3
def lambda_handler(event, context):
client = boto3.client('cloudformation')
stack = client.describe_stacks(StackName='m6bf2b178-3bd2-ac06-88ef-
06545ba3fc74')
print(stack['Stacks'])
Response is:
"[{'StackId': 'arn:aws:cloudformation:us-east-1:*********:stack/m6bf2b178-3bd2-ac06-88ef-06545ba3fc74/10bc3e30-198f-11e8-bec2-50faeaee4499', 'StackName': 'm6bf2b178-3bd2-ac06-88ef-06545ba3fc74', 'Description': 'AWS CloudFormation Sample Template EC2InstanceWithSecurityGroupSample: Create an Amazon EC2 instance running the Amazon Linux AMI. The AMI is chosen based on the region in which the stack is run. This example creates an EC2 security group for the instance to give you SSH access. This template creates an Amazon EC2 instance. You will be billed for the AWS resources used if you create a stack from this template.', 'Parameters': [{'ParameterKey': 'KeyName', 'ParameterValue': 'Test'}, {'ParameterKey': 'InstanceType', 'ParameterValue': 't2.micro'}], 'CreationTime': datetime.datetime(2018, 2, 24, 18, 18, 8, 756000, tzinfo=tzlocal()), 'RollbackConfiguration': {}, 'StackStatus': 'CREATE_COMPLETE', 'DisableRollback': False, 'NotificationARNs': [], 'Outputs': [{'OutputKey': 'InstanceId', 'OutputValue': 'i-04dd2a73e99f8d3d1', 'Description': 'InstanceId of the newly created EC2 instance'}, {'OutputKey': 'PublicIP', 'OutputValue': '34.227.162.158', 'Description': 'Public IP address of the newly created EC2 instance'}, {'OutputKey': 'AZ', 'OutputValue': 'us-east-1a', 'Description': 'Availability Zone of the newly created EC2 instance'}, {'OutputKey': 'PublicDNS', 'OutputValue': 'ec2-34-227-162-158.compute-1.amazonaws.com', 'Description': 'Public DNSName of the newly created EC2 instance'}], 'Tags': [], 'EnableTerminationProtection': False}]"
import boto3
cloudformation = boto3.resource('cloudformation')
stack = cloudformation.Stack('StackName')
print(stack.stack_status)
Since what you get as a response is a list of dictionaries, you can access the stack status with the following line:
stack_status = stack['Stacks'][0]['StackStatus']
print(stack_status)
It should ouput:
CREATE_COMPLETE