I have to write an AWS lambda function in python using boto3. The main aim of the function is that it detects all the unhealthy workspaces in a directory and reboots the workspaces whose state is unhealthy.
I have created a cloudwatch alarm which triggers the SNS and which in turns triggers the lambda.
I have no idea how to iterate through workspaces in a directory using python which will detect the unhealthy state.
Can anybody please provide me the sample code in python so that I can write the lambda.
Thanks
import json
import boto3
client = boto3.client('workspaces')
def lambda_handler(event, context):
statusCode = 200
print("Alarm activated")
DirectoryId = "d-966714f11"
UnhealthyWorkspace = []
if(DirectoryId == 'd-966714f114'):
response = client.describe_workspaces(
WorkspaceIds = (should be in an array)
)
us = response["Contents"]
for i in us:
if(State == 'Unhealthy'):
print(i)
UnhealthyWorkspace.append(i)
response1 = client.reboot_workspaces(
RebootWorkspaceRequests=[
{
'WorkspaceId' : UnhealthyWorkspace
}
]
)
Use describe_workspaces() to retrieve a list of all Workspaces.
Then, loop through the list of Workspace and check for: State = 'UNHEALTHY'
Related
I would like help and or opinion on how to proceed with this Python. The idea is that it is started through an AWS Lambda trigger and takes the new event (file) from S3 and sends it to my sFTP server.
Code:
import boto3
import urllib.parse
import os
from ftplib import FTP_TLS
#Info server SFTP GCore
FTP_HOST = 'valueFTPHOST'
FTP_USER = 'valueFTPUSER'
FTP_PWD = 'valueFTPPASS'
FTP_PORT = 2200
FTP_PATH = 'path'
print('Loading function')
s3 = boto3.client('s3')
def lambda_handler(event, context):
if event and event['Records']:
for record in event['Records']:
sourcebucket = record['s3']['bucket']['name']
sourcekey = record['s3']['object']['key']
#Download in /tmp/
filename = os.path.basename(sourcekey)
download_path = '/tmp/'+ filename
print("printcaminho", download_path)
s3.download_file(sourcebucket, sourcekey, download_path)
print("printcaminho com o arquivo", download_path)
os.chdir("/tmp/")
with FTP_TLS(FTP_HOST, FTP_PORT, FTP_USER, FTP_PWD) as ftps, open(filename, 'rb') as file:
ftps.storbinary(f'STOR {FTP_PATH}{file.name}', file)
#Cleaning /tmp/
os.remove(filename)
Return:
Response
{
"errorMessage": "2022-08-24T14:16:18.627Z 2eed320e-7e94-4f96-9127-bab6a1ebb1cf Task timed out after 3.01 seconds"
}
The code runs without errors but I have the sFTP timeout return.
Any idea how to proceed?
Guys, this is my first question asked here on StackOverFlow. Sorry if I did something wrong. I accept suggestions to improve.
If the lambda logging includes the phrase "task timed out" it means the lambda ran longer than the timeout configuration for the function.
Here is some AWS details on troubleshooting timeouts.
To update the timeout, go to the AWS Management Console > Lambda and click your function. Go to the Configuration Tab > General Configuration. Click Edit and you'll be able to increase your timeout. This will increase the cost of your lambda function.
I tried to create a function that does the following things.
(dont know if its worth mentioning but this function is invoked by another function)
connects to aws resources using boto3
gets a the number of messages available in an sqs queue
counts the number of ec2 instances
evaluates a set of conditions based on the sqs queue and ec2 instances and either do nothing or write to an sns topic.
Basically i want to publish to a message to sns topic every time the sqs queue is high and the number of ec2 instances which are digesting these, is low.
import os
import boto3
import logging
import types
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
# Create session and clients
sts_client = boto3.client('sts')
sqs_client = boto3.client('sqs')
ec2_client = boto3.client('ec2')
sns_client = boto3.client('sns')
# Call the assume_role method of the STSConnection object and pass the role ARN and a role session name.
assumed_role_object = sts_client.assume_role(
RoleArn=os.environ['ROLE_ARN'],
RoleSessionName="AssumeRoleFromCloudOperations"
)
# From the response that contains the assumed role, get the temporary credentials
credentials = assumed_role_object['Credentials']
assumed_role_session = boto3.Session(
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken']
)
# Check the queue size
def sqs():
queue_size = sqs_client.get_queue_attributes(
QueueUrl=os.environ['SQS_QUEUE_URL'],
AttributeNames=['ApproximateNumberOfMessages']
)
messages = int(queue_size["Attributes"]["ApproximateNumberOfMessages"])
return messages
# Count the number of active ec2 instances
def count_instances(ec2):
total_instances = 0
instances = ec2.instances.filter(Filters=[
{
'Instance State': 'instance-state-name',
'Values': ['running'],
'Name': 'tag:Name',
'Values': ['NameOfInstance']
},
])
for _ in instances:
total_instances += 1
return total_instances
print(f"Total number of active scan servers is: {total_instances}")
# Define the SNS Topic which will be integrated with OpsGenie
def sns():
topicArn = os.environ['SNS_ARN']
# Evaluate the set of conditions
def evaluate_conditions(context, event):
sqs()
if messages > int(os.environ['AVG_QUEUE_SIZE']) and count_instances.total_instances > int(os.environ['AVG_NR_OF_EC2_SCAN_SERVERS']):
print('False alert')
logger.info()
elif messages < int(os.environ['AVG_QUEUE_SIZE']) and count_instances.total_instances < int(os.environ['AVG_NR_OF_EC2_SCAN_SERVERS']):
print('False alert')
logger.info()
elif messages < int(os.environ['AVG_QUEUE_SIZE']) and count_instances.total_instances > int(os.environ['AVG_NR_OF_EC2_SCAN_SERVERS']):
print('False alert')
logger.info()
else:
sns.publish(TopicArn=os.environ['SNS_ARN'],
Message='sameple message',
Subject='sample subject')
print("Published to SNS Topic")
the handler is handler.evaluate_conditions
My question is how can i have some structure in this lambda function?
When i run the function i get a naming error:
{
"errorMessage": "name 'messages' is not defined",
"errorType": "NameError",
"stackTrace": [
" File \"/var/task/mdc_alert/handler.py\", line 67, in evaluate_conditions\n if messages > int(os.environ['AVG_QUEUE_SIZE']) and count_instances.total_instances > int(\n"
]
}
So it seems that i cannot use the message variable in the evaluate_conditions() function.
How can i make the "message" and "total_instances" variables usable in the evaluate_conditions() function?
I've written this function completely based on google searches, stackoverflow and boto3 docs since i don't have any experience with programming.
Is this structure any good, or does it need a complete overhaul?
Do i need to change the order of the functions, or maybe create a class?
The immediate issue is that the messages variable is not defined. Your sqs function returns a value but since you're calling it in a void context you're not actually doing anything with that value. You can fix this by changing this line:
sqs()
to this one:
messages = sqs()
I also see some issues with the count_instances function. It's expecting to receive an ec2 variable but you're calling it incorrectly from evaluate_conditions. You could either pass it the ec2_client variable or just use the ec2_client variable directly from within the function.
I suggest renaming your functions to more accurately reflect their return values:
sqs -> sqs_msg_count
count_instances -> running_ec2_count
Making these changes will allow you to refactor evaluate_conditions to shorten the if-then lines, making your code overall easier to read and follow. If you took all these suggestions into account, your code might look something like this:
# Check the queue size
def sqs_msg_count():
messages = sqs_client.get_queue_attributes(
QueueUrl=os.environ['SQS_QUEUE_URL'],
AttributeNames=['ApproximateNumberOfMessages']
)
return int(messages["Attributes"]["ApproximateNumberOfMessages"])
# Count the number of active ec2 instances
def running_instance_count():
running_instances = ec2_client.instances.filter(Filters=[
{
'Instance State': 'instance-state-name',
'Values': ['running'],
'Name': 'tag:Name',
'Values': ['NameOfInstance']
},
])
return len(running_instances)
# Evaluate the set of conditions
def evaluate_conditions(context, event):
sqs_count = sqs_msg_count()
sqs_average = int(os.environ['AVG_QUEUE_SIZE'])
ec2_count = running_instance_count()
ec2_average = int(os.environ['AVG_NR_OF_EC2_SCAN_SERVERS'])
if sqs_count > sqs_average and ec2_count > ec2_average:
print('False alert')
logger.info()
elif sqs_count < sqs_average and ec2_count < ec2_average:
print('False alert')
logger.info()
elif sqs_count < sqs_average and ec2_count > ec2_average:
print('False alert')
logger.info()
else:
sns_client.publish(
TopicArn=os.environ['SNS_ARN'],
Message='sameple message',
Subject='sample subject'
)
print("Published to SNS Topic")
In a Google Cloud function (python 3.7) , I need to fetch the compliance state of all VMs in a given location in a project.
From available google documentation here I could see the REST API format:
https://cloud.google.com/compute/docs/os-configuration-management/view-compliance#view_compliance_state
On searching for the client library here , I found this:
class google.cloud.osconfig_v1alpha.types.ListInstanceOSPoliciesCompliancesRequest(mapping=None, *, ignore_unknown_fields=False, **kwargs)[source]
Bases: proto.message.Message
A request message for listing OS policies compliance data for all Compute Engine VMs in the given location.
parent
Required. The parent resource name.
Format: projects/{project}/locations/{location}
For {project}, either Compute Engine project-number or project-id can be provided.
Type
str
page_size
The maximum number of results to return.
Type
int
page_token
A pagination token returned from a previous call to ListInstanceOSPoliciesCompliances that indicates where this listing should continue from.
Type
str
filter
If provided, this field specifies the criteria that must be met by a InstanceOSPoliciesCompliance API resource to be included in the response.
Type
str
And the response class as:
class google.cloud.osconfig_v1alpha.types.ListInstanceOSPoliciesCompliancesResponse(mapping=None, *, ignore_unknown_fields=False, **kwargs)[source]
Bases: proto.message.Message
A response message for listing OS policies compliance data for all Compute Engine VMs in the given location.
instance_os_policies_compliances
List of instance OS policies compliance objects.
Type
Sequence[google.cloud.osconfig_v1alpha.types.InstanceOSPoliciesCompliance]
next_page_token
The pagination token to retrieve the next page of instance OS policies compliance objects.
Type
str
property raw_page
But I am not sure how to use this information in the python code.
I have written this but not sure if this is correct:
from google.cloud.osconfig_v1alpha.services.os_config_zonal_service import client
from google.cloud.osconfig_v1alpha.types import ListInstanceOSPoliciesCompliancesRequest
import logging
logger = logging.getLogger(__name__)
import os
def handler():
try:
project_id = os.environ["PROJECT_ID"]
location = os.environ["ZONE"]
#list compliance state
request = ListInstanceOSPoliciesCompliancesRequest(
parent=f"projects/{project}/locations/{location}")
response = client.instance_os_policies_compliance(request)
return response
except Exception as e:
logger.error("Unable to get compliance - %s " % str(e))
I could not find any usage example for the client library methods anywhere.
Could someone please help me here?
EDIT:
This is what I am using now:
from googleapiclient.discovery import build
def list_policy_compliance():
projectId = "my_project"
zone = "my_zone"
try:
service = build('osconfig', 'v1alpha', cache_discovery=False)
compliance_response = service.projects().locations(
).instanceOsPoliciesCompliances().list(
parent='projects/%s/locations/%s' % (
projectId, zone)).execute()
return compliance_response
except Exception as e:
raise Exception()
Something like this should work:
from google.cloud import os_config_v1alpha as osc
def handler():
client = osc.OsConfigZonalService()
project_id = "my_project"
location = "my_gcp_zone"
parent = f"projects/{project_id}/locations/{location}"
response = client.list_instance_os_policies_compliances(
parent=parent
)
# response is an iterable yielding
# InstanceOSPoliciesCompliance objects
for result in response:
# do something with result
...
You can also construct the request like this:
response = client.list_instance_os_policies_compliances(
request = {
"parent": parent
}
)
Answering my own question here , this is what I used:
from googleapiclient.discovery import build
def list_policy_compliance():
projectId = "my_project"
zone = "my_zone"
try:
service = build('osconfig', 'v1alpha', cache_discovery=False)
compliance_response = service.projects().locations(
).instanceOsPoliciesCompliances().list(
parent='projects/%s/locations/%s' % (
projectId, zone)).execute()
return compliance_response
except Exception as e:
raise Exception()
I went through the official docs of google cloud but I don't have an idea how to use these to list resources of specific organization by providing the organization id
organizations = CloudResourceManager.Organizations.Search()
projects = emptyList()
parentsToList = queueOf(organizations)
while (parent = parentsToList.pop()) {
// NOTE: Don't forget to iterate over paginated results.
// TODO: handle PERMISSION_DENIED appropriately.
projects.addAll(CloudResourceManager.Projects.List(
"parent.type:" + parent.type + " parent.id:" + parent.id))
parentsToList.addAll(CloudResourceManager.Folders.List(parent))
}
organizations = CloudResourceManager.Organizations.Search()
projects = emptyList()
parentsToList = queueOf(organizations)
while (parent = parentsToList.pop()) {
// NOTE: Don't forget to iterate over paginated results.
// TODO: handle PERMISSION_DENIED appropriately.
projects.addAll(CloudResourceManager.Projects.List(
"parent.type:" + parent.type + " parent.id:" + parent.id))
parentsToList.addAll(CloudResourceManager.Folders.List(parent))
}
You can use Cloud Asset Inventory for this. I wrote this code for performing a sink in BigQuery.
import os
from google.cloud import asset_v1
from google.cloud.asset_v1.proto import asset_service_pb2
def asset_to_bq(request):
client = asset_v1.AssetServiceClient()
parent = 'organizations/{}'.format(os.getEnv('ORGANIZATION_ID'))
output_config = asset_service_pb2.OutputConfig()
output_config.bigquery_destination.dataset = 'projects/{}}/datasets/{}'.format(os.getEnv('PROJECT_ID'),
os.getEnv('DATASET'))
output_config.bigquery_destination.table = 'asset_export'
output_config.bigquery_destination.force = True
response = client.export_assets(parent, output_config)
# For waiting the finish
# response.result()
# Do stuff after export
return "done", 200
if __name__ == "__main__":
asset_to_bq('')
Be careful is you use it, the sink must be done in an empty/not existing table or set the force to true.
In my case, some minutes after the Cloud Scheduler that trigger my function and extract the data to BigQuery, I have a Scheduled Query into BigQuery that copy the data to another table, for keeping the history.
Note: It's also possible to configure an extract in Cloud Storage if you prefer.
I hope that is a starting point for you and for achieving what do you want to do.
I am able to list the project but I also want to list the folder and resources under folder and folder.name and tags and i also want to specify the organization id to resources information from a specific organization
import os
from google.cloud import resource_manager
def export_resource (organizations):
client = resource_manager.Client()
for project in client.list_projects():
print("%s, %s" % (project.project_id, project.status))
I originally had this script working using the socket module, which is why x is defined the way it is. However, I ended up needing to specify the DNS server being used.
This script is currently running successfully in AWS Lambda, however instead of an FQDN/IP address as an output, I am receiving the output defined at the bottom (which is also coming through in the SNS topic).
I can't seem to find any information about what might be going on here. I am hoping someone can shed some light on how I can fix this.
import boto3
import dns.resolver
import dns.query
my_resolver = dns.resolver.Resolver()
my_resolver.nameservers = ['0.0.0.0'] #IP omitted, but internal DNS server is being used
sns = boto3.client('sns')
a = my_resolver.query('crl.godaddy.com')
x = ('gdcrl.godaddy.com.akadns.net', ['crl.godaddy.com'], ['72.167.18.237'])
def cb_crl_check(event=None, context=None):
crlgodaddy(a, x)
def crlgodaddy(a, x):
if a == x:
pass
else:
response = sns.publish(
TopicArn='arn:aws:sns:xxxxxxxxxxxxxx',
Message=('crl.godaddy.com IP address has changed! \n \nThe old
information was: \n {0} \n \nThe new information is: \n {1}').format(x,a),
Subject = '"crl.godaddy.com IP change"')
if '__name__' == '__main__':
cb_crl_check()
AWS Lambda output:
START RequestId: b637c14d-38f0-46cf-83bf-fd1279aff9d8 Version: $LATEST
***<dns.resolver.Answer object at 0x7fb830c2a450>***
END RequestId: b637c14d-38f0-46cf-83bf-fd1279aff9d8
The answer: I needed to use rrset[0] from the dnspython module in order to get this to work. The following is how the code was changed to make it work:
a = str(my_resolver.query('crl.godaddy.com').rrset[0])
x = '72.167.18.237'