I'm new at AWS, so I'm building a code to create a instance from an Image and I want that at the same time that this EC2 is created it run a pyhton code like this:
python /folder/folder2/api_flask.py
Here's my code on boto to create my instance.
import boto3
client = boto3.session('ec2')
client.run_instances(ImageId='ami-id_number_of_img', MinCount=1, MaxCount=1, InstanceType='t2.nano')
Thnks for your help.
run_instances has an option called UserData which allows you to Run commands on your Linux instance at launch.
Thus to run your code, you can try to the following:
import boto3
client = boto3.client('ec2') # not boto3.session('ec2')
client.run_instances(ImageId='ami-id_number_of_img',
MinCount=1,
MaxCount=1,
InstanceType='t2.nano',
UserData='#!/bin/bash\npython /folder/folder2/api_flask.py\n')
Since you mention you are new to AWS, consider using CloudFormation for provisioning AWS Infrastructure. You'll still need leverage UserData as Marcin mentioned.
MyInstance:
Type: AWS::EC2::Instance
Properties:
UserData:
Fn::Base64: !Sub |
python /folder/folder2/api_flask.py
InstanceType: t2.nano
ImageId: ami-id_number_of_img
Why CloudFormation? It'd be more readable, an allows for in-place updates as well as tear downs. You could then launch the stack via boto3 (disclaimer: not tested, but demonstrates the idea):
import boto3
client = boto3.client('cloudformation')
with open('mytemplate.yml', 'r') as f:
response = client.create_stack(
StackName='my-stack',
TemplateBody=f.read())
Related
I am trying to read an excel file from s3 inside an aiflow dag with python, but it does not seem to work. It is very weird because it works when I read it from outside airflow with pd.read_excel(s3_excel_path).
What I did :
Set AWS credential in airflow (this works well as I can list my s3 bucket)
Install pandas, s3fs in my Docker environment where I run Airflow
Try to read the file with pd.read_excel(s3_excel_path)
As I said, it works when I try it outside of Airflow. Moreover, I don't get any error, the dag just continues to run undefinitely (at the step where it is supposed to read the file) and nothing happens, even if I wait 20 minutes.
(I would like to avoir to download the file from s3, process it and then upload it back to s3, which is why I am trying to read it directly from s3)
Note: I does not work with csv as well.
EDIT : Likewise, I can't save my dataframe directly to S3 with df.to_csv('s3_path') in airflow dag while I can do it in python
To read data files stored in S3 using pandas, you have two options, download them using boto3 (or AWS CLI) and read local files, which is the solution you are not locking for, and use s3fs API supported by pandas:
import os
import pandas as pd
AWS_S3_BUCKET = os.getenv("AWS_S3_BUCKET")
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
AWS_SESSION_TOKEN = os.getenv("AWS_SESSION_TOKEN")
key = "path/to/excel/file"
books_df = pd.read_excel(
f"s3://{AWS_S3_BUCKET}/{key}",
storage_options={
"key": AWS_ACCESS_KEY_ID,
"secret": AWS_SECRET_ACCESS_KEY,
"token": AWS_SESSION_TOKEN,
},
)
to use this solution, you need to install s3fs and apache-airflow-providers-amazon
pip install s3fs
pip install apache-airflow-providers-amazon
I am referring the aws-secretsmanager-caching-python documentation and trying to cache the retrieved secret from secrets manager however, for some reason, i am always getting timeout without any helpful errors to troubleshoot this further. I am able to retrieve the secrets properly if i retrieve the secrets from secrets manager (without caching).
My main function in lambda function looks like this:
import botocore
import botocore.session
from aws_secretsmanager_caching import SecretCache, SecretCacheConfig
from cacheSecret import getCachedSecrets
def lambda_handler(event, context):
result = getCachedSecrets()
print(result)
and i have created cacheSecret as following.
from aws_secretsmanager_caching import SecretCache
from aws_secretsmanager_caching import InjectKeywordedSecretString, InjectSecretString
cache = SecretCache()
#InjectKeywordedSecretString(secret_id='my_secret_name', cache=cache, secretKey1='keyname1', secretKey2='keyname2')
def getCachedSecrets(secretKey1, secretKey2):
print(secretKey1)
print(secretKey2)
return secretKey1
In the above code, my_secret_name is the name of the secret created in secret manager and secretKey1 and secretKey1 are the secret key names which have string values.
Error:
{
"errorMessage": "2021-03-31T15:29:08.598Z 01f5ded3-7658-4zb5-ae66-6f300098a6e47 Task timed out after 3.00 seconds"
}
Can someone please suggest what needs to be fixed in the above to make this work. Also, i am not sure where to define the secret_name, secret key names in case if we dont use decorators.
The lambda needs to ack the response within 3 seconds but the code can run longer. The timeout can be configured in the function config: https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html
I have a fast API application. Initially, I was passing my DB URI via ngrok tunnel like this in my SAM template. In this setup Lambda will be using my local machine's PSQL DB.
DbConnnectionString:
Type: String
Default: postgresql://<uname>:<pwd>#x.tcp.ngrok.io:PORT/DB
This is how I read the URI in my Python code
# config.py
DATABASE_URL = os.environ.get('DB_URI')
db_engine = create_engine(DATABASE_URL)
db_session = sessionmaker(autocommit=False, autoflush=False,bind=db_engine)
print(f"Configs initialized for {API_V1_STR}")
# app.py
# 3rd party
from fastapi import FastAPI
# Custom
from config.app_config import PROJECT_NAME, db_engine
from models.db_models import Base
print("Creating all database")
Base.metadata.create_all(bind=db_engine)
app = FastAPI(title=PROJECT_NAME)
print("APP created")
In this setup, everything seems to work as expected.
But whenever I replace the DB URL with RDS DB, suddenly the call gets stuck at create all database step as shown in the image below. when this happens the lambda always times out and throws exceptions.
If I run the code locally using uvicorn this error doesn't occur.
Everything works as expected.
When I use sam local invoke even with RDS URL, the API call works without any issues.
This problem occurs only while deployed in AWS Lambda.
I notice that configs are initialized twice in this setup, Once before START request ID and once after.
I have tried reading up on it but not clear what could I do to fix this. Any help would be much appreciated.
It was my bad!. I didn't pay attention to security groups. It was a connection timeout all along. Once I fixed the port access in Security groups, lambda started working as expected.
I'm running a vanilla AWS lambda function to count the number of messages in my RabbitMQ task queue:
import boto3
from botocore.vendored import requests
cloudwatch_client = boto3.client('cloudwatch')
def get_queue_count(user="user", password="password", domain="<my domain>/api/queues"):
url = f"https://{user}:{password}#{domain}"
res = requests.get(url)
message_count = 0
for queue in res.json():
message_count += queue["messages"]
return message_count
def lambda_handler(event, context):
metric_data = [{'MetricName': 'RabbitMQQueueLength', "Unit": "None", 'Value': get_queue_count()}]
print(metric_data)
response = cloudwatch_client.put_metric_data(MetricData=metric_data, Namespace="RabbitMQ")
print(response)
Which returns the following output on a test run:
Response:
{
"errorMessage": "2020-06-30T19:50:50.175Z d3945a14-82e5-42e5-b03d-3fc07d5c5148 Task timed out after 15.02 seconds"
}
Request ID:
"d3945a14-82e5-42e5-b03d-3fc07d5c5148"
Function logs:
START RequestId: d3945a14-82e5-42e5-b03d-3fc07d5c5148 Version: $LATEST
/var/runtime/botocore/vendored/requests/api.py:72: DeprecationWarning: You are using the get() function from 'botocore.vendored.requests'. This dependency was removed from Botocore and will be removed from Lambda after 2021/01/30. https://aws.amazon.com/blogs/developer/removing-the-vendored-version-of-requests-from-botocore/. Install the requests package, 'import requests' directly, and use the requests.get() function instead.
DeprecationWarning
[{'MetricName': 'RabbitMQQueueLength', 'Value': 295}]
END RequestId: d3945a14-82e5-42e5-b03d-3fc07d5c5148
You can see that I'm able to interact with the RabbitMQ API just fine--the function hangs when trying to post the metric.
The lambda function uses the IAM role put-custom-metric, which uses the policies recommended here, as well as CloudWatchFullAccess for good measure.
Resources on my internal load balancer, where my RabbitMQ server lives, are protected by a VPN, so it's necessary for me to associate this function with the proper VPC/security group. Here's how it's setup right now (I know this is working, because otherwise the communication with RabbitMQ would fail):
I read this post where multiple contributors suggest increasing the function memory and timeout settings. I've done both of these, and the timeout persists.
I can run this locally without any issue to create the metric on CloudWatch in less than 5 seconds.
#noxdafox has written a brilliant plugin that got me most of the way there, but at the end of the day I ended going for a pure lambda-based solution. It was surprisingly tricky getting the cloud watch plugin running with docker, and after I had trouble with the container shutting down its services and stopping processing of the message queue. Additionally, I wanted to be able to normalize queue count by the number of worker services in my ECS cluster, so I was going to need to connect to at least one AWS resource from within my VPC anyhow. I figured best to keep everything simple and in the same place.
import os
import boto3
from botocore.vendored import requests
USER = os.getenv("RMQ_USER")
PASSWORD = os.getenv("RMQ_PASSWORD")
cloudwatch_client = boto3.client(service_name='cloudwatch', endpoint_url="https://MYCLOUDWATCHURL.monitoring.us-east-1.vpce.amazonaws.com")
ecs_client = boto3.client(service_name='ecs', endpoint_url="https://vpce-MYECSURL.ecs.us-east-1.vpce.amazonaws.com")
def get_message_count(user=USER, password=PASSWORD, domain="rabbitmq.stockbets.io/api/queues"):
url = f"https://{user}:{password}#{domain}"
res = requests.get(url)
message_count = 0
for queue in res.json():
message_count += queue["messages"]
print(f"message count: {message_count}")
return message_count
def get_worker_count():
worker_data = ecs_client.describe_services(cluster="prod", services=["worker"])
worker_count = worker_data["services"][0]["runningCount"]
print(f"worker_count count: {worker_count}")
return worker_count
def lambda_handler(event, context):
message_count = get_message_count()
worker_count = get_worker_count()
print(f"msgs per worker: {message_count / worker_count}")
metric_data = [
{'MetricName': 'MessagesPerWorker', "Unit": "Count", 'Value': message_count / worker_count},
{'MetricName': 'NTasks', "Unit": "Count", 'Value': worker_count}
]
cloudwatch_client.put_metric_data(MetricData=metric_data, Namespace="RabbitMQ")
Creating the VPC endpoints was easier that I thought it would be. For Cloudwatch, you want to search for the "monitoring" VPC endpoint during the creation step (not "cloudwatch" or "logs". Searching for "ecs" gets you what you need for the ECS connect.
Once your lambda is us you need to configure the metric and accompanying alerts, and then relate those to an auto-scaling policy, but that's probably beyond the scope of this post. Leave a comment if you have questions on how I worked that out.
Only reason you might want to use a Lambda function to achieve your goal is if you do not own the RabbitMQ cluster. The fact your logic is hanging during communication suggests a network issue mostly due to mis-configured security groups.
If you can change the cluster configuration, I'd suggest you to install and configure the CloudWatch metrics exporter plugin which does most of the heavy-lifting work for you.
If your cluster runs on Docker, I believe the custom Docker file to be the best solution. If you run your Docker instances in AWS via ECS/Fargate, the plugin should be able to automatically infer the credentials from the Task Role through ExAws. Otherwise, just follow the README instructions on how to set the credentials yourself.
I’m trying to get a list of the tables from a database in my aws data catalog. I’m trying to use boto3. I’m running the code below on aws, in a sagemaker notebook. It runs forever (like over 30 minutes) and doesn’t return any results. The test_db only has 4 tables in it. My goal is to run similar code as part of an aws glue etl job, that I would run in an edited aws etl job script. Does anyone see what the issue might be or suggest how to do this?
code:
import boto3
from pprint import pprint
glue = boto3.client('glue', region_name='us-east-2')
response = glue.get_tables(
DatabaseName=‘test_db’
)
print(pprint(response['TableList']))
db = session.resource('dynamodb', region_name="us-east-2")
tables = list(db.tables.all())
print(tables)
resource
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/dynamodb.html