I'm trying to build a jumptable of API methods for a variety of boto3 clients, so I can pass an AWS service name and a authn/authz low-level boto3 client to my utility code and execute the appropriate method to get a list of resources from the AWS service.
I'm not willing to hand-code and maintain a massive if..elif..else statement with >100 clauses.
I have a dictionary of service names (keys) and API method names (values), like this:
jumpTable = { 'lambda' : 'list_functions' }
I'm passed the service name ('lambda') and a boto3 client object ('client') already connected to the right service in the region and account I need.
I use the dict's get() to find the method name for the service, and then use a standard getattr() on the boto3 client object to get a method reference for the desired API call (which of course vary from service to service):
`apimethod = jumpTable.get(service)`
`methodptr = getattr(client, apimethod)`
Sanity-checking says I've got a "botocore.client.Lambda object" for 'client' (that looks OK to me) and a "bound method ClientCreator._create_api_method.._api_call of <botocore.client.Lambda" for the methodptr which reports itself as of type 'method'.
None of the API methods I'm using require arguments. When I invoke it directly:
'response = methodptr()'
it returns a boto3 ClientError, while invoking at through the client:
response = client.methodptr()
returns a boto3 AttributeErrror.
Where am I going wrong here?
I'm locked into boto3, Python3, AWS and have to talk to 100s of AWS services, each of which has a different API method that provides the data I need to gather. To an old C coder, a jump-table seems obvious; a more Pythonic approach would be welcome...
The following works for me:
client = boto3.Session().client("lambda")
methodptr = getattr(client, apimethod)
methodptr()
Note that the boto3.Session() part is required. When calling boto3.client(..) directly, I get a 'UnrecognizedClientException' exception.
Related
I am new to node and typescript. I am working on developing a node library that reaches out to another rest API to get and post data. This library is consumed by a/any UI application to send and receive data from the API service. Now my question is, how do I maintain environment specific configuration within the library? Like for ex:
Consumer calls GET /user
user end point on the consumer side calls a method in the library to get data
But if the consumer is calling the user end point in test environment I want the library to hit the following API Url
for test http://api.test.userinformation.company.com/user
for beta http://api.beta.userinformation.company.com/user
As far as I understand the library is just a reference and is running within the consumer application. Library can for sure get the environment from the consumer, but I do not want the consumer having to specify the full URL that needs to be hit, since that would be the responsibility of the library to figure out.
Note: URL is not the only problem, I can solve that with environment switch within the library, I have some client secrets based on environments which I can neither store in the code nor checkin to source control.
Additional Information
(as per jfriend00's request in comments)
My library has a LibExecutionEngine class and one method in it, which is the entry point of the library:
export class LibExecutionEngine implements ExecutionEngine {
constructor(private environment: Environments, private trailLoader:
TrailLoader) {}
async GetUserInfo(
userId: string,
userGroupVersion: string
): Promise<UserInfo> {
return this.userLoader.loadUserInfo(userId, userGroupVersion)
}
}
export interface ExecutionEngine {
GetUserInfo(userId: string, userGroupVersion: string): Promise<UserInfo>
}
The consumer starts to use the library by creating an instance of the LibraryExecution then calling the getuserinfo for example. As you see the constructor for the class accepts an environment. Once I have the environment in the library, I need to somehow load the values for keys API Url, APIClientId and APIClientSecret from within the constructor. I know of two ways to do this:
Option 1
I could do something like this._configLoader.SetConfigVariables(environment) where configLoader.ts is a class that loads the specific configuration values from files({environment}.json), but this would mean I maintain the above mentioned URL variables and the respective clientid, clientsecret to be able to hit the URL in a json file, which I should not be checking in to source control.
Option 2
I could use dotenv npm package, and create one .env file where I define the three keys, and then the values are stored in the deployment configuration which works perfectly for an independently deployable application, but this is a library and doesn't run by itself in any environment.
Option 3
Accept a configuration object from the consumer, which means that the consumer of the library provides the URL, clientId, and clientSecret based on the environment for the library to access, but why should the responsibility of maintaining the necessary variables for library be put on the consumer?
Please suggest on how best to implement this.
So, I think I got some clarity. Lets call my Library L, and consuming app C1 and the API that the library makes a call out to get user info as A. All are internal applications in our org and have a OAuth setup to be able to communicate, our infosec team provides those clientids and secrets to individual applications, so I think my clarity here is: C1 would request their own clientid and clientsecret to hit A's URL, C1 would then pass in the three config values to the library, which the library uses to communicate with A. Same applies for some C2 in the future.
Which would mean that L somehow needs to accept a full configuration object with all required config values from its consumers C1, C2 etc.
Yes, that sounds like the proper approach. The library is just some code doing what it's told. It's the client in this case that had to fetch the clientid and clientsecret from the infosec team and maintain them and keep them safe and the client also has the URL that goes with them. So, the client passes all this into your library, ideally just once per instance and you then keep it in your instance data for the duration of that instance
What is the best way to get list of service versions in google app engine in flex env? (from service instance in Python 3). I want to authenticate using service account json keys file. I need to find currently default version (with most of traffic).
Is there any lib I can use like googleapiclient.discovery, or google.appengine.api.modules? Or I should build it from scratches and request REST api on apps.services.versions.list using oauth? I couldn't not find any information in google docs..
https://cloud.google.com/appengine/docs/standard/python3/python-differences#cloud_client_libraries
Finally I was able to solve it. Simple things on GAE became big problems..
SOLUTION:
I have path to service_account.json set in GOOGLE_APPLICATION_CREDENTIALS env variable. Then you can use google.auth.default
from googleapiclient.discovery import build
import google.auth
creds, project = google.auth.default(scopes=['https://www.googleapis.com/auth/cloud-platform.read-only'])
service = build('appengine', 'v1', credentials=creds, cache_discovery=False)
data = service.apps().services().get(appsId=APPLICATION_ID, servicesId=SERVICE_ID).execute()
print data['split']['allocations']
Return value is allocations dictionary with versions as keys and traffic percents in values.
All the best!
You can use Google's Python Client Library to interact with the Google App Engine Admin API, in order to get the list of a GAE service versions.
Once you have google-api-python-client installed, you might want to use the list method to list all services in your application:
list(appsId, pageSize=None, pageToken=None, x__xgafv=None)
The arguments of the method should include the following:
appsId: string, Part of `name`. Name of the resource requested. Example: apps/myapp. (required)
pageSize: integer, Maximum results to return per page.
pageToken: string, Continuation token for fetching the next page of results.
x__xgafv: string, V1 error format. Allowed values: v1 error format, v2 error format
You can find more information on this method in the link mentioned above.
I am exploring ways to automate REST API resource and endpoint implementation using flask-restful. In flask-restful, the add_resource(UpdateOrders, '/orders') function links the endpoint to the resource in the framework. UpdateOrders is the endpoint, and also the name of a class containing logic for handling requests sent to '/orders'.
I could manually specify the endpoints and resources like this:
app = Flask(__name__)
api = Api(app)
# only Orders endpoint included for simplicity
class Orders(Resource):
def get(self, item_id):
return order_data[item_id]
def add_resources():
api.add_resource(UpdateOrders, '/orders')
api.add_resource(Orders, '/orders/<item_id>')
api.add_resource(UpdatePayments, '/payments')
api.add_resource(Payments, '/payments/<item_id>')
if __name__ == '__main__':
add_resources()
app.run()
However, both the endpoints and resources will change depending on the use case of the REST API, e.g. instead of '/orders' I could have '/appointments'. I don't know what the use case will be (it is generated from the user's business process choreography).
My initial thought was to start by adding resources dynamically:
def add_resources():
# generate list of endpoint, resource tuples
endpoint_resource = [('UpdateOrders', '/orders'),
('Orders', '/orders/<item_id>'),
('UpdatePayments', '/payments'),
('Payments', '/payments/<item_id>')]
for endpoint, resource in endpoint_resource:
api.add_resource(endpoint, resource)
Of course, this won't work as endpoint here is a string, while add_resource requires a class. So my question is: can/should I convert endpoint from a string to a variable class name so that the API's resources can be created 'dynamically'?
In other words, instead of endpoint being assigned first to string 'UpdateOrders', it will be assigned to <class '__main__.UpdateOrders'>.
Any guidance is appreciated!
I've begun using AWSXRay in order to get more insight into why performance is not ideal in my lambda function. This lambda function runs a gql service meaning it has lots of outbound requests to other lambda functions as well as dynamodb for caching.
I've added tracing to all aws-sdk client calls by utilizing the following in my handler. It mutates the imported AWS module so that all subsequent usage of AWS clients successful includes aws-xray tracing, regardless of what module imports it. Awesome!
import AWS from 'aws-sdk';
import AWSXRay from 'aws-xray-sdk';
AWSXRay.captureAWS(AWS);
Heres an example of the output:
The Problem
The problem is that none of the "Traces" have any annotation regarding the parameters of the requests. Both the annotation and metadata of each trace is empty:
The Hope
The hope is that there is a way to configure the AWSXRay CaptureAWS modifications so that they include the arguments of each aws-client request in the annotations or metadata.
The Question
Is it possible to request that AWSXRay.captureAWS(AWS); includes the parameters passed to the aws sdk client invocations in either the annotations or the metadata of the traces it produces?
The resources section contains high level arguments for some clients e.g. DynamoDB table name. Not all arguments are captured by default. This is because they may contain information that the users do not wish to track in their trace and may also be verbose.
For now opt-in is not available in X-Ray SDK for arbitrary API parameters. As a workaround for now, I would suggest that you wrap your sdk calls in a local subsegment and record the parameters you want to capture as annotations or Metadata for that subsegment. Let me know if you need any help in locating docs that allow you to create your own subsegments.
I recently posted a question about How to allow invoking an AWS Lambda function only from EC2 instances inside a VPC.
I managed to get it working by attaching an IAM role with an "AWS lambda role" policy to the EC2 instances and now I can invoke the lambda function using boto3.
Now, I would like to make the call to the lambda function asynchronously using the asyncio await syntax. I read that the lambda function offers an asynchronous version by setting InvokeType='Event', but that actually makes the call return immediately without getting the result of the function.
Since the function takes some time and I would like to launch many in parallel I would like to avoid blocking the execution while waiting for the function to return.
I tried using aiobotocore but that is only supporting basic 's3' service functionalities.
The best way to solve this (in humble opinion) would be to use the AWS API Gateway service to invoke the lambda function through a GET/POST request that can be easily handled using aiohttp.
Nevertheless I don't manage to make it work.
I added to the EC2 IAM role the policy "AmazonAPIGatewayInvokeFullAccess" but every time I try to:
import requests
r = requests.get('https://url_to_api_gateway_for_function')
I get a forbidden response <Response [403]>.
I created the API Gateway using directly the trigger in the lambda function.
I also tried to edit the API Gateway settings, by adding a post method to the function path and setting the "AWS_IAM" authentication and then deploying it as "prod" deployment...no luck. Still same forbidden response. When I test it through the "test screen on the API gateway, it works fine".
Any idea how to fix this? Am I missing some step?
I managed to solve my issue after some struggling.
The problem is that curl and python modules like python's requests do not sign the http requests with the IAM credentials of the EC2 machine where they are running. The http request to the AWS GATEWAY API must be signed using the AWS v4 signin protocol.
An example is here:
http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
Luckily, to keep things simple, there are some helper modules like requests-aws-sign:
https://github.com/jmenga/requests-aws-sign
At the end the code could look something like:
import aiohttp
import asyncio
from requests_aws_sign import AWSV4Sign
from boto3 import session
session = session.Session()
credentials = session.get_credentials()
region = session.region_name or 'ap-southeast-2'
service = 'execute-api'
url = "get_it_from_api->stages->your_deployment->invoke_url"
auth=AWSV4Sign(credentials, region, service)
async def invoke_func(loop):
async with aiohttp.request('GET', url, auth=auth, loop=loop) as resp:
html = await resp.text()
print(html)
loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))
Hope this will save time to somebody else!
EDIT:
For the sake of completeness and helping others I have to say that the code above does not work due to the fact that requests_aws_sign is not compatible with aiohttp. I was getting some "auth field error".
I manged to solve it by using:
async with session.get(url, headers=update_headers()) as resp:
where update_headers() is a simple function that mimics what requests_aws_sign was doing to the headers (so that I can then set them directly to the request above using the header parameter).
It looks like this:
def update_headers(sim_id):
url = urlparse("get_it_from_api->stages->your_deployment->invoke_url")
path = url.path or '/'
querystring = ''
if url.query:
querystring = '?' + urlencode(parse_qs(url.query), doseq=True)
safe_url = url.scheme + '://' + url.netloc.split(':')[0] + path + querystring
request = AWSRequest(method='GET', url=safe_url)
SigV4Auth(credentials, service, region).add_auth(request)
return dict(request.headers.items())