I'm just starting learning terraform and am trying to setup an elastic search cluster with an API gateway in front of it. I've successfully built the service such that the Elastic Search cluster is built and the output endpoint is passed into the API gateway for the integration request via output variables.
In my initial trials I was using wide open access in the aws_elasticsearch_domain.my_name.access_policies for testing my template code. This worked fine for testing purposes but for real world use I want to use the ARN of the API gateway in the aws_elasticsearch_domain.my_name.access_policies. This is problematic seemingly because the aws_api_gateway_integration.my_name.uri needs the aws_elasticsearch_domain.my_name.endpoint to setup the but aws_elasticsearch_domain.my_name.access_policies needs the aws_api_gateway_deployment.my_name.execution_arn.
I'm guessing this is fairly common but I can't figure out how to achieve this through outputs or variables.
Thanks for any help.
One way to get around this is by creating a route53 endpoint for your elastic search and hard coding the route53 endpoint of elastic search in API gateway there by breaking the circular dependency.
Related
i am trying to connect to an aws region and want to find all resources running in that. the resources can be any thing from the list of services provided by aws (ec2, rds...). Now that i am writing python code to create a client for every service and getting the list. if i have to write the code for all services, it will be huge. please suggest me a best approach to grab the details with python. i cant use the aws config or resource manager as these are not whitelisted yet.
In a hobby side project I am creating an online game where the user can play a card game by implementing strategies. The user can submit his code and it will play against other users strategies. Once the user has submitted his code, the code needs to be run on server side.
I decided that I want to isolate code execution of user submitted code into an AWS lambda function. I want to prevent the code from stealing my AWS credentials, mining cryptocurrency and doing other harmful activity.
My plan is to do following:
Limit code execution time
Prevent any communication to internet & internal services (except trough the return value).
Have a review process in place, which prevents execution of user submitted code before it is considered unharmful
Now I need your advice on how to achieve best isolation:
How do I configure my function, so that it has no internet access?
How do I configure my function, so that it has no access to my internal services?
Do you see any other possible attack vector?
How do I configure my function, so that it has no internet access?
Launch the function into an isolated private subnet within a VPC.
How do I configure my function, so that it has no access to my internal services?
By launching the function inside the isolated private subnet you can configure which services it has access to by controlling them via the security groups and further via Route Table this subnet attached including AWS Network ACLs.
Do you see any other possible attack vector?
There could be multiple attack vectors here :
I would try to answer from the security perspective in AWS Services. The most important would be to add AWS Billing Alerts setup, just in case there is some trouble at least you'll get notified and take necessary action and I am assuming you already have MFA setup for your logins.
Make sure you configure your lambda with the least privilege IAM Role
Create a completely separate subnet dedicated to launching the lambda function
Create security for lambda and control this lambda access to other services in your solution.
Have a separate route table for the subnet where you allow only the selected services or be very specific with corresponding IP addresses as well.
Make sure you use Network ACLs to configure all the outgoing traffic from the subnet by adding ACL as well as an added benefit.
Enable the VPC flow logs and have the necessary Athena queries with analysis in place and add alerts using AWS CloudWatch.
The list can be very long when you want to secure this deployment fully in AWS. I have added just few.
I'd start by saying this is very risky and allowing people to run their own code in your infrastructure can be very dangerous. However, that said, here's a few things:
Limiting Code Execution Time
This is already built in to Lambda. Functions have an execution limit on time which you can configure easily through IaC, the AWS Console or the CLI.
Restricting Internet Access
By default Lambda functions can be thought of as existing outside the constraints of a VPC for more applications. They therefore have internet access. I guess you could put your Lambda function inside a private subnet in a VPC and then configure the networking to not allow connections out except to locations you want.
Restricting Access to Other Services
Assuming that you are referring to AWS services here, Lamdba functions are bound by IAM roles in relation to other AWS services they can access. As long as you don't give the Lambda function access to something in it's IAM role, it won't be able to access those services unless a potentially malicious user provides credentials via some other means such as putting them in plain text in code which could be picked up by an AWS SDK implementation.
If you are referring to other internal services such as EC2 instances or ECS services then you can restrict access using the correct network configuration and putting your function in a VPC.
Do you see any other possible attack vector?
It's hard to say for sure. I'd really advise against this completely without taking some professional (and likely paid and insured) advice. There are new attack vectors that can open up or be discovered daily and therefore any advice now may completely change tomorrow if a new vulnerability is discovered.
I think your best bets are:
Restrict the function timeout to be as low as phyisically possible (allowing for cold starts).
Minimise the IAM policy for the function as far as humanly possible. Careful with logging because I assume you'll want some logs but not allow someone to run GB's of data in to your CloudWatch logs.
Restrict the language used so you are using one language that you're very confident in and that you can audit easily.
Run the lambda in a private subnet in a VPC. You'll likely want a seperate routing table and you will need to audit your security groups and network ACL's closely.
Add alerts and VPC logs so you can be sure that a) if something does happen that shouldn't then it's logged and traceable and b) you are able to automatically get alerted on the problem and rectify it as soon as possible.
Consider who will be reviewing the code. Are they experienced and trained to spot attack vectors?
Seek paid, professional advice so you don't end up with security problems or very large bills from AWS.
We are attempting to get Azure SignalR serverless to with a dotnet core API application. With "default" SignalR, we ran into scaling issues in Azure as server instances behind an API App would continue to receive socket connections even as its CPU increases. There is no way to currently to change load balancing behavior or to take an instance out of traffic. As such, we've looked to use "serverless", however, all documentation points to using Azure Functions. However, given that serverless uses webhooks and such, we should be able to use anything that can take an HTTP request. We already have our APIs setup so getting this to work against out APIs is preferred.
Update 1
Effectively, we're looking for support for serverless that Functions get but for APIs. Functions have triggers and Serverless Hubs to inherit from, etc etc. These things handle negotiate calls and deserialization of negotiation data and all the other things SigR has to do. We're looking for something similar for, I guess, API controllers.
Thanks!
I've been working on developing an API to serve a machine learning model using Azure Machine Learning (AML) webservice deployments on a Kubernetes target as outlined here: https://learn.microsoft.com/en-us/azure/machine-learning/service/how-to-deploy-and-where#prepare-to-deploy
My particular use case requires more than simple scoring of data through the model. The API needs to have multiple endpoints to perform different actions related to the model. For example, an endpoint to upload data, an endpoint to delete data, an endpoint to list existing data, an endpoint to score previously uploaded data, an endpoint to change preprocessing parameters, etc...
I can build all of this logic, but I am struggling with the fact that AML web services only provides one endpoint (The service URI ending in "/score"). Is there a way to add more endpoints to an AML service? For example, I would like to have a way for users to be able to POST, GET, DELETE, PUT "/data", GET "/predictions", and POST, GET, DELETE, PUT "/parameters", etc..
Is there a way to do this in AML or is this not the right tool for what I am trying to accomplish? Is there a better solution within Azure that is more suited for my needs?
Thank you!
Azure ML allows controlled rollout/traffic splitting, but doesn't directly support your API design.
I might need to know more about your use case to make a recommendation. Are you looking at implementing incremental learning? What is the motivation for separate endpoints?
-Andon
Your proposal seems like a stateful web server which is more than a REST API service. For example, you need to keep a piece of logic to maintain "ids" of data: if there are two POST /data calls with different data, and the DELETE /data need to operate on the proper one. This is much more than a single performance optimized machine learning service.
I would recommend you creating a separate server with all these logic pieces and only reach Azure Machine Learning service whenever you need it. You could also build a cache in your service to only call Azure ML service when a new data coming in or the local cache expired. It will save you additional money from Azure :-)
I’ve finished working on my node application which provide a series of APIs endpoint for a mobile application. I’ve made it with node and mongodb as db. Now I’ve reached the point where I should pick the right deployment environment.
Initially I’ll make a private beta but I need to choose a service I can scale easily (I’m not a devop) with the right price balance.
My initial choice is amazon aws (elastic beanstalk?). What’s about the db? I’ve not used dynamodb in order to be more service agnostic but now I don’t know how to create a reliable db infrastructure. Any suggestion to deploy both app and dB in order to make easy scaling in case it will become necessary?