I am looking at using Amazon cloud services (EC2, S3 etc) for hosting. I've been looking at the JSON metadata that can be specified to configure various instances and the complexity has me concerned. Is there a dsl that will generate valid JSON metadata and more importantly validate the entries?
Unfortunately, I drew a blank after searching for this recently. I'm using Amazon Web Services CloudFormation (is that the JSON metadata you're talking about?).
There are a couple of issues with CloudFormation JSON files:
I'm at well over 1,500 lines and it's impossible to read,
You can't express everything the API gives you, notably in the area of Virtual Private Clouds,
There are lots of bugs that are taking a long time to fix, Elastic Load Balancers losing HTTPS information, for example.
So I've been using straight-up API calls in Scala using the Java API. It's actually really nice.
The Java API has a flavor of "setters" starting with with that return this so they can be chained. In Scala, you can use these to act like a poor-man's DSL. So you can do things like
val updateRequest = new UpdateAutoScalingGroupRequest()
.withAutoScalingGroupName(group.getAutoScalingGroupName)
.withAvailabilityZones(subnetAZsOfOurVPC)
.withVPCZoneIdentifier(subnetNamesOfOurVPC)
as.updateAutoScalingGroup(updateRequest)
Other things are easy to do in Scala with the Java API. For example, group all your Subnets by VPC in a Map, simply do
val subnetsByVPC = ec2.describeSubnets(new DescribeSubnetsRequest).getSubnets.groupBy(_.getVpcId)
In case anyone is still looking for the AWS CloudFormation DSL - we've been using the Ruby DSL for CloudFormation:
https://github.com/bazaarvoice/cloudformation-ruby-dsl
This neat project provides a tool to convert your existing CloudFormation template(s) to the Ruby DSL
It will generate valid JSON output
Validating Ruby template entries is similar to validating a regular CloudFormation template (see cfn-validate-template)
Your template becomes Ruby code, so it's easy to have reusable modules (DRY)
You can define local variables
You can have comments in your DSL template
Greatly improved readability
Greatly reduced DSL template size
The CloudFormation request template body size limits are annoying - we have to upload our large CloudFormation templates to S3, then create/update stacks using their S3 URLs.
There is now, though I haven't used it yet: Coffin a CoffeeScript DSL for CloudFormation.
If you're not talking about CloudFormation, but instead more general API talking, then the nicest interface I've found is AWS' own aws-sdk ruby gem. Unlike the other SDKs that they publish which are quite nicely-done-but-crude make-client/make-request/get-response/look-at-result affairs, the ruby SDK wraps a nicer domain-model over the top, so you interact via collections at a higher abstract level.
It also has quite-nice performance features in that you can cache responses to save on round-trips, if you know you don't need fresh responses.
I have CloudFormation templates upwards of 3000 lines. I have found that putting comments in the JSON helps tremendously!! You just have to strip it out before using it. There is a validator that would validate the template and strip out the comments: http://cloudformation-validation.com/
No, as of Feb 2022, AWS still does not provide any domain-specific language for infrastructure as code. They have nothing similar to Azure's Bicep or Terraform's HCL.
This really surprises me, as I generally think of AWS as being more expensive, but ahead of the curve, compared to other major competitors (Azure and GCP).
However, Cloud Formation now allows both JSON and YAML formats. Slight improvement?? IMHO, not really when you have an entire repo that represents your entire cloud stack. If you're using AWS, use Terraform to manage IaC.
Related
Are there any benefits to spawn a new AWKS EKS cluster by using terraform or eksctl?
Are there some long-term maintenance benefits of one vs another?
Well, although I haven't actually tried this out with Terraform, I can definitely say that the eksctl way is not recommended. At least not if you're interested in manageing your infrastructure as code.
With eksctl, most changes to an existing cluster need to be made with specific eksctl commands. Just changing the (declarative) cluster.yaml (or whatever you name) does not apply anything relevant. You want to scale a nodeGroup? Well, please use eksctl scale nodegroup, as changing the size in the YAML file is not applying anything. I think you get the pattern.
It's really sad that, of all companies, Weaveworks, the "inventors" of GitOps, provide a tool that does not even support basic IaC :(
I would highly recommend using terraform. It is declarative and provides an interface that can be used to support all of your infrastructure and not just your EKS cluster(s).
The time and effort you put into learning terraform and implementing it in your pipeline can be easily re-used for other infrastructure needs unlike eksctl.
I am having difficulty in understanding whether to use aws cdk or terraform cdktf. From my work so far, it appears to me that aws cdk has more robust patterns and constructs which will comply with the well architected framework, all available out of the box. Terraform cdktf will require hand crafting many of such constructs and patterns.
An example in point is the construct ecs_patterns.ApplicationLoadBalancedEc2Service which does a lot of heavy lifting in spinning up a industrial strength infrastructure for EC2 based ECS service. I can't find the equivalent of that in Terraform cdktf and it appears I will have to manually assemble and connect all the infra elements.
However, Terraform apparently has the advantage of working with several different cloud vendors, and therefore I want to do due diligence before choosing one of the other.
Therefore, I would like to know if my understanding is correct and if I am not missing something really important. Any other advice / considerations in this matter are highly appreciated
Thanks
The AWS CDK is limited to only AWS as a cloud, whereas you can use CDKTF with any / most of the clouds since terraform providers exist for most of them. The AWS CDK has a diverse ecosystem of constructs that can be used, which is a plus. CDKTF will eventually support these via the CDKTF AWS Adapter.
Need an expert's input who has used cloudformation, sam and serverless framework to deploy nodejs app.
Please advise which is the best path to take I have used serverless framework but not sam or cloudformation, while I agree that it simplifies the process, I wish to learn more on the underlying configuration.
I am leaning towards cloudformation just because both the frameworks transform the code to cloudformation templates. Please correct me if I am wrong and appreciate the best resources to learn the same.
SAM is basically an extension of Cloudformation. If you know SAM, you basically know Cloudformation. SAM can and should be used in conjunction w/ Cloudformation for testing locally.
Serverless is an abstraction layer on top of Cloudformation. It helps to expedite app creation and deployment. It falls short if you are doing more advanced configurations
I always lean towards Cloudformation (or SAM) because it is provided by the CSP (ie AWS). This means everything new will be automatically available, rather than waiting for an abstraction layer (like serverless) to bake in support.
I recommend the Serverless Framework. The advantages are several:
Much easier syntax than CloudFormation
More portable than CloudFormation (you can change between cloud providers)
Ability to write raw CloudFormation inline in serverless.yml if needed
Easy to deploy (just run sls deploy)
There is a sample project at my gitlab profile if you are interested (for GoLang but the principles are the same as for other runtimes) https://gitlab.com/montao/aws-lambda-go-gitlab-sls
LostJon is absolutely correct in that the support for both CloudFormation and SAM is immediately available as it is natively provided by AWS. I am leaning to CloudFormation for the reason that is has a deterministic outcome, which is desired in a DevOps landscape. There are no transformations unless you explicitly add transformations to your CloudFormation stack.
When you deploy a CloudFormation template with SAM, you will be able to view the CloudFormation template in the AWS console, but you can also choose to show the rendered template (e.g. after transformations) so that you can learn what SAM does for you underwater. You could copy the rendered resources and use them as pure CloudFormation. This allows you to speed up development, while not losing the benefit of knowledge of the underlying configuration.
There is this great repository with example implementations of different serverless scenarios.
Right now I'm struggling with the combination of AppSync and Amazon RDS. I tried the implementation of the standalone rds, and the appsync examples provided in the repository. These are working like a charm.
But obviously there are many differences and difficulties if you'd like to combine these technologies. I used the schema, resolver and handler functions from the rds directory and combined it with the appsync lambda implementation. I adjusted the mapping templates and updated the serverless.yml file.
I could successfully deploy the whole appsync service and all resources without any errors. I'm able to access the graphql endpoint from graphiql and do my queries. But when I try it from the appsync console I get null as a response. I guess it has something to do with the mapping templates, but I'm not quite sure.
Has anybody got any suggestions or maybe a working example of this specific combination?
I finally could figure out a working implementation for this specific setup which I want to share with all of you.
Check out my serverless-graphql-appsync-rds repository on GitHub and leave me some feedback!
Note that this repository contains just the source code without any explanations. I'll create a better documentation in the near future.
In AWS Lambda people can create a node.js function and trigger it through events, for example a message, etc..
I wonder how this can work 'under the hood' and how to put something like this together in Haskell. The uploaded functions are basically single function libraries without any main function.
Means on the CLI or via an API you can call any of your functions by name (and with the required input) and you get the output defined by the function signature--or, alternatively an error of course.
Would it be possible to do this in Haskell?
To clarify: what I want to do is for example loading a number of different single function Haskell libraries on a Haskell platform or any other execution context that is in my data center and execute / call them by name via the CLI or an API just in the same way AWS Lambda works with node.js functions.
If you want reproduce the same functionality (functions as services) exists a lot of technologies you can use (soap, rpc, rest, ...). If you unknown it I suggest to you read about.
My ever favorite is SOAP but is so unpopular and Haskell support is limited soap (see related question). SOAP (as others) provide exactly you want.
But you must to concrete your real problem to select the best technology.
If you are looking for distribute your own Haskell code Cloud Haskell may be a good starting point.
If you need more like "web server" then take a look to sodium, elm, ... in a Javascript style or servant (generate client code to some languages).
Anyway, even AWS Lambda require support for each language and you should not expect to find one ubiquitous technology (like HTTP) for RPC.
You may want to check out the "serverless for haskell" framework: http://qmu.li
Not only you can run individual haskell functions as Lambda functions with it but you can also describe your whole AWS infrastructure in haskell. (vs. doing it in CloudFormation json/yaml template), build it all locally and easily deploy to AWS.