I want to ask whether anyone has experience using Terraform to deploy step functions?
I'm experimenting with step functions and need to deploy to multiple environments in a repeatable and auditable manner. I develop my step functions in the AWS console (Workflow Studio) in my sandbox environment but eventually I need to deploy them to my higher envs.
Currently this is done by exporting the step function from the sandbox environment as JSON and putting that into a Terraform module that is used to deploy the solutions. This poses a problem because the sandbox step function is invoking a lambda that lives in the sandbox environment while the other envs have their own lambda deployed of course, which the step function should be calling.
To solve that problem, the step function JSON is actually a template file where the ARNs for the lambdas are replaced with a variable which is then expanded per environment with the appropriate value.
But all this makes for a terrible development experience. Every time a change is made to the step function, I have to export the JSON, copy it into the Terraform module and replace all the sandbox ARNs with the correct template variables.
Does anybody have suggestions on how to streamline this? Are step functions only good for ad-hoc data processing where repeatable and auditable deployments are not needed or am I missing some obvious solution here?
Related
I recently (2 days ago) upgraded the node runtime engine on our Cloud Functions instance from Node 10 to 12. (Not sure that is a factor, but it is a recent change.)
Since the upgrade I have been using the Cloud Functions project without trouble. Today is the first time I have done a deploy SINCE the deployment to change the node engine. After I did the deploy, ALL of the Runtime Environment Variables were deleted except one labeled FIREBASE_CONFIG.
As a test, I added another test environment variable via the Cloud Functions console UI. I refreshed the page to ensure the variable was there. Then, I ran another deploy, using this command:
firebase use {project_name} && firebase deploy --only functions:{function_name}
After the deploy completed, I refreshed the environment variables page and found that the test variable I had created was now missing.
I'm quite stumped. Any ideas? Thank you!
It is true that the Firebase CLI manages enviroment configuration and does not allow us to set the ENV variables of the runtime during deployment. This has been explained on other post as well, like this one.
I guess you are aware of the difference between the Cloud Functions Runtime Variables and the Firebase Environment configuration, so I will just leave it here as a friendly reminder.
Regarding the actual issue (New deployments erasing previously set "Cloud Functions Runtime Variables"), I believe this must have been something they fixed already, because I have tested with version 9.10.2 of the firebase CLI, and I could not replicate the issue on my end.
I recommend checking the CLI version that you have (firebase --version) and, if you still experience the same issue, provide us with the steps you took.
We have a very simple use case--we want to share code with all of our lambdas and we don't want to use webpack.
We can't put relative paths in our package.json files in the lambda folders because when you do sam build twice, it DELETES the shared code and I have no idea why.
Answer requirements:
Be able to debug locally
Be able to run unit tests on business logic (without having to be ran in an AWS sandbox)
Be able to run tests in sam local start-api
Be able to debug the code in the container via sam local invoke
sam build works
sam deploy works
Runs in AWS Lambda in the cloud
TL;DR
Put your shared code in a layer
When referencing shared code in the lambda layer, use a ternary operator when you require(). Check an environment variable that is only set when running in the AWS environment. In this case, we added a short AWS variable in the SAM template, but you can find environment variables that AWS automatically defines, but they will not be as short. This enables you to debug locally outside of the AWS stack, allowing very fast unit tests that test business logic.
let math = require(process.env.AWS ? '/opt/nodejs/common' : '../../layers/layer1/nodejs/common');
let tuc = require(process.env.AWS ? 'temp-units-conv' : '../../layers/layer1/nodejs/node_modules/temp-units-conv');
You shouldn't need to use the ternary operator like that unless within the lambda folder code
Here's a working example that we thought we'd post so that others will have a much easier time of it than we did.
It is our opinion that AWS should make this much easier.
https://github.com/blmille1/aws-sam-layers-template.git
Gotchas
The following gotcha has been avoided in this solution. I am mentioning it because it looked like a straight-forward solution and it took a lot of time before I finally abandoned it.
It is very tempting to add a folder reference in the lambda function's package.json.
//...
"dependencies": {
"common":"file:../../layers/layer1/nodejs/common"
},
//...
If you do that, it will work the first sam build. However, the second time you run sam build, your shared code folder and all subdirectories will be DELETED. This is because when sam builds, it creates an .aws-sam folder. If that folder exists, it performs an npm cleanup, and I think that is what provokes the deleting of the shared code.
I'm working with a React application in GPC, and I have multiple environments [uat|training|staging]. Due to the way GCP is setup each application has a specific application configuration.
Therefore, I want to filter over my GCP app.yaml file to apply environment specific values per my build process. Is there something in NodeJS that allows me to do this? When building my current project I used create-react-app to do the initial build.
At this time, I'm still researching if anyone has done this or if this is an Anti-pattern with NodeJS applications. Not sure if anyone else has run into this problem.
I want to evaluate ARM template file which contains actual values passed by the user before it passed to deployment engine.
Is there any way to do that?
I have started creating evaluation code through PEGJS library of nodejs by using which I can evaluate only particular condition or expression present in AZURE ARM functions but can't evaluate actual template which is passed to deployment engine to create any service.
Also, I have checked sdk azure-rest-client but can't find any way please help me to find the solution for the above issue?
There is no built-in way of doing that, you can use validate deployment api call (its also implemented in different sdk's\cli's) but it doesnt actually guarantee that the template will work, it does some basic sanity checks.
You best bet is to write a script that would deploy the template and a set of tests that would validate the outcode.
I've been using AWS for a while now but am wondering about how to go about developing with Lambda. I'm a big fan of having server-less functions and letting Amazon handle the maintenance and have been using it for a while. My question: Is there a recommended workflow for version control and development?
I understand there's the ability to publish a new version in Lambda. And that you can point to specific versions in a service that calls it, such as API Gateway. I see API Gateway also has some nice abilities to partition who calls which version. i.e. Having a test API and also slowly rolling updates to say 10% of production API calls and scaling up slowly.
However, this feels a bit clunky for an actual version control system. Perhaps the functions are coded locally and uploaded using the AWS CLI and then everything is managed through a third party version control system (Github, Bitbucket, etc)? Can I deploy to new or existing versions of the function this way? That way I can maintain a separation of test and production functions.
Development also doesn't feel as nice through the editor in Lambda. Not to mention using custom packages require to upload anyways. Seems local development is the better solution. Trying to understand others workflows so I can improve mine.
How have you approached this issue in your experience?
I wrote roughly a dozen lambda functions that trigger based on S3 file write event or time, and make a HTTP req to an API to kickstart data processing jobs.
I don't think there's any gold standard. From my research, there are various approaches and frameworks out there. I decided that I didn't want to depend on any kind of frameworks like Serverless nor Apex because I didn't want to learn how to use those things on top of learning about Lambda. Instead I built out improvements organically based on my needs as I was developing a function.
To answer your question, here's my workflow.
Develop locally and git commit changes.
Mock test data and test locally using mocha and chai.
Run a bash script that creates a zip file compressing files to be deployed to AWS lambda.
Upload the zip file to AWS lambda.
You can have version control on your lambda using aws CodeCommit (much simpler than using an external git repository system, although you can do either). Here is a tutorial for setting up a CodePipeline for commit/build/deploy stages: https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-simple-codecommit.html
This example deploys an EC2 instance, so for the deploy portion for a lambda, see here
If you set up a pipeline you can have an initial commit stage, then a build stage that runs your unit tests and packages the code, and then a deploy stage (and potentially more stages if required). It's a very organized way of deploying lambda changes.
I would suggest you to have a look at SAM. SAM is a command line tool and a framework to help you to develop your serverless application. Using SAM, you can test your applications locally before to upload them to the cloud. It also support blue / green deployment and CI/CD workflows, starting automatically from github.
https://github.com/awslabs/aws-sam-cli