As a project, I am trying to create a CI/CD pipeline running inside an AWS Lambda application.
The problem I am facing is that AWS Lambda is missing some tools (for example xargs) that certain applications (for example Gradle) require to run properly:
/tmp/repo/gradlew: line 234: xargs: command not found
Or even more interestingly:
install: apt-get: command not found
How can I install the required tools to build the applications from within an AWS Lambda container?
How can I utilize layers to speed up those containers?
Aka, I assume I need to register that certain cli tools are present in mounted layers.
On windows, I would do this by (ab)using the PATH environment variable, but what is the recommended way to do this in Linux?
And how can I tell tools to look for their dependencies in those layers? to avoid errors like:
ld.gold: error: cannot find -lcurl
The best option as far as I can tell is to create a Docker image containing all the software that you require and provide this to the AWS Lambda service.
There is extensive documentation how to run Docker containers in AWS Lambda:
https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
Personal note: While I like the idea of a challenge or proof-of-concept I'd recommend using one of the many CI/CD services out there instead of building one on your own. I can not think of any upside of this. AWS itself offers CI/CD solutions like AWS CodePipeline etc.
You might want to have a look at the following documentation:
https://aws.amazon.com/getting-started/hands-on/set-up-ci-cd-pipeline/
Related
The answer to Can I configure Linux swap space on AWS Elastic Beanstalk? (from 2016) shows how to configure Linux swap space for an AWS Elastic Beanstalk environment using .ebextensions configuration files.
However, the AWS docs Customizing software on Linux servers has this note for the newer Amazon Linux 2 platforms:
On Amazon Linux 2 platforms, instead of providing files and commands in .ebextensions configuration files, we highly recommend that you use Buildfile. Procfile, and platform hooks whenever possible to configure and run custom code on your environment instances during instance provisioning.
How do I configure swap space using this more modern configuration approach?
Buildfile and Procfile are not suited for that. They serve different purposes - running short and long running commands.
I would use the platform hooks for that. Specifically, prebuild:
Files here run after the Elastic Beanstalk platform engine downloads and extracts the application source bundle, and before it sets up and configures the application and web server.
The rationale is that it's better to create swap now, before the application starts configuring. If the swap creation operation fails, you get notified fast, rather then after you setup your application.
From the SO link, you could put 01_add-swap-space.sh into .platform/hooks/prebuild/ folder. Please make sure that 01_add-swap-space.sh is executable (chmod +x) before you package your appliaction into a zip.
We are taking over a whole application from another company, and they have built the whole pipeline for deploying, but we still don't have access to it. What we know, that there's a lambda function is running triggered by certain SNS messages, and all the code is in Node.js, and the development is in VS Code. We also have issues debugging it locally, but it's a bigger problem, that we need to debug it remotely.
Since I am new in AWS services, I'd really appreciate if somebody could help me in this.
Does it necessary to open a port? How is it possible to connect to a lambda? Do we need serverless to setup? Many unresolved questions.
I don't think there is way you can debug a lambda function remotely. Your best bet is to download the code on local machine, setup the env variables as you have set up on your lambda function and take it from there.
Remember at the end of the day lambda is just a container which is running the code for you. AWS doesn't allow any ssh or connection with those container. In your case you should be able to debug it on local till you have the same env variables. There are other things as well which are lambda specific but considering it is a running code which you have got so you should be able to find out the issue.
Hope it makes sense.
Thundra (https://www.thundra.io/aws-lambda-debugger) has live/remote debugging support for AWS Lambda through its native IDE plugins (VSCode and IntelliJ IDEA).
The way AWS have you 'remote' debug is to execute the lambda locally through Docker as it proxies the requests to the cloud for you, using AWS Toolkit. You have a lambda running on your local computer via docker that can access resources on the cloud, such as databases, api's etc. You can step through debug them using editors like vscode.
I use SAM with a template.yaml . This way, I can pass event data to the handler, reference dependency layers (shared code libraries) and have a deployment manifest to create a Cloudformation stack (release instance with history and resource management).
Debugging can be a bit slow as it compiles, deploys to Docker and invokes, but allows step through debugging and variable inspection.
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-debugging.html
While far from ideal, any console-printing actions would likely get logged to CloudWatch, which you could then access to go through printed data.
For local debugging, there are many Github projects with Dockerfiles which which you can build a docker container locally, just like AWS does when your Lambda is invoked.
I have a docker image which I am running on Google's Cloud Run.
When I want to run the image locally, I have to give my container additional capabilities like the following:
docker run -p 8080:8080 --cap-add=SYS_ADMIN gcr.io/my-project/my-docker-image
Is there a way of configuring Docker's capabilities in Cloud Run?
I stumbled upon this piece of API documentation from Google, but I don't know how to configure my container. I am not even sure that it is relevant to my situation.
Any help would be really appreciated.
Expanding the POSIX capabilities is not an option on Cloud Run or Cloud Run on GKE as they represent expanding the security vulnerabilities of the underlying host.
Adding capabilities is often the easiest way to make something with special system demands work. More complex but frequently doable are modifications within the container environment or to the package configuration to get things working.
If what you're trying to do absolutely requires cap-add, this might be addressed in a feature request to the software package... or it may be a novel use case that Cloud Run cannot support but may in the future with your feedback.
We are using chef to deploy all of our stacks.
I need to build a runbook for each environment we deploy.
I have been parsing the environment, node and recipe files but the more information I need to extract, the more complex it becomes because I am converging the attributes in my application.
I would like to use the converged-attributes.json file produced by our chef deployment without deploying any code because we can't deploy production to the build runbooks.
We also plan to build the runbook before the environment exists to provide configuration information to the DevOps team (e.g. memory requirements, ports, etc.).
Is there a way to use any of the chef/knife components or libraries to do the following?
Converge the attributes for each node
Write the converged attributes to a location my application can access on Mac OSX.
Quit before attempting to access any servers
This is not possible in the generic case. Chef is executable code at heart and the only way to fully compute the side effects is to actually execute it. This is what chef-client does, you can't "converge" the node externally so step 3 doesn't really make any sense. You could try to use Why Run mode but we really don't recommend it and are probably going to remove the feature as it does more harm than good most of the time. Roles and environments are static data so you can parse and manipulate those, but cookbooks are code and have to be run in-place to know exactly what they will do.
guys,
For various projects, I'm creating single Docker environments. Each Docker container consists of Debian, Nginx, Node.js, etc. and is going to use by developers as well as in production via Google Cloud's Kubernetes. Since the Node.js/module version should be everywhere the same, I would like to restrict the access to certain npm commands (somehow). Often developers work with different Node.js and project modules and that caused a lot of trouble in the past. With the Docker containers, I can provide environments with everything you need for a project. To finish this step, I would like to restrict the npm command execution and only allow arguments like install, test, etc.
Please drop me a comment if you know how to resolve this :)
Cheers
It is almost impossible to limit your developers to run some commands in the container if they have an access to Dockerfiles and can somehow change a build flow.
But, because container providing isolation and you can build a custom container for which application based on your basic image, it can be not a big problem if the version of any package for one application will be changed somehow, as an example in a build step, because it will not affect other apps. They just have different containers.
So, you will not have a problem with compatibility like when you using one server with many application which using a shared environment.
The only one thing you need to do - make sure that nobody change container which you using as a base image.