I'm working on this cloud project where we have several development repositories in GitHub and in each we have the overlays containing config files that are specific for a local K8S cluster, a dev Azure cluster and a prod Azure cluster.
In order to have different repos for these envs we use a repo with a kustomization file for each service that fetches the overlay of the dev/test/prod and uses it as it's base.
However the issue is managing this resources since we don't want to share the dev repos to possible clients or other end users in order for them to deploy these services into their K8S environment but not giving them permissions will imply that they will not be able to fetch these overlays and bases and deploy them.
What is the best practice in order to have a protected and restrictive dev repos and yet be able to do the deployment operation?
I know this is a abstract question but I've never dealt with organization of repos in a scale like this.
To clarify I am posting Community Wiki answer.
The solution you suggested in comment's section:
We will have the deployments/namespaces/services manifests in the same repo as the application source code as well the an overlay with a customization with the necessary resources to fully deploy in the dev environment.
As for test/ prod environments we created a structure to add an overlay per app with the same resource files but with the env details in the files to be used as configmaps.
And a customization using the dev repository as the base. Unfortunately this will imply that the cluster admin will have access to all repos of an application.
Related
As I'm pretty new to Terraform, I'm not sure if Terraform is suitable for the problem I'd like to solve. Here is the scenario...
We are using Keycloak as our Identity & Access Management tool. Currently, we're running dedicated Keycloak instances in different environments:
(locally - on each developer's machine)
dev
preproduction
production
The configuration is done completely via the web UI which is, no surprise, cumbersome and error-prone.
Example
For adding a Keycloak client or adding roles, the workflow is similar to the following:
A developer makes the configuration changes on her local Keycloak instance.
If things are working, the same configuration needs to be applied to the dev instance.
Then on the preproduction instance...
Finally, on the production instance.
I was able to create a basic Terraform main.tf which successfully performs all the configuration on my local machine. But thinking this further, I have some difficulties...
The above workflow is not "cloud-centric", that is, it's not our goal to apply the same Keycloak configuration to different environments, but rather applying different Keycloak configurations depending on their stage to a dedicated environment. For example, a Keycloak role app_admin may exist in the dev stage but not yet in the preprod and prod stage.
The most basic question:
Is Terraform a suitable tool to cover the above workflow?
We'd like to allow our developers to automatically deploy their changes to a kubernetes cluster by merging their code and k8s ressources in a git repo which is watched by ArgoCD. The release management teams would responsible to manage the ArgoCD config and setting up new apps as well as for creation of namespaces, roles and role bindings on the cluster while the devs should be able to deploy their applications through GitOps without the need to interact with the cluster directly. Devs might have read access on the cluster for debugging purposes.
Now the question: in theory it would be possible that a dev would create a new yaml and specify a rolebinding ressource, which binds his/her account to a cluster admin role. As ArgoCD has cluster admin rights, this would be a way to escalate privileges for the dev. (or an attacker impersonating a developer)
Is there a way to restrict, which k8s ressources are allowed to be created through ArgoCD.
EDIT:
According to the docs, this is possible per project using clusterResourceWhitelist.
Is it possible to do that globally?
You are right about Argo CD project. The project CRD supports allowing/denying K8S resources using clusterResourceWhitelist, clusterResourceBlacklist etc fields. The sample project definition is also available in Argo CD documentation.
In order to restrict the list of managed resources globally, you can specify the resource.exclusions/resource.inclusions field in the argocd-cm ConfigMap. The example is available here.
Each developer on our team forks the production repository, then opens MRs from their fork to merge into the master branch of the production one.
We have a fixed number of test environments which are shared by all developers. Eventually we hope to move to ephemeral environments for each MR but at the moment this is not possible.
We utilize the Environments / Deployments functionality of Gitlab: https://docs.gitlab.com/ee/ci/environments/ and the production repository is the central location where we see what is deployed to each environment.
The problem is that when a developer opens a MR they frequently choose to manually deploy to one of our shared test environments. But since the pipeline for their branch runs in their fork it records the Environment / Deploy information there rather than in production. This makes it impossible for us to know who deployed what to each test environment as that information is recorded in random developer forks rather than in the centralized location.
We only deploy to production hosts from the production repository (it is disabled in developer forks) so that information is centralized and up to date. But it is a headache for developers to determine who has deployed something to one of the shared test environments and they frequently overwrite each others' changes by accident.
Is there any way for us to allow test deploys from developer branches, but centralize the information about what is deployed in each environment in the production repository?
I have a project that consists of an Azure webapp, a PostgreSQL on Azure, and multiple Azure functions for background ETL workflows. I also have a local Python package that I need to access from both the webapp and the Azure functions.
How can I structure configuration and script deployment for those resources from a single git repo?
Any suggestions or pointers to good examples or tutorials would be very appreciated.
All the Azure tutorials that I've seen are only for small and simple projects.
For now, I've hand-written an admin.py script that does e.g. the webapp and function deployments by creating a Python package, creating ZIP files for each resource and doing ZIP deployments. This is getting messy, and now I want to have QA and PROD versions, and I need to pass secrets so that the DB is reachable, and it's getting more complex. Is there either a nice way to structure this packaging / deployment, or a tool to help with it? For me, putting everything in Kubernetes is not the solution, at least the DB already exists. Also, Azure DevOps is not an option, we are using Gitlab CI, so eventually I want to have a solution that can run on CI/CD there.
Not sure if this will help complete but here we go.
Instead of using a hand-written admin.py script, try using a yaml pipeline flow. For Gitlab, they have https://docs.gitlab.com/ee/ci/yaml/ that you can use to get started. From what you've indicated, I would recommend having several job steps in your yaml pipeline that will build and package your web and function apps. For deployment, you can make use of environments. Have a look at https://docs.gitlab.com/ee/ci/multi_project_pipelines.html as well which illustrates how you can create downstream pipelines.
From a deployment standpoint, the current integration I've found between Azure and GitLab leaves me with two recommendations:
Leverage the script command of yaml to continue zipping your artifacts use Azure CLI (I would assume you can install the tools during the pipeline) to zip deploy.
Keep your code inside the GitLab repo and utilize Azure Pipelines to handle the CI/CD for you.
I hope you find this helpful.
I saw projects hosted on Gitlab with Gitlab AutoDevOps connect to a separate GKE cluster, with every project connecting to its own GKE cluster.
I saw there is something called monorepo in Gitlab which can be used to deploy everything on a single cluster but clusters being expensive to create for a single project, what was the reason behind Gitlab going for one cluster per project instead of say, one namespace per project or something else?
instead of say, one namespace per project or something else?
Actually... GitLab 13.5 (October 2020) has now:
Customizable namespaces for GitLab Managed Clusters
Users of GitLab Managed Clusters can now choose to use a single namespace per project, or a single namespace per environment, when creating clusters.
While single namespaces per project simplified finding review apps, separate namespaces per environment can provide additional security.
See Documentation and Issue.