What would be the best place for cross-environment Terraform configuration files in Terragrunt? Let's say that the directory structure looks like this:
└── live
├── terragrunt.hcl
├── prod
│ ├── app
│ │ └── terragrunt.hcl
│ ├── mysql
│ │ └── terragrunt.hcl
│ └── vpc
│ └── terragrunt.hcl
├── qa
│ ├── app
│ │ └── terragrunt.hcl
│ ├── mysql
│ │ └── terragrunt.hcl
│ └── vpc
│ └── terragrunt.hcl
└── stage
├── app
│ └── terragrunt.hcl
├── mysql
│ └── terragrunt.hcl
└── vpc
└── terragrunt.hcl
All nested terragrunt.hcl files are sourcing corresponding modules, eg. prod/app/terragrunt.hcl has a source in modules/app where terraform configs live:
modules
├── app
│ └── ...terraform configs
├── mysql
│ └── ...terraform configs
└── vpc
└── ...terraform configs
This works fine. But where in this directory structure should common terraform configs live (terragrunt documentation says about common terragrunt config to dry the configuration)? Modules seems to be a good place for reusable (among environments) configurations...
Related
I am a little stuck on the best way to deliver the following deployments.
For clarity, child terragrunt.hcl files are pulling from a private repo.
.
├── env01
│ ├── resource_group
│ │ └── terragrunt.hcl
│ ├── virtual_network
│ │ └── terragrunt.hcl
│ └── subnet
│ └── terragrunt.hcl
├── env02
│ ├── resource_group
│ │ └── terragrunt.hcl
│ ├── virtual_network
│ │ └── terragrunt.hcl
│ └── subnet
│ └── terragrunt.hcl
└── terragrunt.hcl
How can I, in the above terragrunt layout, merge the child terragrunt.hcl files into a single state file, at present, we have a state file per terragrunt.hcl per resource, which, while works, feels overkill for me.
Do I need to do this at the terraform end? Creating modules for the env01/env02 deployments or is there a way to achieve some kind of terraform merge when deploying?
Hope that makes sense...
I'm new to Terraform and have been spending some time reading up and had a couple of questions related to the best way to structure the code. My plan is to store my Terraform code all in a single repository for all my projects. The infrastructure (Azure) consists of Virtual Machines and an AKS cluster. 98% of our VMs are all exactly the same except for subscription (uat/prod), size, resource group, name and maybe few differences in data disks. My first idea was to create a single environment for virtual machines with multiple .tfvars files for each of the VMs I want to manage? Like this:
└── terraform/
├── virtual_machines/
│ ├── main.tf
│ ├── variables.tf
│ ├── outputs.tf
│ ├── vm1.tfvars
│ ├── vm2.tfvars
│ └── vm3.tfvars
└── aks/
└── aks_project/
├── main.tf
├── variables.tf
└── outputs.tf
Then I can just specify what .tfvars files to point to when applying?
The second idea would be to put the shared vm code in a module then create a directory for each VM like this:
└── terraform/
├── virtual_machines/
│ ├── modules/
│ │ └── virtual_machines/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ └── outputs.tf
│ ├── vm1/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ └── outputs.tf
│ ├── vm2/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ └── outputs.tf
│ └── vm3/
│ ├── main.tf
│ ├── variables.tf
│ └── outputs.tf
└── aks/
└── aks_project/
├── main.tf
├── variables.tf
└── outputs.tf
With each VM directory sourcing from the modules and including the variable values needed for each?
Are either of these approaches make sense for what I'm looking to do? What would you suggest for or use for organizing your terraform code?
Your second approach looks like a better start, as it applies Hashicorp's recommendation of using modules whenever it's possible.
Then you can simply define each VM in a separate .tf file (or group all of them in one), each one of these files should use your local VM module that sets your defaults, and then provide multiple tfvars for each environment.
So you would have something like:
.
├── main.tf
└── virtual_machines
├── modules
│ └── virtual_machines
│ ├── README.md
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
├── dev.tfvars
├── uat.tfvars
├── prod.tfvars
├── variables.tf
├── locals.tf
├── vm1.tf
└── vm2.tf
Where vm1.tf is using your local module and sets its custom parameters, for example:
module "vm1" {
source = "./modules/virtual_machines"
name = "my-vm1-name"
environment = var.environment
}
My var.environment here is defined by my tfvars file which will be picked up for each environment, if I'm on dev I would run my plan or apply with dev.tfvars.
Read more at:
https://cloud.google.com/docs/terraform/best-practices-for-terraform#module-structure
https://www.hashicorp.com/blog/structuring-hashicorp-terraform-configuration-for-production#considerations-for-defining-modules
https://developer.hashicorp.com/terraform/tutorials/modules/pattern-module-creation?in=terraform%2Fmodules&product_intent=terraform
https://www.terraform-best-practices.com/examples/terraform
Also, you may want to consider options like terragrunt for this scenario of configuration DRY.
I hope this answer helps, it's based on my opinions and my experience. You may have a different use case, and a different setup, and you may have other answers from smarter people suggesting different approaches since there's nothing such as a "Best terraform project structure".
I am facing a big problem while deploying to vercel via now.
I am able to successfully execute yarn run build and yarn run start, but when I am trying to deploy it via now.
I get this error
Error: You have 'doc' in your headerLinks, but no 'docs' folder exists one level up from 'website' folder. Did you run `docusaurus-init` or `npm run examples`? If so, make sure you rename 'docs-examples-from-docusaurus' to 'docs'.
2020-08-24T18:48:36.566Z at forEach (/vercel/58999d95/node_modules/docusaurus/lib/core/nav/HeaderNav.js:253:15)
2020-08-24T18:48:36.566Z at Array.forEach (<anonymous>)
2020-08-24T18:48:36.566Z at HeaderNav.renderResponsiveNav (/vercel/58999d95/node_modules/docusaurus/lib/core/nav/HeaderNav.js:248:17)
2020-08-24T18:48:36.566Z at HeaderNav.render (/vercel/58999d95/node_modules/docusaurus/lib/core/nav/HeaderNav.js:325:19)
2020-08-24T18:48:36.566Z at processChild (/vercel/58999d95/node_modules/react-dom/cjs/react-dom-server.node.development.js:3134:18)
2020-08-24T18:48:36.567Z at resolve (/vercel/58999d95/node_modules/react-dom/cjs/react-dom-server.node.development.js:2960:5)
2020-08-24T18:48:36.567Z at ReactDOMServerRenderer.render (/vercel/58999d95/node_modules/react-dom/cjs/react-dom-server.node.development.js:3435:22)
2020-08-24T18:48:36.567Z at ReactDOMServerRenderer.read (/vercel/58999d95/node_modules/react-dom/cjs/react-dom-server.node.development.js:3373:29)
2020-08-24T18:48:36.567Z at renderToStaticMarkup (/vercel/58999d95/node_modules/react-dom/cjs/react-dom-server.node.development.js:4004:27)
2020-08-24T18:48:36.567Z at renderToStaticMarkupWithDoctype (/vercel/58999d95/node_modules/docusaurus/lib/server/renderUtils.js:16:48)
2020-08-24T18:48:36.607Z error Command failed with exit code 1.
Here is my file structure.
example
├── Dockerfile
├── docker-compose.yml
├── docs
│ ├── doc1.md
│ ├── doc2.md
│ ├── doc3.md
│ ├── exampledoc4.md
│ └── exampledoc5.md
└── website
├── README.md
├── blog
│ ├── 2016-03-11-blog-post.md
│ ├── 2017-04-10-blog-post-two.md
│ ├── 2017-09-25-testing-rss.md
│ ├── 2017-09-26-adding-rss.md
│ └── 2017-10-24-new-version-1.0.0.md
├── core
│ └── Footer.js
├── package.json
├── pages
│ └── en
│ ├── help.js
│ ├── index.js
│ └── users.js
├── sidebars.json
├── siteConfig.js
├── static
│ ├── css
│ │ └── custom.css
│ └── img
│ ├── favicon.ico
│ ├── oss_logo.png
│ ├── undraw_code_review.svg
│ ├── undraw_monitor.svg
│ ├── undraw_note_list.svg
│ ├── undraw_online.svg
│ ├── undraw_open_source.svg
│ ├── undraw_operating_system.svg
│ ├── undraw_react.svg
│ ├── undraw_tweetstorm.svg
│ └── undraw_youtube_tutorial.svg
└── yarn.lock
Done in 0.57s.
This is actually generated via npx docusaurus-init, but still unable to deploy.
Any help would be highly appreciated :)
Fixed it by doing npm install and yarn run build
My team uses a Puppet architecture which currently accommodates a single application in multiple environments (vagrant, staging, production.)
We now want to expand the scope of this setup to support additional applications. Many of them will use a subset of the existing modules we've already defined, and others will call for new modules to be defined (which may or may not be shared.)
What is the most appropriate Puppet architecture, for supporting multiple environments of multiple applications?
In such an architecture, each application would amount to a module, presumably. What's the best way of (file-) structurally differentiating between a module which is an application, and a module which is a dependency of one or more modules?
Could it be as simple as adding a third modules folder under a top-level applications folder, for example? Or is there a better tiering strategy?
Research so far hasn't turned up any best-practice examples / boilerplates, e.g. via example42 or puppetlabs on GitHub.
Our file structure:
puppet
├── environments
│ ├── production → manifests → init.pp
│ ├── staging → manifests → init.pp
│ └── vagrant → manifests → init.pp
├── hiera.yaml
├── hieradata
│ ├── accounts.yaml
│ ├── common.yaml
│ └── environments
│ ├── production.yaml
│ ├── staging.yaml
│ └── vagrant.yaml
├── modules
│ ├── acl [..]
│ ├── newrelic [..]
│ ├── nginx [..]
│ └── puma [..]
└── vendor
├── Puppetfile
├── Puppetfile.lock
└── modules [..]
I'm sure there are a lot of opinions on what the 'most appropriate' solution for this is, but I'll give you mine.
Puppet is actually designed to support multiple applications in multiple environments right out of the box, with some notable caveats:
All common dependencies (within a single environment) must be pinned to the same version
So if you have three applications that need Apache, you can only have one Apache module
All applications can be referenced using a distinctive name
I.E. If you have three different node.js applications that require their own module, you would need three uniquely named modules (or manifests) for them
You are willing to tackle the upkeep/maintenance of updating dependencies for multiple applications simultaneously
If app1 needs to update an Apache module dependency, you're willing to make sure that apps 2-* remain compatible
The other thing to keep in mind is that Puppet's terminology for 'environment' is an acknowledged misnomer. Most well operated environments I have seen actually have distinct Puppet masters in each of their true 'environments' (vagrant/dev/stage/prod) in order to avoid the perils of environment leakage as well as test out upgrades to the Puppet infrastructure (You should have somewhere to test an upgrade to your Puppet version that doesn't instantly impact your production)
Therefore, this frees up the Puppet 'environment directories' to operate free of the true 'environment' concept, and should be considered 'a collection of modules at a particular revision' instead of an 'environment'. You do still need to be cognizant of environment leakage, but this does open up a potential avenue for splitting up your modules.
Another concept you will want to keep in mind is Roles and Profiles (Well discussed by Gary Larizza, Adrien Thebo, and Craig Dunn). These help enable separating business logic from technology management modules. You can then handle dependency ordering and business oriented logic separate from the code/modules for managing individual components.
With all of these concepts in place, here are two architectural layouts that may be a good fit in your use case:
Environments by application
puppet
├── environments (Managed by r10k/code manager)
│ ├── app1
│ │ └── modules
│ │ ├── profiles [..]
│ │ └── app1_specific_component [..]
│ ├── app2
│ │ └── modules
│ │ ├── profiles [..]
│ │ └── app2_specific_component [..]
│ └── app3
│ └── modules
│ ├── profiles [..]
│ └── app3_specific_component [..]
├── hiera.yaml
├── hieradata
│ ├── accounts.yaml
│ ├── common.yaml
│ └── applications
│ ├── app1
│ │ ├── default.yaml
│ │ └── environments (server environments)
│ │ ├── vagrant
│ │ │ └── roles
│ │ │ ├── role1.yaml
│ │ │ ├── role2.yaml
│ │ │ └── role3.yaml
│ │ ├── stg
│ │ │ └── roles
│ │ │ ├── role1.yaml
│ │ │ ├── role2.yaml
│ │ │ └── role3.yaml
│ │ └── prd
│ │ └── roles
│ │ ├── role1.yaml
│ │ ├── role2.yaml
│ │ └── role3.yaml
│ ├── app2
│ │ ├── default.yaml
│ │ └── environments
│ │ ├── vagrant
│ │ │ └── roles
│ │ │ ├── role1.yaml
│ │ │ ├── role2.yaml
│ │ │ └── role3.yaml
│ │ ├── stg
│ │ │ └── roles
│ │ │ ├── role1.yaml
│ │ │ ├── role2.yaml
│ │ │ └── role3.yaml
│ │ └── prd
│ │ └── roles
│ │ ├── role1.yaml
│ │ ├── role2.yaml
│ │ └── role3.yaml
│ └── app3
│ ├── default.yaml
│ └── environments
│ ├── vagrant
│ │ └── roles
│ │ ├── role1.yaml
│ │ ├── role2.yaml
│ │ └── role3.yaml
│ ├── stg
│ │ └── roles
│ │ ├── role1.yaml
│ │ ├── role2.yaml
│ │ └── role3.yaml
│ └── prd
│ └── roles
│ ├── role1.yaml
│ ├── role2.yaml
│ └── role3.yaml
├── modules (These are common to all environments, to prevent leakage)
│ ├── acl [..]
│ ├── newrelic [..]
│ ├── nginx [..]
│ └── puma [..]
└── vendor
├── Puppetfile
├── Puppetfile.lock
└── modules [..]
Environments as a 'release' (for iteration on Puppet code over time)
puppet
├── environments (Managed by r10k/code manager)
│ ├── release_1
│ │ └── modules
│ │ ├── profiles [..]
│ │ ├── app1_specific_component [..]
│ │ ├── app2_specific_component [..]
│ │ ├── app2_specific_component [..]
│ │ ├── acl [..] (v1)
│ │ ├── newrelic [..]
│ │ ├── nginx [..]
│ │ └── puma [..]
│ ├── release_2
│ │ └── modules
│ │ ├── profiles [..]
│ │ ├── app1_specific_component [..]
│ │ ├── app2_specific_component [..]
│ │ ├── app2_specific_component [..]
│ │ ├── acl [..] (v1.1)
│ │ ├── newrelic [..]
│ │ ├── nginx [..]
│ │ ├── puma [..]
│ │ └── some_new_thing_for_release_2 [..]
│ └── release_3
│ └── modules
│ ├── profiles [..]
│ ├── app1_specific_component [..]
│ ├── app2_specific_component [..]
│ ├── app2_specific_component [..]
│ ├── acl [..] (v2.0)
│ ├── newrelic [..]
│ ├── nginx [..]
│ ├── puma [..]
│ ├── some_new_thing_for_release_2 [..]
│ └── some_new_thing_for_release_3 [..]
├── hiera.yaml
├── hieradata
│ ├── accounts.yaml
│ ├── common.yaml
│ ├── environments
│ │ ├── release_1.yaml
│ │ ├── release_2.yaml
│ │ └── release_3.yaml
│ └── roles
│ ├── role1
│ │ ├── default.yaml
│ │ ├── environments (server environments)
│ │ │ ├── vagrant
│ │ │ │ ├── defaults.yaml
│ │ │ │ └── release (optional, only if absolutely necessary)
│ │ │ │ ├── release_1.yaml
│ │ │ │ ├── release_2.yaml
│ │ │ │ └── release_3.yaml
│ │ │ ├── stg
│ │ │ │ ├── defaults.yaml
│ │ │ │ └── release (optional)
│ │ │ │ ├── release_1.yaml
│ │ │ │ ├── release_2.yaml
│ │ │ │ └── release_3.yaml
│ │ │ └── prd
│ │ │ ├── defaults.yaml
│ │ │ └── release (optional)
│ │ │ ├── release_1.yaml
│ │ │ ├── release_2.yaml
│ │ │ └── release_3.yaml
│ ├── role2
│ │ ├── default.yaml
│ │ └── environments
│ │ ├── vagrant
│ │ │ └── defaults.yaml
│ │ ├── stg
│ │ │ └── defaults.yaml
│ │ └── prd
│ │ └── defaults.yaml
│ └── role3
│ └── default.yaml
├── modules (Anything with ruby libraries should go here to prevent leakage)
│ ├── stdlib [..]
└── vendor
├── Puppetfile
├── Puppetfile.lock
└── modules [..]
Keep in mind that the nesting order (release/environment/role etc...) is flexible based on what makes the most sense for your implementation (and some can be eliminated if you're not going to use them).
I encourage you to take this information as merely a starting point, and not a concrete 'do this for instant success'. Having a highly skilled Puppet Architect work with you to understand your precise needs and environments will end up in a far better tuned and appropriate solution than the assumptions and 'cookie cutter' type solutions you are likely to find online (including mine).
I'm using gulp-pretty-url to keep page URLs clean in a project, such that a file src/about.html will be output as app/about/index.html. Some files end up in deeper structures though, like src/series-460.html to app/series/460/index.html. Of course, each of the HTML files references Javascript and CSS files, and they need to use relative paths.
How can I rebase relative asset paths in files as they're being run through a gulp task? Here's the file structure I'm working with:
project
├── app
│ ├── about
│ │ └──index.html
│ ├── assets
│ │ ├── css
│ │ │ └── styles.css
│ │ └── js
│ │ └── scripts.js
│ └── index.html
└── src
├── assets
│ ├── css
│ │ └── styles.css
│ └── js
│ └── scripts.js
├── about.html
└── index.html