Puppet architecture for multiple environments of multiple applications - puppet

Question

My team uses a Puppet architecture which currently accommodates a single application in multiple environments (vagrant, staging, production.)
We now want to expand the scope of this setup to support additional applications. Many of them will use a subset of the existing modules we've already defined, and others will call for new modules to be defined (which may or may not be shared.)
What is the most appropriate Puppet architecture, for supporting multiple environments of multiple applications?
In such an architecture, each application would amount to a module, presumably. What's the best way of (file-) structurally differentiating between a module which is an application, and a module which is a dependency of one or more modules?
Could it be as simple as adding a third modules folder under a top-level applications folder, for example? Or is there a better tiering strategy?
Research so far hasn't turned up any best-practice examples / boilerplates, e.g. via example42 or puppetlabs on GitHub.
Our file structure:
puppet
├── environments
│   ├── production → manifests → init.pp
│   ├── staging → manifests → init.pp
│   └── vagrant → manifests → init.pp
├── hiera.yaml
├── hieradata
│   ├── accounts.yaml
│   ├── common.yaml
│   └── environments
│   ├── production.yaml
│   ├── staging.yaml
│   └── vagrant.yaml
├── modules
│   ├── acl [..]
│   ├── newrelic [..]
│   ├── nginx [..]
│   └── puma [..]
└── vendor
├── Puppetfile
├── Puppetfile.lock
└── modules [..]

Answer 1

I'm sure there are a lot of opinions on what the 'most appropriate' solution for this is, but I'll give you mine.
Puppet is actually designed to support multiple applications in multiple environments right out of the box, with some notable caveats:
All common dependencies (within a single environment) must be pinned to the same version
So if you have three applications that need Apache, you can only have one Apache module
All applications can be referenced using a distinctive name
I.E. If you have three different node.js applications that require their own module, you would need three uniquely named modules (or manifests) for them
You are willing to tackle the upkeep/maintenance of updating dependencies for multiple applications simultaneously
If app1 needs to update an Apache module dependency, you're willing to make sure that apps 2-* remain compatible
The other thing to keep in mind is that Puppet's terminology for 'environment' is an acknowledged misnomer. Most well operated environments I have seen actually have distinct Puppet masters in each of their true 'environments' (vagrant/dev/stage/prod) in order to avoid the perils of environment leakage as well as test out upgrades to the Puppet infrastructure (You should have somewhere to test an upgrade to your Puppet version that doesn't instantly impact your production)
Therefore, this frees up the Puppet 'environment directories' to operate free of the true 'environment' concept, and should be considered 'a collection of modules at a particular revision' instead of an 'environment'. You do still need to be cognizant of environment leakage, but this does open up a potential avenue for splitting up your modules.
Another concept you will want to keep in mind is Roles and Profiles (Well discussed by Gary Larizza, Adrien Thebo, and Craig Dunn). These help enable separating business logic from technology management modules. You can then handle dependency ordering and business oriented logic separate from the code/modules for managing individual components.
With all of these concepts in place, here are two architectural layouts that may be a good fit in your use case:
Environments by application
puppet
├── environments (Managed by r10k/code manager)
│ ├── app1
│ │ └── modules
│ │ ├── profiles [..]
│ │ └── app1_specific_component [..]
│ ├── app2
│ │ └── modules
│ │ ├── profiles [..]
│ │ └── app2_specific_component [..]
│ └── app3
│ └── modules
│ ├── profiles [..]
│ └── app3_specific_component [..]
├── hiera.yaml
├── hieradata
│ ├── accounts.yaml
│ ├── common.yaml
│ └── applications
│ ├── app1
│ │ ├── default.yaml
│ │ └── environments (server environments)
│ │ ├── vagrant
│ │ │ └── roles
│ │ │ ├── role1.yaml
│ │ │ ├── role2.yaml
│ │ │ └── role3.yaml
│ │ ├── stg
│ │ │ └── roles
│ │ │ ├── role1.yaml
│ │ │ ├── role2.yaml
│ │ │ └── role3.yaml
│ │ └── prd
│ │ └── roles
│ │ ├── role1.yaml
│ │ ├── role2.yaml
│ │ └── role3.yaml
│ ├── app2
│ │ ├── default.yaml
│ │ └── environments
│ │ ├── vagrant
│ │ │ └── roles
│ │ │ ├── role1.yaml
│ │ │ ├── role2.yaml
│ │ │ └── role3.yaml
│ │ ├── stg
│ │ │ └── roles
│ │ │ ├── role1.yaml
│ │ │ ├── role2.yaml
│ │ │ └── role3.yaml
│ │ └── prd
│ │ └── roles
│ │ ├── role1.yaml
│ │ ├── role2.yaml
│ │ └── role3.yaml
│ └── app3
│ ├── default.yaml
│ └── environments
│ ├── vagrant
│ │ └── roles
│ │ ├── role1.yaml
│ │ ├── role2.yaml
│ │ └── role3.yaml
│ ├── stg
│ │ └── roles
│ │ ├── role1.yaml
│ │ ├── role2.yaml
│ │ └── role3.yaml
│ └── prd
│ └── roles
│ ├── role1.yaml
│ ├── role2.yaml
│ └── role3.yaml
├── modules (These are common to all environments, to prevent leakage)
│ ├── acl [..]
│ ├── newrelic [..]
│ ├── nginx [..]
│ └── puma [..]
└── vendor
├── Puppetfile
├── Puppetfile.lock
└── modules [..]
Environments as a 'release' (for iteration on Puppet code over time)
puppet
├── environments (Managed by r10k/code manager)
│ ├── release_1
│ │ └── modules
│ │ ├── profiles [..]
│ │ ├── app1_specific_component [..]
│ │ ├── app2_specific_component [..]
│ │ ├── app2_specific_component [..]
│ │ ├── acl [..] (v1)
│ │ ├── newrelic [..]
│ │ ├── nginx [..]
│ │ └── puma [..]
│ ├── release_2
│ │ └── modules
│ │ ├── profiles [..]
│ │ ├── app1_specific_component [..]
│ │ ├── app2_specific_component [..]
│ │ ├── app2_specific_component [..]
│ │ ├── acl [..] (v1.1)
│ │ ├── newrelic [..]
│ │ ├── nginx [..]
│ │ ├── puma [..]
│ │ └── some_new_thing_for_release_2 [..]
│ └── release_3
│ └── modules
│ ├── profiles [..]
│ ├── app1_specific_component [..]
│ ├── app2_specific_component [..]
│ ├── app2_specific_component [..]
│ ├── acl [..] (v2.0)
│ ├── newrelic [..]
│ ├── nginx [..]
│ ├── puma [..]
│ ├── some_new_thing_for_release_2 [..]
│ └── some_new_thing_for_release_3 [..]
├── hiera.yaml
├── hieradata
│ ├── accounts.yaml
│ ├── common.yaml
│ ├── environments
│ │ ├── release_1.yaml
│ │ ├── release_2.yaml
│ │ └── release_3.yaml
│ └── roles
│ ├── role1
│ │ ├── default.yaml
│ │ ├── environments (server environments)
│ │ │ ├── vagrant
│ │ │ │ ├── defaults.yaml
│ │ │ │ └── release (optional, only if absolutely necessary)
│ │ │ │ ├── release_1.yaml
│ │ │ │ ├── release_2.yaml
│ │ │ │ └── release_3.yaml
│ │ │ ├── stg
│ │ │ │ ├── defaults.yaml
│ │ │ │ └── release (optional)
│ │ │ │ ├── release_1.yaml
│ │ │ │ ├── release_2.yaml
│ │ │ │ └── release_3.yaml
│ │ │ └── prd
│ │ │ ├── defaults.yaml
│ │ │ └── release (optional)
│ │ │ ├── release_1.yaml
│ │ │ ├── release_2.yaml
│ │ │ └── release_3.yaml
│ ├── role2
│ │ ├── default.yaml
│ │ └── environments
│ │ ├── vagrant
│ │ │ └── defaults.yaml
│ │ ├── stg
│ │ │ └── defaults.yaml
│ │ └── prd
│ │ └── defaults.yaml
│ └── role3
│ └── default.yaml
├── modules (Anything with ruby libraries should go here to prevent leakage)
│ ├── stdlib [..]
└── vendor
├── Puppetfile
├── Puppetfile.lock
└── modules [..]
Keep in mind that the nesting order (release/environment/role etc...) is flexible based on what makes the most sense for your implementation (and some can be eliminated if you're not going to use them).
I encourage you to take this information as merely a starting point, and not a concrete 'do this for instant success'. Having a highly skilled Puppet Architect work with you to understand your precise needs and environments will end up in a far better tuned and appropriate solution than the assumptions and 'cookie cutter' type solutions you are likely to find online (including mine).