In Terraform is there some way to refer to the set of instances of a module?
I have several workloads that each need very similar (but still separate) infrastructure, and I also want to configure another item of infrastructure to be used in common between them.
For example, say each needs several pieces of infrastructure (AWS S3 bucket, SQS queue, and IAM role..) but with mostly equivalent attributes. I want to achieve this without code duplication (e.g., by writing a Terraform module to be reused for each instance, with input variables for name prefixes and specific IAM policies).
Is there a Terraform syntax for then making a reference to all of those instances in a single resource, and with minimal boilerplate? (Maybe something analogous to a classmethod, to define a common resource to only be created once, even if multiple instances of that module get created?) For example, to make a shared kubernetes config-map that contains an index of the generated addresses (bucket names or SQS URLs), as a mechanism for passing those addresses to the containerised workloads that will use them? Another example might be setting up a single load balancer or DNS server with rules referring individually to every service from this group.
Or does the problem have to be approached in the other direction, by starting with a list of parameter sets, and somehow looping over that list to create the infrastructure? (Requiring every instance of this kind to be specified together in the same file?)
The Terraform terminology for modularity is a child module that is called from multiple places in the configuration. The call uses the module block (where parameter values are passed). Since modules are directories, the child module will be defined somewhere outside of the configuration root module directory tree. The calling module can access output values exported from the child module.
You can use a single module call to create multiple instances, by putting a for_each argument in the module block, and passing a map (or set) through this meta-argument. The other argument expressions in the block can use the each object to refer to the particular for_each element corresponding to the current instance, as Terraform iterates through them. From outside of the module block (elsewhere in the calling module), the output values can themselves be accessed like a map.
You can use [for .. in .. : .. if ..] expressions to filter and transform maps. (Some transformations can also be performed more concisely by splat expressions.) Some resources (such as kubernetes_config_map) have arguments that take maps directly. (Also, some data or resources can accept a sequence of nested blocks, which can be generated from a map by using dynamic block syntax.)
Note, do not use the older feature count as an alternative to for_each. Otherwise, there is a documented tendency to produce unintended destruction and recreation of other infrastructure if one element of the list has to be decommissioned. Similarly, passing a list-derived set, instead of a map, to for_each can make indexing more cumbersome.
Thus for the OP's example, the approach would be to first create a parameterised child module defining the nearly-duplicated parts of the infrastructure (e.g., bucket, queue, role, etc). Make sure the child module has either inputs or outputs for any aspect that needs customisation (for example, output a handle for the created bucket resource, or at least for its auto-generated globally-unique name). Then have a module in your configuration that creates your whole collection of instances (by a single module block that sources the child module and using for_each). The customisation of individual instances (e.g., some having additional policies or other specific infrastructure) can be achieved by a combination of the parameter sets initially passed to the call, and supplementary resource blocks that each refer to outputs from an individual instance. Furthermore, the outputs can also be referred to collectively, for example parsing the module call outputs to form a list of bucket names (or addresses) and then passing this list to another resource (i.e., to a k8s config map). Again this must be done from the calling module; the child module does not have access to a list of instances of itself.
Related
I am not able to figure out what's the difference between data source block and output block in terms of functionality because both are used for getting information about that resource from the console like id, public_ip etc. Can anyone please help me in understanding this because I could'nt find out a suitable resource for this
I have tried to search online for this difference but couldnt find the actual answer.
data essentially represents a dependency on an object that isn't managed by the current Terraform configuration but the current Terraform configuration still needs to make use of it. Mechanically that typically means making a Get or Read request to a specific API endpoint and then exporting the data from that API response in the resulting attributes.
output represents is one of the two ways that data can flow from one module into another. variable blocks represent data moving from the parent module into the child module, and output blocks represent data moving from the child module out to the parent.
There is no strong relationship between these two concepts but one way they sometimes connect is if you use the tfe_outputs data source belonging to the hashicorp/tfe provider, or if you use the terraform_remote_state data source from the terraform.io/builtin/terraform provider. Both of those data sources treat the output values from the root module of some other Terraform configuration as the external object to fetch, and so you can use these as one way to use the results from one configuration as part of another configuration, as long as the second configuration will be run in a context that has access to the state of the first.
NOTE: I'm using the v2 SDK.
In my provider my 'resource' isn't a single API call.
My resource is actually multiple 'things'.
For example...
resource "my_resource" "example" {
foo {
...
}
bar {
...
}
baz {
...
}
}
The resource and each of the nested blocks are all separate 'things' that each have their own API calls.
So when 'creating' this resource I need to actually make multiple API calls. One API call to create the resource itself, then I need to make an API call to create a 'foo', then another API for 'bar', 'baz' etc. Finally, once those nested things are created I need to call my API one last time to activate my main resource.
The problem I've found is that if there's an error in the creation of one of the nested blocks, I'm finding the state is getting messed up and reflecting the 'planned' diff even though I return an error from the API call as part of the Create step.
I'm interested to know how other people are handling errors in a provider that has a structure like this?
I've tried using Partial(). I've also tried to trigger another Read of each 'thing' but although the final state data looks to be correct (when printing it as part of a debug run with trace logs), once I've done a read, because my 'Create' function has to return an error, the state data that's read is dropped and the original planned diff is persisted (I've even stopped returning an error altogether and tried to return just the result of the Read, which is successful, and STILL the state reflects the planned diff rather than the modified state after a Read).
Since you mentioned Partial I'm assuming for this answer that you are using the older SDKv2 rather than the modern Terraform provider framework.
The programming model for SDKv2 is for the action functions like Create to receive a mutable value representing the planned values, encapsulated in a schema.ResourceData object, and then the provider will modify that value through that wrapping object to make it describe the object that was actually created (or updated).
Terraform Core itself expects a provider to respond to the "apply" request by returning the closest possible representation of what was actually created in the remote system. If the value is returned without an error then Terraform will require that the object conforms to the plan and will raise an error saying that there's a bug in the provider if not. If the provider returns a value and an error then Terraform Core will propagate that error to the user and save whatever value was returned, as long as it matches the schema of the resource type.
Unfortunately this mismatch in models between Terraform Core and the SDK makes the situation you've described quite tricky: if you don't call d.Set at all in your Create function then by default the SDK will just return whatever values were in the plan, even if parts of it weren't actually created yet. To make your provider behave in the way that Terraform is expecting you'd need to do something like this:
At the beginning of Create, decode all of the nested block data into some local variables of data types that are useful for making the API calls you intend to make. For example, you might at this step decode the data from the ResourceData object into whatever struct types the underlying SDK expects.
Before you take any other actions, use d.Set to remove all of the blocks of the types that will require separate requests each. This means you'll need to pass an empty value of whatever type is appropriate for the type you chose for that block's value.
In your loop where you're gradually creating the separate objects that each block represents, gradually append the results into a growing set of objects representing the blocks you've already successfully created. Each time you add a new item to that set, call d.Set again to reset the attribute representing the appropriate block type to now include the object that you created.
If you get to the end without any errors then your attributes should now again describe all of the objects requested in the configuration and you can return without an error. If you encounter an error partway through then you can return that error and the SDK will automatically also return the partially-updated value encapsulated inside the ResourceData object.
If you return an accurate description of which of the blocks were created and exclude the ones that weren't then on the next plan the SDK logic should notice that some of the blocks declared in the configuration aren't present in the prior state and so it should propose to update the object to include those additional blocks. Your provider's Update function can then follow a similar principle as above to gradually append only the nested objects it successfully created, so that it'll once again return a complete set if successful or a partial set in case of any errors.
SDKv2 was optimized for the more common case where a single resource block represents a single remote API call, and so its default behavior deals with either fully-successful or fully-failed responses. Dealing with partial failure requires more subtlety that is difficult to represent in that SDK's API.
The newer Terraform Plugin Framework has a different design for these operations which separates the request data from the response data, thereby making it less confusing to return only a partial result. The Resource interface has a Create method which has a request object containing the config and the plan and a response object containing a representation of the final state.
It pre-populates the response state with the planned values similarly to SDKv2 to still handle that common case of entirely-failing vs. entirely-succeeding, but it does also allow totally overwriting that default with a locally-constructed object representing a partial result, to better support situations like yours where one resource in Terraform is representing a number of different fallible calls to the underlying API.
I'm having difficulty finding the documentation I need to work with Resources and resourcePools - there is no "resource API documentation" that I can find.
I would like to programatically create static resources (of a custom type) and then add these resources to a resourcePool. When creating the resources, I'd like to be able to specify their property values prior to adding them to the resourcePool. In my mind the code would look something like this:
Room myRoom;
myRoom = new Room("redRoom", 20);
addTo_myResourcePool(myRoom);
myRoom = new Room("greenRoom", 10);
addTo_myResourcePool(myRoom);
Does anyone know if there are ways to achieve this end?
This is a bit of a blind spot in AnyLogic. It can only be done indirectly:
Create an empty agent population with your agent type
Tell the resource pool to use that agent type
set the resource pool capacity as you need. The pool will create agents for your in that population (if the capacity is larger than the current number of resources)
If you want to manually create a resource, you must call myResourcePool.set_Capacity(myResourcePool.getCapacity()+1)
Destroying 1 resource works vice-versa.
Also, make sure to "destroy resources on capacity decrease" so the agents are destroyed from the population
There's a trick I use for this, that works sometimes, but I wouldn't generalize it as a final solution...I only do it when i have a few different unit characteristics, which seems to be your case.
step 1: create a population of resourcePools... each resource pool will correspond to one kind of agent characteristics, and all resourcePools use the same resource (agent) type
step 2: in the on new unit of the resource pool, you will use the index of the resourcePool population to generate the unit with particular characteristics... then you can just do something like resourcePool.get(i).set_capacity(whatever) in order to generate a resource unit with the exact characteristics you want
step 3: when you seize the resource, you will use alternatives... each resourcePool from the resourcePool population will be 1 option among the alternatives to use... you will need to create a function that retourns a ResourcePool[][]
step 4: you will use conditions to select the unit based on its characteristics (customize resource selection)
One option is to create a population of your resource agent, which I assume is of type room based on your code.
Then you have a function that will add a new agent to the population and return it to the caller.
And now you only need to add this to the new resource unit call in the resource pool object
Do not change the "Add Units to" option since we are already doing this in the function.
I tested this in a small model by having two buttons to increase and decrease the capacity during execution using
resourcePool.set_capacity(max(0, resourcePool.size() + 1));
and
remove_myResource(myResource.get(myResource.size()-1));
resourcePool.set_capacity(max(0, myResource.size()));
There are several related points here.
AnyLogic is designed such that you only change the agents in a resource pool via changing the capacity of the pool. This can be done directly via the pool's set_capacity function (if the pool's capacity is set to be defined "Directly"), but is more commonly done indirectly by linking the pool's capacity to a schedule (to represent shift patterns or any other pre-defined changes over time in capacity).
Resource pools will keep agents by default when the capacity decreases (i.e., they exist but are 'unavailable'), but you can set it to delete them. What you'll want will depend on what the resources and any resource-specific attributes/state represent.
If you need to address/access the agents in the resource pool independently from their use as seized/released resources you would have the pool store them in a custom population (rather than put them in the default population along with all other agents not created in a custom population) via the "Add units to" setting. Then you can access that population explicitly (e.g., loop through it) whenever you need to. Otherwise this aspect is not necessary.
If your resource pool is of custom agents with parameters (or other state that needs initialising in a particular way), the standard approach is to have direct/indirect capacity increases create the unit (with default parameter values) and then do any follow-on initialisation in the "On new unit" action (where you have access to the newly-created unit via the agent keyword).
You can alternatively do it via a dynamic expression for the "New resource unit" (as in Jaco's answer) but this doesn't particularly gain you anything (though it's equally valid). Yes, it's a bit more 'object-oriented', but there are loads of barriers AnyLogic puts up to doing things in a proper OO way anyway — as you say, the more 'obvious' OO approach would just be to have methods on the resource pool which add/remove agents. Basically, the standard 'design pattern' on blocks which create agents, like a Resource Pool or a Source block, is to let it create them with default info and then customise it in the appropriate action.
You could also use the "On startup" action of the resource pool agent as well / instead, but normally you would be using information from the process-containing agent to determine what state the new resource agent should have, which makes the "On startup" action less useful.
Finally(!), if you have an unchanging set of resource pool agents and it's easier to initialise them together (e.g., because there is some database input data for all of them that can thus just be looped through once), then just do that in your "On startup" action of the agent containing the resource pool (which runs after the resource pool has initialised and created default-state agents). That will require them in a custom population so you can loop through just those agents. But your question seemed to imply you were concerned about dynamic additions of resource agents.
But, as some of my other comments indicated, there are various subtleties/restrictions in how resource pools work (really based on them currently being designed as a pool where the explicit identity of each individual is not fully 'controllable') which mean that what you actually need may go beyond this.
A couple of examples:
If you wanted your resources to be explicit individual staff with info tracked across shifts (e.g., with their state changing to reflect experience or the history of tasks they've conducted), you have the complication that you can't control which resources (if you didn't delete them on capacity decrease) 'come back' when the capacity changes, so you may need more complex workarounds such as having separate resource pools for each distinct shift and seizing from resource sets including all such pools — only one of them (the active shift one) will have non-zero capacity at any given time unless the shifts overlap.
If capacity decreases (representing end-of-shift), you can't control / determine which resource agents are chosen to go off-shift. Plus there are issues regarding resource agents which are still active finishing a task before they actually 'leave' (given that shift-ends weren't set to pre-empt) — because they still exist until they finish, if you increase the capacity again whilst they're working AnyLogic won't create a new resource agent (it will just add this existing agent back as an 'active' agent in the pool). So that makes some other things harder...
Question: Is it thread-safe to use static variables (as a shared storage between orchestrations) or better to save/retrieve data to durable-entity?
There are couple of azure functions in the same namespace: hub-trigger, durable-entity, 2 orchestrations (main process and the one that monitors the whole process) and activity.
They all need some shared variables. In my case I need to know the number of main orchestration instances (start new or hold on). It's done in another orchestration (monitor)
I've tried both options and ask because I see different results.
Static variables: in my case there is a generic List, where SomeMyType holds the Id of the task, state, number of attempts, records it processed and other info.
When I need to start new orchestration and List.Add(), when I need to retrieve and modify it I use simple List.First(id_of_the_task). First() - I know for sure needed task is there.
With static variables I sometimes see that tasks become duplicated for some reason - I retrieve the task with List.First(id_of_the_task) - change something on result variable and that is it. Not a lot of code.
Durable-entity: the major difference is that I add List on a durable entity and each time I need to retrieve it I call for .CallEntityAsync("getTask") and .CallEntityAsync("saveTask") that might slow done the app.
With this approach more code and calls is required however it looks more stable, I don't see any duplicates.
Please, advice
Can't answer why you would see duplicates with the static variables approach without the code, may be because list is not thread safe and it may need ConcurrentBag but not sure. One issue with static variable is if the function app is not always on or if it can have multiple instances. Because when function unloads (or crashes) the state would be lost. Static variables are not shared across instances either so during high loads it wont work (if there can be many instances).
Durable entities seem better here. Yes they can be shared across many concurrent function instances and each entity can only execute one operation at a time so they are for sure a better option. The performance cost is a bit higher but they should not be slower than orchestrators since they perform a lot of common operations, writing to Table Storage, checking for events etc.
Can't say if its right for you but instead of List.First(id_of_the_task) you should just be able to access the orchestrators properties through the client which can hold custom data. Another idea depending on the usage is that you may be able to query the Table Storages directly with CloudTable class for the information about the running orchestrators.
Although not entirely related you can look at some settings for parallelism for durable functions Azure (Durable) Functions - Managing parallelism
Please ask any questions if I should clarify anything or if I misunderstood your question.
could you help me to understand
For example I have incremental changes to infrastructure, like [A] -> [B] -> [C], where [A] separately can be one server named i, [B] separately can be second server named j, and [C] separately can be third server named k. In total there should be 3 servers. Every state can be described as [A] = x, x + [B] = y, y + [C] = z where x, y, z are states in the middle.
My question are
How to organize incremental infrastructure changes for multiple modules in Terraform without affecting previous modules?
Is it possible to rollback changes in the middle of the chain eg. [B] and get x-state or we should follow chain from the last module [C] to required in the middle [B]?
At this time Terraform only considers two states[1]: the "current state" (the result of the previous Terraform run along with any "drift" in the mean time) and the "desired state" (described in configuration). Terraform's task is to identify the differences between these two states and determine which API actions are needed to move resources from their current state to the desired state.
This means that any multi-step transition cannot be orchestrated within Terraform alone. In your example, to add server j you would add it alone to the Terraform configuration, and run Terraform to create that server. You can then add server k to the configuration and run Terraform again. To automate this, an external process would need to generate these progressive configuration changes.
An alternative approach -- though not recommended for everyday use, since it can cause confusion in a collaborative environment where others can't easily see how this state was reached -- is to use the -target argument to specify one or more specific resources to operate on. In principle this allows adding both servers j and k to configuration but then using -target with only the resource representing j.
There is no formal support for rollback in Terraform. Terraform merely sees this as another state transition, "rolling forward". For example, if after creating server k you wish to revert to state [A], you would remove the configuration for server k (by reverting in version control, perhaps) and apply again. Terraform doesn't have any awareness of the fact that this is a "rollback", but it can see that the configuration no longer contains server k and thus know that it needs to be destroyed to reach the desired state.
One of your questions is about "affecting previous modules". In general, if no changes are made to a resource in your configuration (either the config changed, or the resource itself changed outside of Terraform's purview) then Terraform should not try to update it. If it did, that would be considered a bug. However, for larger systems it can be useful to split infrastructure across multiple top-level Terraform configurations that are each applied separately. If a remote backend is in use (recommended for collaborative environments) then the terraform_remote_state data source can be used to access the output values of one configuration from within another, thus allowing the creation of a tree or DAG of Terraform configurations. This adds complexity, so should be weighed carefully, but has the advantage of decreasing the "blast radius" of a particular change by taking unrelated resources out of consideration altogether.
[1] I am using "state" here in the general sense you used it, which is distinct from Terraform's physical idea of "state", a data file that contains a record of which resources existed at the last run.