I was given a link to an OpenAPI 3.0.1 definition hosted on SwaggerHub, and was told to deploy it. On the Terraform side, I see way too many resources that confuse me, I'm not sure which one to use. What's the most straightforward way to deploy an API gateway via Terraform that's already all configured in an OpenAPI definition? Is there a resource that would simply let me provide an OpenAPI definition URL to the API gateway, or would I have to copy paste the actual JSON somewhere?
The AWS API Gateway service has two main usage patterns:
Directly specify individual resources, methods, requests, integrations, and responses as individual objects in the API Gateway API.
Submit an OpenAPI definition of the entire API as a single unit and have API Gateway itself split that out into all of the separate objects in API Gateway's data model.
Since the underlying API supports both models, it can be hard to see initially which parts are relevant to each usage pattern. The Terraform provider for AWS follows the underlying API design, and so that confusion appears there too.
It sounds like you are intending to take the second path I described above, in which case the definition in Terraform is comparatively straightforward, and in particular it typically involves only a single Terraform resource to define the API itself. (You may need to use others to "deploy" the API, etc, but that seems outside of the scope of your current question.)
The api_gateway_rest_api resource type is the root resource type for defining an API Gateway REST API, and for the OpenAPI approach is the only one required to define your entire API surface, by specifying the OpenAPI definition in its body argument:
resource "aws_api_gateway_rest_api" "example" {
name = "example"
body = file("${path.module}/openapi.json")
}
In the above example I've assumed that you've saved the API definition in JSON format in an openapi.json file in the same directory as the .tf file which would contain the resource configuration. I'm not familiar with SwaggerHub, but if there is a Terraform provider available for it which has a data source for retrieving the definition directly from that system then you could potentially combine those, but the principle would be the same; it would only be the exact expression for the body argument that would change.
The other approach with the resources/etc defined explicitly via the API Gateway API would have a separate resource for each of API Gateway's separate object types describing an API, which makes for a much more complicated Terraform configuration. However, none of those need be used (and indeed, none should be used, to avoid conflicts) when you have defined your API using an OpenAPI specification.
NOTE: The above is about API Gateway REST APIs, which is a separate offering from "API Gateway v2", which offers so-called "HTTP APIs" and "WebSocket APIs". As far as I know, API Gateway v2 doesn't support OpenAPI definitions and therefore I've assumed you're asking about the original API Gateway, and thus "REST APIs".
Related
I want to serve the response from a different http endpoint. As per the API-gateway documentation we can achieve this by creating a new resource in api-gateway, for which the Integration type should be HTTP, and we can also define "Endpoint URL" which will take the request for further processing.
Refer this image to understand how it works.
I have added this record manually inside api-gateway. Is there any way to define it in the serverless.yml file, so that on every deployment, this proxy resource will also get created.
I am using below link to create/assign Standard Azure policies for my Management Groups. The problem is my org. has already created Management Groups and Subscriptions manually using Azure Portal. Can i still create / apply policies using ESLZ TF code and apply to these manually created Management groups using TF code:
https://github.com/Azure/terraform-azurerm-caf-enterprise-scale
When i see the code archetype (policy) is very tightly coupled to MG creation ?
locals.management_groups.tf:
"${local.root_id}-landing-zones" = {
archetype_id = "es_landing_zones"
parameters = local.empty_map
access_control = local.empty_map
}
archetype_id is the policy.
The enterprise scale example follows what they call a "supermodule approach".
The example from locals.management_groups.tf that you posted there is just the configuration layer of the module. If you want CAF to adopt an existing management group instead of creating a new one, you have to terraform import it. That requires you to know the resource address, which you can derive from the the actual resource definition.
So you're looking for resource "azurerm_management_group" and that's in resources.management_groups.tf. That file declares multiple management group resources level_1 to level_6, and they're using for_each to generate the resources from the module's configuration model.
So after you've somehow traced your way through the modules' configuration model and determined where you want your existing management group to end up in the CAFs model, you can run an import as described in terraform's docs and the azurerm provider's docs like this:
terraform import azurerm_management_group.level_3["my-root-demo-corp"] providers/Microsoft.Management/managementGroups/group1
However, even though all of that is technically possible, I'm not sure this is a good approach. The enterprise scale supermodule configuration model is super idiosyncratic to trace through - it's stated goal is to replace "infrastructure as code" with "infrastructure as data". You may find one of the approaches described in Azure Landing Zone for existing Azure infrastructure more practical for your purposes.
You could import the deployed environment into the TF state and proceed from there.
However, do you see an absolute necessity ? IMO, setting up the LZ is a one-time activity. However, one can argue that ensuring compliance across active & DR LZs is a good use-case for a TF based LZ deployment.
I have been using Terraform now for some months, and I have reached the point where my infrastructure is all base in Terraform files and I now have better control of the resources in our multiple accounts.
But I have a big problem. If someone makes a "manual" alteration of any Terraformed resource, it is easy to detect the change.
But what happens if the resource was not created using Terraform? I just don't know how to track any new resource or changes in them if the resource was not created using Terraform.
A key design tradeoff for Terraform is that it will only attempt to manage objects that it created or that you explicitly imported into it, because Terraform is often used in mixed environments where either some objects are managed by other software (like an application deployment tool) or the Terraform descriptions are decomposed into multiple separate configurations designed to work together.
For this reason, Terraform itself cannot help with the problem of objects created outside of Terraform. You will need to solve this using other techniques, such as access policies that prevent creating objects directly, or separate software (possibly created in-house) that periodically scans your cloud vendor accounts for objects that are not present in the expected Terraform state snapshot(s).
Access policies are typically the more straightforward path to implement, because preventing objects from being created in the first place is easier than recognizing objects that already exist, particularly if you are working with cloud services that create downstream objects as a side-effect of their work, as we see with (for example) autoscaling controllers.
Martin's answer is excellent and explains that Terraform can't be the arbiter of this as it is designed to play nicely both with other tooling and with itself (ie across different state files).
He also mentioned that access policies (although these have to be cloud/provider specific) are a good alternative to this so this answer will instead provide some options here for handling this with AWS if you do want to enforce this.
The AWS SDKs and other clients, including Terraform, all provide a user agent header in all requests. This is recorded by CloudTrail and thus you can search through CloudTrail logs with your favourite log searching tools to look for API actions that should be done via Terraform but don't use Terraform's user agent.
The other option that uses the user agent request header is to use IAM's aws:UserAgent global condition key which will block any requests that don't match the user agent header that's defined. An example IAM policy may look like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1598919227338",
"Action": [
"dlm:GetLifecyclePolicies",
"dlm:GetLifecyclePolicy",
"dlm:ListTagsForResource"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "Stmt1598919387700",
"Action": [
"dlm:CreateLifecyclePolicy",
"dlm:DeleteLifecyclePolicy",
"dlm:TagResource",
"dlm:UntagResource",
"dlm:UpdateLifecyclePolicy"
],
"Effect": "Allow",
"Resource": "*",
"Condition": {
"StringLike": {
"aws:UserAgent": "*terraform*"
}
}
}
]
}
The above policy allows the user, group or role it is attached to to be able to perform read only tasks to any DLM resource in the AWS account. It then allows any client with a user agent header including the string terraform to perform actions that can create, update or delete DLM resources. If a client doesn't have terraform in the user agent header then any requests to modify a DLM resource will be denied.
Caution: It's worth noting that clients can override the user agent string and so this shouldn't be relied on as a foolproof way of preventing access to things outside of this. The above mentioned techniques are mostly useful to get an idea about the usage of other tools (eg the AWS Console) in your account where you would prefer changes to be made by Terraform only.
The AWS documentation to the IAM global condition keys has this to say:
Warning
This key should be used carefully. Since the aws:UserAgent
value is provided by the caller in an HTTP header, unauthorized
parties can use modified or custom browsers to provide any
aws:UserAgent value that they choose. As a result, aws:UserAgent
should not be used to prevent unauthorized parties from making direct
AWS requests. You can use it to allow only specific client
applications, and only after testing your policy.
The Python SDK, boto, covers how the user agent string can be modified in the configuration documentation.
I haven't executed it but my idea has always been that this should be possible with a consistent usage of tags. A first naive
provider "aws" {
default_tags {
tags = {
Terraform = "true"
}
}
}
should be sufficient in many cases.
If you fear rogue developers will add this tag manually so as to hide their hacks, you could convolute your terraform modules to rotate the tag value over time to unpredictable values, so you could still search for inappropriately tagged resources. Hopefully the burden for them to overcome such mechanism will defeat the effort of simply terraforming a project. (Not for you)
On the downside, many resources will legitimately be not terraformed, e.g. DynamoDB tables or S3 items. A watching process should somehow whitelist what is allowed to exist. Not computational resources, that's for sure.
Tuning access policies and usage of CloudTrail as #ydaetskcoR suggests might be unsuitable to assess the extent of unterraformed legacy infrastructure, but are definitely worth the effort anyway.
This Reddit thread https://old.reddit.com/r/devops/comments/9rev5f/how_do_i_diff_whats_in_terraform_vs_whats_in_aws/ discusses this very topic, with some attention gathered around the sadly archived https://github.com/dtan4/terraforming , although it feels too much IMHO.
I have already installed my azure environment with a VM, a storage account and a data base server, and it works fine, but now i want to export the ARM template in order to automate the whole proccess to my customers. The problem is that when exporting this message shows up:
error
So the question is how do i export an ARM template correctly from Azure with all my resources without having to do much fixing my final template?
that is expected. some resource types cannot be exported. you'd have to take a look at the api definition and use that to export those (say at resources.azure.com)
Simple: use bicep. Azure Resource Manager's transpiler (it's a sweeter syntax that solves your needs)
In Bicep the directive you're looking for in your Azure Resource Manager "ARM" template is keyword "existing"
https://github.com/Azure/bicep/blob/main/docs/tutorial/05-loops-conditions-existing.md
keyword "existing" lets you reference a resource without a complete definition.
Otherwise you need to provide the entire ARM definition for the object.
Export failures per resource type occur when a given resource types schema is not available. We are looking into how we can autogenerate schemas for Azure resource providers and onboarding them to this new process, improving the overall success of the Export Template API.
I'm trying to build a small program to change the autoscale settings for our Azure WebApps, using the Microsoft.WindowsAzure.Management.Monitoring and Microsoft.WindowsAzure.Management.WebSites NuGet packages.
I have been roughly following the guide here.
However, we are interested in scaling WebApps / App Services rather than Cloud Services, so I am trying to use the same code to read the autoscale settings but providing a resource ID for our WebApp. I have already got the credentials required for making a connection (using a browser window popup for Active Directory authentication, but I understand we can use X.509 management certificates for non-interactive programs).
This is the request I'm trying to make. Credentials already established, and an exception is thrown earlier if they're not valid.
AutoscaleClient autoscaleClient = new AutoscaleClient(credentials);
var resourceId = AutoscaleResourceIdBuilder.BuildWebSiteResourceId(webspaceName: WebSpaceNames.NorthEuropeWebSpace, serverFarmName: "Default2");
AutoscaleSettingGetResponse get = autoscaleClient.Settings.Get(resourceId); // exception here
The WebApp (let's call it "MyWebApp") is part of an App Service Plan called "Default2" (Standard: 1 small), in a Resource Group called "WebDevResources", in the North Europe region. I expect that my problem is that I am using the wrong names to build the resourceId in the code - the naming conventions in the library don't map well onto what I can see in the Azure Portal.
I'm assuming that BuildWebSiteResourceId is the correct method to call, see MSDN documentation here.
However the two parameters it takes are webspaceName and serverFarmName, neither of which match anything in the Azure portal (or Google). I found another example which seemed to be using the WebApp's geo region for webSpaceName, so I've used the predefined value for North Europe where our app is hosted.
While trying to find the correct value for serverFarmName in the Azure Portal, I found the Resource ID for the App Service Plan, which looks like this:
/subscriptions/{subscription-guid}/resourceGroups/WebDevResources/providers/Microsoft.Web/serverfarms/Default2
That resource ID isn't valid for the call I'm trying to make, but it does support the idea that a 'serverfarm' is the same as an App Service Plan.
When I run the code, regardless of whether the resourceId parameters seem to be correct or garbage, I get this error response:
<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">
{"Code":"SettingNotFound","Message":"Could not find the autoscale settings."}
</string>
So, how can I construct the correct resource ID for my WebApp or App Service Plan? Or alternatively, is there a different tree I should be barking up to programatially manage WebApp scaling?
Update:
The solution below got the info I wanted. I also found the Azure resource explorer at resources.azure.com extremely useful to browse existing resources and find the correct names. For example, the name for my autoscale settings is actually "Default2-WebDevResources", i.e. "{AppServicePlan}-{ResourceGroup}" which I wouldn't have expected.
There is a preview service https://resources.azure.com/ where you can inspect all your resources easily. If you search for autoscale in the UI you will easily find the settings for your resource. It will also show you how to call the relevant REST Api endpoint to read or update that resorce.
It's a great tool for revealing a lot of details for your deployed resources and it will actually give you an ARM template stub for the resource you are looking at.
And to answer your question, you could programmatically call the REST API from a client with updated settings for autoscale. The REST API is one way of doing this, the SDK another and PowerShell a third.
The guide which you're following is based on the Azure Service Management model, aka Classic mode, which is deprecated and only exists mainly for backward compatibility support.
You should use the latest
Microsoft.Azure.Insights nuget package for getting the autoscale settings.
Sample code using the nuget above is as below:
using Microsoft.Azure.Management.Insights;
using Microsoft.Rest;
//... Get necessary values for the required parameters
var client = new InsightsManagementClient(new TokenCredentials(token));
client.AutoscaleSettings.Get(resourceGroupName, autoScaleSettingName);
Besides, the autoscalesettings is a resource under the "Microsoft.Insights" provider and not under the "Microsoft.Web" provider, which explains why you are not able to find it with your serverfarm resourceId.
See the REST API Reference below for getting the autoscale settings.
GET
https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/microsoft.insights/autoscaleSettings/{autoscale-setting-name}?api-version={api-version}