In Cloudformation YAML, use a Ref in a multiline string (? use Fn:Sub) - string

Imagine you have a aws resource such as
Resources:
IdentityPool:
Type: "AWS::Cognito::IdentityPool"
Properties:
IdentityPoolName: ${self:custom.appName}_${self:provider.stage}_identity
CognitoIdentityProviders:
- ClientId:
Ref: UserPoolClient
The Ref for "AWS::Cognito::IdentityPool" returns the id of this resource. Now lets say I want to reference that id in a multiline string. I've tried
Outputs:
AmplifyConfig:
Description: key/values to be passed to Amplify.configure(config);
Value: |
{
'aws_cognito_identity_pool_id': ${Ref: IdentityPool}, ##<------ Error
'aws_sign_in_enabled': 'enable',
'aws_user_pools_mfa_type': 'OFF',
}
I've also tried to use Fn:Sub but without luck.
AmplifyConfig:
Description: key/values to be passed to Amplify.configure(config);
Value:
Fn::Sub
- |
{
'aws_cognito_identity_pool_id': '${Var1Name}',
'aws_sign_in_enabled': 'enable',
}
- Var1Name:
Ref: IdentityPool
Any way to do this?

Using a pipe symbol | in YAML turns all of the following indented lines into a multi-line string.
A pipe, combined with !Sub will let you use:
your resources Ref return value easily like ${YourResource}
their Fn::GetAtt return values with just a period ${YourResource.TheAttribute}
any Pseudo Parameter just as is like ${AWS:region}
As easy as !Sub |, jumping to the next line and adding proper indentation. Example:
Resources:
YourUserPool:
Type: AWS::Cognito::UserPool
Properties:
UserPoolName: blabla
Outputs:
AmplifyConfig:
Description: key/values to be passed to Amplify.configure(config);
Value: !Sub |
{
'aws_cognito_identity_pool_id': '${YourUserPool}',
'aws_sign_in_enabled': 'enable',
'aws_user_pools_mfa_type': 'OFF',
}
AdvancedUsage:
Description: use Pseudo Parameters and/or resources attributes
Value: !Sub |
{
'aws_region': '${AWS::Region}',
'user_pool_arn': '${YourUserPool.Arn}',
}

I found out how to do this using Join
AmplifyConfig:
Description: key/values to be passed to Amplify.configure(config);
Value:
Fn::Join:
- ''
- - "{"
- "\n 'aws_cognito_identity_pool_id':"
- Ref : IdentityPool
- "\n 'aws_user_pools_id':"
- Ref : UserPool
- "\n 'aws_user_pools_web_client_id':"
- Ref : UserPoolClient
- ",\n 'aws_cognito_region': '${self:provider.region}'"
- ",\n 'aws_sign_in_enabled': 'enable'"
- ",\n 'aws_user_pools': 'enable'"
- ",\n 'aws_user_pools_mfa_type': 'OFF'"
- "\n}"
This works but it's kinda ugly. I'm going to leave this answer unaccepted for a while to see if anyone can show how to do this with Fn::Sub.

Using YAML you could compose this simply:
Outputs:
AmplifyConfig:
Description: key/values to be passed to Amplify.configure(config);
Value: !Sub '
{
"aws_cognito_identity_pool_id": "${IdentityPool}",
"aws_sign_in_enabled": "enable",
"aws_user_pools_mfa_type": "OFF",
}'

Leaving this here as I encountered a Base64 encoding error when doing something similar and this question came up when searching for
a solution.
In my case I was a using multi line string + !Sub to populate UserData and receiving the following error in AWS Cloudformation.
Error:
Invalid BASE64 encoding of user data. (Service: AmazonEC2; Status
Code: 400; Error Code: InvalidUserData.Malformed; Request ID: *;
Proxy: null)
Solution:
Can be solved by combining two built in Cloudformation functions; Fn::Base64 and !Sub:
UserData:
Fn::Base64: !Sub |
#!/bin/bash
echo ${SomeVar}

Related

How to provide expression as value inside an ssm document?

I would like to add a server to an ausostaling-group using SSM document, if the group has n instances running - i want to have (n+1).
Since this stack is managed by cloudformation, i just need to increase the 'DesiredCapacity' variable and update the stack. so i created a document with 2 steps:
get the current value of 'DesiredCapacity'
update stack with value of 'DesiredCapacity' + 1
I didnt find a way to express this simple operation, i guess im doing something wrong ...
SSM Document:
schemaVersion: '0.3'
parameters:
cfnStack:
description: 'The cloudformation stack to be updated'
type: String
mainSteps:
- name: GetDesiredCount
action: 'aws:executeAwsApi'
inputs:
Service: cloudformation
Api: DescribeStacks
StackName: '{{ cfnStack }}'
outputs:
- Selector: '$.Stacks[0].Outputs.DesiredCapacity'
Type: String
Name: DesiredCapacity
- name: UpdateCloudFormationStack
action: 'aws:executeAwsApi'
inputs:
Service: cloudformation
Api: UpdateStack
StackName: '{{ cfnStack }}'
UsePreviousTemplate: true
Parameters:
- ParameterKey: WebServerCapacity
ParameterValue: 'GetDesiredCount.DesiredCapacity' + 1 ### ERROR
# ParameterValue: '{{GetDesiredCount.DesiredCapacity}}' + 1 ### ERROR (trying to concat STR to INT)
# ParameterValue: '{{ GetDesiredCount.DesiredCapacity + 1}}' ### ERROR
There is a way to do calculation inside an SSM document using python runtime.
The additional python step do the following:
Python runtime get variables via the the 'InputPayload' property
The 'current' (str) key added to the event object
The python function script_handler called
The 'current' extracted using event['current']
Converting string to int and adding 1
return a dictionary with the 'desired_capacity' key and value as string
expose the output ($.Payload.desired_capacity referred to the 'desired_capacity' of the returned dictionary)
schemaVersion: '0.3'
parameters:
cfnStack:
description: 'The cloudformation stack to be updated'
type: String
mainSteps:
- name: GetDesiredCount
action: 'aws:executeAwsApi'
inputs:
Service: cloudformation
Api: DescribeStacks
StackName: '{{ cfnStack }}'
outputs:
- Selector: '$.Stacks[0].Outputs.DesiredCapacity'
Type: String
Name: DesiredCapacity
- name: Calculate
action: 'aws:executeScript'
inputs:
Runtime: python3.6
Handler: script_handler
Script: |-
def script_handler(events, context):
desired_capacity = int(events['current']) + 1
return {'desired_capacity': str(desired_capacity)}
InputPayload:
current: '{{ GetDesiredCount.DesiredCapacity }}'
outputs:
- Selector: $.Payload.desired_capacity
Type: String
Name: NewDesiredCapacity
- name: UpdateCloudFormationStack
action: 'aws:executeAwsApi'
inputs:
Service: cloudformation
Api: UpdateStack
StackName: '{{ cfnStack }}'
UsePreviousTemplate: true
Parameters:
- ParameterKey: WebServerCapacity
ParameterValue: '{{ Calculate.NewDesiredCapacity}}'

How to iterate all the variables from a variables template file in azure pipeline?

test_env_template.yml
variables:
- name: DB_HOSTNAME
value: 10.123.56.222
- name: DB_PORTNUMBER
value: 1521
- name: USERNAME
value: TEST
- name: PASSWORD
value: TEST
- name: SCHEMANAME
value: SCHEMA
- name: ACTIVEMQNAME
value: 10.123.56.223
- name: ACTIVEMQPORT
value: 8161
and many more variables in the list.
I wanted to iterate through all the variables in the test_env_template.yml using a loop to replace the values in a file, Is there a way to do that rather than calling each values separately like ${{ variables.ACTIVEMQNAME}} as the no. of variables in the template is dynamic.
In short no. There is no easy way to get your azure pipeline variables specific to tamplet variable. You can get env variables, but there you will get regular env variables and pipeline variables mapped to env variables.
You can get it via env | sort but I'm pretty sure that this is not waht you want.
You can't display variables specific to template but you can get all pipeline variables in this way:
steps:
- pwsh:
Write-Host "${{ convertToJson(variables) }}"
and then you will get
{
system: build,
system.hosttype: build,
system.servertype: Hosted,
system.culture: en-US,
system.collectionId: be1a2b52-5ed1-4713-8508-ed226307f634,
system.collectionUri: https://dev.azure.com/thecodemanual/,
system.teamFoundationCollectionUri: https://dev.azure.com/thecodemanual/,
system.taskDefinitionsUri: https://dev.azure.com/thecodemanual/,
system.pipelineStartTime: 2021-09-21 08:06:07+00:00,
system.teamProject: DevOps Manual,
system.teamProjectId: 4fa6b279-3db9-4cb0-aab8-e06c2ad550b2,
system.definitionId: 275,
build.definitionName: kmadof.devops-manual 123 ,
build.definitionVersion: 1,
build.queuedBy: Krzysztof Madej,
build.queuedById: daec281a-9c41-4c66-91b0-8146285ccdcb,
build.requestedFor: Krzysztof Madej,
build.requestedForId: daec281a-9c41-4c66-91b0-8146285ccdcb,
build.requestedForEmail: krzysztof.madej#hotmail.com,
build.sourceVersion: 583a276cd9a0f5bf664b4b128f6ad45de1592b14,
build.sourceBranch: refs/heads/master,
build.sourceBranchName: master,
build.reason: Manual,
system.pullRequest.isFork: False,
system.jobParallelismTag: Public,
system.enableAccessToken: SecretVariable,
DB_HOSTNAME: 10.123.56.222,
DB_PORTNUMBER: 1521,
USERNAME: TEST,
PASSWORD: TEST,
SCHEMANAME: SCHEMA,
ACTIVEMQNAME: 10.123.56.223,
ACTIVEMQPORT: 8161
}
If you prefix them then you can try to filter using jq.

Ansible - Access output of a shell command with with_items

I wrote a python script which gets executed via my ansible playbook and returns the following output via stdout:
- { username: ansible, admin: yes}
- { username: test, admin: no }
The output then should get saved in the variable "users" and with the "with_items" (or the newer "loop" conditional) I want to iterate through the variable in order to assign the right permissions for each user separately:
- name: execute python script
command: "python3 /tmp/parse_string.py --user_permissions={{ user_permissions }}"
register: output
- name: register
set_fact:
users: "{{ output.stdout }}"
- name: output
debug: msg="{{ users }}"
- name: Add user to group -admin
user:
name={{ item.username }}
groups=admin
append=yes
state=present
when: "item.admin == yes"
with_items: '{{users}}
However when launching the playbook it says that the variable "users" has no attribute "username".
TASK [create_users : output] ***************************************************
ok: [ansible] => {
"msg": "- { username: ansible, admin: yes }\n- { username: test, admin: no }\n- { username: test2, admin: no }"
}
TASK [create_users : Add user to group -admin ***************
fatal: [ansible]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'username'\n\nThe error appears to be in '***': line 29, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \n ^ here\n"}
Can anyone help me with this case?
BR
You are setting your users var to a string. It happens that this string is a yaml representation of a datastructure but ansible has no clue about that at this point.
To achieve your requirement, you need to parse that string as yaml and register the result. Luckily, there is a from_yaml filter for that purpose
You just have to modify your set_fact task as the following and everything should work as expected:
- name: register
set_fact:
users: "{{ output.stdout | from_yaml }}"

why does my swagger-server generate errors?

I got following errors about the swagger yaml when starting index.js in nodejs-server from "Generate Server" on Swagger Editor.
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/2: Not a valid parameter definition
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/2: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/2: Missing required property: $ref
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/1: Not a valid parameter definition
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/1: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/1: Missing required property: $ref
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/0: Not a valid parameter definition
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/0: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1deleteScenario~1/delete/parameters/0: Missing required property: $ref
#/paths/~1botManagement~1stopScenario~1/post/parameters/2: Not a valid parameter definition
#/paths/~1botManagement~1stopScenario~1/post/parameters/2: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1stopScenario~1/post/parameters/2: Missing required property: $ref
#/paths/~1botManagement~1stopScenario~1/post/parameters/1: Not a valid parameter definition
#/paths/~1botManagement~1stopScenario~1/post/parameters/1: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1stopScenario~1/post/parameters/1: Missing required property: $ref
#/paths/~1botManagement~1stopScenario~1/post/parameters/0: Not a valid parameter definition
#/paths/~1botManagement~1stopScenario~1/post/parameters/0: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1stopScenario~1/post/parameters/0: Missing required property: $ref
#/paths/~1botManagement~1restartScenario~1/post/parameters/2: Not a valid parameter definition
#/paths/~1botManagement~1restartScenario~1/post/parameters/2: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1restartScenario~1/post/parameters/2: Missing required property: $ref
#/paths/~1botManagement~1restartScenario~1/post/parameters/1: Not a valid parameter definition
#/paths/~1botManagement~1restartScenario~1/post/parameters/1: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1restartScenario~1/post/parameters/1: Missing required property: $ref
#/paths/~1botManagement~1restartScenario~1/post/parameters/0: Not a valid parameter definition
#/paths/~1botManagement~1restartScenario~1/post/parameters/0: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1restartScenario~1/post/parameters/0: Missing required property: $ref
#/paths/~1botManagement~1startScenario~1/post/parameters/2: Not a valid parameter definition
#/paths/~1botManagement~1startScenario~1/post/parameters/2: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1startScenario~1/post/parameters/2: Missing required property: $ref
#/paths/~1botManagement~1startScenario~1/post/parameters/1: Not a valid parameter definition
#/paths/~1botManagement~1startScenario~1/post/parameters/1: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1startScenario~1/post/parameters/1: Missing required property: $ref
#/paths/~1botManagement~1startScenario~1/post/parameters/0: Not a valid parameter definition
#/paths/~1botManagement~1startScenario~1/post/parameters/0: Not a valid parameter definition
#: Missing required property: schema
#: Missing required property: type
#/paths/~1botManagement~1startScenario~1/post/parameters/0: Missing required property: $ref
yaml file is below.
swagger: "2.0"
info:
description: I wrote something but can't describe it.
version: 1
title: I wrote something but can't describe it.
termOfService: I wrote something but can't describe it.
contact:
email: xxxx#xxxx
license:
name: Apache 2.0
url: http://www.apache.org/licenses/LICENSE-2.0.html
host: xxxx.xxxx.xxxx.xxxx
basePath: /v1
tags:
- name: I wrote something but can't describe it.
schemes:
- https
paths:
/XXXXXXXXXX:
get:
summary: I wrote something but can't describe it.
description: I wrote something but can't describe it.
responses:
200:
description: Successful response
schema:
type: object
properties:
pid:
type: integer
format: int64
example: 0
name:
type: string
example: sample
pm2_env:
type: object
example: {username:sample, windowsHide:true...}
pm_id:
type: integer
format: int64
example: 0
monit:
type: object
example: {memory:0, cpu:0}
404:
description: API not found
500:
description: Internal server error
/xxxxx/xxxxx:
post:
summary: I wrote something but can't describe it.
description: I wrote something but can't describe it.
parameters:
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: true
type: string
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: false
type: string
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: false
type: string
responses:
200:
description: Successful response
schema:
type: object
properties:
scenario:
type: string
example: sample.js
status:
type: string
example: I wrote something but can't describe it.
400:
description: I wrote something but can't describe it.
404:
description: "API not found"
500:
description: "Internal server error"
/xxxxx/xxxxx/:
post:
summary: I wrote something but can't describe it.
description:
I wrote something but can't describe it.
parameters:
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: true
type: string
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: false
type: string
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: false
type: string
responses:
200:
description: Successful response
schema:
type: object
properties:
scenario:
type: string
example: sample.js
status:
type: string
example: I wrote something but can't describe it.
400:
description: Bad request
404:
description: API not found
500:
description: Internal server error
/xxxxx/xxxxx/:
post:
summary: I wrote something but can't describe it.
description:
I wrote something but can't describe it.
parameters:
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: true
type: string
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: false
type: string
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: false
type: string
responses:
200:
description: Successful response
schema:
type: object
properties:
scenario:
type: string
example: sample.js
status:
type: string
example: I wrote something but can't describe it.
400:
description: Bad request
404:
description: API not found
500:
description: Internal server error
/xxxxx/xxxxx/:
delete:
summary: I wrote something but can't describe it.
description:
I wrote something but can't describe it.
parameters:
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: true
type: string
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: false
type: string
- name: xxxxx
in: formData
description: I wrote something but can't describe it.
required: false
type: string
responses:
200:
description: Successful response
schema:
type: object
properties:
scenario:
type: string
example: sample.js
status:
type: string
example: I wrote something but can't describe it.
400:
description: Bad request
404:
description: API not found
500:
description: Internal server error
It looks the missing property of "$ref".
I know what this property is used. But I don't need any reference to other docs.
In this case, how can I solve this problems?
Your definition has syntax errors.
1) version requires a string value, so it should be version: '1', not version: 1. Without quotes 1 is parsed as a number.
2) example: {username:sample, windowsHide:true...} is not valid YAML, it should be
example:
username: sample
windowsHide: true
or using JSON syntax:
example: {"username": "sample", "windowsHide": true}
Same for other inline object examples.
3) Operations that use in: formData parameters should specify application/x-www-form-urlencoded or multipart/form-data in the consumes keyword:
consumes:
- application/x-www-form-urlencoded
It's hard to say what else might be wrong without seeing your actual definition. You can paste your YAML into https://editor.swagger.io to check for syntax errors, or if your definition has a public URL, use the online validator
http://online.swagger.io/validator/debug?url=YOUR_URL

LambdaFunction - Value of property Variables must be an object with String (or simple type) properties

I am using serverless to deploy my Lambda based application. It was deploying just fine, and then it stopped for some reason. I paired down the entire package to the serverless.yml below and one function in the handler - but I keep getting this error:
Serverless Error ---------------------------------------
An error occurred: TestLambdaFunction - Value of property Variables must be an object with String (or simple type) properties.
Stack Trace --------------------------------------------
Here is the serverless.yml
# serverless.yml
service: some-api
provider:
name: aws
runtime: nodejs6.10
stage: prod
region: us-east-1
iamRoleStatements:
$ref: ./user-policy.json
environment:
config:
region: us-east-1
plugins:
- serverless-local-dev-server
- serverless-dynamodb-local
- serverless-step-functions
package:
exclude:
- .gitignore
- package.json
- README.md
- .git
- ./**.test.js
functions:
test:
handler: handler.test
events:
- http: GET test
resources:
Outputs:
NewOutput:
Description: Description for the output
Value: Some output value
Test Lambda Function in Package
#handler.test
module.exports.test = (event, context, callback) => {
callback(null, {
statusCode: 200,
body: JSON.stringify({
message: 'sadfasd',
input: event
})
})
}
Turns out, this issue does not have any relationship to the Lambda function. Here is the issue that caused the error.
This does NOT work:
environment:
config:
region: us-east-1
This DOES work:
environment:
region: us-east-1
Simply put, I don't think you can have more than one level in your yaml environment variables.
Even if you try sls print as a sanity check, this issue will not pop up. Only in sls deploy.
You have been warned, and hopefully saved!
Other thing that might cause this kind of error is using invalid yaml syntax.
It's easy to get confused about this.
Valid syntax for environment variables
environment:
key: value
Invalid syntax for environment variables
environment:
- key: value
Notice little dash in the code below?
In yaml syntax - means array and therefore, code below is interpreted as array, not object.
So that's why error tells "Value of property Variables must be an object with String (or simple type) properties."
This can be easily fixed by removing - in front of all keys.

Resources