Google labs provided me the next code as a .yaml file to create a virtual machine that allow me to run Android studio and get some info from the Android studio emulator API to logstash / elasticsearch.
To be honest I´m new and not sure why I´m getting the error. I think is related to text in '' and/or {} but not sure.
Any help more than apreciate!!!
resources:
- type: gcp-types/compute-v1:instances
name: dev-vm
properties:
zone: {{ properties["zone"] }}
machineType: zones/{{ properties["zone"] }}/machineTypes/n1-highmem-4
disks:
- boot: true
autoDelete: true
initializeParams:
sourceImage: projects/qwiklabs-resources/global/images/vnc-android-pie-tensorflow
networkInterfaces:
- network: global/networks/default
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
serviceAccounts:
- email: default
scopes:
- 'https://www.googleapis.com/auth/cloud.useraccounts.readonly'
- 'https://www.googleapis.com/auth/devstorage.read_only'
- 'https://www.googleapis.com/auth/logging.write'
- 'https://www.googleapis.com/auth/monitoring.write'
tags:
items:
- "vnc"
metadata:
items:
- key: startup-script-url
value: https://storage.googleapis.com/cloud-training/gsp296/setup.sh
- name: vnc-firewall-rule
type: compute.v1.firewall
properties:
sourceRanges: [0.0.0.0/0]
allowed:
- IPProtocol: tcp
ports:
- 5901
- 6901
Related
I want to use templates in my local backstage and have been trying to publish a create-react-app to my gitlab instance, but keep getting the following
22022-10-21T22:36:25.000Z HTTPError: Response code 400 (Bad Request)
I have added the integration within my app-config yaml file. There is some connection happening, but seams that I have missed a required attribute for this request. For example, the title for a merge request was not given. A missed a required attribute for this request. For example, the title for a merge request was not given. Wondering whats another required attribute that I am missing for when publishing to gitlab.
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
name: create-react-app-template
title: Create React App Template
description: Create a new React website project
tags:
- react
- cra
spec:
owner: web#example.com
type: website
parameters:
- title: Provide some simple information
required:
- component_id
- owner
properties:
component_id:
title: Name
type: string
description: Unique name of the component
ui:field: EntityNamePicker
description:
title: Description
type: string
description: Help others understand what this website is for.
owner:
title: Owner
type: string
description: Owner of the component
ui:field: OwnerPicker
ui:options:
allowedKinds:
- Group
- title: Choose a location
required:
- repoUrl
properties:
repoUrl:
title: Repository Location
type: string
ui:field: RepoUrlPicker
ui:options:
allowedHosts:
- gitlab.example.com
steps:
- id: template
name: Fetch Skeleton + Template
action: fetch:template
input:
url: ./skeleton
copyWithoutRender:
- .github/workflows/*
values:
component_id: ${{ parameters.component_id }}
description: ${{ parameters.description }}
destination: ${{ parameters.repoUrl | parseRepoUrl }}
owner: ${{ parameters.owner }}
- id: publish
name: Publish
action: publish:gitlab
input:
allowedHosts:
- gitlab.example.com
description: This is ${{ parameters.component_id }}
repoUrl: ${{ parameters.repoUrl }}
title: Creating catalog-info.yaml ${{ parameters.name }} for backstage
- id: register
name: Register
action: catalog:register
input:
repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
catalogInfoPath: "/catalog-info.yaml"
output:
links:
- title: Repository
url: ${{ steps.publish.output.remoteUrl }}
- title: Open in catalog
icon: catalog
entityRef: ${{ steps.register.output.entityRef }}
When I want to go through the template of the Create React App it creates a
Repo Url -> gitlab.example.com?owner=maxbojorquez&repo=template
I have set up the airflow in Azure Cloud (Azure Container Apps) and attached an Azure File Share as an external mount/volume
1. I ran **airflow init service**, it had created the airflow.cfg and `'webserver_config.py'` file in the **AIRFLOW_HOME (/opt/airflow)**, which is actually an azure mounted file system
2. I ran **airflow webserver service**, it had created the `airflow-webserver.pid` file in the **AIRFLOW_HOME (/opt/airflow)**, which is actually an azure mounted file system
Now the problem is all the files created above are created with root user&groups, not as airflow user(50000),
I have also set the env variable AIRFLOW_UID to 50000 during the creation of the container app. due to this my webservers are not starting, throwing the below error
PermissionError: [Errno 1] Operation not permitted: '/opt/airflow/airflow-webserver.pid'
Note: Azure Containers Apps does not allow use root/sudo commands, otherwise I could solve this problem with simple chown commands
Another problem is the airflow configurations passed through environment variables are never picked up by Docker, eg
- name: AIRFLOW__API__AUTH_BACKENDS
value: 'airflow.api.auth.backend.basic_auth'
Attached screenshot for reference
Your help is much appreciated!
YAML file that I use to create my container app:
id: /subscriptions/1234/resourceGroups/<my-res-group>/providers/Microsoft.App/containerApps/<app-name>
identity:
type: None
location: eastus2
name: webservice
properties:
configuration:
activeRevisionsMode: Single
registries: []
managedEnvironmentId: /subscriptions/1234/resourceGroups/<my-res-group>/providers/Microsoft.App/managedEnvironments/container-app-env
template:
containers:
- command:
- /bin/bash
- -c
- exec /entrypoint airflow webserver
env:
- name: AIRFLOW__API__AUTH_BACKENDS
value: 'airflow.api.auth.backend.basic_auth'
- name: AIRFLOW__CELERY__BROKER_URL
value: redis://:#myredis.redis.cache.windows.net:6379/0
- name: AIRFLOW__CELERY__RESULT_BACKEND
value: db+postgresql://user:pass#postres-db-servconn.postgres.database.azure.com/airflow?sslmode=require
- name: AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION
value: 'true'
- name: AIRFLOW__CORE__EXECUTOR
value: CeleryExecutor
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
value: postgresql+psycopg2://user:pass#postres-db-servconn.postgres.database.azure.com/airflow?sslmode=require
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
value: postgresql+psycopg2://user:pass#postres-db-servconn.postgres.database.azure.com/airflow?sslmode=require
- name: AIRFLOW__CORE__LOAD_EXAMPLES
value: 'false'
- name: AIRFLOW_UID
value: 50000
image: docker.io/apache/airflow:latest
name: wsr
volumeMounts:
- volumeName: randaf-azure-files-volume
mountPath: /opt/airflow
probes: []
resources:
cpu: 0.25
memory: 0.5Gi
scale:
maxReplicas: 3
minReplicas: 1
volumes:
- name: randaf-azure-files-volume
storageName: randafstorage
storageType: AzureFile
resourceGroup: RAND
tags:
tagname: ws-only
type: Microsoft.App/containerApps
Because ACI doesn't support scaling, we deploy multiple container groups containing an Azure DevOps agent, a selenium grid hub and a selenium grid node. To try and speed things up I've tried to deploy the container groups with an additional node, identical to the first only being started on port 6666 instead of port 5555. I can see the two nodes register with the grid without issue but when I execute the same batch of tests with the additional node and without they take the exact same amount of time. How do I go about finding out what's going on here?
My ACI yaml:
apiVersion: 2018-10-01
location: australiaeast
properties:
containers:
- name: devops-agent
properties:
image: __AZUREDEVOPSAGENTIMAGE__
resources:
requests:
cpu: 0.5
memoryInGb: 1
environmentVariables:
- name: AZP_URL
value: __AZUREDEVOPSPROJECTURL__
- name: AZP_POOL
value: __AGENTPOOLNAME__
- name: AZP_TOKEN
secureValue: __AZUREDEVOPSAGENTTOKEN__
- name: SCREEN_WIDTH
value: "1920"
- name: SCREEN_HEIGHT
value: "1080"
volumeMounts:
- name: downloads
mountPath: /tmp/
- name: selenium-hub
properties:
image: selenium/hub:3.141.59-xenon
resources:
requests:
cpu: 1
memoryInGb: 1
ports:
- port: 4444
- name: chrome-node
properties:
image: selenium/node-chrome:3.141.59-xenon
resources:
requests:
cpu: 1
memoryInGb: 2
environmentVariables:
- name: HUB_HOST
value: localhost
- name: HUB_PORT
value: 4444
- name: SCREEN_WIDTH
value: "1920"
- name: SCREEN_HEIGHT
value: "1080"
volumeMounts:
- name: devshm
mountPath: /dev/shm
- name: downloads
mountPath: /home/seluser/downloads
- name: chrome-node-2
properties:
image: selenium/node-chrome:3.141.59-xenon
resources:
requests:
cpu: 1
memoryInGb: 2
environmentVariables:
- name: HUB_HOST
value: localhost
- name: HUB_PORT
value: 4444
- name: SCREEN_WIDTH
value: "1920"
- name: SCREEN_HEIGHT
value: "1080"
- name: SE_OPTS
value: "-port 6666"
volumeMounts:
- name: devshm
mountPath: /dev/shm
- name: downloads
mountPath: /home/seluser/downloads
osType: Linux
diagnostics:
logAnalytics:
workspaceId: __LOGANALYTICSWORKSPACEID__
workspaceKey: __LOGANALYTICSPRIMARYKEY__
volumes:
- name: devshm
emptyDir: {}
- name: downloads
emptyDir: {}
ipAddress:
type: Public
ports:
- protocol: tcp
port: '4444'
#==================== remove this section if not pulling images from private image registries ===============
imageRegistryCredentials:
- server: __IMAGEREGISTRYLOGINSERVER__
username: __IMAGEREGISTRYUSERNAME__
password: __IMAGEREGISTRYPASSWORD__
#========================================================================================================================
tags: null
type: Microsoft.ContainerInstance/containerGroups
When I run my tests locally against a docker selenium grid either from Visual Studio or via dotnet vstest, my tests run in parallel across all available nodes and complete in half the time.
I'm trying to configure Lambda#edge functions using CloudFormation. After deploying the template everything looks find in the console, however the lambda functions listening to origin-request events are not being triggered.
Strange enough, a viewer-request event does manage to trigger a function.
What am I doing wrong? How can I make the origin-request event to work?
Here's my template:
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Deployment of Lambda#edge functions
Parameters:
Stage:
Type: String
AllowedValues:
- staging
- production
Default: staging
Description: Stage that can be added to resource names
CodeBucket:
Type: String
Description: The S3 Bucket name for latest code upload
CodeKey:
Type: String
Description: The S3 Key for latest code upload
# Mappings:
# AliasMap:
# staging:
# Alias: "staging-app.achrafsouk.com"
# production:
# Alias: "app.achrafsouk.com"
Resources:
CFDistribution:
Type: AWS::CloudFront::Distribution
Properties:
DistributionConfig:
Enabled: true
Logging:
Bucket: XXX.s3.amazonaws.com
IncludeCookies: false
Prefix: !Sub ${Stage}
PriceClass: PriceClass_100
Comment:
!Sub "Lambda#Edge - ${Stage}"
# Aliases:
# - !FindInMap [AliasMap,
# Ref: Stage, Alias]
Origins:
- CustomOriginConfig:
OriginProtocolPolicy: https-only
DomainName: !Sub "${Stage}.defaultOrigin.example.com"
Id: defaultOrigin
OriginCustomHeaders:
- HeaderName: X-Node-Env
HeaderValue: !Sub "${Stage}"
- CustomOriginConfig:
OriginProtocolPolicy: https-only
DomainName: !Sub "${Stage}.another.example.com"
Id: rateLimitsOrigin
OriginCustomHeaders:
- HeaderName: X-Node-Env
HeaderValue: !Sub "${Stage}"
DefaultCacheBehavior:
ForwardedValues:
Headers:
- CloudFront-Viewer-Country
- X-Request-Host
- User-Agent
QueryString: true
ViewerProtocolPolicy: https-only
TargetOriginId: defaultOrigin
DefaultTTL: 0
MaxTTL: 0
LambdaFunctionAssociations:
# - EventType: viewer-request
# LambdaFunctionARN:
# Ref: ViewerRequestLambdaFunction.Version
- EventType: origin-request
LambdaFunctionARN:
Ref: OriginRequestLambdaFunction.Version
CacheBehaviors:
- ForwardedValues:
Headers:
- CloudFront-Viewer-Country
- X-Request-Host
- User-Agent
QueryString: true
ViewerProtocolPolicy: https-only
DefaultTTL: 60
MaxTTL: 60
TargetOriginId: rateLimitsOrigin
PathPattern: "/rate-limit*"
LambdaFunctionAssociations:
- EventType: origin-request
LambdaFunctionARN:
Ref: GetRateLimitLambdaFunction.Version
# ViewerRequestLambdaFunction:
# Type: AWS::Serverless::Function
# Properties:
# CodeUri:
# Bucket: !Sub ${CodeBucket}
# Key: !Sub ${CodeKey}
# Role: !GetAtt LambdaEdgeFunctionRole.Arn
# Runtime: nodejs10.x
# Handler: src/functions/viewerRequest.handler
# Timeout: 5
# AutoPublishAlias: live
OriginRequestLambdaFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri:
Bucket: !Sub ${CodeBucket}
Key: !Sub ${CodeKey}
Role: !GetAtt LambdaEdgeFunctionRole.Arn
Runtime: nodejs10.x
Handler: src/functions/originRequest.handler
Timeout: 5
AutoPublishAlias: live
GetRateLimitLambdaFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri:
Bucket: !Sub ${CodeBucket}
Key: !Sub ${CodeKey}
Role: !GetAtt LambdaEdgeFunctionRole.Arn
Runtime: nodejs10.x
Handler: src/functions/getRateLimit.handler
Timeout: 5
AutoPublishAlias: live
LambdaEdgeFunctionRole:
Type: "AWS::IAM::Role"
Properties:
Path: "/"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: "AllowLambdaServiceToAssumeRole"
Effect: "Allow"
Action:
- "sts:AssumeRole"
Principal:
Service:
- "lambda.amazonaws.com"
- "edgelambda.amazonaws.com"
Outputs:
# ViewerRequestLambdaFunctionVersion:
# Value: !Ref ViewerRequestLambdaFunction.Version
OriginRequestLambdaFunctionVersion:
Value: !Ref OriginRequestLambdaFunction.Version
GetRateLimitLambdaFunctionVersion:
Value: !Ref GetRateLimitLambdaFunction.Version
CFDistribution:
Description: Cloudfront Distribution Domain Name
Value: !GetAtt CFDistribution.DomainName
So, after painful hours of trial and error the solution was staring me in the face:
CloudFront distribution was emitting and error saying it could not connect to the origin, and it was right - because I've specified dummy domains in the origins definition of the CloudFront distribution.
My logic was that if my lambda#edge functions intercept the requests before reaching the origin, then it shouldn't matter if the origin's domains is real or not.
As it turns out, it's partially true. Because it seams that:
viewer-request event is triggered before trying to connect to the origin, and there for it can by fake.
origin-request event is triggered only after trying to connect to the origin, so you'd need to specify something real.
Hope my wasited hours will contribute to someone else exploring the same path of reasoning.
In case it helps someone in future with this problem - I noticed that origin-response (which I accidentally selected instead of origin-request) events were (mostly) working for me, but there were still a large number of OriginConnectError in the logs, and the "Percentage of Viewer Requests by Result Type" report was showing a high error rate.
It turns out it was only working at all because I had my origin connection set to https-only, but my origin (an empty s3 bucket) only accepted http connections - and for some reason the events were still being triggered.
Changing the origin connection to the correct http-only completely stopped things working for origin-response, until I also corrected the event type to origin-request
So, in general, check the OriginProtocolPolicy is http-only if you're using an s3 bucket origin. If it's wrong, things might still appear to work sometimes, but not always, and you'll end up with high error rates.
I'm experiencing difficulties regarding accessing external resources from nodes (RHEL) configured in a VM Scale Set.
To sketch the environment I'm trying to describe using Azure Resource Manager Templates, I'm looking to create:
1 common virtualNetwork
1 Frontend VM (running RHEL, and is working as intended)
1 Cluster (vmss) running 2 nodes (RHEL)
Nodes are spawned in the same private subnet as the frontend VM
1 loadbalancer should work as a NAT gateway (but it's not working this way)
The loadbalancer has an external IP, inboundNatPool (which works), backendAddressPool (in which nodes are successfully registered)
the Network Security Group manages access to ports (set to allow all outbound connections)
As a footnote, I'm comfortable writing up AWS cloudformation files in YAML, so I'm handling Azure Resource Manager Templates in a similar way, for the sake of readability and the added functionality of adding comments in my template.
An Example of my vmss config (short snippet)
... #(yaml-template is first converted to json and than deployed using the azure cli)
# Cluster
# -------
# Scale Set
# ---------
# | VM Scale Set can not connect to external sources
# |
- type: Microsoft.Compute/virtualMachineScaleSets
name: '[variables(''vmssName'')]'
location: '[resourceGroup().location]'
apiVersion: '2017-12-01'
dependsOn:
- '[variables(''vnetName'')]'
- '[variables(''loadBalancerName'')]'
- '[variables(''networkSecurityGroupName'')]'
sku:
capacity: '[variables(''instanceCount'')]' # Amount of nodes to be spawned
name: Standard_A2_v2
tier: Standard
# zones: # If zone is specified, no sku can be chosen
# - '1'
properties:
overprovision: 'true'
upgradePolicy:
mode: Manual
virtualMachineProfile:
networkProfile:
networkInterfaceConfigurations:
- name: '[variables(''vmssNicName'')]'
properties:
ipConfigurations:
- name: '[variables(''ipConfigName'')]'
properties:
loadBalancerBackendAddressPools:
- id: '[variables(''lbBackendAddressPoolsId'')]'
loadBalancerInboundNatPools:
- id: '[variables(''lbInboundNatPoolsId'')]'
subnet:
id: '[variables(''subnetId'')]'
primary: true
networkSecurityGroup:
id: '[variables(''networkSecurityGroupId'')]'
osProfile:
computerNamePrefix: '[variables(''vmssName'')]'
adminUsername: '[parameters(''sshUserName'')]'
# adminPassword: '[parameters(''adminPassword'')]'
linuxConfiguration:
disablePasswordAuthentication: True
ssh:
publicKeys:
- keyData: '[parameters(''sshPublicKey'')]'
path: '[concat(''/home/'',parameters(''sshUserName''),''/.ssh/authorized_keys'')]'
storageProfile:
imageReference: '[variables(''clusterImageReference'')]'
osDisk:
caching: ReadWrite
createOption: FromImage
...
The Network Security Group referenced from the template above is:
# NetworkSecurityGroup
# --------------------
- type: Microsoft.Network/networkSecurityGroups
name: '[variables(''networkSecurityGroupName'')]'
apiVersion: '2017-10-01'
location: '[resourceGroup().location]'
properties:
securityRules:
- name: remoteConnection
properties:
priority: 101
access: Allow
direction: Inbound
protocol: Tcp
description: Allow SSH traffic
sourceAddressPrefix: '*'
sourcePortRange: '*'
destinationAddressPrefix: '*'
destinationPortRange: '22'
- name: allow_outbound_connections
properties:
description: This rule allows outbound connections
priority: 200
access: Allow
direction: Outbound
protocol: '*'
sourceAddressPrefix: 'VirtualNetwork'
sourcePortRange: '*'
destinationAddressPrefix: '*'
destinationPortRange: '*'
And the loadbalancer, where I assume the error should be, is described as:
# Loadbalancer as NatGateway
# --------------------------
- type: Microsoft.Network/loadBalancers
name: '[variables(''loadBalancerName'')]'
apiVersion: '2017-10-01'
location: '[resourceGroup().location]'
sku:
name: Standard
dependsOn:
- '[variables(''natIPAddressName'')]'
properties:
backendAddressPools:
- name: '[variables(''lbBackendPoolName'')]'
frontendIPConfigurations:
- name: LoadBalancerFrontEnd
properties:
publicIPAddress:
id: '[variables(''natIPAddressId'')]'
inboundNatPools:
- name: '[variables(''lbNatPoolName'')]'
properties:
backendPort: '22'
frontendIPConfiguration:
id: '[variables(''frontEndIPConfigID'')]'
frontendPortRangeStart: '50000'
frontendPortRangeEnd: '50099'
protocol: tcp
I keep reading articles about configuring a SNAT with port masquerading, but I'm missing relevant examples of such setup.
Any help is greatly appreciated.
It took a lot of searching but the article from Azure about Azure Load Balancer outbound Connections (Scenario #2) stated a load-balancing rule (and complementary Health Probe) was necessary for SNAT to function.
the new code for the load balancer became:
...
- type: Microsoft.Network/loadBalancers
name: '[variables(''loadBalancerName'')]'
apiVersion: '2017-10-01'
location: '[resourceGroup().location]'
sku:
name: Standard
dependsOn:
- '[variables(''natIPAddressName'')]'
properties:
backendAddressPools:
- name: '[variables(''lbBackendPoolName'')]'
frontendIPConfigurations:
- name: LoadBalancerFrontEnd
properties:
publicIPAddress:
id: '[variables(''natIPAddressId'')]'
probes: # Needed for loadBalancingRule to work
- name: '[variables(''lbProbeName'')]'
properties:
protocol: Tcp
port: 22
intervalInSeconds: 5
numberOfProbes: 2
loadBalancingRules: # Needed for SNAT to work
- name: '[concat(variables(''loadBalancerName''),''NatRule'')]'
properties:
disableOutboundSnat: false
frontendIPConfiguration:
id: '[variables(''frontEndIPConfigID'')]'
backendAddressPool:
id: '[variables(''lbBackendAddressPoolsId'')]'
probe:
id: '[variables(''lbProbeId'')]'
protocol: tcp
frontendPort: 80
backendPort: 80
...