I'm using gitlab pipelines to update configuration and code of my AWS Lambda functions. I noticed that when you perform AWS cli requests to update the Lambda like:
aws lambda update-function-configuration --region $your_region --function-name $your_lambda_arn --handler $your_lambda_handler --runtime $your_lambda_runtime
OR
aws lambda update-function-code --region $your_region --function-name $your_lambda_arn --s3-bucket $your_s3_bucket --s3-key $your_project_name.zip
Both requests responses in gitlab console reveals the environment variables (like db connection string) and I would like to keep it secret. Is there any way, any flag or whatsoever to keep these values hidden in response?
Regards
Could perhaps use jq to exclude the output? Otherwise use an SDK (e.g. boto3 or aws-sdk-go) and don't print the entire response.
$ cat ~/Desktop/Me.json
{
"FunctionName": "my-function",
"LastModified": "2019-09-26T20:28:40.438+0000",
"RevisionId": "e52502d4-9320-4688-9cd6-152a6ab7490d",
"MemorySize": 256,
"Version": "$LATEST",
"Role": "arn:aws:iam::123456789012:role/service-role/my-function-role-uy3l9qyq",
"Timeout": 3,
"Runtime": "nodejs10.x",
"TracingConfig": {
"Mode": "PassThrough"
},
"Environment": {
"foo": "bar"
},
"CodeSha256": "5tT2qgzYUHaqwR716pZ2dpkn/0J1FrzJmlKidWoaCgk=",
"Description": "",
"VpcConfig": {
"SubnetIds": [],
"VpcId": "",
"SecurityGroupIds": []
},
"CodeSize": 304,
"FunctionArn": "arn:aws:lambda:us-west-2:123456789012:function:my-function",
"Handler": "index.handler"
}
# no Environment key
$ cat ~/Desktop/Me.json | jq 'del(.Environment)'
{
"FunctionName": "my-function",
"LastModified": "2019-09-26T20:28:40.438+0000",
"RevisionId": "e52502d4-9320-4688-9cd6-152a6ab7490d",
"MemorySize": 256,
"Version": "$LATEST",
"Role": "arn:aws:iam::123456789012:role/service-role/my-function-role-uy3l9qyq",
"Timeout": 3,
"Runtime": "nodejs10.x",
"TracingConfig": {
"Mode": "PassThrough"
},
"CodeSha256": "5tT2qgzYUHaqwR716pZ2dpkn/0J1FrzJmlKidWoaCgk=",
"Description": "",
"VpcConfig": {
"SubnetIds": [],
"VpcId": "",
"SecurityGroupIds": []
},
"CodeSize": 304,
"FunctionArn": "arn:aws:lambda:us-west-2:123456789012:function:my-function",
"Handler": "index.handler"
}
I've found two additional solutions for that case:
Gitlab itself has mechanism of hiding secrets, to achieve that you can setup environment variable in gitlab cicd settings as protected, in that case console output should be hidden where needed
You can filter out the aws cli response
https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html
Regards
Related
"MetricAlarms": [
{
"AlarmName": "AWS_CPUUtilisation_Alarm",
"MetricName": "CPUUtilization",
"Namespace": "AWS/EC2",
"Statistic": "Average",
"Dimensions": [
{
"Name": "InstanceId",
"Value": "i-XXXXXXXXX"
}
],
"Period": 300,
"EvaluationPeriods": 1,
},
From the above data how can i get only instance id by using below command I am getting name and value:
aws cloudwatch describe-alarms --query "MetricAlarms[*].{Instanceid:Dimensions}" --output table
You're almost there:
aws cloudwatch describe-alarms --query "MetricAlarms[*].{Instanceid:Dimensions}[*].Instanceid[*].Value"
You might find it useful to play around with a tool like jmespath terminal or jmespath tutorial.
I'm trying to deploy a Windows-based Container from a private repository within an Azure Container Instance using the Azure Portal and I'm not sure whether I use the "Command override" at the "Advanced" section properly (probably I do not). The thing is I've to pass an argument during runtime, which sets the value of a License Server, so that a specific application, which needs to establish a connection to the License Server, can start up.
In general, the run command for the container would look like:
docker run IMAGE:TAG -LicenseServer Port#Host
My entrypoint within the Dockerfile is a Powershell Script "Start.ps1", which requests the corresponding value of the mentioned License Server.
I've read the manual and therefore I've inserted following string to override and to pass the argument:
[ "cmd", "Start.ps1", "-LicenseServer", "<Port>#<Hostname>"]
After deploying the ACI, the Container gets the state "running" for a few seconds, after that, it's terminated again. According to logs, it didn't work anyway.
So I wonder, what would be the proper way to deploy the container to get it running?
Thank you a lot in advance!
In addition to my question, to get more context:
ACI was created within Azure Portal:
I've used following settings see JSON view:
{
"properties": {
"sku": "Standard",
"provisioningState": "Succeeded",
"containers": [
{
"name": "<name>",
"properties": {
"image": "<image name>",
"command": [
"powershell",
"Start.ps1",
"-LicenseServer",
"<port>#<host>"
],
"ports": [
{
"protocol": "TCP",
"port": 80
}
],
"environmentVariables": [],
"instanceView": {
"restartCount": 1,
"currentState": {
"state": "Terminated",
"finishTime": "2021-04-28T06:06:22.2263538Z",
"detailStatus": "Container stopped per client request"
},
"previousState": {
"state": "Waiting",
"detailStatus": "CrashLoopBackOff: Back-off restarting failed"
}
},
"resources": {
"requests": {
"memoryInGB": 8,
"cpu": 1
}
}
}
}
],
"initContainers": [],
"imageRegistryCredentials": [
{
"server": "<login server>",
"username": "<user>"
}
],
"restartPolicy": "OnFailure",
"ipAddress": {
"ports": [
{
"protocol": "TCP",
"port": 80
}
],
"type": "Public",
"dnsNameLabel": "mycontainerdns",
"fqdn": "mycontainerdns.westeurope.azurecontainer.io"
},
"osType": "Windows",
"instanceView": {
"events": [],
"state": "Stopped"
}
},
"id": "/subscriptions/<subscription id>",
"name": "<aci name>",
"type": "Microsoft.ContainerInstance/containerGroups",
"location": "westeurope",
"tags": {}
}
Actually, the cmd just tell you when you need to connect to the windows container instance, you need to use the command:
az container exec -g resource_group_name -n container_group_name --container-name container_name --exec-command "cmd"
But when you want to overwrite the CMD command, you need to pass the arguments like this:
["powershell", "Start.ps1", "-LicenseServer", "<Port>#<Hostname>"]
It means you need to execute the Powershell script in the cmd terminal.
I finally found the solution. The command string, provided within "Command override" was wrong.
I've tried several versions, but it now worked with following:
[ "powershell", "C:/Windows/Scripts/Start.ps1", "-LicenseServer", "<port>#<host>" ]
Now I get logs and the running state of the container within the ACI deployment.
Before, I've tried as suggested in the first answer: (among others)
["powershell", "Start.ps1", "-LicenseServer", "<Port>#<Hostname>"]
But that seems not to work within ACI, as "Start.ps1" script couldn't be found ALTHOUGH I've set the working directory within the Dockerfile and of course it works within my Rancher deployment (by just providing "-LicenseServer PortatHost").
So, as conclusion you've to provide the full path to your file when it serves as Entrypoint within the Container.
Thank you a lot anyway for your help!
I have seen lots of answer of how to debug a lambda function offline in vscode and I have got that working, to the extent I can set breakpoint and step through it.
However I am unsure how to specify the payload input for the lambda function for testing.
{
"configurations": [
{
"type": "aws-sam",
"request": "direct-invoke",
"name": "Downloads:charge.handler (nodejs10.x)",
"invokeTarget": {
"target": "code",
"projectRoot": "",
"lambdaHandler": "charge.handler"
},
"lambda": {
"runtime": "nodejs10.x",
"payload": {},
"environmentVariables": {}
}
}
]
}
It seems whatever I put into the payload json field, I only ever see an empty param object when my lambda function runs. I also have an ssm key saved on the aws server. Will that automatically be available to my locally debugged lambda function which I have setup with with SAM CLI, Docker and AWS CLI?
Any help would be greatly appreciated.
Thanks,
Greg
Okay so I wasn't specifying the payload properly.
Should be:
"payload": { "json": { "body": {
"item1": 1, "item2": 2, ...
}}}
Current situation: I currently have a working web app bot with LUIS integration (NODE.js). I want to add a QnA maker to the bot. I have created a QnA maker via the Azure bot service and created a knowledge base for it to use.
Issue: When adding the qna maker details to the bot and running with nodemon ./index.js, I get the error "Error: The encrypted value is not a valid format". I've tested and this error is thrown when it tries to read the hostname value during
botConfig = BotConfiguration.loadSync(BOT_FILE, process.env.botFileSecret);
When pasting the hostname into the browser Azure shows me a "Your App Service app is up and running" page, indicating the hostname is fine.
Questions:
How do I debug this further? Could it be something to do with how the QnA maker is set up?
Both the QnA maker and Knoledge base is published - is there something I have to add manually to the config of the bot via the Azure portal to get it to recognise the QnA maker?
A lot of the documentation is based on v3 of the bot framework and have no idea if it's still applicable.
QnA snippet in Bot file (some values omitted, not sure how sensitive they are):
{
"type": "qna",
"name": "pathqna",
"KbId": "OMITTED",
"subscriptionId": "OMITTED",
"endpointKey": "OMITTED",
"hostname": "https://pathqna.azurewebsites.net",
"id": "7"
}
Documentation I've looked at:
https://learn.microsoft.com/en-gb/azure/bot-service/bot-builder-tutorial-dispatch?view=azure-bot-service-4.0&tabs=javascript
https://learn.microsoft.com/en-us/azure/cognitive-services/qnamaker/tutorials/create-qna-bot
https://github.com/Microsoft/botbuilder-tools/blob/master/packages/MSBot/docs/sample-bot-file.json
https://learn.microsoft.com/en-us/azure/cognitive-services/QnAMaker/how-to/troubleshooting-runtime#how-to-get-latest-qnamaker-runtime-updates
Full bot file with the empty padlock value (all OMITTED values have real keys and name has been changed to Test):
{
"name": “Test”,
"padlock": "",
"version": "2.0",
"services": [
{
"tenantId": "OMITTED",
"subscriptionId": "OMITTED",
"resourceGroup": OMITTED,
"serviceName": OMITTED,
"type": "abs",
"name": OMITTED,
"id": "1"
},
{
"connectionString": "OMITTED",
"tenantId": "OMITTED",
"subscriptionId": "OMITTED",
"resourceGroup": OMITTED,
"serviceName": "patha048",
"type": "blob",
"id": "2"
},
{
“appId": "OMITTED",
"appPassword": “OMITTED”,
"endpoint": "http://localhost:3978/api/messages",
"type": "endpoint",
"name": "development",
"id": "3"
},
{
"appId": "OMITTED",
"appPassword": “OMITTED”,
"endpoint": "https://path-a048.azurewebsites.net/api/messages",
"type": "endpoint",
"name": "production",
"id": "4"
},
{
"instrumentationKey": “OMITTED”,
"applicationId": “OMITTED”,
"apiKeys": {},
"tenantId": "OMITTED",
"subscriptionId": "OMITTED",
"resourceGroup": OMITTED,
"serviceName": "Patht6r6m4",
"type": "appInsights",
"id": "5"
},
{
"appId": “OMITTED”,
"authoringKey": “OMITTED”,
"version": "0.1",
"region": "westus",
"type": "luis",
"name": "BasicBotLuisApplication",
"id": "6"
},
{
"type": "qna",
"name": "pathqna",
"id": "7",
"kbId": “OMITTED”,
"subscriptionKey": "OMITTED",
"endpointKey": “OMITTED”,
"hostname": "https://pathqna.azurewebsites.net"
}
]
}
Found the solution - Use msbot cli to add the QnA maker instead of adding manually as the file is encrypted and loses it's decryption if you don't use the msbot/emulator shrug
I removed the qna snippet and ran this command (have added the generic values to preserve the real values):
msbot connect qna --secret <botFileSecret> --name pathqna --kbId <KB-ID> --subscriptionKey <SUB_KEY> --endpointKey <ENDPOINT_KEY> --hostname "https://pathqna.azurewebsites.net" --bot Path.bot
This preserved the padlock value and added it successfully.
Although the information you have provided is not enough to provide a solution, you may please check on the following steps.
"Error: The encrypted value is not a valid format"
*Please check your bot secret keys once again.
Then, in your bot file, try removing the padlock value.
"padlock": ""
Also, I assume in your actual code, you have replaced 'OMITTED' with the real time keys that you have obtained from QnAMaker portal.*
Provide a screenshot of the error if possible.
I use circleci to build and push the application to aws. I have now managed to create and register a new task definition in the circleci config using the aws cli. This works well. The problem or case i´am having is how do I overwrite the placeholder values using the cli?
Here is how i read the task definition:
aws ecs register-task-definition --cli-input-json file://.circleci/taskdefinition.json
The task definition file:
{
"containerDefinitions": [
{
"cpu": 10,
"environment": [
{
"name": "Secret_api_key",
"value": "placeholder_value"
}
],
"image": "<Image>",
"name": "app-dev"
}
],
"placementConstraints": [],
"memory": "512",
"family": "pp-dev",
"networkMode": "bridge"
}
What I want to know is, how do I update the placeholder_value which I have retrieved from aws secrets manager?
Use sed to replace the placeholders with secret valuables from private environment variables.