Normally, to test my aws lambda functions, Id do this:
IMG_FILE='/some/image/loc//img.jpg' &&\
jo img=%"$IMG_FILE" | curl -X POST -H 'Content-Type: application/json' -d #- "$LAMBDA_HOST" >> output.bin
Id like to replicate the same with aws lambda invoke function like so:
IMG_FILE='/some/image/loc//img.jpg' &&\
jo img=%"$IMG_FILE" |aws lambda invoke --payload XXXXXX --function-name funky_func output.bin
How do I pass the jo output to the --payload param? :(
Oh, looking at the documentation, it appears that --payload can take a file:// parameter. So you can do:
IMG_FILE='/some/image/loc/img.jpg' &&
jo img=%"$IMG_FILE" > img.json &&
aws lambda invoke --payload file://img.json \
--function-name funky_func output.bin
...but it's possible you'll hit some limitation on command line length depending on the size of your image.
It's possible -- but I'm unable to test this at the moment -- that if you were to provide a file argument of file:///dev/stdin that you could provide the JSON on stdin. Maybe worth a try.
Related
I am trying to put logs to AWS CloudWatch logs via AWS CLI using a bash script:
#!/bin/bash
EVENT_TIME=$(date +%s%3N)
LOG_LEVEL=6
EVENT_SOURCE=myapp
MESSAGE=1
OUTPUT=$(jq -n \
--arg EventTime "$EVENT_TIME" \
--arg LogLevel "$LOG_LEVEL" \
--arg EventSource "$EVENT_SOURCE" \
--arg Message "$MESSAGE" \
'{EventTime:$EventTime,LogLevel:$LogLevel,EventSource:$EventSource,Message:$Message}')
MESSAGE="$OUTPUT"
aws logs put-log-events --log-group-name test --log-stream-name local --log-events timestamp=$(date +%s%3N),message=$MESSAGE
but I am getting error:
Error parsing parameter '--log-events': Expected: '<double quoted>', received: '<none>'
for input:
timestamp=1654692489664,message="{
The command works fine, if I remove the JSON message to a simple string. It should be an issue with quoting but not sure where the problem is. Any idea?
The message parameter needs to be a string containing the json, not the direct json created with jq.
Something like this should work:
#!/bin/bash
EVENT_TIME=$(date +%s000)
LOG_LEVEL=6
EVENT_SOURCE=myapp
MESSAGE=1
OUTPUT=$(jq -n \
--arg EventTime "$EVENT_TIME" \
--arg LogLevel "$LOG_LEVEL" \
--arg EventSource "$EVENT_SOURCE" \
--arg Message "$MESSAGE" \
'{EventTime:$EventTime,LogLevel:$LogLevel,EventSource:$EventSource,Message:$Message}')
LOG_MESSAGE=$(echo $OUTPUT | sed 's/"/\\"/g')
aws logs put-log-events --log-group-name test --log-stream-name local --log-events timestamp=$(date +%s000),message=\""$LOG_MESSAGE"\"
Also, if you plan to use the put-log-events like this, you will need to provide the --sequence-token for consecutive puts. See here: https://docs.aws.amazon.com/cli/latest/reference/logs/put-log-events.html
Might be best to setup CloudWatch agent to publish the logs.
Issue
I am trying to use the Gitlab yaml API linting tool on an enterprise instance of Gitlab. However, I am getting an empty response (not just an empty json object, like absolutely zero output).
Steps to duplicate
I am using a stripped version of the sample .yaml file shown on the gitlab CI/CD tutorial page. The file is shown here:
build-job:
stage: build
script:
- echo "Hello, $GITLAB_USER_LOGIN!"
deploy-prod:
stage: deploy
script:
- echo "This job deploys something from the $CI_COMMIT_BRANCH branch."
I am using the 1 line curl command as shown on the CI Linting API page.
If I use the command as given (replacing only the filename), I get
$ jq --null-input --arg yaml "$(<.gitlab-ci.yml)" '.content=$yaml' \
| curl "https://gitlab.mycompany.com/api/v4/ci/lint?include_merged_yaml=true" \
--header 'Content-Type: application/json' \
--data #-
I get the output {"message":"401 Unauthorized"}, which is to be expected as the API call requires an API key. I generate an API key in my profile and try again:
$ export TOKEN='xxxxxxxxxx'
$ jq --null-input --arg yaml "$(<.gitlab-ci.yml)" '.content=$yaml' \
| curl "https://gitlab.mycompany.com/api/v4/ci/lint?include_merged_yaml=true" \
--header "Content-Type: application/json PRIVATE-TOKEN=${TOKEN}" \
--data #-
When I run this, the output shows nothing. This is confirmed by a pipe to wc -c which outputs 0.
My expected output should be:
{
"status": "valid",
"errors": [],
"warnings": []
}
Questions:
Why is no response a result of me using my valid API key (This is with a newly generated key)?
How can I fix this, and receive the expected output shown above?
Make sure your token as the api scope, as illustrated here.
Without that scope, you would get a 401 Unauthorized, which would not be parsed parsed by jq at all.
I am currently working on some code to interact with images on the Google Container Registry. I have working code both using plain curl and also httpx. I am trying to build a package without 3rd party dependencies. My curiosity is around a particular endpoint from which I get a successful response in curl and httpx but a 401 Unauthorized using urllib.request.
The bash script that demonstrates what I'm trying to achieve is the following. It retrieves an access token from the registry API, then uses that token to verify that the API indeed runs version 2 and tries to access a particular Docker image configuration. I'm afraid that in order to test this, you will need access to a private GCR image and a digest for one of the tags.
#!/usr/bin/env bash
set -eu
token=$(gcloud auth print-access-token)
image=...
digest=sha256:...
get_token() {
curl -sSL \
-G \
--http1.1 \
-H "Authorization: Bearer ${token}" \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
--data-urlencode "scope=repository:$1:pull" \
--data-urlencode "service=gcr.io" \
"https://gcr.io/v2/token" | jq -r '.token'
}
echo "---"
echo "Retrieving access token."
access_token=$(get_token ${image})
echo
echo "---"
echo "Testing version 2 capability with access token."
curl -sSL \
--http1.1 \
-o /dev/null \
-w "%{http_code}" \
-H "Authorization: Bearer ${access_token}" \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
https://gcr.io/v2/
echo
echo "---"
echo "Retrieving image configuration with access token."
curl -vL \
--http1.1 \
-o /dev/null \
-w "%{http_code}" \
-H "Authorization: Bearer ${access_token}" \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
"https://gcr.io/v2/${image}/blobs/${digest}"
I additionally created two Jupyter notebooks demonstrating my solutions in httpx and bare urllib.request. The httpx one works perfectly while somehow urllib fails on the image configuration request. I'm running out of ideas trying to spot the difference. If you run the notebook yourself, you will see that the called URL contains a token as a query parameter (is this a security issue?). When I open that link I can actually successfully download the data myself. Maybe urllib still passes along the Authorization header with the Bearer token making that last call fail with 401 Unauthorized?
Any insights are greatly appreciated.
I did some investigation and I believe the difference is that the last call to "https://gcr.io/v2/${image}/blobs/${digest}" actually contains a redirect. Inspecting the curl and httpx calls showed me that both do not include the Authorization header in the second, redirected request, whereas in the way that I set up the urllib.request in the notebook, this header is always included. It's a bit odd that this leads to a 401 but now I know how to address it.
Edit: I can now confirm that by building a urllib.request.Request instance and unlike in the linked notebook, add the authorization header with the request's add_unredirected_header method, everything works as expected.
I do have a simple AWS API Gateway implementation protected by an AWS_IAM Authorization.
I just want to test from command line via cURL :
curl --location --request GET 'https://<API_ID>.execute-api.eu-west-1.amazonaws.com/stage?type=type&category=category&lc=lc&passprhase=passprhase&product=product'
--header 'Authorization: AWS4-HMAC-SHA256 Credential=<AWS_ACCESS_KEY>/20200127/eu-west-1/execute-api/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<AWS_SECRET_ACCESS_KEY>' --header 'Content-Type: application/json' \
--data-raw '{"query":"","variables":{}}'
but keep getting the follow error :
Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header.
Can someone advice what am I doing wrong ?
AWS V4 signature authentication is supported in curl starting from version 7.75, so you should be able to call your AWS resource this way:
curl --location --request GET 'https://$API-ID.execute-api.eu-west-1.amazonaws.com/stage?type=type&category=category&lc=lc&passprhase=passprhase&product=product' \
--header 'Content-Type: application/json' \
--user $ACCESS_KEY:$SECRET_KEY \
--aws-sigv4 "aws:amz" \
--data-raw '{"query":"","variables":{}}'
Note that you may need to add in the --aws-sigv4 value your region and service.
For example: --aws-sigv4 "aws:amz:eu-west-2:execute-api"
You can find more documentation here: https://curl.se/libcurl/c/CURLOPT_AWS_SIGV4.html
And the documentation for the CLI option here: https://curl.se/docs/manpage.html#--aws-sigv4
AWS_IAM authorization uses Sigv4 and its calculation process requires values certain headers - Date being one of them. You are passing x-amz-date as a part of the "SignedHeaders" field, but not actually passing it with the other headers.
One way to create the right curl command to invoke an API with AWS_IAM would be to use Postman application. Add in the API URL and select "AWS Signature" under Authorization tab. You can then select the "Code" option and get the full curl command which would look something like this -
curl -X POST \
https://$API-ID.execute-api.$AWS_REGION.amazonaws.com/$STAGE/$RESOURCE \
-H 'authorization: AWS4-HMAC-SHA256 Credential=$ACCESS_KEY/20200128/$AWS_REGION/execute-api/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=$SIGNATURE_VALUE' \
-H 'cache-control: no-cache' \
-H 'content-type: application/x-www-form-urlencoded' \
-H 'host: API-ID.execute-api.$AWS_REGION.amazonaws.com' \
-H 'postman-token: 15f9498e-95b7-f22b-eed9-016cdea07424' \
-H 'x-amz-date: $DATE_STAMP'
Create a Canonical Request for Signature Version 4
I could suggest to use awscurl which is much easier.
To install awscurl click here. For documentation you can refer here.
Example to call apigateway to call lambda for POST query is below.
awscurl --service execute-api -X POST -d '{ "alias" : "xyx", "type" : "LDAP" }' https://.execute-api.us-west-2.amazonaws.com/Prod/user/groups/get --region us-west-2 --access_key ACCESS_KEY --secret_key mfBl0YJRsXDue4C5F5B6rz1eUpQpA8uC24RtSnsg --security_token SECURITY_TOKEN
What is the difference between this two curl calls:
curl -X POST -d "/path/file.txt" http://api-path
curl -X POST -d "#/path/file.txt" http://api-path
The first form sends the string "/path/file.txt" as data, probably not what you want. The # makes curl interpret it as a filename to read from. See http://curl.haxx.se/docs/manpage.html, search for the "--data" option.