I have used the aws encryption cli to encrypt a zip file (a database backup actually) as follows:
aws-encryption-cli -vv --encrypt \
--input "backup.zip" \
--wrapping-keys key=caf...854 region=us-east-1 \
--encryption-context purpose=dbbackup \
--metadata-output backup.metadata \
--output backup.zip.enc
This works great, and produces the encrypted backup file. However I can't seem to decrypt using the same basic pattern.
aws-encryption-cli \
-vv \
--decrypt \
--input backup.zip.enc \
--wrapping-keys key=caf...854 region=us-east-1 \
--output backup.zip \
--metadata-output backup.zip.metadatadec
Produces an error:
2020-11-13 15:04:04,580 - MainThread - aws_encryption_sdk.key_providers.base - DEBUG - IncorrectMasterKeyError("Provided data key provider MasterKeyInfo(provider_id='aws-kms', key_info=b'arn:aws:kms:us-east-1:...:key/caf...854') does not match Master Key provider MasterKeyInfo(provider_id='aws-kms', key_info=b'caf...854')",) raised when attempting to decrypt data key with master key MasterKeyInfo(provider_id='aws-kms', key_info=b'caf...854')
It looks to me like it's complaining that the KMS key ID doesn't match because on decryption it uses the full ARN (arn:aws:kms:us-east-1...) when on encryption it only used the ID (caf...854). I'm not sure how to change the behavior to use the same thing on both cases, or if something else is the problem?
FWIW, this seems to work if I specify the full ARN for the wrapping key key when I encrypt and decrypt. I'm not sure why it would succeed on the encryption but fail on decryption when I use only the ID - this feels like a bug (should either fail for both or work for both).
Related
I have 1 .zip.asc file ,inside that 3 txt files are there. I am trying to decrypt the test.zip.asc using below code but not working ,though while using tool it is working fine.
Databricks code
%sh
#Apply looping for each file in the source folder.
for entry in /dbfs/mnt/acc/Temp/*
do
#Create input variable
input=test.zip.asc
echo $input
#create output variable
output=test.zip
echo=$output
#import private key
gpg --no---batch --import /dbfs/mnt/acc/pgp/PrivateKey.asc
#start decryption
gpg --no--tty --batch --yes --ignore-mdc-error --pinentry-mode=loopback --passphrase-fd 1 --passphrase-file /dbfs/mnt/acc/pgp/Passphrase.txt --output /dbfs/mnt/acc/out/$output --decrypt /dbfs/mnt/acc/input/$input
done
output I'm getting
test.zip.asc
test.zip
gpg:no valid openPGP data found.
gpg:Total number proessed:0
gpg:enrypted with RSA key ,ID A48FAF69A4E7B316
gpg:decryption failed : No secret key
I am trying to put logs to AWS CloudWatch logs via AWS CLI using a bash script:
#!/bin/bash
EVENT_TIME=$(date +%s%3N)
LOG_LEVEL=6
EVENT_SOURCE=myapp
MESSAGE=1
OUTPUT=$(jq -n \
--arg EventTime "$EVENT_TIME" \
--arg LogLevel "$LOG_LEVEL" \
--arg EventSource "$EVENT_SOURCE" \
--arg Message "$MESSAGE" \
'{EventTime:$EventTime,LogLevel:$LogLevel,EventSource:$EventSource,Message:$Message}')
MESSAGE="$OUTPUT"
aws logs put-log-events --log-group-name test --log-stream-name local --log-events timestamp=$(date +%s%3N),message=$MESSAGE
but I am getting error:
Error parsing parameter '--log-events': Expected: '<double quoted>', received: '<none>'
for input:
timestamp=1654692489664,message="{
The command works fine, if I remove the JSON message to a simple string. It should be an issue with quoting but not sure where the problem is. Any idea?
The message parameter needs to be a string containing the json, not the direct json created with jq.
Something like this should work:
#!/bin/bash
EVENT_TIME=$(date +%s000)
LOG_LEVEL=6
EVENT_SOURCE=myapp
MESSAGE=1
OUTPUT=$(jq -n \
--arg EventTime "$EVENT_TIME" \
--arg LogLevel "$LOG_LEVEL" \
--arg EventSource "$EVENT_SOURCE" \
--arg Message "$MESSAGE" \
'{EventTime:$EventTime,LogLevel:$LogLevel,EventSource:$EventSource,Message:$Message}')
LOG_MESSAGE=$(echo $OUTPUT | sed 's/"/\\"/g')
aws logs put-log-events --log-group-name test --log-stream-name local --log-events timestamp=$(date +%s000),message=\""$LOG_MESSAGE"\"
Also, if you plan to use the put-log-events like this, you will need to provide the --sequence-token for consecutive puts. See here: https://docs.aws.amazon.com/cli/latest/reference/logs/put-log-events.html
Might be best to setup CloudWatch agent to publish the logs.
in GPG while we are doing encryption
gpg --cipher-algo AES256 --symmetric ./.abcd.csv
It open a prompt
Enter Passphrase
Can we change that too
Enter Passphrase for xyz:
The gpg PIN entry is handled by an external program or device, so there is no universal mean to control the prompt of a PIN, unless you force gpg into batch mode, and force PIN entry to loop back to the caller script, so you have full control of it.
Note that this is of questionable security, because while taking control of the pin entry, you are also responsible for all the security, and a shell script is the least secure environment to handle secret data like a PIN.
Here it is how you would control the PIN prompt.
(Know the security concerns before using this):
read -r -s -p 'Enter Passphrase for xyz: ' secret
gpg \
--cipher-algo AES256 \
--pinentry-mode loopback \
--batch \
--passphrase "$secret" \
--symmetric \
./.abcd.csv
printf '\n'
I followed the steps in this article, http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html#mon-scripts-systems to monitor my server's memory usage and submit the data to CloudWatch.
One of the metrics that can be used is --mem-used-incl-cache-buff which collects and sends the MemoryUsed metrics, reported in megabytes. This option reports used in cache and buffers, as well as memory allocated by applications and the operating system. This memory metric is important compare to other memory metrics because this metric collects memory used in cache and buffers as well. The other memory metrics simply collect free and used memory to determine whether or not my server is running out of memory.
Unfortunately this is the output when I tried to run the script:
[root#ip-172-31-10-167 ~]# ~/aws-scripts-mon/mon-put-instance-data.pl --mem-used-incl-cache-buff --verify --verbose
ERROR: No metrics specified for collection and submission to CloudWatch.
For more information, run 'mon-put-instance-data.pl --help'
I've followed the prerequisite in the documentation to ensure that all required tools are installed first. What did I miss?
I think what you need is:
~/aws-scripts-mon/mon-put-instance-data.pl --mem-used-incl-cache-buff --mem-used
--mem-used flag will tell the script that you want memory used and --mem-used-incl-cache-buff tells it to include cache and buffers.
To verify this you can run these 2 commands and compare the output:
~/aws-scripts-mon/mon-put-instance-data.pl --mem-used --verify --verbose
~/aws-scripts-mon/mon-put-instance-data.pl --mem-used-incl-cache-buff --mem-used --verify --verbose
I use this in my custom metrics. As mentioned by #Tartaglia, you need to enable the mem-used switch to report memory statistics. You can view the Perl script and see how this is computed.
Without buffers and cache:
mon-put-instance-data.pl --mem-used
Output:
Payload:{"MetricData":[{"MetricName":"MemoryUsed","Unit":"Megabytes","Value":4915.6640625,"Timestamp":1501356451}
With buffers and cache:
mon-put-instance-data.pl --mem-used-incl-cache-buff --mem-used
Output:
Payload:{"MetricData":[{"MetricName":"MemoryUsed","Unit":"Megabytes","Value":5690.55078125,"Timestamp":1501356486}
Try something like this:
If you want to send custom data
/usr/local/bin/aws cloudwatch put-metric-data \
--namespace "TomcatOpenFiles" \
--dimensions INSTANCE_ID="`curl -s http://169.254.169.254/latest/meta-data/instance-id`" \
--metric-name "TomcatOpenFiles" \
--value $Tomcat_Openfiles \
--region ap-south-1
Note: 169.254.169.254 is where you can retrieve instance metadata as documented here
If you want to send AWS defined data
/opt/aws-scripts-mon/mon-put-instance-data.pl \
--mem-util \
--mem-used \
--mem-avail \
--disk-space-util \
--disk-space-used \
--disk-space-avail \
--memory-units=gigabytes \
--disk-space-units=gigabytes \
--disk-path=/ \
--from-cron
I have the following POST call I need to make. However, even if I provided the right client id and secret id, my call is getting rejected.
curl POST https://login.microsoftonline.com/f02....e3/oauth2/token
-H 'Content-Type: application/x-www-form-urlencoded' --data 'grant_type=authorization_code&redirect_uri=https://requestb.in/ac&
source=https://graph.microsoft.com&client_id=1e1....-913d9
&client_secret=YmbSFYz.....4Uk=&scope=mail.read&code=AaAAA........on0a569'
This is the error I receive:
curl: (6) Could not resolve host: POST
{"error":"invalid_client","error_description":"AADSTS70002:
Error validating credentials. AADSTS50012: Invalid client secret is
provided.\r\nTrace ID: 78d...a2b\r\nCorrelation ID:
01....ab2\r\nTimestamp: 2016-12-14 01:46:47Z","error_codes":[70002,50012],"timestamp":"2016-12-14 01:46:47Z","trace_id":"78d....a2b","correlation_id":"018.....ab2"}
How could I resolve this ?
EDIT: I am trying to achieve the second section(i.e getting token) in this documentation
The post you provided is leveraging AAD V2 endpoint. But according your code snippet, you were using V1 endpoint https://login.microsoftonline.com/f02....e3/oauth2/token. For acquire access token via V1 endpoint, you can refer to https://graph.microsoft.io/en-us/docs/authorization/app_authorization for more details.
For the V2 authorization endpoint, you may check out the endpoints you are using:
GET https://login.microsoftonline.com/common/oauth2/v2.0/authorize?...
POST https://login.microsoftonline.com/common/oauth2/v2.0/token
And also it is required a v2.0 ad application:
This article assumes a v2.0 registration, so you'll register your app on the Application Registration Portal.
It was due to client_secret. It may contain special characters.
The encodeURIComponent() function encodes a URI component.
This function encodes special characters. In addition, it encodes the following characters: , / ? : # & = + $ #
Use the below one:
encodeURIComponent(client_secret);
In my case:
in configs You have to use the client secret "Value", not ID.
This Value is visible only when it is generated. So you can copy/paste it, in this moment.
See: https://learn.microsoft.com/en-us/answers/questions/370508/getting-34invalid-client-secret-is-provided34-erro.html
I am using the npm package:
Microsoft Azure Active Directory Passport.js Plug-In
with the v1 tenant-specific endpoint.
I was initially receiving the same error message after successfully logging into Microsoft: "Invalid Client Secret".
I was using the "Application ID" as the Client Secret. Then I discovered that you need to go onto the "Settings" tab in the Microsoft Azure Portal
Microsoft Azure
and create a new "Key". It doesn't matter what you name the key. When you click the "Save" button, the value of the key will be filled in the web form. Copy this right away, since it will not be displayed again if the web page is refreshed.
This is the "Client Secret" that you need to add to your configuration.
I hade the same problem today and with help of #muthu I sorted it out.
When client secret is generated with app registration in azure. The secret is plain. But when to use it in the rest call you must urlencode it.
Use an encoder like visual code or other. But it must be encoded else you ge the error
AADSTS7000215: Invalid client secret is provided.
You need to base 64 encode client_secret in the POST body.
For example (using curl):
# Authn details
LoginURL='https://login.microsoftonline.com'
TenantDomain='********.onmicrosoft.com'
ClientID='********'
ClientSecret='********'
# Endpoint details
Resource='https://graph.microsoft.com'
TenantGUID="********"
# Authenticate with OAuth v1
URL="$LoginURL/$TenantDomain/oauth2/token?api-version=1.0"
json=`
curl \
-s \
-k \
-X POST \
-H "Content-Type: application/x-www-form-urlencoded" \
-u "$ClientID:$ClientSecret" \
-d "grant_type=client_credentials" \
-d "resource=$Resource" \
-d "client_id=$ClientID" \
--data-urlencode "client_secret=$ClientSecret" \
$URL \
| python -m json.tool
`
access_token=`echo $json | python -c 'import sys, json; print json.load(sys.stdin)["access_token"]'`
token_type=`echo $json | python -c 'import sys, json; print json.load(sys.stdin)["token_type"]'`
#echo "access_token:$access_token"
#echo "token_type:$token_type:"
# Access resource
URL="$Resource/v1.0/directoryRoles/$TenantGUID/members"
curl \
-s \
-k \
-X GET \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "Authorization: $token_type $access_token" \
$URL \
| python -m json.tool