is there an elegant way to create a RDS instance without providing the credentials in plain text?
I already created a secret, but I can't find any documentation about how to use a secret when creating a database.
My current solution is saving the password in a file and reading it into a variable:
$ vim password.txt
$ pw=$(cat password.txt)
$ aws rds create-db-instance \
# all the arguments \
--master-user-password $pw
Update: I figured another way:
$ read -s -p "enter password: " pw
$ aws rds ... --master-user-password $pw
openejb-core jar is a library that can be used to cipher and decipher text. You can use this library to cipher the actual password and store the ciphered password in the text file 'password.txt'.
While retrieving the password into 'pw' variable, you can decipher it again and use it further. Hence, the actual password is never visible, even with root access.
I am trying to initialize hashicorp vault with api.
payload.json:
{
"secret_shares": 2,
"secret_threshold": 2,
"pgp_keys": [
"sample_pub_1.asc",
"sample_pub_2.asc"
]
}
When I execute below command:
curl --request PUT --data #payload.json http://localhost:8200/v1/sys/init
Getting below error:
{"errors":["invalid seal configuration: error decoding given PGP key:> illegal base64 data at input byte 6"]}
But If I initialize Vault with command line everything works fine:
vault operator init -key-shares=2 -key-threshold=2 -pgp-keys="sample_pub_1.asc, sample_pub_2.asc"
Below is the output:
Unseal Key 1: wcBMA3EOVuvAOcf6AQgAhm4FpWmwhU9BV8jaYLuL8gyzhpQgPs76ByF/v5XFYj3PKaVcm2U3nzn7dablYMheGkFbOytCQ8G1guJrQ756+9t0dotAzghEeCUDwwIEU5lqENR/nAJXT4NvgrycASfS5OKv7vT6L/L8GokiHPSdBdouI3/4GyHKygrqaAUjbmNDUrgXDEfaWMkNv58yJE1tVpjYuKh8pWqNqJgSyDdkrXUt4AEY6yLTsLaCkOuu0j1sVE5D3huE0UV8u59qDwf9FusACgoyhyqyNwXqMn0pGet0hebk9pY/F293LWUr2zp7otbo5sX7H+086pl4plxUHo3GN/vhYnfwIVvJbCz68tLgAeShAyKfKCX6kFz+QX3LCkLW4UOS4Dngt+EvS+B24sFDtIvgPOZPueiPgmxwj5Evv0GzPm8iDvnrxvaWU1vxl+YlP1f3PI+rs+oBCndlBL0lVpFTDbtVLrFb/szxj+rRwKnz0oOZ4Djh2ufgDeQIlx/Zw6T1y+41jj8ITwn24uo5ChnhmeIA
Unseal Key 2: wcBMA0Eqk591x+XTAQgAQDi/V1yGqHlrSDWJ2SLeiqaLzpNrcbC5cB4+CxzkZsiVgjrd5YitULL7TmWkYl43VSPxs0hnk3O+uRUp+D4gVjUWykpGmdhwB5CiWHwNMzuIUE1E/UQVMQ39OLnG+C3VgzRe8HNhEr6pkbLFgSQLlr/459bgEyAPrW/U1PrHrH+hf+eE0me5/hUQWBfqcZHeY1Zp3A97UUBV54yarSgfRyqluiXe5j6L2X0qw5Ge/luRR8xPygyx3JqlIlPCj6liMS/to/yRGOKvYAXjXdflpBeodx3cSUOMmBTrzk1h6otM8LR28dp7qS+0qX0mWHFAfElMrJh9tz5PpWY5WIdyJNLgAeQO/Bvz4UMJLBAzEcCy2G7L4fDf4I/gSuHJROC+4lgAeaTgv+bRQCmvD4SEQYUJX3IP0cwMgskkys/vwUMqnZSNMS0gXayRvVSJ5YAHsynuixp+aE1x6UBiEw6cjVX7wwfupvdg4PTh4GPgG+TjTQDF06tQigWmpxio1JAa4jdU9uzhWbMA
Initial Root Token: aa3cccce-5ac8-4aef-7d50-6fab16e89ee4
You might not have base64-encoded your pgp_keys; according to the official Vault documentation when using curl init:
The keys must be base64-encoded from their original binary representation
You have to encode the original binary output in Base64.
On my Mac:
gpg2 --export sample#stackoverflow.com | base64 -b 0
Or on Linux:
gpg2 --export sample#stackoverflow.com | base64 -w 0
There is a difference of use between the vault command and HTTP API.
While the command can read a file from disk and allows the file name as a parameter as your example shows correctly:
vault operator init -key-shares=2 -key-threshold=2 -pgp-keys="sample_pub_1.asc, sample_pub_2.asc"
The HTTP API needs the base64 key in this field instead the file name.
{ "secret_shares": 2, "secret_threshold": 2, "pgp_keys": [
"mQGNBF/ALSkBDADoIAuvrTNbvkuZZTl28Pwh7i0aa.......",
"mQGNBF/ALSkBDADoIAuvrTNbvkuZZTl28Pwh7i0aa......."
]
}
That's why the error indicates the bad byte in the position number 6 because is trying to decode the string "sample_pub_1.asc" as a base64 valid value.
best regards
I have file that ends by -comps.xml and has the following form:
http://some/url/<sha256sum>-<2 chars>-x86_64-comps.xml
sha256sum is alphanumeric string of 65 length.
For example:
http://some/url/0dae8d32824acd9dbdf7ed72f628152dd00b85e4bd802e6b46e4d7b78c1042a3-c6-x86_64-comps.xml
How I can download this file using curl?
I've found solution using wget:
wget --recursive --level=1 --no-parent --no-directories --accept '*-comps.xml' --directory-prefix=. http://some/url
Assuming that you already know the filename, to download the contents of the file, then simply use
curl -O http://some/url/0dae8d32824acd9dbdf7ed72f628152dd00b85e4bd802e6b46e4d7b78c1042a3-c6-x86_64-comps.xml
If you are looking to somehow predetermine the file name based on an SHA256 of the file's contents, then you will need to either already have access to these contents to be able to determine the SHA256 part of the URL, or to have access to an alternative source for this information.
I'm using GNUPG to encrypt my ascii files.
I learnt to generate a key, also how to use the it to encrypt and decrypt a file.
There are two ways I used:
gpg -d foo.txt.gpg
and
gpg --output foo.txt --decrypt
foo.txt.gpg
I realized the first method will display the decrypted file on the screen, for example when I executed the command over SSH.
With regard to the second method, I concerned if it will leave a trace on the local pc - the
foo.txt file.
Most importantly, I don't know how to edit the contents of the foo file on the fly.
Ideally, I would like to open the file over SSH use nano/pico, type my passphrase to decrypt, then edit the file, save it and encrypt it. I very much like to avoid save any files to the local disk.
Any comments are welcome.
Thank you in advance.
One way is using vim. See this page and this related question.
If you need more flexibility or don't want to use vim, writing a short program to read the decrypted text coming from STDOUT, edit to your liking, and then re-encrypt isn't too difficult. For example, you could use this minimal Python code (104 lines!) to give you the bare bones editor, and then add the stream reading and writing functionality yourself.
One thing to bear in mind is that holding unencrypted data in memory is no guarantee that it wont find its way to disk. If the system in question is under heavy load any unencrypted data may be written to the swap partition. Similarly, if the system is put into sleep mode, the state of any suspended processes will be stored to disk. If your program is running on a embedded system, it's conceivable that your memory and "disk" are one and the same.
The mlock() system call will protect allocated memory from getting swapped to disk. However, this requires administrative privileges and limits you to a low-level language where you are directly responsible for memory management.
That said, it is prudent to avoid creating files with unencrypted data. Just know that this doesn't offer you 100% safety if the underlying system is compromised.
I wrote a python script to solve this (for Linux only). It works by decrypting the file into /dev/shm to ensure that the unencrypted data is never written to disk (although it is possible for any of the programs using the data to be swapped to disk; this is almost always a concern).
This has some benefits over some of the other posted answers:
Only need to type the password once
Works with any editor
Here is the code:
#!/usr/bin/python
import os, sys, subprocess, getpass, stat, shutil
editor = 'nano'
dataFile = sys.argv[1]
## make a backup of the encrypted file
bakFile = dataFile+'-gpgedit_backup'
shutil.copy(dataFile, bakFile)
dstat = os.stat(dataFile)
## create temporary directory in tmpfs to work from
tmpDir = '/dev/shm/gpgedit'
n = 0
while True:
try:
os.mkdir(tmpDir+str(n))
break
except OSError as err:
if err.errno != 17: ## file already exists
raise
n += 1
tmpDir += str(n)
os.chmod(tmpDir, stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR)
try:
## Get password
passwd = getpass.getpass()
## decrypt file
tmpFile = os.path.join(tmpDir, 'data')
cmd = "gpg -d --passphrase-fd 0 --output %s %s" % (tmpFile, dataFile)
proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(passwd)
proc.stdin.close()
if proc.wait() != 0:
raise Exception("Error decrypting file.")
## record stats of tmp file
stat = os.stat(tmpFile)
## invoke editor
os.system('%s %s' % (editor, tmpFile))
## see whether data has changed
stat2 = os.stat(tmpFile)
if stat.st_mtime == stat2.st_mtime and stat.st_size == stat2.st_size:
raise Exception("Data unchanged; not writing encrypted file.")
## re-encrypt, write back to original file
cmd = "gpg --yes --symmetric --passphrase-fd 0 --output %s %s" % (dataFile, tmpFile)
proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(passwd)
proc.stdin.close()
if proc.wait() != 0:
raise Exception("Error encrypting file.")
except:
## If there was an error AND the data file was modified, restore the backup.
dstat2 = os.stat(dataFile)
if dstat.st_mtime != dstat2.st_mtime or dstat.st_size != dstat2.st_size:
print "Error occurred, restored encrypted file from backup."
shutil.copy(bakFile, dataFile)
raise
finally:
shutil.rmtree(tmpDir)
os.remove(bakFile)
Where is gnupg plugin - exactly for this point for example
Inspired by Luke's answer, I wrote a Python script myself. Hopefully, somebody will find this useful. Here are the core features:
uses temporary file under /dev/shm using a secure method to generate tempfile
creates backup file in case of failures
both encryption modes (public key/symmetric)
create a new file on-the-fly
choose your editor through environment variables
Further information can be found in the script itself. It currently won't work on any non *nix-machine.
To install the script just put it in any directory on your path and make it executable.
Get it now!
Warning: Backup your data! The script comes without any warranty!
I have spent countless hours on this quest, too: simply encrypt a text file with a passphrase with simple open+read/write access. I didn't want to deal with private/public keys nor keyrings bound to an OS login, blah, blah, blah. File encryption with passphrase only is so simple and so universal and perfect for a simple text file to hold passwords. No bloat nor complication of a database-driven solution like KeePass, etc. (which also requires data entry into multiple GUI elements rather than just typing your passwords in a searchable text file). The gold standard on Windows is Steganos LockNote. How to do it on Linux? Surprisingly very difficult to find, but...
I finally converged on the recommendation I consider best: cream. http://cream.sourceforge.net/ Cream is a facade to vim to make it more-user-friendly ... useful for other family members (I am a Linux geek at work comfortable with vi[m], but I needed something more accessible for my family).
Just enter:
"vim -x yourfile.txt"
It will be saved as encrypted with a passphrase.
You can use vim or cream at this point:
"vim yourfile.txt" or "cream yourfile.txt".
Either one will natively open "yourfile.txt" and prompt for the passphrase and transparently allow edits and re-saving as encrypted. FINALLY the quest has been completed !!!!
An alternative is to have a tmp filesystem in ram using tmpfs then when you power off it's gone for ever.
If your editor can read input from a pipe, and save to a pipe, then you can actually use the version of gpg that decrypts to stdout and encrypts from stdin. Unfortunately, for nano, reading from a pipe is only planned for 2.4. E.g. for gvim, you can bind decryption and encryption (through pipes) to a key.
To open gpg files, editing them and then ecrypt/save again use:
kgpg
icon in systray has option: Editor...
Press on it, then open the gpg file, then on the bottom there is a button to decrypt it and voila you have your file in the editor, after you made any changes just press Encrypt and then save it.
Just today I have found out about a way of doing all that in vim!
here is the link: full howto on setting up vim for gpg files
works like a charm, just in that tutorial, the link to the plugin is url to a page so not to wget it, but go to the page and select the one you want to download.
I detest vi, so i had to make up some glue around nano. This is what i came up with. Downside is that you have to enter password again when encrypting.
alias file_ed="gpg file.txt.gpg; nano file.txt; gpg -c --force-mdc -o file.txt.gpg_temp file.txt; mv file.txt.gpg_temp file.txt.gpg; rm file.txt"
It isn't very secure from the filesystem point of view, but I fear other users and myself, not root.
viencrypt by Paul Tarjan is a script for editing GPG encrypted files on the fly.
Using the editor joe ( aka Joe's Own Editor ) in a command similar to
gpg --decrypt foo.txt.gpg | joe - | gpg --armor --recipient name#example.com --encrypt > bar.txt.gpg
will do what you're looking for.
The - in joe - tells joe to take its input from stdin and to write its output to stdout when you save the file (hit ctrl+k and then x to save). Joe will initially display some crufty output from gpg; this can be cleared by hitting ctrl+r to refresh the screen.
I use > bar.txt.gpg to specify the output file instead of --output bar.txt.gpg because the --output flag causes gpg to open an interactive dialogue if you're overwriting the output file, and this confuses joe.
Here is a slight improvement to #Luke's answer. It makes two small improvements:
It avoids the stack trace if the file is unmodified during the edit session.
It restores the original gpg file if re-encryption back to the original gpg file was attempted, which is a little safer than checking the modification dates of the edit file.
#!/usr/bin/python
# Downloaded from https://stackoverflow.com/questions/1510105/gnupg-how-to-edit-the-file-without-decrypt-and-save-to-local-disk-first/12289967#12289967
# and then slightly improved.
import os, sys, subprocess, getpass, stat, shutil
editor = 'nano'
dataFile = sys.argv[1]
## make a backup of the encrypted file
bakFile = dataFile+'-gpgedit_backup'
shutil.copy(dataFile, bakFile)
dstat = os.stat(dataFile)
## create temporary directory in tmpfs to work from
tmpDir = '/dev/shm/gpgedit'
n = 0
while True:
try:
os.mkdir(tmpDir+str(n))
break
except OSError as err:
if err.errno != 17: ## file already exists
raise
n += 1
tmpDir += str(n)
os.chmod(tmpDir, stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR)
reEncrypted = False
try:
## Get password
passwd = getpass.getpass()
## decrypt file
tmpFile = os.path.join(tmpDir, 'data')
cmd = "gpg -d --cipher-algo AES256 --passphrase-fd 0 --output %s %s" % (tmpFile, dataFile)
proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(passwd)
proc.stdin.close()
if proc.wait() != 0:
raise Exception("Error decrypting file.")
## record stats of tmp file
stat = os.stat(tmpFile)
## invoke editor
os.system('%s %s' % (editor, tmpFile))
## see whether data has changed
stat2 = os.stat(tmpFile)
if stat.st_mtime == stat2.st_mtime and stat.st_size == stat2.st_size:
print "Data unchanged; not re-writing encrypted file."
else:
## re-encrypt, write back to original file
reEncrypted = True
cmd = "gpg --yes --symmetric --cipher-algo AES256 --passphrase-fd 0 --output %s %s" % (dataFile, tmpFile)
proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(passwd)
proc.stdin.close()
if proc.wait() != 0:
raise Exception("Error encrypting file.")
except:
## If there was an error AND re-encryption was attempted, restore the backup.
if reEncrypted:
print "Error occurred; restoring encrypted file from backup."
shutil.copy(bakFile, dataFile)
raise
finally:
shutil.rmtree(tmpDir)
os.remove(bakFile)
I would have posted these suggested improvements as comments to #Luke's answer -- which I like a lot -- but did not have enough reputation points to do so. :(
I wrote a shell script to edit files encrypted with gpg.
Call it like ./editgpg.sh path/to/.secrets.gpg, edit with vim then close with ':q!'.
Vim flag '-n' opens file only in memory. No temp files.
#!/usr/bin/env bash
# decrypt CRYPT_FILE, pipe to vim, and encrypt againg whent type ':q!'
edit_crypt_file() {
echo "Enter your gpg encrypted file passphrase,
edit it with vim, then close the editor with ':q!'."
# first argument is a file encrypted with gpg
CRYPT_FILE=$1
# get password user input
local pass
read -sp "Password:" pass
gpg_flags='--batch --yes'
vim_flags='- -n -u NONE --not-a-term'
# vim command to run before exit with ':q!'
vim_autocmd=":autocmd VimLeave * :%! tee | gpg $gpg_flags --passphrase $pass -o $CRYPT_FILE -c"
gpg $gpg_flags --passphrase $pass -d $CRYPT_FILE | vim $vim_flags -c "$vim_autocmd"
# restart agent in order to lose kept password
gpgconf --kill gpg-agent
unset pass
}
edit_crypt_file $1