I am trying to make my expect script more robust and to be able to handle more situations to be more automated. Currently my script works fine, however, there are times where I will be asked to add the RSA keys to known_hosts, I want this to default to yes all the time. My server doesn't always ask for the keys, once added then after awhile it wont ask until you delete the keys or switch gatways. After looking online i have tried to add this (commented code) in my working code and after added that it stalls at password input screen if RSA key had been added already.
So my question is, is there a way to handle this situation lets say if RSA has been added already, it will just skip to the password line?
[user#gateway my_direcotry]$ cat loadItTest
#!/usr/bin/expect -f
set timeout 600
set user root
set host 1.1.1.1
set pass pass
spawn ssh $user#$host
#expect {
# -re "RSA key fingerprint" {send "yes\r"}
#}
expect "assword:"
send "$pass\r"
expect "#"
Sample output:
[root#gateway my_direcotry]# loadItTest
spawn ssh root#1.1.1.1
root#1.1.1.1's password:
This is where the exp_continue command comes into play, to essentially create a "loop" within the expect command:
spawn ssh $user#$host
expect {
"RSA key fingerprint" {send "yes\r"; exp_continue}
"assword:" {send "$pass\r"}
}
expect "#"
If you see the "RSA key" pattern, answer "yes" but then keep waiting for the "assword" pattern. If you don't see "assword" before "RSA key", that's OK.
The body for the "assword" pattern does not contain exp_continue. After you send the password, the enclosing expect command will return, and the next command is to expect your prompt.
Note I removed the -re option: there are no regex-special characters in that pattern.
I recently updated and reformatted my /etc/hosts file and would like to run through every system (roughly 1000) to refresh my known_hosts file. I'm looking at using an expect script to send "yes" to the RSA fingerprint question. Simple enough. However, I know some of these systems are completely new to me and my password has not been set. This creates two possibilities:
"yes" is sent to the RSA fingerprint question and I'm logged into
the server. I'll then need to send an exit to close the connection
before moving onto the next host. Or...
"yes" is sent to the RSA fingerprint question and I'm presented with
the prompts to update my password starting with the current and
followed by the new password entered twice. The connection will
automatically close after the password is updated moving onto the
next host.
I think I have a basic grasp of the concept of "if/else" in expect, but I don't fully understand how to nest them, if there is a better way, or if I'm completely off-base to begin with.
This is what I have right now:
set file1 [open [lindex $argv 0] r]
set pw1 [exec cat /home/user/.pw1.txt]
set pw2 [exec cat /home/user/.pw2.txt]
while {[gets $file1 host] != -1} {
puts $host
spawn -noecho "ssh $host"
expect {
"continue connecting"{
send "yes\r"
expect {
"current" {
send $pw2\r
} "New password" {
send $pw1\r
} "Retype new password" {
send $pw1\r
}
}
expect "msnyder"
send "exit\r"
}
interact
}
The file1 variable is the list of hosts to run the script against.
I know it isn't accurate because it errors on line 22. But, I have no idea what needs to be fixed.
Two errors I spotted:
missing close brace, probably for the "continue connecting" block
missing space before the open brace of "continue connecting". Tcl (hence Expect) is very sensitive to whitespace as it is parsed into words before the commands are evaluated. For the very few gory details, see the 12 syntax rules of Tcl.
Your code might look like:
while {[gets $file1 host] != -1} {
puts $host
spawn -noecho "ssh $host"
expect {
"continue connecting" {
send "yes\r"
expect {
"current" {
send -- $pw2\r
exp_continue
}
"New password" {
send -- $pw1\r
exp_continue
}
"Retype new password" {
send -- $pw1\r
exp_continue
}
msnyder
}
send "exit\r"
}
}
interact
}
Notes:
exp_continue is used to "loop" back up to the expect statement: in this case, you will expect to see all of "current", "new" and "retype", so you don't want to bail out until you see your prompt.
get into the habit of typing send -- something. Without the double dash, you'll be surprised the day someone types in a password with a leading dash.
You can probably avoid having the script run like a human and just spawn an expected to fail ssh connection which will automatically accept the host RSA key and not bother with prompting for a password; you can do that later for new systems where you need to initiate a password.
Temporarily try adding this to your ~/.ssh/config file until your script is finished:
Host *
Protocol 2
PasswordAuthentication 0
StrictHostKeyChecking 'no'
CheckHostIP 'no'
UserKnownHostsFile ~/.ssh/known_hosts_new
Then when the new know_hosts_new file is loaded up, you can replace the default ~/.ssh/known_hosts and remove the UserKnownHostsFile line from your config.
This question already has an answer here:
Best way to soft brute-force your own GPG/PGP passphrase?
(1 answer)
Closed 8 years ago.
I have forgotten my passphrase for my gpg key on linux. Can someone please help me write a simple script to use bruteforce to crack the key? I remember some of the words which MIGHT be in the passphrase, so hopefully, it will not take long for my computer to bruteforce it.
All is not lost if I can't recover the passphrase, it just means I will not be able to work on my project for the next 10 days until I get back to work to get another copy of the files, but this time with a new key for which I will remember to passphrase.
However, it will be nice to be able to work on my project in these 10 days.
Maybe something like:
#!/bin/bash
#
# try all word in words.txt
for word in $(cat words.txt); do
# try to decrypt with word
echo "${word}" | gpg --passphrase-fd 0 --no-tty --decrypt somegpgfile.gpg --output somegpgfile;
# if decrypt is successfull; stop
if [ $? -eq 0 ]; then
echo "GPG passphrase is: ${word}";
exit 0;
fi
done;
exit 1;
1) The script won't be simple, at least how you envisage "simple."
2) It will take a long time - that's the point of using pass phrases over simple passwords. Taking the time to write such a script, incorporating your words which may or may not be in the phrase plus a stab at iterating will probably take over ten days.
3) You probably will forget the next passphrase too.
4) Ooops!
Sorry dude, time to start a new project (at least to while away the next ten days - I suggest a passphrase cracker as an ideal distraction.)
Merry Christmas!
-Oisin
Tersmitten's answer may be out of date.
echo "${word}" | gpg --passphrase-fd 0 -q --batch --allow-multiple-messages --no-tty --output the_decrypted_file -d /some/input/file.gpg;
I used the above line with gpg 2.0.20 and libcrypt 1.5.2 to achieve the desired results.
I have a client and server system that regularly run scheduled tasks and communicate through xml files that have been encrypted by gpg. All required public keys have been successfully exchanged between the client and server. The encryption and decryption calls are being done from a batch file.
encrypt syntax
gpg.exe --batch --yes --recipient %1 --output %4 --passphrase %5 --local-user %2 --sign --encrypt %3
decrypt syntax
gpg.exe --batch --yes --output %3 --passphrase %4 --decrypt %2 2>%1
The client creates a xml file, encrypts it with gpg using server public key, signs with private key and uploads it to the server's ftp site. Server regularly checks for new files in ftp folder. For any new file it decrypts using gpg and then processes the xml inside the file.
For some of the xml files that the server tries to decrypt, I receive an error as follows:
gpg: block_filter 00AA8400: read error (size=7841,a->size=395)
gpg: mdc_packet with invalid encoding
gpg: decryption failed: invalid packet
gpg: block_filter: pending bytes!
The point to note is that this is not happening with all the files but with only some files. I haven't been able to find any commonality between the files that it fails on.
Is anyone familiar to what this error means? any suggestions to help track this down are welcome.
Finally figured it out. gpg was not the culprit here. when the server was checking for files in the specified folder, it was using the Append(fileHandler) method on Delphi to test if the file could be opened. But this method had a peculiar condition as if it found the ascii character 26 (i.e. CTRL+z) in the last 128 byte block of the file, it would remove everything from that character till the end of the file. This caused some part of the encrypted files to be deleted and subsequently caused the above error when decrypting it through gpg. After I replaced the Append method with Reset(fileHandler), encrypted files were no longer modified and decryption works perfectly.
I'm using GNUPG to encrypt my ascii files.
I learnt to generate a key, also how to use the it to encrypt and decrypt a file.
There are two ways I used:
gpg -d foo.txt.gpg
and
gpg --output foo.txt --decrypt
foo.txt.gpg
I realized the first method will display the decrypted file on the screen, for example when I executed the command over SSH.
With regard to the second method, I concerned if it will leave a trace on the local pc - the
foo.txt file.
Most importantly, I don't know how to edit the contents of the foo file on the fly.
Ideally, I would like to open the file over SSH use nano/pico, type my passphrase to decrypt, then edit the file, save it and encrypt it. I very much like to avoid save any files to the local disk.
Any comments are welcome.
Thank you in advance.
One way is using vim. See this page and this related question.
If you need more flexibility or don't want to use vim, writing a short program to read the decrypted text coming from STDOUT, edit to your liking, and then re-encrypt isn't too difficult. For example, you could use this minimal Python code (104 lines!) to give you the bare bones editor, and then add the stream reading and writing functionality yourself.
One thing to bear in mind is that holding unencrypted data in memory is no guarantee that it wont find its way to disk. If the system in question is under heavy load any unencrypted data may be written to the swap partition. Similarly, if the system is put into sleep mode, the state of any suspended processes will be stored to disk. If your program is running on a embedded system, it's conceivable that your memory and "disk" are one and the same.
The mlock() system call will protect allocated memory from getting swapped to disk. However, this requires administrative privileges and limits you to a low-level language where you are directly responsible for memory management.
That said, it is prudent to avoid creating files with unencrypted data. Just know that this doesn't offer you 100% safety if the underlying system is compromised.
I wrote a python script to solve this (for Linux only). It works by decrypting the file into /dev/shm to ensure that the unencrypted data is never written to disk (although it is possible for any of the programs using the data to be swapped to disk; this is almost always a concern).
This has some benefits over some of the other posted answers:
Only need to type the password once
Works with any editor
Here is the code:
#!/usr/bin/python
import os, sys, subprocess, getpass, stat, shutil
editor = 'nano'
dataFile = sys.argv[1]
## make a backup of the encrypted file
bakFile = dataFile+'-gpgedit_backup'
shutil.copy(dataFile, bakFile)
dstat = os.stat(dataFile)
## create temporary directory in tmpfs to work from
tmpDir = '/dev/shm/gpgedit'
n = 0
while True:
try:
os.mkdir(tmpDir+str(n))
break
except OSError as err:
if err.errno != 17: ## file already exists
raise
n += 1
tmpDir += str(n)
os.chmod(tmpDir, stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR)
try:
## Get password
passwd = getpass.getpass()
## decrypt file
tmpFile = os.path.join(tmpDir, 'data')
cmd = "gpg -d --passphrase-fd 0 --output %s %s" % (tmpFile, dataFile)
proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(passwd)
proc.stdin.close()
if proc.wait() != 0:
raise Exception("Error decrypting file.")
## record stats of tmp file
stat = os.stat(tmpFile)
## invoke editor
os.system('%s %s' % (editor, tmpFile))
## see whether data has changed
stat2 = os.stat(tmpFile)
if stat.st_mtime == stat2.st_mtime and stat.st_size == stat2.st_size:
raise Exception("Data unchanged; not writing encrypted file.")
## re-encrypt, write back to original file
cmd = "gpg --yes --symmetric --passphrase-fd 0 --output %s %s" % (dataFile, tmpFile)
proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(passwd)
proc.stdin.close()
if proc.wait() != 0:
raise Exception("Error encrypting file.")
except:
## If there was an error AND the data file was modified, restore the backup.
dstat2 = os.stat(dataFile)
if dstat.st_mtime != dstat2.st_mtime or dstat.st_size != dstat2.st_size:
print "Error occurred, restored encrypted file from backup."
shutil.copy(bakFile, dataFile)
raise
finally:
shutil.rmtree(tmpDir)
os.remove(bakFile)
Where is gnupg plugin - exactly for this point for example
Inspired by Luke's answer, I wrote a Python script myself. Hopefully, somebody will find this useful. Here are the core features:
uses temporary file under /dev/shm using a secure method to generate tempfile
creates backup file in case of failures
both encryption modes (public key/symmetric)
create a new file on-the-fly
choose your editor through environment variables
Further information can be found in the script itself. It currently won't work on any non *nix-machine.
To install the script just put it in any directory on your path and make it executable.
Get it now!
Warning: Backup your data! The script comes without any warranty!
I have spent countless hours on this quest, too: simply encrypt a text file with a passphrase with simple open+read/write access. I didn't want to deal with private/public keys nor keyrings bound to an OS login, blah, blah, blah. File encryption with passphrase only is so simple and so universal and perfect for a simple text file to hold passwords. No bloat nor complication of a database-driven solution like KeePass, etc. (which also requires data entry into multiple GUI elements rather than just typing your passwords in a searchable text file). The gold standard on Windows is Steganos LockNote. How to do it on Linux? Surprisingly very difficult to find, but...
I finally converged on the recommendation I consider best: cream. http://cream.sourceforge.net/ Cream is a facade to vim to make it more-user-friendly ... useful for other family members (I am a Linux geek at work comfortable with vi[m], but I needed something more accessible for my family).
Just enter:
"vim -x yourfile.txt"
It will be saved as encrypted with a passphrase.
You can use vim or cream at this point:
"vim yourfile.txt" or "cream yourfile.txt".
Either one will natively open "yourfile.txt" and prompt for the passphrase and transparently allow edits and re-saving as encrypted. FINALLY the quest has been completed !!!!
An alternative is to have a tmp filesystem in ram using tmpfs then when you power off it's gone for ever.
If your editor can read input from a pipe, and save to a pipe, then you can actually use the version of gpg that decrypts to stdout and encrypts from stdin. Unfortunately, for nano, reading from a pipe is only planned for 2.4. E.g. for gvim, you can bind decryption and encryption (through pipes) to a key.
To open gpg files, editing them and then ecrypt/save again use:
kgpg
icon in systray has option: Editor...
Press on it, then open the gpg file, then on the bottom there is a button to decrypt it and voila you have your file in the editor, after you made any changes just press Encrypt and then save it.
Just today I have found out about a way of doing all that in vim!
here is the link: full howto on setting up vim for gpg files
works like a charm, just in that tutorial, the link to the plugin is url to a page so not to wget it, but go to the page and select the one you want to download.
I detest vi, so i had to make up some glue around nano. This is what i came up with. Downside is that you have to enter password again when encrypting.
alias file_ed="gpg file.txt.gpg; nano file.txt; gpg -c --force-mdc -o file.txt.gpg_temp file.txt; mv file.txt.gpg_temp file.txt.gpg; rm file.txt"
It isn't very secure from the filesystem point of view, but I fear other users and myself, not root.
viencrypt by Paul Tarjan is a script for editing GPG encrypted files on the fly.
Using the editor joe ( aka Joe's Own Editor ) in a command similar to
gpg --decrypt foo.txt.gpg | joe - | gpg --armor --recipient name#example.com --encrypt > bar.txt.gpg
will do what you're looking for.
The - in joe - tells joe to take its input from stdin and to write its output to stdout when you save the file (hit ctrl+k and then x to save). Joe will initially display some crufty output from gpg; this can be cleared by hitting ctrl+r to refresh the screen.
I use > bar.txt.gpg to specify the output file instead of --output bar.txt.gpg because the --output flag causes gpg to open an interactive dialogue if you're overwriting the output file, and this confuses joe.
Here is a slight improvement to #Luke's answer. It makes two small improvements:
It avoids the stack trace if the file is unmodified during the edit session.
It restores the original gpg file if re-encryption back to the original gpg file was attempted, which is a little safer than checking the modification dates of the edit file.
#!/usr/bin/python
# Downloaded from https://stackoverflow.com/questions/1510105/gnupg-how-to-edit-the-file-without-decrypt-and-save-to-local-disk-first/12289967#12289967
# and then slightly improved.
import os, sys, subprocess, getpass, stat, shutil
editor = 'nano'
dataFile = sys.argv[1]
## make a backup of the encrypted file
bakFile = dataFile+'-gpgedit_backup'
shutil.copy(dataFile, bakFile)
dstat = os.stat(dataFile)
## create temporary directory in tmpfs to work from
tmpDir = '/dev/shm/gpgedit'
n = 0
while True:
try:
os.mkdir(tmpDir+str(n))
break
except OSError as err:
if err.errno != 17: ## file already exists
raise
n += 1
tmpDir += str(n)
os.chmod(tmpDir, stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR)
reEncrypted = False
try:
## Get password
passwd = getpass.getpass()
## decrypt file
tmpFile = os.path.join(tmpDir, 'data')
cmd = "gpg -d --cipher-algo AES256 --passphrase-fd 0 --output %s %s" % (tmpFile, dataFile)
proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(passwd)
proc.stdin.close()
if proc.wait() != 0:
raise Exception("Error decrypting file.")
## record stats of tmp file
stat = os.stat(tmpFile)
## invoke editor
os.system('%s %s' % (editor, tmpFile))
## see whether data has changed
stat2 = os.stat(tmpFile)
if stat.st_mtime == stat2.st_mtime and stat.st_size == stat2.st_size:
print "Data unchanged; not re-writing encrypted file."
else:
## re-encrypt, write back to original file
reEncrypted = True
cmd = "gpg --yes --symmetric --cipher-algo AES256 --passphrase-fd 0 --output %s %s" % (dataFile, tmpFile)
proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(passwd)
proc.stdin.close()
if proc.wait() != 0:
raise Exception("Error encrypting file.")
except:
## If there was an error AND re-encryption was attempted, restore the backup.
if reEncrypted:
print "Error occurred; restoring encrypted file from backup."
shutil.copy(bakFile, dataFile)
raise
finally:
shutil.rmtree(tmpDir)
os.remove(bakFile)
I would have posted these suggested improvements as comments to #Luke's answer -- which I like a lot -- but did not have enough reputation points to do so. :(
I wrote a shell script to edit files encrypted with gpg.
Call it like ./editgpg.sh path/to/.secrets.gpg, edit with vim then close with ':q!'.
Vim flag '-n' opens file only in memory. No temp files.
#!/usr/bin/env bash
# decrypt CRYPT_FILE, pipe to vim, and encrypt againg whent type ':q!'
edit_crypt_file() {
echo "Enter your gpg encrypted file passphrase,
edit it with vim, then close the editor with ':q!'."
# first argument is a file encrypted with gpg
CRYPT_FILE=$1
# get password user input
local pass
read -sp "Password:" pass
gpg_flags='--batch --yes'
vim_flags='- -n -u NONE --not-a-term'
# vim command to run before exit with ':q!'
vim_autocmd=":autocmd VimLeave * :%! tee | gpg $gpg_flags --passphrase $pass -o $CRYPT_FILE -c"
gpg $gpg_flags --passphrase $pass -d $CRYPT_FILE | vim $vim_flags -c "$vim_autocmd"
# restart agent in order to lose kept password
gpgconf --kill gpg-agent
unset pass
}
edit_crypt_file $1