I have a bash script, which runs correctly in my system:
uname -a
Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux
But I need it to work in a Redhat 7.2 chroot:
Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 unknown
The same code executes correctly on the first one, but when running it on 7.2 first it doesn't recognize sed -i (just the -i argument). Commenting some lines y run into another problem:
bash: command substitution: line 1: syntax error near unexpected token `<<<'
The thing is, that this script need to be executed in a remote machine with a debian 7.2 (that's why i'm testing it in a chroot with the same distro), so it's no solution to install modules/upgrades to make it runnable.
A sample code:
#!/bin/bash
...
count=0
while read line; do
if echo "$line" | grep -q ')'
then
((count++))
comas=`grep -o "," <<< "$line" | wc -l`
num=`grep -o "byRef" <<< "$line" | wc -l`
...
sed -i 's/shortInteger/int/g' "test.h"
Any ideas?
Thanks.
Edit:
These are the commands that cause me trouble:
comas=`grep -o "," <<< "$line" | wc -l`
sed -i 's/shortInteger/int/g' "test.h"
Edit 2:
GNU bash, version 2.05.8 --> "here string" (<<<) doesn't exist
grep (GNU grep) 2.4.2 --> -o option doesn't exist
GNU sed version 3.02 --> -i option doesn't exist
Finally I got it with the following...
For the versions i have which i can't update:
sed solution
sed 's/shortInteger/int/g' "test.h" > test.temp.h;
mv "test.temp.h" "test.h"
This was pointed out by #sorontar. Thanks a lot!
To find special characters or substrings inside a string
comas=$(echo "$line" | tr " " "\n" | grep -c ",")
According to my source files, there's a pattern so after any comma I have an space. So tr " " "\n" separates substrings between spaces making newlines, then I can use grep -c "," to count each line with a comma inside.
I've successfully used the following sed command to search/replace text in Linux:
sed -i 's/old_link/new_link/g' *
However, when I try it on my Mac OS X, I get:
"command c expects \ followed by text"
I thought my Mac runs a normal BASH shell. What's up?
EDIT:
According to #High Performance, this is due to Mac sed being of a different (BSD) flavor, so my question would therefore be how do I replicate this command in BSD sed?
EDIT:
Here is an actual example that causes this:
sed -i 's/hello/gbye/g' *
If you use the -i option you need to provide an extension for your backups.
If you have:
File1.txt
File2.cfg
The command (note the lack of space between -i and '' and the -e to make it work on new versions of Mac and on GNU):
sed -i'.original' -e 's/old_link/new_link/g' *
Create 2 backup files like:
File1.txt.original
File2.cfg.original
There is no portable way to avoid making backup files because it is impossible to find a mix of sed commands that works on all cases:
sed -i -e ... - does not work on OS X as it creates -e backups
sed -i'' -e ... - does not work on OS X 10.6 but works on 10.9+
sed -i '' -e ... - not working on GNU
Note Given that there isn't a sed command working on all platforms, you can try to use another command to achieve the same result.
E.g., perl -i -pe's/old_link/new_link/g' *
I believe on OS X when you use -i an extension for the backup files is required. Try:
sed -i .bak 's/hello/gbye/g' *
Using GNU sed the extension is optional.
This works with both GNU and BSD versions of sed:
sed -i'' -e 's/old_link/new_link/g' *
or with backup:
sed -i'.bak' -e 's/old_link/new_link/g' *
Note missing space after -i option! (Necessary for GNU sed)
Had the same problem in Mac and solved it with brew:
brew install gnu-sed
and use as
gsed SED_COMMAND
you can set as well set sed as alias to gsed (if you want):
alias sed=gsed
Or, you can install the GNU version of sed in your Mac, called gsed, and use it using the standard Linux syntax.
For that, install gsed using ports (if you don't have it, get it at http://www.macports.org/) by running sudo port install gsed. Then, you can run sed -i 's/old_link/new_link/g' *
Your Mac does indeed run a BASH shell, but this is more a question of which implementation of sed you are dealing with. On a Mac sed comes from BSD and is subtly different from the sed you might find on a typical Linux box. I suggest you man sed.
Insead of calling sed with sed, I do ./bin/sed
And this is the wrapper script in my ~/project/bin/sed
#!/bin/bash
if [[ "$OSTYPE" == "darwin"* ]]; then
exec "gsed" "$#"
else
exec "sed" "$#"
fi
Don't forget to chmod 755 the wrapper script.
Sinetris' answer is right, but I use this with find command to be more specific about what files I want to change. In general this should work (tested on osx /bin/bash):
find . -name "*.smth" -exec sed -i '' 's/text1/text2/g' {} \;
In general when using sed without find in complex projects is less efficient.
I've created a function to handle sed difference between MacOS (tested on MacOS 10.12) and other OS:
OS=`uname`
# $(replace_in_file pattern file)
function replace_in_file() {
if [ "$OS" = 'Darwin' ]; then
# for MacOS
sed -i '' -e "$1" "$2"
else
# for Linux and Windows
sed -i'' -e "$1" "$2"
fi
}
Usage:
$(replace_in_file 's,MASTER_HOST.*,MASTER_HOST='"$MASTER_IP"',' "./mysql/.env")
Where:
, is a delimeter
's,MASTER_HOST.*,MASTER_HOST='"$MASTER_IP"',' is pattern
"./mysql/.env" is path to file
As the other answers indicate, there is not a way to use sed portably across OS X and Linux without making backup files. So, I instead used this Ruby one-liner to do so:
ruby -pi -e "sub(/ $/, '')" ./config/locales/*.yml
In my case, I needed to call it from a rake task (i.e., inside a Ruby script), so I used this additional level of quoting:
sh %q{ruby -pi -e "sub(/ $/, '')" ./config/locales/*.yml}
Here's how to apply environment variables to template file (no backup need).
1. Create template with {{FOO}} for later replace.
echo "Hello {{FOO}}" > foo.conf.tmpl
2. Replace {{FOO}} with FOO variable and output to new foo.conf file
FOO="world" && sed -e "s/{{FOO}}/$FOO/g" foo.conf.tmpl > foo.conf
Working both macOS 10.12.4 and Ubuntu 14.04.5
Here is an option in bash scripts:
#!/bin/bash
GO_OS=${GO_OS:-"linux"}
function detect_os {
# Detect the OS name
case "$(uname -s)" in
Darwin)
host_os=darwin
;;
Linux)
host_os=linux
;;
*)
echo "Unsupported host OS. Must be Linux or Mac OS X." >&2
exit 1
;;
esac
GO_OS="${host_os}"
}
detect_os
if [ "${GO_OS}" == "darwin" ]; then
sed -i '' -e ...
else
sed -i -e ...
fi
sed -ie 's/old_link/new_link/g' *
Works on both BSD & Linux with gnu sed
Please share more ideas to get software version from bash command and use it as variable later.
su --version
su (GNU coreutils) 5.97
Copyright etc.
and create variable of the result of it.
Something like I tried below.
su --version >/tmp/temp.txt
if [ -f /tmp/temp.txt ]; then
elv=`cat /tmp/temp.txt | gawk 'BEGIN {FS="(GNU coreutils)"} {print $2}' | gawk 'BEGIN {FS="."} {print $1}'`
#Version String. Just a shortcut to be used later
els=el$elv
else
echo "Unable to determine version. I can't continue"
exit 1
fi
if [ `rpm -qa | egrep -c -i "^mysql-"` -gt 0 ]; then
cat << EOF
It appears that the distro-supplied version of MySQL is at least partially installed,
or a prior installation attempt failed.
Please remove these packages, as well as their dependencies (often postfix), and then
retry this script:
$(rpm -qa | egrep -i "^mysql-")
EOF
exit 1
fi
trying to setup a bash script to look for the version in a binary file all the versions are included so I am thinking I have to strip out the AUDIT_TRAIL_#_##_A-Z,1-9
This is what I have so far any suggestions
#!/bin/bash
echo searching fmx files AUDIT_TRAIL FOR FORM VERSION
for file in `/bin/ls *.fmx`
do
current_release=`strings $file |sed "s/\"AUDIT_TRAIL/NOTNEEDED/" |grep -i "AUDIT_TRAIL_8" |sed "s/AUDIT\_TRAIL\_/AUDIT\-TRAIL /" |sed "s/\_/\./g" |sed "s/\.[A-Z,a-z]/\.00/" |awk '{print substr($0,0,18)}' |awk '{print $2}'`
export form=$(echo "$file" | cut -f1 -d'.')
export dbform=`echo $form |awk '{print toupper($0)}'`
echo "FORMNAME" $form
sqlplus -s /nolog <<EOF
connect system/xxx#xxxxx
set echo on
whenever oserror exit 88
whenever sqlerror exit 1
spool forms.lst
select GURAOBJ_CURRENT_VERSION from bansecr.GURAOBJ where GURAOBJ_OBJECT = '$dbform';
spool off
exit
EOF
echo $file $current_release
done
OUTPUT
bash-4.1$ ./find_current_release_fmx_db.shl
+ ./find_current_release_fmx_db.shl
./find_current_release_fmx_db.shl: line 1: !/bin/bash: No such file or directory
searching fmx files AUDIT_TRAIL FOR FORM VERSION
FORMNAME peaempl
FROM THE DATABASE
GURAOBJ_CU
----------
8.11.2
peaempl.fmx
FROM THE COMPILED FORM
8.0 8.0 8.0.00 8.0.00 8.1.0. 8.1.0. 8.2.00 8.2.00 8.3 8.3 8.4 8.4 8.7.1 8.7.1 8.7.1. 8.7.1. 8.8.0. 8.8.0. 8.8.1. 8.8.1. **THE ONE I NEED 8.11.2** 8.11.2 8.1.0. 8.8.0. 8.7.1. 8.11.2 8.0.00 8.2.00 8.0 8.3 8.4 8.8.1. 8.7.1
DESIRED OUTPUT
8.11.2
Any help would be greatly appreciated
Let's start by tidying up your script to not use so many pipes and commands, to be more robust, etc.:
#!/bin/bash
printf 'searching fmx files AUDIT_TRAIL FOR FORM VERSION\n'
for file in *.fmx
do
current_release=$(strings "$file" | awk '<something>')
form="${file%%.}"
dbform="${form^^}"
printf 'FORMNAME %s\n' "$form"
sqlplus -s /nolog <<EOF
connect system/xxx#xxxxx
set echo on
whenever oserror exit 88
whenever sqlerror exit 1
spool forms.lst
select GURAOBJ_CURRENT_VERSION from bansecr.GURAOBJ where GURAOBJ_OBJECT = '$dbform';
spool off
exit
EOF
printf '%s %s\n' "$file" "$current_release"
done
Now all we need to know is what the <something> in the awk command should be. I couldn't figure it out from your chain of piped commands as they seem to be adding stuff and then removing it again and escaping things that shouldn't need to be escaped, etc. but once you show the sample output of strings and what you want the awk script to convert that into it should be obvious.
I wrote a linux bash script with tee and grep to log and timestamp the actions I take in my various ssh sessions. It works, but the logged lines are mixed together sometimes and are full of control characters. How can I properly escape control and other characters not visible in the original sessions and log each line separately?
I am learning bash and the linux interface, so any other suggestions to improve the script would be extremely welcome!
Here is my script (used as a wrapper for the ssh command):
#! /bin/bash
logfile=~/logs/ssh.log
desc="sshlog ${#}"
tab="\t"
format_line() {
while IFS= read -r line; do
echo -e "$(date +"%Y-%m-%d %H:%M:%S %z")${tab}${desc}${tab}${line}"
done
}
echo "[START]" | format_line >> ${logfile}
# grep is used to filter out command line output while keeping commands
ssh "$#" | tee >(grep -e '\#.*\:.*\$' --color=never --line-buffered | format_line >> ${logfile})
echo "[END]" | format_line >> ${logfile}
And here is a screenshot of the jarbled output in the log file:
A note on the solution: Tiago's answer took care of the nonprinting characters very well. Unfortunately, I just realized that the jumbling is being caused by backspaces and using the up and down keys for command completion. That is, the characters are being piped to grep as soon as they appear, and not line-by-line. I will have to ask about this in another question.
Update: I figured out a way to (almost always) handle up/down completion, backspace completion, and control characters.
You can remove those characters with:
perl -lpe 's/[^[:print:]]//g'
Not filtered:
perl -e 'for($i=0; $i<=255; $i++){print chr($i);}' | cat -A
^#^A^B^C^D^E^F^G^H^I$
^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_ !"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~^?M-^#M-^AM-^BM-^CM-^DM-^EM-^FM-^GM-^HM-^IM-^JM-^KM-^LM-^MM-^NM-^OM-^PM-^QM-^RM-^SM-^TM-^UM-^VM-^WM-^XM-^YM-^ZM-^[M-^\M-^]M-^^M-^_M- M-!M-"M-#M-$M-%M-&M-'M-(M-)M-*M-+M-,M--M-.M-/M-0M-1M-2M-3M-4M-5M-6M-7M-8M-9M-:M-;M-<M-=M->M-?M-#M-AM-BM-CM-DM-EM-FM-GM-HM-IM-JM-KM-LM-MM-NM-OM-PM-QM-RM-SM-TM-UM-VM-WM-XM-YM-ZM-[M-\M-]M-^M-_M-`M-aM-bM-cM-dM-eM-fM-gM-hM-iM-jM-kM-lM-mM-nM-oM-pM-qM-rM-sM-tM-uM-vM-wM-xM-yM-zM-{M-|M-}M-~M-^?
Filtered:
perl -e 'for($i=0; $i<=255; $i++){print chr($i);}' | perl -lpe 's/[^[:print:]]//g' | cat -A
$
!"#$%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~$
Explanation:
I am printing the whole ASCII table with:
perl -e 'for($i=0; $i<=255; $i++){print chr($i);}'
I am identifying non printable chars with:
cat -A
I am filtering non printable chars with:
perl -lpe 's/[^[:print:]]//g'
Edit: It seems to me that you need to remove ANSI color chars:
Example:
perl -MTerm::ANSIColor -e 'print colored("yellow on_magenta","yellow on_magenta"),"\n"'| sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" | perl -lpe 's/[^[:print:]]//g'
Adapting to your code:
format_line() {
while IFS= read -r line; do
line=$(sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" <<< "$line")
line=$(perl -lpe 's/[^[:print:]]//g' <<< "$line")
echo -e "$(date +"%Y-%m-%d %H:%M:%S %z")${tab}${desc}${tab}${line}"
done
}
I also edited your grep command:
ssh "$#" | tee >(grep -Po '(?<=\$).*' --color=never --line-buffered | format_line >> ${logfile})
Below the output of my test:
2014-06-26 10:11:10 +0100 sshlog tiago#localhost [START]
2014-06-26 10:11:15 +0100 sshlog tiago#localhost whoami
2014-06-26 10:11:16 +0100 sshlog tiago#localhost exit
2014-06-26 10:11:16 +0100 sshlog tiago#localhost [END]
While writing your own script is a great learning experience, you can also use script to record everything printed on your terminal to a file.
The resulting file will still contains the control characters but there are multiple ways to get rid of them as described in How to clean up output of linux 'script' command.