So I am creating a variable and I want to echo it with an addition to the end like so:
I have a file: Filename-08-10-2017.txt
I create a variable:
myvariable=Filename*.txt
When I echo that variable:
echo $myvariable
it outputs Filename-08-10-2017.txt
But I want to change the name to .zip
So I am trying to go:
echo $myvariable.zip and have it output Filename-08-10-2017.txt.zip
however it outputs:
Filename*.txt.zip
How do I go about having it output the way I want?
Thanks.
EDIT:
I kind of figured it out.
I saved a new variable as $($myvariable) which saved the output.
The file name comes from glob expansion. If you want to iterate all files by using glob expansion, you can :
myvariable=Filename*.txt
for f in $myvariable; do echo $f; done
If you want to "disable" the glob expansion, e.g. get literial Filename*.txt by echo $myvariable, you can either set -f or just wrap the variable by double quote: echo "$myvariable".
To maniuplate the text you can do something like:
for f in $myvariable; do echo $f"whatever_like_zip"; done
if you want to do some text substitution, you can
for ... echo ${f/%txt/zip} ; done
It will change all txt file name to zip.
Also if you want to rename the file, change the above echo... into
mv "$f" "yourNewNameHere"
Anyway by reading your question I'm not quite clear, what do you really want.
Yuo can use sed
try this
file="test.txt"
newext=$(echo "$file" | sed -e "s|txt|zip|g")
echo $newext
Related
I have the output file day by day:
linux-202105200900-foo.direct.tar.gz
The date and time string, ex: 202105200900 will change every day.
I need to manually rename these files to
linux-202105200900x86-foo.direct.tar.gz
( insert a short string x86 after date/time )
any bash script can help to do this?
If you're always inserting the string "x86" at character #18 in the string, you may use that command:
var="linux-202105200900-foo.direct.tar.gz"
var2=${var:0:18}"x86"${var:18}
echo $var2
The 2nd line means: "assign to variable var2 the first 18 characters of var, followed by x86 followed by the rest of the variable var"
If you want to insert "x86" just before the last hyphen in the string, you may write it like this:
var="linux-202105200900-foo.direct.tar.gz"
var2=${var%-*}"x86-"${var##*-}
echo $var2
The 2nd line means: "assign to variable var2:
the content of the variable var after removing the shortest matching pattern "-*" at the end
the string "x86-"
the content of the variable var after removing the longest matching pattern "*-" at the beginning
In addition to the very good answer by #Jean-Loup Sabatier another, perhaps more general way would simply be to replace the second occurrence of '-' with x86- which you can do with sed. Let's say you have:
fname=linux-202105200900-foo.direct.tar.gz
You can update that with:
fname="$(sed 's/-/x86-/2' <<< "$fname")"
Which simply uses a command substitution with sed and a herestring to modify fname assigning the modified result back to fname.
Example Use/Output
$ fname=linux-202105200900-foo.direct.tar.gz
fname="$(sed 's/-/x86-/2' <<< "$fname")"
echo $fname
linux-202105200900x86-foo.direct.tar.gz
Do you need this?
❯ dat=$(date '+%Y%m%d%H%M%S'); echo ${dat}
20210520170336
❯ filename="linux-${dat}x86-foo.direct.tar.gz"; echo ${filename}
linux-20210520170336x86-foo.direct.tar.gz
I wanted to go as simple as possible, considering only the timestamp is going to change, this script should do it. Just run it inside the folder where files are located and you'll get all of them renamed with x86.
#!/bin/bash
for file in $(ls); do
replaced=$(echo $file | sed 's|-foo|x86-foo|g')
mv $file $replaced
done
This is my output
filip#filip-ThinkPad-T14-Gen-1:~/test$ ls
linux-202105200900-foo.direct.tar.gz linux-202105201000-foo.direct.tar.gz linux-202105201100-foo.direct.tar.gz
filip#filip-ThinkPad-T14-Gen-1:~/test$ ./../development/bash-utils/bulk-rename.sh
filip#filip-ThinkPad-T14-Gen-1:~/test$ ls
linux-202105200900x86-foo.direct.tar.gz linux-202105201000x86-foo.direct.tar.gz linux-202105201100x86-foo.direct.tar.gz
Simply iterate through all the files in current folder and pipeline result to sed to replace regex -foo with x86-foo, then rename file with mv command.
As David mentioned in comment, if you're worried that there could be multiple occurrences of -foo then you can just replace g as global to 1 as first occurrence and that's it!
There is also the rename utility (https://man7.org/linux/man-pages/man1/rename.1.html), you could use:
rename -v 0-foo.direct.tar.gz 0x86-foo.direct.tar.gz *
which results in
`linux-202105200900-foo.direct.tar.gz' -> `linux-202105200900x86-foo.direct.tar.gz'
`linux-202205200900-foo.direct.tar.gz' -> `linux-202205200900x86-foo.direct.tar.gz'
`linux-202305200900-foo.direct.tar.gz' -> `linux-202305200900x86-foo.direct.tar.gz'
In addition to the very good answer by #David C. Rankin, just adding it in a loop and renaming the files
# !/usr/bin/bash
for file in `ls linux* 2>/dev/null` # Extract all files starting with linux
do
echo $file
fname="$(sed 's/-/x86-/2' <<< "$file")"
mv "$file" "$fname" # Rename file
done
Output recieved :
linux-202105200900x86-foo.direct.tar.gz
deploy.sh
USERNAME="Tom"
PASSWORD="abc123"
FILE="config.conf"
sed -i "s/\PLACEHOLDER_USERNAME/$USERNAME/g" $FILE
sed -i "s/\PLACEHOLDER_PASSWORD/$PASSWORD/g" $FILE
config.conf
deloy="PLACEHOLDER_USERNAME"
pass="PLACEHOLDER_PASSWORD"
This file puts my variables defined in deploy into my config file. I can't source the file so I want put my variables in this way.
Question
I want a command that is generic to work for all placeholder variables using some sort of while loop rather than needing one command per variable. This means any term starting with placeholder_ in the file will try to be replaced with the value of the variable defined already in deploy.sh
All variables should be set and not empty. I guess if there is the ability to print a warning if it can't find the variable that would be good but it isn't mandatory for this.
Basically, use shell code to write a sed script and then use sed -i .bak -f sed.script config.conf to apply it:
trap "rm -f sed.script; exit 1" 0 1 2 3 13 15
for var in USERNAME PASSWORD
do
echo "s/PLACEHOLDER_$var/${!var}/"
done > sed.script
sed -i .bak -f sed.script config.conf
rm -f sed.script
trap 0
The main 'tricks' here are:
knowing that ${!var} expands to the value of the variable named by $var, and
knowing that sed will take a script full of commands via -f sed.script, and
knowing how to use trap to ensure temporary files are cleaned up.
You could also use sed -e "s/.../.../" -e "s/.../.../" -i .bak config.conf too, but the script file is easier, I think, especially if you have more than 2 values to substitute. If you want to go down this route, use a bash array to hold the arguments to sed. A more careful script would use at least $$ in the script file name, or use mktemp to create the temporary file.
Revised answer
The trouble is, although much closer to being generic, it is still not generic since I have to manually put in what variables I want to change. Can it not be more like "for each placeholder_, find the variable in deploy.sh and add that variable, so it can work for any number of variables.
So, find what the variables are in the configuration file, then apply the techniques of the previous answer to solve that problem:
trap "rm -f $tmp; exit 1" 0 1 2 3 13 15
for file in "$#"
do
for var in $(sed 's/.*PLACEHOLDER_\([A-Z0-9_]*\).*/\1/' "$file")
do
value="${!var}"
[ -z "$value" ] && { echo "$0: variable $var not set for $file" >&2; exit 1; }
echo "s/PLACEHOLDER_$var/$value/"
done > $tmp
sed -i .bak -f $tmp "$file"
rm -f $tmp
done
trap 0
This code still pulls the values from the environment. You need to clarify what is required if you want to extract the settings from the shell script, but it can be done — the script will have to be sufficiently self-aware to find its source so it can search it for the names. But the basics are in this answer; the rest is a question of tinkering until it does what you need.
#!/bin/ksh
TemplateFile=$1
SourceData=$2
(sed 's/.*/#V0r:PLACEHOLDER_&:r0V#/' ${SourceData}; cat ${TemplateFile}) | sed -n "
s/$/²/
H
$ {
x
s/^\(\n *\)*//
# also reset t flag
t varxs
:varxs
s/^#V0r:\([a-zA-Z0-9_]\{1,\}\)=\([^²]*\):r0V#²\(\n.*\)\"\1\"/#V0r:\1=\2:r0V#²\3\2/
t varxs
# clean the line when no more occurance in text
s/^[^²]*:r0V#²\n//
# and next
t varxs
# clean the marker
s/²\(\n\)/\1/g
s/²$//
# display the result
p
}
"
call like this: YourScript.ksh YourTemplateFile YourDataSourceFile where:
YourTemplateFile is the file that contain the structure with generic value like deloy="PLACEHOLDER_USERNAME"
YourDataSourceFile is the file that contain all the peer Generic value = specific value like USERNAME="Tom"
I have a folder that is full of .bak files and some other files also. I need to remove the extension of all .bak files in that folder. How do I make a command which will accept a folder name and then remove the extension of all .bak files in that folder ?
Thanks.
To remove a string from the end of a BASH variable, use the ${var%ending} syntax. It's one of a number of string manipulations available to you in BASH.
Use it like this:
# Run in the same directory as the files
for FILENAME in *.bak; do mv "$FILENAME" "${FILENAME%.bak}"; done
That works nicely as a one-liner, but you could also wrap it as a script to work in an arbitrary directory:
# If we're passed a parameter, cd into that directory. Otherwise, do nothing.
if [ -n "$1" ]; then
cd "$1"
fi
for FILENAME in *.bak; do mv "$FILENAME" "${FILENAME%.bak}"; done
Note that while quoting your variables is almost always a good practice, the for FILENAME in *.bak is still dangerous if any of your filenames might contain spaces. Read David W.'s answer for a more-robust solution, and this document for alternative solutions.
There are several ways to remove file suffixes:
In BASH and Kornshell, you can use the environment variable filtering. Search for ${parameter%word} in the BASH manpage for complete information. Basically, # is a left filter and % is a right filter. You can remember this because # is to the left of %.
If you use a double filter (i.e. ## or %%, you are trying to filter on the biggest match. If you have a single filter (i.e. # or %, you are trying to filter on the smallest match.
What matches is filtered out and you get the rest of the string:
file="this/is/my/file/name.txt"
echo ${file#*/} #Matches is "this/` and will print out "is/my/file/name.txt"
echo ${file##*/} #Matches "this/is/my/file/" and will print out "name.txt"
echo ${file%/*} #Matches "/name.txt" and will print out "/this/is/my/file"
echo ${file%%/*} #Matches "/is/my/file/name.txt" and will print out "this"
Notice this is a glob match and not a regular expression match!. If you want to remove a file suffix:
file_sans_ext=${file%.*}
The .* will match on the period and all characters after it. Since it is a single %, it will match on the smallest glob on the right side of the string. If the filter can't match anything, it the same as your original string.
You can verify a file suffix with something like this:
if [ "${file}" != "${file%.bak}" ]
then
echo "$file is a type '.bak' file"
else
echo "$file is not a type '.bak' file"
fi
Or you could do this:
file_suffix=$(file##*.}
echo "My file is a file '.$file_suffix'"
Note that this will remove the period of the file extension.
Next, we will loop:
find . -name "*.bak" -print0 | while read -d $'\0' file
do
echo "mv '$file' '${file%.bak}'"
done | tee find.out
The find command finds the files you specify. The -print0 separates out the names of the files with a NUL symbol -- which is one of the few characters not allowed in a file name. The -d $\0means that your input separators are NUL symbols. See how nicely thefind -print0andread -d $'\0'` together?
You should almost never use the for file in $(*.bak) method. This will fail if the files have any white space in the name.
Notice that this command doesn't actually move any files. Instead, it produces a find.out file with a list of all the file renames. You should always do something like this when you do commands that operate on massive amounts of files just to be sure everything is fine.
Once you've determined that all the commands in find.out are correct, you can run it like a shell script:
$ bash find.out
rename .bak '' *.bak
(rename is in the util-linux package)
Caveat: there is no error checking:
#!/bin/bash
cd "$1"
for i in *.bak ; do mv -f "$i" "${i%%.bak}" ; done
You can always use the find command to get all the subdirectories
for FILENAME in `find . -name "*.bak"`; do mv --force "$FILENAME" "${FILENAME%.bak}"; done
I am trying to replace the prefix of all files in the directory with another prefix (renaming).
This is my script
# Script to rename the files
#!/bin/bash
for file in $1*;
do
mv $file `echo $file | sed -e 's/^$1/$2/'`;
done
Upon executing the script with
rename.sh BIT SIT
I get the following errors
mv: `BITfile.h' and `BITFile.h' are the same file
mv: `BITDefs.cpp' and `BITDefs.cpp' are the same file
mv: `BITDefs.h' and `BITDefs.h' are the same file
Seems like sed is treating $1 and $2 as the same value, but when I print those variables on another line it shows that they are different.
As Roman Newaza says, you can use " instead of ' to tell Bash that you want variables to be expanded. However, in your case, it would be safest to write:
for file in "$1"* ; do
mv -- "$file" "$2${file#$1}"
done
so that weird characters in filenames, or in your script parameters, cannot cause any problems.
You can also use parameter expansion to replace the prefix of all files in a directory.
for file in "$1"*;
do
mv ${file} ${file/#$1/$2}
done
Use double quotes instead:
# ...
mv "$file" `echo $file | sed -e "s/^$1/$2/"`
# ...
And learn Quotes and escaping in Bash.
When you don't use double quotes, the variables won't expand.
I will rather use this
#!/bin/bash
for file in $1*;
do
mv "$file" "$1${file:${#2}}"
done
where
${file:${#2}
means substring, from length of the argument 2 to the end
I would like to copy the contents of a variable (here called var) into a file.
The name of the file is stored in another variable destfile.
I'm having problems doing this. Here's what I've tried:
cp $var $destfile
I've also tried the same thing with the dd command... Obviously the shell thought that $var was referring to a directory and so told me that the directory could not be found.
How do I get around this?
Use the echo command:
var="text to append";
destdir=/some/directory/path/filename
if [ -f "$destdir" ]
then
echo "$var" > "$destdir"
fi
The if tests that $destdir represents a file.
The > appends the text after truncating the file. If you only want to append the text in $var to the file existing contents, then use >> instead:
echo "$var" >> "$destdir"
The cp command is used for copying files (to files), not for writing text to a file.
echo has the problem that if var contains something like -e, it will be interpreted as a flag. Another option is printf, but printf "$var" > "$destdir" will expand any escaped characters in the variable, so if the variable contains backslashes the file contents won't match. However, because printf only interprets backslashes as escapes in the format string, you can use the %s format specifier to store the exact variable contents to the destination file:
printf "%s" "$var" > "$destdir"
None of the answers above work if your variable:
starts with -e
starts with -n
starts with -E
contains a \ followed by an n
should not have an extra newline appended after it
and so they cannot be relied upon for arbitrary string contents.
In bash, you can use "here strings" as:
cat <<< "$var" > "$destdir"
As noted in the comment by Ash below, #Trebawa's answer (formulated in the same room as mine!) using printf is a better approach than cat.
All of the above work, but also have to work around a problem (escapes and special characters) that doesn't need to occur in the first place: Special characters when the variable is expanded by the shell. Just don't do that (variable expansion) in the first place. Use the variable directly, without expansion.
Also, if your variable contains a secret and you want to copy that secret into a file, you might want to not have expansion in the command line as tracing/command echo of the shell commands might reveal the secret. Means, all answers which use $var in the command line may have a potential security risk by exposing the variable contents to tracing and logging of the shell.
For variables that are already exported, use this:
printenv var >file
That means, in case of the OP question:
printenv var >"$destfile"
Note: variable names are case sensitive.
Warning: It is not a good idea to export a variable just for the sake of printing it with printenv. If you have a non-exported script variable that contains a secret, exporting it will expose it to all future child processes (unless unexported, for example using export -n).
If I understood you right, you want to copy $var in a file (if it's a string).
echo $var > $destdir
When you say "copy the contents of a variable", does that variable contain a file name, or does it contain a name of a file?
I'm assuming by your question that $var contains the contents you want to copy into the file:
$ echo "$var" > "$destdir"
This will echo the value of $var into a file called $destdir. Note the quotes. Very important to have "$var" enclosed in quotes. Also for "$destdir" if there's a space in the name. To append it:
$ echo "$var" >> "$destdir"
you may need to edit a conf file in a build process:
echo "db-url-host=$POSTGRESQL_HOST" >> my-service.conf
You can test this solution with running before export POSTGRESQL_HOST="localhost"