My code works, but not in the way I want to exactly. Basically, what my code does is that it looks through the current directory, searches for folders, and within those folders it looks at the files. If one of the file extensions is the value of the $Example variable, then it should delete all other files with the same beginning file name, regardless of extension and rename the one with the $Example extension to the same name, just without the $Example extension. Here is the code:
#!/bin/sh
set -o errexit
Example=dummy
for d in *; do
if test "$(ls -A "$d" 2>/dev/null)"; then
if [ $(ls -1 ${d}/*.$Example 2>/dev/null | wc -l) -ge 1 ]; then
cd $(pwd)/$d;
for f in *.$Example; do
fileName="${f%.$Example}";
mv "$f" "${f%.$Example}";
#tr "\r" "\n" < "${f%.$Example}" > "${f%.$Example}"
listToDelete=$(find -name "$fileName.*");
for g in $listToDelete; do
rm $g;
done;
done;
cd ..;
fi;
fi;
done
The files being used have been created in VIM, so are supposed to have Linux formatting, rather than Windows formatting. For some reason or other, once the extension has been stripped, using this code, it gets formatted with \r, and the file fails to run. I added the comment where my temporary solution is located, but I was wondering if either there is some way to alter the mv function to keep the Linux formatting, or maybe there is another way to achieve what I want. Thanks
The files being used have been created in VIM, so are supposed to have Linux formatting, rather than Windows formatting.
That has no impact on the line separator being used.
But I can think of 3 different possible causes.
ViM substitution
The \r and \n may have been mixed up or the Windows line separator (\r\n) may have been striped out incorrectly. If you're just trying to convert the Windows line separator at some point, convert them to to Unix line separated files with dos2unix instead of sed or vim magic if possible and then edit them. If the files are adding or replacing line separators using ViM, remember that in ViM line separators are searched for using \n and replaced with \r since ViM just loads file data into a buffer. e.g. %s/\n/somestring/ and %s/somestring/\r/g.
Cygwin
If you're using Cygwin, I'm pretty sure it defaults to ViM using Windows line separators, but you can change it to Unix line separators. Don't remember how to off the top of my head though.
ViM default line separators
Not sure how this would have happened if you're on a GNU/Linux system, but the default line separator to be used in ViM can be specified to Unix using :e ++ff=unix
This answer does not address the issue with a possible stray \r in the script source code, but rather gives an alternative way of doing what I believe is what the user wants to achieve:
#!/bin/bash
ext='dummy'
tmpfile=$(mktemp)
find . -type f -name "*.$ext" \
-execdir bash -x -c \
'cp "$3" "$2" &&
tee "${3%.$1}"* <"$2" >/dev/null' bash "$ext" "$tmpfile" {} \;
rm -f "$tmpfile"
This finds all files with the given extension and for each of them executes
cp "$3" "$2" && tee "${3%.$1}"* <"$2" >/dev/null
$1 will be the extension without the dot.
$2 will be the name of a temporary file.
$3 will be the found file.
The command will first copy the found file to a temporary file, then it will feed this temporary file through tee which will duplicate its contents to all files with the same prefix (${3%.$1} will strip the extension from the found filename and ${3%.$1}* will expand to all files in the same directory that has the same prefix).
Most modern implementations of find supports -execdir which works like -exec with the difference that the given utility will be executed in the directory of the found name. Also, {} will be the pathless basename of the found thing.
Given the following files in a directory:
$ ls test/
f1.bar f1.foo f2.bar f2.foo f3.bar f3.foo
f1.dummy f1.txt f2.dummy f2.txt f3.dummy f3.txt
This script does the following:
$ bash -x script.sh
+ ext=dummy
++ mktemp
+ tmpfile=/tmp/tmp.9v5JMAcA12
+ find . -type f -name '*.dummy' -execdir bash -x -c 'cp "$3" "$2" &&
tee "${3%.$1}"* <"$2" >/dev/null' sh dummy /tmp/tmp.9v5JMAcA12 '{}' ';'
+ cp f1.dummy /tmp/tmp.9v5JMAcA12
+ tee f1.bar f1.dummy f1.foo f1.txt
+ cp f2.dummy /tmp/tmp.9v5JMAcA12
+ tee f2.bar f2.dummy f2.foo f2.txt
+ cp f3.dummy /tmp/tmp.9v5JMAcA12
+ tee f3.bar f3.dummy f3.foo f3.txt
+ rm -f /tmp/tmp.9v5JMAcA12
Remove -x from the invocation of bash (and from bash on the command line) to disable tracing.
I used bash rather than sh because my sh can't grok ${3%.$1} properly.
I am currently trying to list the size of all files in a directory which is passed as the first argument to the script, but the -f option in Linux is not working, or am I missing something.
Here is the code :
for tmp in "$1/*"
do
echo $tmp
if [ -f "$tmp" ]
then num=`ls -l $tmp | cut -d " " -f5`
echo $num
fi
done
How would I fix this problem?
I think the error is with your glob syntax which doesn't work in either single- or double-quotes,
for tmp in "$1"/*; do
..
Do the above to expand the glob outside the quotes.
There are couple more improvements possible in your script,
Double-quote your variables to prevent from word-splitting, e.g. echo "$temp"
Backtick command substitution `` is legacy syntax with several issues, use the $(..) syntax.
The [-f "filename"] condition check in linux is for checking the existence of a file and it is a regular file. For reference, use this text as reference,
-b FILE
FILE exists and is block special
-c FILE
FILE exists and is character special
-d FILE
FILE exists and is a directory
-e FILE
FILE exists
-f FILE
FILE exists and is a regular file
-g FILE
FILE exists and is set-group-ID
-G FILE
FILE exists and is owned by the effective group ID
I suggest you try with [-e "filename"] and see if it works.
Cheers!
At least on the command line, this piece of script does it:
for tmp in *; do echo $tmp; if [ -f $tmp ]; then num=$(ls -l $tmp | sed -e 's/ */ /g' | cut -d ' ' -f5); echo $num; fi; done;
If cut uses space as delimiter, it cuts at every space sign. Sometimes you have more than one space between columns and the count can easily go wrong. I'm guessing that in your case you just happened to echo a space, which looks like nothing. With the sed command I remove extra spaces.
i have a directory with a lot of sub-directories including files.
For each WAV file i would like to rename WAV file by adding creation date (date when file WAV has been firstly created) at the beginning of the file (without changing timestamps of file itself).
Next step would be to convert the WAV file to MP3 file, so i will save hard drive space.
for that purpose, i'm trying to create a bash script but i'm having some issues.
I want to keep the same structure as original directory and therefore i was thinking of something like:
for file in `ls -1 *.wav`
do name=`stat -c %y $file | awk -F"." '{ print $1 }' | sed -e "s/\-//g" -e "s/\://g" -e "s/[ ]/_/g"`.wav
cp -r --preserve=timestampcp $dir_original/$file $dir_converted/$name
done
Don't use ls to generate a list of file names, just let the shell glob them (that's what ls *.wav does anyway):
for file in ./*.wav ; do
I think you want the timestamp in the format YYYYMMDD_HHMMSS ?
You could use GNU date with stat to have a somewhat neater control of the output format:
epochtime=$(stat -c %Y "$file" )
name=$(date -d "#$epochtime" +%Y%m%d_%H%M%S).wav
stat -c %Y (or %y) gives the last modification date, but you can't really get the date of the file creation on Linux systems.
That cp looks ok, except for the stray cp at the end of timestampcp, but that must be a typo. If you do *.wav, the file names will be relative to current directory anyway, so no need to prefix with $dir_original/.
If you want to walk through a whole subdirectory, use Bash's globstar feature, or find. Something like this:
shopt -s globstar
cd "$sourcedir"
for file in ./**/*.wav ; do
epochtime=$(stat -c %Y "$file" )
name=$(date -d "#$epochtime" +%Y%m%d_%H%M%S).wav
dir=$(dirname "$file")
mkdir -p "$target/$dir"
cp -r --preserve=timestamp "$file" "$target/$dir/$name"
done
The slight inconvenience here is that cp can't create the directories in the path, so we need to use mkdir there. Also, I'm not sure if you wanted to keep the original filename as part of the resulting one, this would remove it and just replace the file names with the timestamp.
I did some experimenting with the calculation of name to see if I could get it more succinctly, and came up with this:
name=$(date "+%Y%m%d_%H%M%S" -r "$file")
I wanted to append all file names in that folder with the date they were created , and below works perfectly.
#############################
#!/bin/sh
for file in `ls *.JPG`;
do
mv -f "$file" "$(date -r "$file" +"%Y%m%d_%H_%M_%S")_"$file".jpg"
done
##############################
I need to add the following line to the end of a config file:
include "/configs/projectname.conf"
to a file called lighttpd.conf
I am looking into using sed to do this, but I can't work out how.
How would I only insert it if the line doesn't already exist?
Just keep it simple :)
grep + echo should suffice:
grep -qxF 'include "/configs/projectname.conf"' foo.bar || echo 'include "/configs/projectname.conf"' >> foo.bar
-q be quiet
-x match the whole line
-F pattern is a plain string
https://linux.die.net/man/1/grep
Edit:
incorporated #cerin and #thijs-wouters suggestions.
This would be a clean, readable and reusable solution using grep and echo to add a line to a file only if it doesn't already exist:
LINE='include "/configs/projectname.conf"'
FILE='lighttpd.conf'
grep -qF -- "$LINE" "$FILE" || echo "$LINE" >> "$FILE"
If you need to match the whole line use grep -xqF
Add -s to ignore errors when the file does not exist, creating a new file with just that line.
Try this:
grep -q '^option' file && sed -i 's/^option.*/option=value/' file || echo 'option=value' >> file
Using sed, the simplest syntax:
sed \
-e '/^\(option=\).*/{s//\1value/;:a;n;ba;q}' \
-e '$aoption=value' filename
This would replace the parameter if it exists, else would add it to the bottom of the file.
Use the -i option if you want to edit the file in-place.
If you want to accept and keep white spaces, and in addition to remove the comment, if the line already exists, but is commented out, write:
sed -i \
-e '/^#\?\(\s*option\s*=\s*\).*/{s//\1value/;:a;n;ba;q}' \
-e '$aoption=value' filename
Please note that neither option nor value must contain a slash /, or you will have to escape it to \/.
To use bash-variables $option and $value, you could write:
sed -i \
-e '/^#\?\(\s*'${option//\//\\/}'\s*=\s*\).*/{s//\1'${value//\//\\/}'/;:a;n;ba;q}' \
-e '$a'${option//\//\\/}'='${value//\//\\/} filename
The bash expression ${option//\//\\/} quotes slashes, it replaces all / with \/.
Note: Just trapped into a problem. In bash you may quote "${option//\//\\/}", but in the sh of busybox, this does not work, so you should avoid the quotes, at least in non-bourne-shells.
All combined in a bash function:
# call option with parameters: $1=name $2=value $3=file
function option() {
name=${1//\//\\/}
value=${2//\//\\/}
sed -i \
-e '/^#\?\(\s*'"${name}"'\s*=\s*\).*/{s//\1'"${value}"'/;:a;n;ba;q}' \
-e '$a'"${name}"'='"${value}" $3
}
Explanation:
/^\(option=\).*/: Match lines that start with option= and (.*) ignore everything after the =. The \(…\) encloses the part we will reuse as \1later.
/^#?(\s*'"${option//////}"'\s*=\s*).*/: Ignore commented out code with # at the begin of line. \? means «optional». The comment will be removed, because it is outside of the copied part in \(…\). \s* means «any number of white spaces» (space, tabulator). White spaces are copied, since they are within \(…\), so you do not lose formatting.
/^\(option=\).*/{…}: If matches a line /…/, then execute the next command. Command to execute is not a single command, but a block {…}.
s//…/: Search and replace. Since the search term is empty //, it applies to the last match, which was /^\(option=\).*/.
s//\1value/: Replace the last match with everything in (…), referenced by \1and the textvalue`
:a;n;ba;q: Set label a, then read next line n, then branch b (or goto) back to label a, that means: read all lines up to the end of file, so after the first match, just fetch all following lines without further processing. Then q quit and therefore ignore everything else.
$aoption=value: At the end of file $, append a the text option=value
More information on sed and a command overview is on my blog:
https://marc.wäckerlin.ch/computer/stream-editor-sed-overview-and-reference
If writing to a protected file, #drAlberT and #rubo77 's answers might not work for you since one can't sudo >>. A similarly simple solution, then, would be to use tee --append (or, on MacOS, tee -a):
LINE='include "/configs/projectname.conf"'
FILE=lighttpd.conf
grep -qF "$LINE" "$FILE" || echo "$LINE" | sudo tee --append "$FILE"
Here's a sed version:
sed -e '\|include "/configs/projectname.conf"|h; ${x;s/incl//;{g;t};a\' -e 'include "/configs/projectname.conf"' -e '}' file
If your string is in a variable:
string='include "/configs/projectname.conf"'
sed -e "\|$string|h; \${x;s|$string||;{g;t};a\\" -e "$string" -e "}" file
If, one day, someone else have to deal with this code as "legacy code", then that person will be grateful if you write a less exoteric code, such as
grep -q -F 'include "/configs/projectname.conf"' lighttpd.conf
if [ $? -ne 0 ]; then
echo 'include "/configs/projectname.conf"' >> lighttpd.conf
fi
another sed solution is to always append it on the last line and delete a pre existing one.
sed -e '$a\' -e '<your-entry>' -e "/<your-entry-properly-escaped>/d"
"properly-escaped" means to put a regex that matches your entry, i.e. to escape all regex controls from your actual entry, i.e. to put a backslash in front of ^$/*?+().
this might fail on the last line of your file or if there's no dangling newline, I'm not sure, but that could be dealt with by some nifty branching...
Here is a one-liner sed which does the job inline. Note that it preserves the location of the variable and its indentation in the file when it exists. This is often important for the context, like when there are comments around or when the variable is in an indented block. Any solution based on "delete-then-append" paradigm fails badly at this.
sed -i '/^[ \t]*option=/{h;s/=.*/=value/};${x;/^$/{s//option=value/;H};x}' test.conf
With a generic pair of variable/value you can write it this way:
var=c
val='12 34' # it handles spaces nicely btw
sed -i '/^[ \t]*'"$var"'=/{h;s/=.*/='"$val"'/};${x;/^$/{s//c='"$val"'/;H};x}' test.conf
Finally, if you want also to keep inline comments, you can do it with a catch group. E.g. if test.conf contains the following:
a=123
# Here is "c":
c=999 # with its own comment and indent
b=234
d=567
Then running this
var='c'
val='"yay"'
sed -i '/^[ \t]*'"$var"'=/{h;s/=[^#]*\(.*\)/='"$val"'\1/;s/'"$val"'#/'"$val"' #/};${x;/^$/{s//'"$var"'='"$val"'/;H};x}' test.conf
Produces that:
a=123
# Here is "c":
c="yay" # with its own comment and indent
b=234
d=567
As an awk-only one-liner:
awk -v s=option=value '/^option=/{$0=s;f=1} {a[++n]=$0} END{if(!f)a[++n]=s;for(i=1;i<=n;i++)print a[i]>ARGV[1]}' file
ARGV[1] is your input file. It is opened and written to in the for loop of theEND block. Opening file for output in the END block replaces the need for utilities like sponge or writing to a temporary file and then mving the temporary file to file.
The two assignments to array a[] accumulate all output lines into a. if(!f)a[++n]=s appends the new option=value if the main awk loop couldn't find option in file.
I have added some spaces (not many) for readability, but you really need just one space in the whole awk program, the space after print.
If file includes # comments they will be preserved.
Here's an awk implementation
/^option *=/ {
print "option=value"; # print this instead of the original line
done=1; # set a flag, that the line was found
next # all done for this line
}
{print} # all other lines -> print them
END { # end of file
if(done != 1) # haven't found /option=/ -> add it at the end of output
print "option=value"
}
Run it using
awk -f update.awk < /etc/fdm_monitor.conf > /etc/fdm_monitor.conf.tmp && \
mv /etc/fdm_monitor.conf.tmp /etc/fdm_monitor.conf
or
awk -f update.awk < /etc/fdm_monitor.conf | sponge /etc/fdm_monitor.conf
EDIT:
As a one-liner:
awk '/^option *=/ {print "option=value";d=1;next}{print}END{if(d!=1)print "option=value"}' /etc/fdm_monitor.conf | sponge /etc/fdm_monitor.conf
use awk
awk 'FNR==NR && /configs.*projectname\.conf/{f=1;next}f==0;END{ if(!f) { print "your line"}} ' file file
sed -i 's/^option.*/option=value/g' /etc/fdm_monitor.conf
grep -q "option=value" /etc/fdm_monitor.conf || echo "option=value" >> /etc/fdm_monitor.conf
here is an awk one-liner:
awk -v s="option=value" '/^option/{f=1;$0=s}7;END{if(!f)print s}' file
this doesn't do in-place change on the file, you can however :
awk '...' file > tmpfile && mv tmpfile file
Using sed, you could say:
sed -e '/option=/{s/.*/option=value/;:a;n;:ba;q}' -e 'aoption=value' filename
This would replace the parameter if it exists, else would add it to the bottom of the file.
Use the -i option if you want to edit the file in-place:
sed -i -e '/option=/{s/.*/option=value/;:a;n;:ba;q}' -e 'aoption=value' filename
sed -i '1 h
1 !H
$ {
x
s/^option.*/option=value/g
t
s/$/\
option=value/
}' /etc/fdm_monitor.conf
Load all the file in buffer, at the end, change all occurence and if no change occur, add to the end
The answers using grep are wrong. You need to add an -x option to match the entire line otherwise lines like #text to add will still match when looking to add exactly text to add.
So the correct solution is something like:
grep -qxF 'include "/configs/projectname.conf"' foo.bar || echo 'include "/configs/projectname.conf"' >> foo.bar
Using sed: It will insert at the end of line. You can also pass in variables as usual of course.
grep -qxF "port=9033" $light.conf
if [ $? -ne 0 ]; then
sed -i "$ a port=9033" $light.conf
else
echo "port=9033 already added"
fi
Using oneliner sed
grep -qxF "port=9033" $lightconf || sed -i "$ a port=9033" $lightconf
Using echo may not work under root, but will work like this. But it will not let you automate things if you are looking to do it since it might ask for password.
I had a problem when I was trying to edit from the root for a particular user. Just adding the $username before was a fix for me.
grep -qxF "port=9033" light.conf
if [ $? -ne 0 ]; then
sudo -u $user_name echo "port=9033" >> light.conf
else
echo "already there"
fi
I elaborated on kev's grep/sed solution by setting variables in order to reduce duplication.
Set the variables in the first line (hint: $_option shall match everything on the line up until the value [including any seperator like = or :]).
_file="/etc/ssmtp/ssmtp.conf" _option="mailhub=" _value="my.domain.tld" \
sh -c '\
grep -q "^$_option" "$_file" \
&& sed -i "s/^$_option.*/$_option$_value/" "$_file" \
|| echo "$_option$_value" >> "$_file"\
'
Mind that the sh -c '...' just has the effect of widening the scope of the variables without the need for an export. (See Setting an environment variable before a command in bash not working for second command in a pipe)
You can use this function to find and search config changes:
#!/bin/bash
#Find and Replace config values
find_and_replace_config () {
file=$1
var=$2
new_value=$3
awk -v var="$var" -v new_val="$new_value" 'BEGIN{FS=OFS="="}match($1, "^\\s*" var "\\s*") {$2=" " new_val}1' "$file" > output.tmp && sudo mv output.tmp $file
}
find_and_replace_config /etc/php5/apache2/php.ini max_execution_time 60
If you want to run this command using a python script within a Linux terminal...
import os,sys
LINE = 'include '+ <insert_line_STRING>
FILE = <insert_file_path_STRING>
os.system('grep -qxF $"'+LINE+'" '+FILE+' || echo $"'+LINE+'" >> '+FILE)
The $ and double quotations had me in a jungle, but this worked.
Thanks everyone
Try:
LINE='include "/configs/projectname.conf"'
sed -n "\|$LINE|q;\$a $LINE" lighttpd.conf >> lighttpd.conf
Use the pipe as separator and quit if $LINE has been found. Otherwise, append $LINE at the end.
Since we only read the file in sed command, I suppose we have no clobber issue in general (it depends on your shell settings).
Using only sed I'd suggest the following solution:
sed -i \
-e 's#^include "/configs/projectname.conf"#include "/configs/projectname.conf"#' \
-e t \
-e '$ainclude "/configs/projectname.conf"' lighttpd.conf
s replace the line include "/configs/projectname.conf with itself (using # as delimiter here)
t if the replacement was successful skip the rest of the commands
$a otherwise jump to the last line and append include "/configs/projectname.conf after it
Almost all of the answers work but not in all scenarios or OS as per my experience. Only thing that worked on older systems and new and different flavours of OS is the following.
I needed to append KUBECONFIG path to bashrc file if it doesnt exist. So, what I did is
I assume that it exists and delete it.
with sed I append the string I want.
sed -i '/KUBECONFIG=/d' ~/.bashrc
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc
I needed to edit a file with restricted write permissions so needed sudo. working from ghostdog74's answer and using a temp file:
awk 'FNR==NR && /configs.*projectname\.conf/{f=1;next}f==0;END{ if(!f) { print "your line"}} ' file > /tmp/file
sudo mv /tmp/file file