I am on a Solaris 8 box that does not support -i option for sed, so I am using the following from a google search on the topic:
# find . -name cancel_submit.cgi | while read file; do
> sed 's/ned.dindo.com\/confluence\/display\/CESDT\/CETS+DocTools>DOC Team/wwwin-dev.dindo.com\/Eng\/CntlSvcs\/InfoFrwk\/GblEngWWW\/Public\/index.html>EDCS Team/g' ${file} > ${file}.new
> mv ${file}.new ${file}
> done
This works except it messes up file permissions and group:owner.
How can I retain the original information?
You may use 'cat'.
cat ${file}.new > ${file} && rm ${file}.new
cp -p preserves the stuff you want. Personally I would do this (to imitate sed -i.bak):
...
cp -p ${file} ${file}.bak
sed 's/..../g' ${file}.bak > ${file}
...
You could add rm ${file}.bak to the end if desired, in which case you wouldn't technically need the -p in the cp line above. But with the above you can do mv ${file}.bak ${file} to recover if the replacement goes awry.
Related
I have files in a directory such as
FILE1.docx.txt
FILE2.docx.txt
FILE3.docx.txt
FILE4.docx.txt
FILE5.docx.txt
And I would like to remove .docx from all of them to make the final output such as
FILE1.txt
FILE2.txt
FILE3.txt
FILE4.txt
FILE5.txt
How do I do this?
With Parameter Expansion and mv
for f in *.docx.txt; do
echo mv -vn "$f" "${f%%.*}.${f##*.}"
done
The one-liner
for f in *.docx.txt; do echo mv -vn "$f" "${f%%.*}.${f##*.}"; done
Remove the echo if you think the output is correct, to rename the files.
Should work in any POSIX compliant shell, without any script.
With bash, enable the nullglob shell option so the glob *.docx.txt will not expand as literal *.docx.txt if there are no files ending with .docx.txt
#!/usr/bin/env bash
shopt -s nullglob
for f in *.docx.txt; do
echo mv -vn "$f" "${f%%.*}.${f##*.}"
done
UPDATE: Thanks to #Léa Gris add nullglob change the glob to *.docx.txt and add -n to mv, Although -n and -v is not defined by POSIX as per https://pubs.opengroup.org/onlinepubs/9699919799/utilities/mv.html It should be in both GNU and BSD mv
Just run this python script in the same folder that contains files:
import os
for file in os.listdir(os.getcwd()):
aux = file.split('.')
if len(aux) == 3:
os.rename(file, aux[0] + '.' + aux[2])
you can make use of sed and bash like this:
for i in *.docx.txt
do
mv "$i" "`echo $i | sed 's/.docx//'`"
done
Situation:
I need a bash script that deletes all files in the current folder, except all the files mentioned in a file called ".rmignore". This file may contain addresses relative to the current folder, that might also contain asterisks(*). For example:
1.php
2/1.php
1/*.php
What I've tried:
I tried to use GLOBIGNORE but that didn't work well.
I also tried to use find with grep, like follows:
find . | grep -Fxv $(echo $(cat .rmignore) | tr ' ' "\n")
It is considered bad practice to pipe the exit of find to another command. You can use -exec, -execdir followed by the command and '{}' as a placeholder for the file, and ';' to indicate the end of your command. You can also use '+' to pipe commands together IIRC.
In your case, you want to list all the contend of a directory, and remove files one by one.
#!/usr/bin/env bash
set -o nounset
set -o errexit
shopt -s nullglob # allows glob to expand to nothing if no match
shopt -s globstar # process recursively current directory
my:rm_all() {
local ignore_file=".rmignore"
local ignore_array=()
while read -r glob; # Generate files list
do
ignore_array+=(${glob});
done < "${ignore_file}"
echo "${ignore_array[#]}"
for file in **; # iterate over all the content of the current directory
do
if [ -f "${file}" ]; # file exist and is file
then
local do_rmfile=true;
# Remove only if matches regex
for ignore in "${ignore_array[#]}"; # Iterate over files to keep
do
[[ "${file}" == "${ignore}" ]] && do_rmfile=false; #rm ${file};
done
${do_rmfile} && echo "Removing ${file}"
fi
done
}
my:rm_all;
If we assume that none of the files in .rmignore contain newlines in their name, the following might suffice:
# Gather our exclusions...
mapfile -t excl < .rmignore
# Reverse the array (put data in indexes)
declare -A arr=()
for file in "${excl[#]}"; do arr[$file]=1; done
# Walk through files, deleting anything that's not in the associative array.
shopt -s globstar
for file in **; do
[ -n "${arr[$file]}" ] && continue
echo rm -fv "$file"
done
Note: untested. :-) Also, associative arrays were introduced with Bash 4.
An alternate method might be to populate an array with the whole file list, then remove the exclusions. This might be impractical if you're dealing with hundreds of thousands of files.
shopt -s globstar
declare -A filelist=()
# Build a list of all files...
for file in **; do filelist[$file]=1; done
# Remove files to be ignored.
while read -r file; do unset filelist[$file]; done < .rmignore
# Annd .. delete.
echo rm -v "${!filelist[#]}"
Also untested.
Warning: rm at your own risk. May contain nuts. Keep backups.
I note that neither of these solutions will handle wildcards in your .rmignore file. For that, you might need some extra processing...
shopt -s globstar
declare -A filelist=()
# Build a list...
for file in **; do filelist[$file]=1; done
# Remove PATTERNS...
while read -r glob; do
for file in $glob; do
unset filelist[$file]
done
done < .rmignore
# And remove whatever's left.
echo rm -v "${!filelist[#]}"
And .. you guessed it. Untested. This depends on $f expanding as a glob.
Lastly, if you want a heavier-weight solution, you can use find and grep:
find . -type f -not -exec grep -q -f '{}' .rmignore \; -delete
This runs a grep for EACH file being considered. And it's not a bash solution, it only relies on find which is pretty universal.
Note that ALL of these solutions are at risk of errors if you have files that contain newlines.
This line do perfectly the job
find . -type f | grep -vFf .rmignore
If you have rsync, you might be able to copy an empty directory to the target one, with suitable rsync ignore files. Try it first with -n, to see what it will attempt, before running it for real!
This is another bash solution that seems to work ok in my tests:
while read -r line;do
exclude+=$(find . -type f -path "./$line")$'\n'
done <.rmignore
echo "ignored files:"
printf '%s\n' "$exclude"
echo "files to be deleted"
echo rm $(LC_ALL=C sort <(find . -type f) <(printf '%s\n' "$exclude") |uniq -u ) #intentionally non quoted to remove new lines
Test it online here
Alternatively, you may want to look at the simplest format:
rm $(ls -1 | grep -v .rmignore)
I want to read a file and and parse out filenames and remove them. In my case this means removing everything after the first tab for each line in the file to get the filenames and then calling rm -i on the files.
This is what I have so far but it just removes them all without prompting...if I add the -i to xargs rm it gives me a wall of text without letting me choose y/n
while IFS=' ' read -r line; do
#echo ${line%*}
sed -e 's/\t.*$//' | xargs rm
done < $1
The problem is that rm -i asks for yes/no on stdin. You redirect to the while loop and pipe to xargs, both of which will override stdin for rm -i.
You can rewrite to avoid xargs and also use a different FD for your loop:
while IFS=$'\t' read -u 3 -r file _
do
rm -i "$file"
done 3< yourfile.txt
You can avoid rm -i and use xargs -p for prompting use for each file to be deleted:
cut -f1 file | xargs -n1 -p rm
I have a script that queries the Twitter API for several queries, and then writes the raw data to a file with the query in the name, plus a timestamp. I'd like to have a script that, given the list of query strings (regexs?) and for all files in a folder, if one of the query strings is a substring in that file, move it to a specific folder. Right now I have just a script with just a few dozen mv commands, but I'd like a simpler and more maintainable version. Here's an example of what I'm doing now:
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*femin*/home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*patriarchy* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*yesallwomen* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*womanpower* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
I would use a for loop:
for i in femin patriarchy yesallwomen womanpower; do
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*$i* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
done
That way the list is in the first line so it is easy to amend.
I would isolate data (the words to be moved to feminism) and code.
When you have more keywords (feminism and so), you can make files with keywords and check these keywordfiles for the files you are considering to move.
With ${fromdir} where the files come from, ${todir} where you want them and ${keyfiledir} with the keywords, you get something like
for keyfile in ${keyfiledir}/*; do
key="${keyfile##*/}"
find $from -type f | sed 's#.*/##' | while read -r file; do
echo "${file}" | grep -q -f "${keyfiledir}"/"${key}" && mv "${from}"/"${file}" "${to}"/"${key}"
done
done
How does that work? I tested the solution above with the following script.
from=fromdir
to=todir
keyfiledir=keyfiledir
rm -rf ${from} ${to} ${keyfiledir}
mkdir ${from} ${to} ${keyfiledir}
mkdir ${to}/feminism ${to}/so
touch ${from}/yesallwomen ${from}/women ${from}/some_femin ${from}/"help move"
cat <<# > ${keyfiledir}/feminism
femin
patriarchy
yesallwomen
womanpower
#
touch ${from}/yesallwomen ${from}/women ${from}/some_femin
cat <<# > ${keyfiledir}/so
stack
exchange
help
#
test ! -d "${from}" && echo " Wrong dir ${from}" && exit 1
test ! -d "${to}" && echo " Wrong dir ${to}" && exit 1
test ! -d "${keyfiledir}" && echo " Wrong dir ${keyfiledir}" && exit 1
for keyfile in ${keyfiledir}/*; do
key="${keyfile##*/}"
find $from -type f | sed 's#.*/##' | while read -r file; do
echo "${file}" | grep -q -f "${keyfiledir}"/"${key}" && mv "${from}"/"${file}" "${to}"/"${key}"
done
done
echo "Not moved"
ls ${from}
echo "Moved"
ls -R ${to}
A simple combination of mv and egrep should suffice. egrep can take a pattern list from a file (and then you get to use full regexp syntax, not just glob syntax.) Make sure to exclude the name of the target folder.
cd /home/nick/TwitterSearchToDatabase/queries_for_amita
mv $(ls | egrep -f patterns.txt | grep -v '^feminism$') feminism
I have a problem with this linux shell script.
#! /bin/bash
find /sdcard/ -type f -iname "*.srt" -print >> /sdcard/files
count=`wc -l /sdcard/files |cut -d'/' -f1`
for (( c=1; c<=$count; c++ ))
do
line=`sed -n ''$c'p' /sdcard/files`
cat "$line" | sed -e 's/č/c/g' > "$line".srt""
rm "$line"
done
rm /sdcard/files
I know this isnt the best way to do this but thats all i can do with my knowlage
As you can see it finds all srt files and then replaces all "č" charactes with "c". But it doesnt work with files i downloaded
However when i make a new file and write "č" inside (with my keyboard), it replaces it as it should. I dont understand why?
I think we discovered the cause, now the fix:
vim somefile.srt -c ":set bomb" -c ":set fileencoding=utf-8" -c ":wq"
There's also a dirty way
echo -e "\xC2\xA0" >> somefile.srt
I tried iconv tool which is supposed to do the conversion, but it didn't help.