The bash script I wrote is supposed to modify my text files. The problem is the speed of operation. There are 4 lines of each file I want to modify.
This is my bash script to modify all .txt files in a given folder:
srcdir="$1" //source directory
cpr=$2 //given string argument
find $srcdir -name "*.txt" | while read i; do
echo "#############################"
echo "$i"
echo "Custom string: $cpr"
echo "#############################"
# remove document name and title
sed -i 's_document=.*\/[0-9]\{10\}\(, User=team\)\?__g' $i
# remove document date
sed -i 's|document date , [0-9]\{2\}\/[0-9]\{2\}\/[0-9]\{4\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\} MDT||g' $i
# remove document id
sed -i 's|document id = 878h67||g' $i
# replace new producer
sed_arg="-i 's|Reproduced by $cpr|john smith|g' $i"
eval sed "$sed_arg"
done
I dont know how to concatinate all my sed commands in one command or two, so the job would be done faster ( I think! )
I have tried the OR operator for regex | but no success.
Have you tried
sed -i -e 's/pattern/replacement/g' -e 's/pattern1/replace1/g' file
sed -i '
s_document=.*\/[0-9]\{10\}\(, User=team\)\?__g;
s|document date , [0-9]\{2\}\/[0-9]\{2\}\/[0-9]\{4\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\} MDT||g;
s|document id = 878h67||g;
s|Reproduced by '"$cpr"'|john smith|g' $i
Related
What's the best way to find and replace the values of all keys in a file without precising the keys ?
Example :
originalFile.txt ------------------------------------> newFile.txt
key1=a key1=$key1
key2=b key2=$key2
key3=c key3=$key3
I came up with this but I'm not able to do it dynamically for any new added key :
sed '/^key1/s/=.*$/= $key1/' newFile.txt
awk -vFS="=" -v OFS="=" '$2="$"$1' originalFile.txt
Using sed: using back referencing capture the left of = in \1 and right of = in \2:
sed -r 's/(^[^=]+)*=(.*)/\1=$\1/g' originalFile.txt
key1=$key1
key2=$key2
key3=$key3
This should do it:
sed -re 's/^(key[[:digit:]]+)=.*$/\1=\$\1/' originalFile.txt
It works because (since the -r options is used) a pair of parentheses captures its match. In the substitution pattern, the \1 reproduces the text matched.
this script extract the password from env and include it in the new file:
while read line; do
password=$(echo $line | sed "s/\(.*\)=.*/\1/g")
newpassword="$(printenv $password)"
if [ -n "$newpassword" ]; then
echo "$password=$newpassword"
else
echo $line
fi
done <originalFile.txt >newFile.txt
i wanna find all the json files file specific directory and combine it one parent result that will contain all the result data
I am trying this
find . -name *data.json | tee result.json
It is finding all the data but it is saving its path in result.json,how can i get the data inside all the file and get a combine JSON of all result
this command is saving a result.json file which look like this
./17-10-2018/ghatkopar/17-18/data.json
./17-10-2018/ghatkopar/18-19/data.json
./17-10-2018/ghatkopar/10-11/data.json
./17-10-2018/ghatkopar/11-12/data.json
./17-10-2018/ghatkopar/15-16/data.json
./17-10-2018/ghatkopar/19-20/data.json
./17-10-2018/ghatkopar/14-15/data.json
./17-10-2018/ghatkopar/12-13/data.json
./17-10-2018/ghatkopar/20-21/data.json
./17-10-2018/mulund-west/16-17/data.json
./17-10-2018/mulund-west/21-22/data.json
./17-10-2018/mulund-west/13-14/data.json
./17-10-2018/mulund-west/data.json
./17-10-2018/mulund-west/17-18/data.json
./17-10-2018/mulund-west/18-19/data.json
./17-10-2018/mulund-west/10-11/data.json
./17-10-2018/mulund-west/11-12/data.json
./17-10-2018/mulund-west/15-16/data.json
./17-10-2018/mulund-west/19-20/data.json
./17-10-2018/mulund-west/14-15/data.json
./17-10-2018/mulund-west/12-13/data.json
./17-10-2018/mulund-west/20-21/data.json
./17-10-2018/bhandup/16-17/data.json
./17-10-2018/bhandup/21-22/data.json
./17-10-2018/bhandup/13-14/data.json
./17-10-2018/bhandup/data.json
./17-10-2018/bhandup/17-18/data.json
./17-10-2018/bhandup/18-19/data.json
./17-10-2018/bhandup/10-11/data.json
./17-10-2018/bhandup/11-12/data.json
./17-10-2018/bhandup/15-16/data.json
You need to cat each file, not output its name. You also need to output a comma before each json except the first one, and the enclosing square brackets:
#! /bin/bash
printf [
separator=""
find . -name '*data.json' -print0 | while IFS= read -d '' -r j ; do
printf $separator
separator=,
cat "$j"
done
printf ]
This command will search all data*.json files in current and child folders and combine them as a results.json ( array )
find . -name data\*.json -type f | xargs cat | sed -e s/}/},/g -e \$s/,\$/]/ -e 1s/^/[/ > results.json
I have done basic testing using below commands
echo {\"name\":\"Alex\", age:24, city:\"Tokyo\"} > sample.json
for i in {1..10} ; do cp sample.json data$i.json ; done
find . -name data\*.json -type f | xargs cat | sed -e s/}/},/g -e \$s/,\$/]/ -e 1s/^/[/ > results.json
If you find its not working for a specific case, I will try to fix it. But as of now it will not work for nested json or json with array.
In my script I am taking a text file and splitting into sections. Before doing any splitting, I am reformatting the name of the text file. PROBLEM: Creating a folder/directory and naming it the formatted file name. This is where segments are placed. However the script breaks when the text file has spaces in it. But that is the reason I am trying to reformat the name first and then do the rest of the operations. How could I do so in that sequence?
execute script: text_split.sh -s "my File .txt" -c 2
text_split.sh
# remove whitespace and format file name
FILE_PATH="/archive/"
find $FILE_PATH -type f -exec bash -c 'mv "$1" "$(echo "$1" \
| sed -re '\''s/^([^-]*)-\s*([^\.]*)/\L\1\E-\2/'\'' -e '\''s/ /_/g'\'' -e '\''s/_-/-/g'\'')"' - {} \;
sleep 1
# arg1: path to input file / source
# create directory
function fallback_out_file_format() {
__FILE_NAME=`rev <<< "$1" | cut -d"." -f2- | rev`
__FILE_EXT=`rev <<< "$1" | cut -d"." -f1 | rev`
mkdir -p $FILE_PATH${__FILE_NAME};
__OUT_FILE_FORMAT="$FILE_PATH${__FILE_NAME}"/"${__FILE_NAME}-part-%03d.${__FILE_EXT}"
echo $__OUT_FILE_FORMAT
exit 1
}
# Set variables and default values
OUT_FILE_FORMAT=''
# Grab input arguments
while getopts “s:c” OPTION
do
case $OPTION in
s) SOURCE=$(echo "$OPTARG" | sed 's/ /\\ /g' ) ;;
c) CHUNK_LEN="$OPTARG" ;;
?) usage
exit 1
;;
esac
done
if [ -z "$OUT_FILE_FORMAT" ] ; then
OUT_FILE_FORMAT=$(fallback_out_file_format $SOURCE)
fi
Your script takes a filename argument, specified with -s, then modifies a hard-coded directory by renaming the files it contains, then uses the initial filename to generate an output directory and filename. It definitely sounds like the workflow should be adjusted. For instance, instead of trying to correct all the bad filenames in /archive/, just fix the name of the file specified with -s.
To get filename and extension, use bash's string manipulation ability, as shown in this question:
filename="${fullfile##*/}"
extension="${filename##*.}"
name="${filename%.*}"
You can trim whitespace from the input string using tr -d ' '.
You can then join this to your FILE_PATH variable with something like this:
FILE_NAME=$(echo $1 | tr -d ' ')
FILE_PATH="/archive/"
FILE_PATH=$FILE_PATH$FILE_NAME
You can escape the space using a back slash \
Now the user may not always provide with the back slash, so the script can use sed to convert all (space) to \
sed 's/ /\ /g'
you can obtain the new directory name as
dir_name=`echo $1 | sed 's/ /\ /g'
I have written the code for replacing content but quite confused with replacing file name, both in the same script. I want my script to run this way:
suppose I give input : ./myscript.sh abc xyz
abc is my string to be replaced
xyz is the string to replace with
If some directory or any subdirectory or file in it has name abc, that should also change to xyz.
Here is my code:
filepath="/misc/home3/babitasandooja/fol1"
for file in $(grep -lR $1 $filepath)
do
sed -i "s/$1/$2/g" $file
echo "Modified: " $file
done
Now How should I code to replace the filename as well.
I tried:
if( *$1*==$file )
then
rename $1 $2 $filepath
fi
and
find -iname $filepath -exec mv $1 $2 \;
but any of then is not working correctly. What should I do? Which approach should I take?
Any help would be appreciated.
Thanks :)
#!/bin/bash
dir=$1 str=$2 rep=$3
while IFS= read -rd '' file; do
sed -i "s/$str/$rep/g" -- "$file"
base=${file##*/} dir=${file%/*}
[[ $base == *"$str"* ]] && mv "$file" "$dir/${base//$str/$rep}"
done < <(exec grep -ZFlR "$str" "$dir")
Usage:
bash script.sh dir string replacement
Note: rename would also rename the directory part.
grep -Z makes it produce null-delimited outputs. i.e. it produces output which is composed of filenames in which everything is separated by 0x00.
-d '' makes read read input delimited by 0x00; -r prevents
backslashes to be interpreted; and IFS= prevents word splitting with IFS.
IFS= read -rd '' makes read read input with delimiter of `0
base=${file##*/} removes the directory part. It's the same as base=$(basename "$file")
dir=${file%/*} removes the file part.
[[ $base == *"$str"* ]] checks if the filename has something that could be renamed. && makes the command that follows it execute if the previous returns zero (true) code. Think of it as a single link if statement.
"$dir/${base//$str/$rep}" forms the new filename. ${base//$str/$rep} replaces anything in $base that matches value of $str and replace it with value of $rep.
I have the following script and for some reason it is not working
find . -name '*---*' | while read fname3
do
new_fname3=`echo $fname3 | tr "---" "-"`
if [ -e $new_fname3 ]
then
echo "File $new_fname3 already exists. Not replacing $fname3"
else
echo "Creating new file $new_fname3 to replace $fname3"
mv "$fname3" $new_fname3
fi
done
However if I use
find . -name '*---*' | while read fname3
do
new_fname3=`echo $fname3 | tr "-" "_"`
if [ -e $new_fname3 ]
then
echo "File $new_fname3 already exists. Not replacing $fname3"
else
echo "Creating new file $new_fname3 to replace $fname3"
mv "$fname3" $new_fname3
fi
done
The script works but I end up with 3 underscores "_" how can I replace the 3 dashes "---" with a single dash?
Thanks,
Have a look at man tr. tr will just replace single characters.
Use something like perl -wpe "s/---/-/" instead.
Also have a look at man 1p rename. It is doing pretty much exactly what you want:
rename 's/---/-/' *---*
I believe you need to change the tr for a sed substitution:
tr '---' '-' should be changed to sed -e 's/---/-/g
As an example of the difference:
$ echo "a---b" | tr '---' '-'
tr: unrecognised option '---'
try `tr --help' for more information
$ echo "a---b" | sed -e 's/---/-/g'
a-b