I have a range of directories from 2010 to 2017 and a sub-directory in them from 1 to 12. There is a file in each sub-directory, I need to add a line to each of these files.
This is part of my script:
#!/bin/bash
mkdir -p test/201{0..7}/{1..12}/
touch test/201{0..7}/{1..12}/file_{0..9}.txt
echo "42" | tee test/201{0..7}/{1..12}/file_{0..9}.txt
Append to files:
echo "42" | tee -a test/201{0..7}/{1..12}/file_{0..9}.txt
In your script, it was created a range of directories from 2010 to 2017 and a range of sub-directories that goes from 1 to 12 in each directory. You also created nine files in each sub-directory at the end.
So, to append a new line in each of these files you should use the echo command to create the contents of your line and redirect it to the tee command to add it to all your files at once, as follows:
echo "new line" | tee -a test/201{0..7}/{1..12}/file_{0..9}.txt
That should be enough.
Related
Lets say i have multiple filesnames e.g. R014-20171109-1159.log.20171109_1159.
I want to create a shell script which creates for every given date a folder and moves the files matching the date to it.
Is this possible?
For the example a folder "20171109" should be created and has the file "R014-20171109-1159.log.20171109_1159" on it.
Thanks
This is a typical application of a for-loop in bash to iterate thru files.
At the same time, this solution utilizes GNU [ shell param substitution ].
for file in /path/to/files/*\.log\.*
do
foldername=${file#*-}
foldername=${foldername%%-*}
mkdir -p "${foldername}" # -p suppress errors if folder already exists
[ $? -eq 0 ] && mv "${file}" "${foldername}" # check last cmd status and move
done
Since you want to write a shell script, use commands. To get date, use cut cmd like ex:
cat 1.txt
R014-20171109-1159.log.20171109_1159
cat 1.txt | cut -d "-" -f2
Output
20171109
is your date and create folder. This way you can loop and create as many folders as you want
Its actually quite easy(my Bash syntax might be a bit off) -
for f in /path/to/your/files*; do
## Check if the glob gets expanded to existing files.
## If not, f here will be exactly the pattern above
## and the exists test will evaluate to false.
[ -e "$f" ] && echo $f > #grep the file name for "*.log."
#and extract 8 charecters after "*.log." .
#Next check if a folder exists already with the name of 8 charecters.
#If not { create}
#else just move the file to that folder path
break
done
Main idea is from this post link. Sorry for not providing the actual code as i havent worked anytime recently on Bash
Below commands can be put in script to achieve this,
Assign a variable with current date as below ( use --date='n day ago' option if need to have an older date).
if need to get it from File name itself, get files in a loop then use cut command to get the date string,
dirVar=$(date +%Y%m%d) --> for current day,
dirVar=$(date +%Y%m%d --date='1 day ago') --> for yesterday,
dirVar=$(echo $fileName | cut -c6-13) or
dirVar=$(echo $fileName | cut -d- -f2) --> to get from $fileName
Create directory with the variable value as below, (-p : create directory if doesn't exist.)
mkdir -p ${dirVar}
Move files to directory to the directory with below line,
mv *log.${dirVar}* ${dirVar}/
I want to monitor a directories + sub dir and get the first 7 characters of file that created in the folder or sub folder. I'm using these codes for my problem :
inotifywait -mqr /path/ -e create |
while read path file ; do
line=$(head -c 7 $file) ;
echo $line ;
done;
but when I run the code it won't print only succeeded for some text file (first 7 character = "setruk"), and not print anythings for some text file (first 7 characters = "kukuryu"). Can anyone help me ? pleasee ?
For me, this is the output of inotifywait:
$ inotifywait -mqr . -e create
./test/ CREATE five
./test/ CREATE six
./test/ CREATE seven
I don't know if you get the CREATE part, but you're probably missing the path to the directory. You likely want:
inotifywait -mqr /path/ -e create |
while read path action file ; do
line=$(head -c 7 "$path$file") ;
echo $line ;
done;
If you don't have the CREATE part in the line, just change the head command to include the $path part of the filename:
line=$(head -c 7 "$path$file") ;
What's probably happening is that files in sub-directories are not getting their characters printed but files in the main directory are.
I need help copying content from various files to others (same name and format, different path).
For example, $HOME/initial/baby.desktop has text which I need to write into $HOME/scripts/baby.desktop. This is very simple for a single file, but I have 2500 files in $HOME/initial/ and the same number in $HOME/scripts/ with corresponding names (same names and format). I want append (copy) the content of file in path A to path B (which have the same name and format), to the end of file in path B without erase the content of file in path B.
Example content of $HOME/initial/*.desktop to final $HOME/scripts/*.desktop. I tried the following, but it don't work:
cd $HOME/initial/
for i in $( ls *.desktop ); do egrep "Icon" $i >> $HOME/scripts/$i; done
Firstly, I would backup $HOME/initial and $HOME/scripts, because there is lots of scope for people misunderstanding your question. Like this:
cd $HOME
tar -cvf initial.tar initial
tar -cvf scripts.tar scripts
That will put all the files in $HOME/initial into a single tarfile called initial.tar and all the files in $HOME/scripts into a single tarfile called scripts.tar.
Now for your question... in general, if you want to put the contents of FileB onto the end of FileA, the command is
cat FileB >> FileA
Note the DOUBLE ">>" which means "append" rather than single ">" which means overwrite.
So, I think you want to do this:
cd $HOME/initial/baby.desktop
cat SomeFile >> $HOME/scripts/baby.desktop/SomeFile
where SomeFile is the name of any file you choose to test with. I would test that has worked and then, if you are happy with that, go ahead and run the same command inside a loop:
cd $HOME/initial/baby.desktop
for SOURCE in *
do
DESTINATION="$HOME/scripts/baby.desktop/$SOURCE"
echo Appending "$SOURCE" to "$DESTINATION"
#cat "$SOURCE" >> "$DESTINATION"
done
When the output looks correct, remove the "#" at the start of the penultimate line and run it again.
I solved it, if some people want learn how to resolve is very simple:
using Sed
I need only the match (or pattern) line "Icon=/usr/share/some_picture.png into $HOME/initial/example.desktop to other with same name and format $HOME/scripts/example.desktop, but I had a lot of .desktop files (2500 files)
cd $HOME/initial
STRING_LINE=`grep -l -R "Icon=" *.desktop`
for i in $STRING_LINE; do sed -ne '/Icon=/ p' $i >> $HOME/scripts/$i ; done
_________
If you need only copy all to other file with same name and format
using cat
cd $HOME/initial
STRING_LINE=`grep -l -R "Icon=" *.desktop`
for i in $STRING_LINE; do cat $i >> $HOME/scripts/$i ; done
I am having trouble with several bits of code, I am no expert in Linux Bash programming unfortunately so I have tried unsuccessfully to find something that works for my task all day and was hoping you could help guide me in the right direction.
I have many large files that I would like to split according to the third field within each of them, I would like to keep the header in each of the sub-files, and save the created sub-files in new directories created from the root names of the files.
The initial files stored in the original directory are:
Downloads/directory1/Levels_CHG_Lab_S_sample1.txt
Downloads/directory1/Levels_CHG_Lab_S_sample2.txt
Downloads/directory1/Levels_CHG_Lab_S_sample3.txt
and so on..
Each of these files have 200 columns, and column 3 contains values from 1 through 10.
I would like to split each of the files above based on the value of this column, and store the subfiles in subfolders, so for example sub-folder "Downloads/directory1/sample1" will contain 10 files (with the header line) derived by splitting the file Downloads/directory1/Levels_CHG_Lab_S_sample1.txt.
I have tried now many different steps for these steps, with no success.. I must be making this more complicated than it is since the code I have tried looks aweful…
Here is the code I am trying to work from:
FILES=Downloads/directory1/
for f in $FILES
do
# Create folder with root name by stripping file names
fname=${echo $f | sed 's/.txt//;s/Levels_CHG_Lab_S_//'}
echo "Creating sub-directory [$fname]"
mkdir "$fname"
# Save the header
awk 'NR==1{print $0}' $f > header
# Split each file by third column
echo "Splitting file $f"
awk 'NR>1 {print $0 > $3".txt" }' $f
# Move newly created files in sub directory
mv {1..10}.txt $fname # I have no idea how to do specify the files just created
# Loop through the sub-files to attach header row:
for subfile in $fname
do
cat header $subfile >> tmp_file
mv -f tmp_file $subfile
done
done
All these steps seem very complicated to me, I would very much appreciate if you could help me solve this in the right way. Thank you very much for your help.
-fra
You have a few problems with your code right now. First of all, at no point do you list the contents of your downloads directory. You are simply setting the FILES variable to a string that is the path to that directory. You would need something like:
FILES=$(ls Downloads/directory1/*.txt)
You also never cd to the Downloads/directory1 folder, so your mkdir would create directories in cwd; probably not what you want.
If you know that the numbers in column 3 always range from 1 to 10, I would just pre-populate those files with the header line before you split the file.
Try this code to do what you want (untested):
BASEDIR=Downloads/directory1/
FILES=$(ls ${BASEDIR}/*.txt)
for f in $FILES; do
# Create folder with root name by stripping file names
dirname=$(echo $f | sed 's/.txt//;s/Levels_CHG_Lab_S_//')
dirname="${BASENAME}/${dirname}/"
echo "Creating sub-directory [$dirname]"
mkdir "$dirname"
# Save the header to each file
HEADER_LINE=$(head -n1 $f)
for i in {1..10}; do
echo ${HEADER_LINE} > ${dirname}/${i}.txt
done
# Split each file by third column
echo "Splitting file $f"
awk -v dirname=${dirname} 'NR>1 {filename=dirname$3".txt"; print $0 >> filename }' $f
done
I have two set of folders that have files with the same filenames and structure. The folder structure is something like this:
\outputfolder\
|---\folder1\
| |---file1.txt
| |---file2.txt
|
|---\folder2\
|---file1.txt
|---file2.txt
So what I need to do is to combine (append) all the files with the same name in these folders (file1.txt with file1.txt etc.) into another file inside the outputfolder. After getting these combined files I also need to create a tar.gz file from all of these combined files.
How can I accomplish this in a Linux based command line environment? The folder name (folder1 and folder2 etc) is variable so this needs to be given but the files need not and it should automatically combine all the files with the same name.
Also, these files have headers for column names, so I would need to remove that as well while appending.
Here's some code to get you started
topdir=outputfolder
dir1=folder1
dir2=folder2
for f in $topdir/$dir1/*.txt
do
outf=$topdir/`basename $f .txt`-concat.txt
cp $f $outf
sed -e '1 d' $topdir/$dir2/`basename $f` >> $outf
done
tar czf foo.tar.gz $topdir/*-concat.txt
Edit: added the part removing the header of the 2nd file.
find . -name 'file1.txt' | xargs cat >file1_concat.txt
This will work even if there are some files only in folder1 and some files only in folder2:
concat_files() {
for dir in "$#"; do
for file in "$dir"/*; do
this=$(basename "$file")
{ [[ -f "$this" ]] && sed 1d "$file" || cat "$file"; } >> "$this"
done
done
tar zcvf allfiles.tar.gz *
}
concat_files folder1 folder2
It will work if you have more than 2 folders for your concatenation job.
I assume you want to keep the header in the resulting file.
Have you tried the cat command (concatenation)?
cat file1 file2 >> outputfile
Might want to chuck this in a small bash script to go through the directory. This should start you off.
Best of luck.
Leo