I want to create symlinks to all files in 'myfiles' which are not already linked to and specify the destination folder for the just-created symlinks.
I am using the following cmd, successfully, to generate the list of existing links, which point to 'myfolder' :
find ~/my-existing-links/ -lname '*/myfiles/*' -printf "%f\n" > results.txt
And I'm using the following cmd to reverse match i.e. to list the files in myfiles which are not linked to:
ls ~/myfiles | grep -vf results.txt > results2.txt
So, results2.txt has a list of the files, each of which I now want to create a new symlink to.... in a folder called ~/newlinks .
I know it is possible to feed 'ln -s' a file list using the find / exec combination i.e.
find ~/myfiles/ -exec ln -s {} -t ~/newlinks \; -print
.... but that would be the unfiltered file list in myfiles. I want to use the filtered list.
Any ideas how I can do this? I'm going to be adding files to myfiles regularly and so will periodically visit the folder for the purpose of generating symlinks for all the new files so I can divi the links up logically(rather than change the original filename).
Try with xargs:
cat results2.txt | xargs -I{} ln -s {} ~/newlinks
You can use xargs to apply the links, so that your composite command might look like this:
find ~/myfiles/ | grep -vf results.txt | xargs make-my-links
and make-my-links would be a script something like this:
#!/bin/sh
for source in "$#"
do
ln -s "$source" -t ~/newlinks
done
The separate script and loop are used with xargs because it does not accept a command-template, but will (default) send as many of the inputs as it thinks will fit on a command-line.
So, you have 3 entities of type directory:
~/myfiles/: contains your files.
~/my-existing-links/: contains links to files from ~/myfiles/.
~/newlinks/: contains links to new files from ~/myfiles/
To me, the third entity is rather unnecessary. Why the new links aren't created directly in ~/my-existing-links/?
I would only use a script to update the list of links in ~/my-existing-links/, whenever new files are added in ~/myfiles/:
update_v1.sh
#!/bin/bash
for f in $(find ~/myfiles -type f); do
ln -sf "$f" "~/my-existing-links/$(basename $f)"
done
update_v2.sh
find ~/myfiles -type f -exec sh -c \
'for f; do ln -sf "$f" "~/my-existing-links/${f#*/}"; done' sh {} +
update_print.sh
#!/bin/bash
for f in $(find ~/myfiles -type f); do
if [[ ! -L "~/my-existing-links/${f#*/}" ]]; then
echo "Link not existing for $f"
fi
done
Thanks, Thomas and pasaba... I found a way to do it:
So I did the following from ~/newlinks :
while read line; do ln -s "$line" "${line##*/}" ; done < ~/myfiles/results2.txt
Thanks again for your time.
Related
I would like to preface this with I am a complete noob with scripting. So I have a situation where I need to manually look for a phone number that could live in one of hundreds of files.
so the logs live in the following directory.
/actlogs/sbclogger_archive
The logs file names are in directories numbered 01-31 inside of that directory and all the files are zipped.
Inside of those numbered directories are tons of files but the only ones I want to search are "sipd.logthenthedate.gz" and "sipmsg.logthenthedate.gz".
So I need to look in all the files in the following directory.
"/actlogs/sbclogger_archive"
Which has 31 directories labeled "01-31"
Then in each 01-31 there is hundreds of files the only ones I want to look are are "sipd.logthenthedate.gz" and "sipmsg.logthenthedate.gz".
The script I am using is below, please let me know what I could do to make this work.
#!/bin/bash
read -p "Enter a phone number: " text
read -p "Enter directory of log file's, Hint it should be /actlogs/sbclogger_archive: " directory
#arr=( $(find $directory -type f -exec grep -l "$text" {} \; | sort -r) )
#find $directory -type f -exec grep -qe "$text" {} \; -exec bash -c '
file=$(find $directory -type f -name 'sipd.log*' -exec grep -qe "$text" {} \; -exec bash -c 'select f; do echo $f; break; done' find-sh {} +;)
if [ -z "$file" ]; then
echo "No matches found."
else
echo "select tool:"
tools=("nano" "less" "vim" "quit")
select tool in "${tools[#]}"
do
case $tool in
"quit")
break
;;
*)
$tool $file
break
;;
esac
done
fi
This would give you the list of files matching:
find \( -name 'sipd.log[0-9]*.gz' -o -name 'sipmsg.log[0-9]*.gz' \) \
-exec sh -c 'gunzip -c {}| grep -m1 -q 888333' \; -print
./18/sipd.log20200118.gz
./7/sipd.log20200107.gz
Note: -m1 tells grep to stop after first match, since you need only the file name in this case, it's enough.
If you have zgrep, you can shorten it to:
find \( -name 'sipd.log[0-9]*.gz' -o -name 'sipmsg.log[0-9]*.gz' \) \
-exec zgrep -l '888333' {} \;
./18/sipd.log20200118.gz
./7/sipd.log20200107.gz
Also, some of the tools you are suggesting do not support gzip files (nano and some variants of less for example). In which case you might need to decompress the file and compress it again when done.
And, you might want to consider a loop if you want to "quit". Feeding the file list to the tool doesn't make sense.
Note: AFAIK zgrep doesn't do recursive:
DESCRIPTION
Zgrep invokes grep on compressed or gzipped files. These grep options will cause zgrep to terminate with an
error code:
(-[drRzZ]|--di*|--exc*|--inc*|--rec*|--nu*). All other options specified are passed directly to grep. If no file is specified, then
the
standard input is decompressed if necessary and fed to grep. Otherwise the given files are uncompressed if necessary and fed to
grep.
so zgrep -rl "$text" "$directory" or zgrep -rl --include 'simpd.log*.gz' "$test" {01..31} won't work except if you have a special zgrep
As you must unzip before using your tool, i would divide the problem in two blocks.
Firstly, i would expand the paths you need (looking under <directory> for the phone <text>), and then iterate to apply the tool (because some tools like vim or nano cannot be piped).
Try something like this:
#!/bin/bash
#...
# text/directory input stuff
#...
tmpdir=$(mktemp -d)
trap 'rm -rf ${tmpdir}' EXIT
while IFS= read -r file; do
unzipped=${tmpdir}/$(basename "${file}" .gz)
gunzip -c "${file}" > "${unzipped}"
${tool} "${unzipped}"
done < <(zgrep -lw "${text}" "${directory}"/{01..31}/{sipd.logthenthedate.gz,sipmsg.logthenthedate.gz} 2>/dev/null)
Above is the proposed invert-form by Charles Duffy following this Bash FAQ.
If you prefer to iterate an array, you could build in this way:
# shellcheck disable=SC2207
files=( $(zgrep -lw "${text}" "${directory}"/{01..31}/{sipd.logthenthedate.gz,sipmsg.logthenthedate.gz} 2>/dev/null) )
for file in "${files[#]}"; do
# etc.
as in our particular case, the files to match have no spaces in their names and shellcheck warning is not so important (hidden above).
BRs
I am currently working on an auto grading script for a class project. It has to be able to search any number of given directories lets say
for example
usr/autograder/jdoe/
jdoe contains two files house.c and readme.txt.
I need to create a file in jdoe called jdoe.pdf
Currently i'm using this line of code below to get the path to where i need to create the file. Where $1 is user input of the path containing the projects the auto grader will grade.
find $1 -name "*.txt" -exec sh -c "dirname {}"
When I try adding /somename.pdf to the end of this statement I get readme.txt/somename.pdf
along with another -exec to get the name for the file.
\; -exec sh -c "dirname {} xargs -n 1 basename" \;
I'm having problems combining these two into one working statement.
I'm new to unix programming and would appreciate any advice or help even if it means re-writing the code using different unix tools.
The main question here is how do I create files in a path other than the directory I call my script from. Thanks in advance.
How about this?
find "$1" -name "*.txt" -exec bash -c 'd=$(dirname "$1"); touch $d"/"$(basename "$d").pdf' - {} \;
You can create files in another path using change directory command (cd).
If you start your script in usr/autograder/script and want to change to usr/autograder/jdoe you can change directory with shell command cd ../jdoe (relative) or cd usr/autograder/jdoe (absolute).
Now you are in the directory of usr/autograder/jdoe and you are able to create files in this directory, for example gedit readme.txt will open gedit and creates the file in usr/autograder/jdoe.
The simplest way is to loop over the files returned by find and then do whatever you need to do.
For example:
find "$1" -type f -name "*.txt" -print0 | while IFS= read -r -d $'\0' filename; do
dir=$(dirname "$filename")
# create pdf file
touch "$dir/${dir##*/}.pdf"
done
(Note the use of find -print0 to correctly handle filenames containing whitespace and newline characters.)
Is this what you are looking for?
function process_file {
dir=$(dirname "$1")
name=$(basename "$1")
echo name is $name and dir is $dir;
cd "$dir"
touch "${dir##*/}.pdf" # or anything else
}
# export the function, so that it is known in the child processes
export -f process_file
find . -name '*.txt' -exec bash -c "process_file '{}'" \;
I am new to Linux. I am trying to write a shell script which will move files to certain folders based on their extension, like for example in my downloads folder, I have all files of mixed file types. I have written the following script
mv *.mp3 ../Music
mv *.ogg ../Music
mv *.wav ../Music
mv *.mp4 ../Videos
mv *.flv ../Videos
How can I make it run automatically when a file is added to this folder? Now I have to manually run the script each time.
One more question, is there any way of combining these 2 statements
mv *.mp3 ../../Music
mv *.ogg ../../Music
into a single statement? I tried using || (C programming 'or' operator) and comma but they don't seem to work.
There is no trigger for when a file is added to a directory. If the file is uploaded via a webpage, you might be able to make the webpage do it.
You can put a script in crontab to do this, on unix machines (or task schedular in windows). Google crontab for a how-to.
As for combining your commands, use the following:
mv *.mp3 *.ogg ../../Music
You can include as many different "globs" (filenames with wildcards) as you like. The last thing should be the target directory.
Two ways:
find . -name '*mp3' -or -name '*ogg' -print | xargs -J% mv % ../../Music
find . -name '*mp3' -or -name '*ogg' -exec mv {} ../Music \;
The first uses a pipe and may run out of argument space; while the second may use too many forks and be slower. But, both will work.
Another way is:
mv -v {*.mp3,*.ogg,*.wav} ../Music
mv -v {*.mp4,*.flv} ../Videos
PS: option -v shows what is going on (verbose).
I like this method:
#!/bin/bash
for filename in *; do
if [[ -f "$filename" ]]; then
base=${filename%.*}
ext=${filename#$base.}
mkdir -p "${ext}"
mv "$filename" "${ext}"
fi
done
incron will watch the filesystem and perform run commands upon certain events.
You can combine multiple commands on a single line by using a command separator. The unconditional serialized command separator is ;.
command1 ; command2
You can use for loop to traverse through folders and subfolders inside the source folder.
The following code will help you move files in pair from "/source/foler/path/" to "/destination/fodler/path/". This code will move file matching their name and having different extensions.
for d in /source/folder/path/*; do
ls -tr $d |grep txt | rev | cut -f 2 -d '.' | rev | uniq | head -n 4 | xargs -I % bash -c 'mv -v '$d'/%.{txt,csv} /destination/folder/path/'
sleep 30
done
I have the following problem using UNIX Commands. I wish to go through a large number of files and convert them using a command that converts them. My idea is to work like this: command *.fileending > *.newfileending
The problem is that I wish to keep the file-names and only replace the file-ending. Thus filename.fileending should become filename.newfileending. How do I achieve this?
Use a for loop:
for file in *.krn; do
hum2mid "$file" -o "${file%.krn}.mid"
done
In a single line: for file in *.krn; do hum2mid "$file" -o "${file%.krn}.mid"; done
To apply the command to files and subdirectories recursively, use the find|xargs pattern:
find -type f -name '*.krn' -print0 \
| xargs -0 -n1 sh -c 'hum2mid "$1" -o "/destination/dir/$(basename ${1%.krn}.mid)"' -
Note that this will overwrite already converted files, if a file from another directory has the same name.
rename .fileending .newfileending *
#!/bin/bash
ls -1 *.fileending | while read i; do
command "$i" > "${i/%.fileending/.newfileending}"
done
if you need process 'weird' filenames ( like with embedded '\n', for example ), you can use following trick:
create file foo.sh:
#!/bin/bash
command "$1" > "${1/%.fileending/.newfileending}"
, then do chmod +x foo.sh and finally find . -maxdepth 1 -a -type f -a -name '*.fileending' -print0 | xargs -0 -n 1 -J '%' ./foo.sh "%"
I want to recursively search a directory tree and get the 10 most recently modified files.
For each one of these files, i want to create a symlink in my /home/mostrecent/ directory.
I know i could solve this with a scripting language, but I'm a bit miffed that I can't do it with a linux command!
So far i have this:
find /home/myfiles -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail -n 10 | cut -c 32-
How do i create a symlink in /home/mostrecent for each one of these files, without using a scripting language?
Actually, bash is a scripting language, more than capable of doing that sort of stuff even from the command line :-)
Assuming that the command you posted works (and it seems to, based on my cursory testing), you can just do:
i=0
for f in $(CMD) ; do
ln -s $f $HOME/recent$i
((i++))
done
Or, as a one-liner:
i=0;for f in $(CMD);do ln -s $f $HOME/recent$i;((i++));done
This will create the files recent0 through recent9 in your home directory, which are symlinks to the most recent files.
Obviously, you should put your actual command where I've put the marker text CMD above. I've used the marker just so it formats nicely here on SO.
As Jan Hudec points out in a comment, that will only work for files without spaces, evil things in my opinion :-)
But, since people seem to use them, you can use the safer:
i=0
CMD | while read f; do
ln -s $f $HOME/recent$i
((i++))
done
And, again, the one-liner version:
i=0;CMD|while read f;do ln -s $f $HOME/recent$i;((i++));done
I solved this with sed.
All hail sed!
find /home/myfiles/ -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail -n 10 | cut -c 21- | sed -e "s/^/ln -s \"/" -e "s/$/\"/" -e "s/$/ \"\/home\recent\/\"/" | sh
If i pipe sed to cat instead of sh, this is the output:
ln -s "/home/myfiles/1.simplest" "/home/recent/"
ln -s "/home/myfiles/2.with space" "/home/recent/"
ln -s "/home/myfiles/3.with'apostraphe" "/home/recent/"
ln -s "/home/myfiles/4.with'apostrophe space" "/home/recent/"
Thanks for your help.
Create symlinks to several file types modified in the last 24 hrs, with the same filenames (but a different path of course)
Thanks to Pax, Jan and Jon, with a little modification...
Make a 'recent' directory
mkdir ~/recent
Create 'getrecentfiles.sh' and add...
#!/usr/bin/bash
find $HOME -mtime 0 -name \*.txt -print -o \
-mtime 0 -name \*.pdf -print -o \
-mtime 0 -name \*.extensionname -print -o | while read f; do
ln -s $f $HOME/recent/
done
filters:
-mtime (n*24hrs) is time since last modified (n=1 shows only files modified bw 24-48hrs ago)
-o is the OR operator for multiple files (default is AND)
Change it to executable, add it to your startup scripts and make a shortcut to ~/recent on your desktop, to have the latest files you want on hand!