I am trying to move specific files from one folder to another. Would the below work?
mkdir test
touch test1.sh
touch test2.sh
touch test3.sh
mkdir test2
find test/ | xargs -I% mv % test2
I think this can work:
find ./ -name "test*.sh" | xargs -I% mv % test2
There is somethin odd in your example:
If test1 does not contain any subdirectories, or if you want to move the subdirectories as they are, you could simply do a
mv test1/* test2
(Note that this would (by default) not move entries which start with a period. If this is a problem, you either should consider not using Posix shell but, say, bash or Zsh, or indeed could use find, for the safe side with the -prune option) .
The problem starts with subdirectories. The output of find contains all directories along with the files at the end. The mv inside the xargs would then move, say, a directory test1/foo, and if it later wants to process a file test1/foo/bar/baz.txt, the file is not here anymore. The overall effect would be that you would have moved all the subdirectory (as in my first solution which does not need find), but get in addition plenty of error messages.
Related
This is a little anecdote from earlier on why not running root is vital.
I was sorting my home directory and deleted a few compressed files I had, I wrote
ls . | grep -P 'zip|tar|7z' | xargs rm and thought, hey I could also write this as rm -r $(ls . | grep -P '...') I suppose.
The second part I didn't mean to use it since there was nothing to delete, it was morelike a mental exercise, I wrote it next to the last command with a 'divider' to visually compare them.
ls . | grep -P 'zip|tar|7z' | xargs rm **//** rm -r $(ls . | grep -P '...')
Being **//** the "divider" and ... the mental "substitute" for 'zip|tar..'
I thought this wouldn't run but to my surprise, it acted as rm -r /and tried to delete everything, luckily permissions saved me and nothing was deleted.
But I'm curious why it'd work that way,
my guess is that rm **//** somehow translated to rm / but I'm not sure.
In the zsh shell, **//** would expand to all names under / as well to all names below the current directory (recursively).
From an empty directory on my system:
$ echo **//**
/altroot /bin /boot /bsd /bsd.booted /bsd.rd /bsd.sp /dev /etc /extra /home /mnt /root /sbin /sys /tmp /tmp_mnt /usr /var /vol
Why? Well, **/ matches all directories recursively under the current directory. More importantly, it matches the current directory, but since the current directory's name is not available inside the current directory, there's no entry returned for that.
However, when you add a / to that to create **//, then you get a lone / back for the current directory. Again in an empty directory:
$ echo **//
/
Then, if you add a further ** to make **//**, you pick up all names from the root directory, together with all names from the current directory and below (directory names from the current directory and below would occur twice in the list).
Your xargs is calling
rm **//** rm -r $(ls . | grep -P '...')
If you're using GNU rm, it will helpfully rearrange the command line so that it is interpreted the same as
rm -r **//** rm $(ls . | grep -P '...')
What this does should now be clear.
If you want to delete all regular files in the current directory that have filename suffixes .zip, .tar or .7z, use
rm ./*.(zip|tar|7z)(.)
in the zsh shell. If want to do that recursively down into subdirectories, use
rm ./**/*.(zip|tar|7z)(.)
The glob qualifier (.) makes the globbing pattern only match regular files. You could even restrict it to files above a certain size, say 10MB, with ./**/*.(zip|tar|7z)(.Lm+10).
One difference is that the ls ... | xargs .... solution also works if there are really a lot of files involved, while your rm $( .... ) might produce a argument list too long error. But if this is not an issue in your case, an even simpler attempt would be (assuming here Zsh; I don't understand why you tagged this bash, since you explicitly refer to Zsh only in your question)
rm *(zip|tar|7z)*(N)
which would express your original statement; I believe however that you really meant
rm -- *.(zip|tar|7z)(N)
because the solution you posted would also remove a file tarpit.txt, for instance. The (N) flag is a frail attempt to treat the case, that you don't have any file matching the pattern. Without the (N), you would get an error message from Zsh, and rm would receive the unexpanded file pattern, and, since it is unlikely that a file of this name exists, would output a second error message. By using (N), Zsh would simply pass nothing in this case (without complaining), and in fact rm would be invoked without arguments. Of course you would then get a rm: missing operand on stderr, and if you don't like this, you can filter this message.
UPDATES:
As Kusalananda has pointed out in his/her comment, omitting the (N) would, by default, make zsh only print an error message, if no files match the pattern, but not cause rm to be invoked.
Also added the -- flag to rm to allow removal of, i.e., a file called -rf.tar.
I have 2 folders: folder_a & folder_b. In each of these folders there are a bunch of files. I am trying to use sed to move all of these files out of these folders and into my current working directory I am currently in.
My folder structure looks like this:
mytest:
a:
1.txt
2.txt
3.txt
b:
4.txt
5.txt
The command I am trying to use is:
find . -type d ! -iname '*.*' # find all folders other than root
| sed -r 's/.*/&\/*/' # add '/*' to each of the arguments
| sed -r 'p;s/.*/./' # output: a/* . b/* .
| xargs -n 2 mv # should be creating two commands: 'mv a/* .' and 'mv b/* .'
Unfortunately I get an error:
mv: cannot stat './aaa/*': No such file or directory
I also get the same error when I try this other strategy (using ls instead of mv):
for dir in */; do
ls $dir;
done;
Even if I use sed to replace the spaces in each directory name with '\ ', or surround the directory names with quotes I get the same error.
I'm not sure if these 2 examples are related in my misunderstanding of bash but they both seem to demonstrate my ignorance of how bash translates the output from one command into the input of another command.
Can anyone shed some light on this?
Update: Completely rewritten.
As #EtanReisner and #melpomene have noted, mv */* . or, more specifically, mv a/* b/* . is the most straightforward solution, but you state that this is in part a learning exercise, so the remainder of the answer shows an efficient find-based solution and explains the problem with the original command.
An efficient find-based solution
Generally, if feasible, it's best and most efficient to let find itself do the work, without involving additional tools; find's -exec action is like a built-in xargs, with {} representing the path at hand (with terminator \;) / all paths (with +):
find . -type f -exec echo mv -t . {} +
To be safe, his will just print the mv commands that would be executed; remove the echo to actually execute them.
This will execute a single[1] mv command to which all matching files are passed, and -t . moves them all to the current dir.
[1] If the resulting command line is too long (which is unlikely), it is split up into multiple commands, just as with xargs.
Operating on files (-type f) bypasses the need for globbing, as find will then enumerate all files for you (it also bypasses the need to exclude . explicitly).
Note that this solution works on entire subtrees, not just (immediate) subdirectories.
It's tempting to consider turning on Bash 4's globstar option and using mv */** ., but that won't work, because it will attempt to move directories as well, not just the files in them.
A caveat re -exec with +: it only works if {} - the placeholder for all paths - is the token immediately before the +.
Since you're on Linux, we can satisfy this condition by specifying the target folder for mv with option -t before the {}; on BSD-based systems such as OSX, you could not do that, because mv doesn't support -t there, so you'd have to use terminator \;, which means that mv is called once for every path, which is obviously much slower.
Why your command didn't work:
As #EtanReisner points out in a comment, xargs invokes the command specified without (implicitly) involving a shell, so globbing won't work; you can verify this with the following command:
echo '*' | xargs echo # -> '*' - NO globbing
If we leave the globbing issue aside, additional work would have been necessary to make your xargs command work correctly with folder names with embedded spaces (or other shell metacharacters):
find . -mindepth 1 -type d |
sed -r "s/.*/'&'\/* ./" | # -> '<input-path>'/* . (including single-quotes)
xargs -n 2 echo mv # NOTE: still won't work due to lack of globbing
Note how the (combined) sed command now produces a single output line '<input-path>'/* ., with the input path enclosed in embedded single-quotes, which is required for xargs to recognize <input-path> as a single argument, even if it contains embedded spaces.
(If your filenames contain single-quotes, you'd have to do more work; also note that since now all arguments for a given dir. are on a single line, you could use xargs -L 1 ....)
Also note how -mindepth 1 (only process paths at the subdirectory level or below) is used to skip processing of . itself.
The only way to make globbing happen is to get the shell involved:
find . -mindepth 1 -type d |
sed -r "s/.*/'&'\/* ./" | # -> '<input-path>'/* . (including single-quotes)
xargs -I {} sh -c 'echo mv {}' # works, but is inefficient
Note the use of xargs' -I option to treat each input line as its own argument ({} is a self-chosen placeholder for the input).
sh -c invokes the (default) shell to execute the resulting command, at which globbing does happen.
However, overall, this is quite inefficient:
A pipeline with 3 segments is used.
A shell instance is invoked for every input path, which in turn calls the mv utility.
Compare this to the efficient find-only solution above, which (typically) creates only 2 processes in total.
I want to copy recursively files to its parent folder for a specific file extension. For example:
./folderA/folder1/*.txt to ./folderA/*.txt
./folderB/folder2/*.txt to ./folderB/*.txt
etc.
I checked cp and find commands but couldn't get it working.
I suspect that while you say copy, you actually mean to move the files up to their respective parent directories. It can be done easily using find:
$ find . -name '*.txt' -type f -execdir mv -n '{}' ../ \;
The above command recurses into the current directory . and then applies the following cascade of conditionals to each item found:
-name '*.txt' will filter out only files that have the .txt extension
-type f will filter out only regular files (eg, not directories that – for whatever reason – happen to have a name ending in .txt)
-execdir mv -n '{}' ../ \; executes the command mv -n '{}' ../ in the containing directory where the {} is a placeholder for the matched file's name and the single quotes are needed to stop the shell from interpreting the curly braces. The ; terminates the command and again has to be escaped from the shell interpreting it.
I have passed the -n flag to the mv program to avoid accidentally overwriting an existing file.
The above command will transform the following file system tree
dir1/
dir11/
file3.txt
file4.txt
dir12/
file2.txt
dir2/
dir21/
file6.dat
dir22/
dir221/
dir221/file8.txt
file7.txt
file5.txt
dir3/
file9.dat
file1.txt
into this one:
dir1/
dir11/
dir12/
file3.txt
file4.txt
dir2/
dir21/
file6.dat
dir22/
dir221/
file8.txt
file7.txt
dir3/
file9.dat
file2.txt
file5.txt
To get rid of the empty directories, run
$ find . -type d -empty -delete
Again, this command will traverse the current directory . and then apply the following:
-type d this time filters out only directories
-empty filters out only those that are empty
-delete deletes them.
Fine print: -execdir is not specified by POSIX, though major implementations (at least the GNU and BSD one) support it. If you need strict POSIX compliance, you'll have to make do with the less safe -exec which would need additional thought to be applied correctly in this case.
Finally, please try your commands in a test directory with dummy files, not your actual data. Especially with the -delete option of find, you can loose all your data quicker than you might imaging. Read the man page and, if that is not enough, the reference manual of find. Never blindly copy shell commands from random strangers posted on the internet if you don't understand them.
$cp ./folderA/folder1/*.txt ./folderA
Try this commnad
Run something like this from the root(ish) directory:
#! /bin/bash
BASE_DIR=./
new_dir() {
LOC_DIR=`pwd`
for i in "${LOC_DIR}"/*; do
[[ -f "${i}" ]] && cp "${i}" ../
[[ -d "${i}" ]] && cd "${i}" && new_dir
cd ..
done
return 0
}
new_dir
This will search each directory. When a file is encountered, it copies the file up a directory. When a directory is found, it will move down into the directory and start the process over again. I think it'll work for you.
Good luck.
I am very new to the unix. I ran the following command.
ls -l | xargs rm -rf bark.*
and above command removed every directory in the folder.
Can any one explained me why ?
The -r argument means "delete recursively" (ie descend into subdirectories). The -f command means "force" (in other words, don't ask for confirmation). -rf means "descend recursively into subdirectories without asking for confirmation"
ls -l lists all files in the directory. xargs takes the input from ls -l and appends it to the command you pass to xargs
The final command that got executed looked like this:
rm -rf bark.* <output of ls -l>
This essentially removed bark.* and all files in the current directory. Moral of the story: be very careful with rm -rf. (You can use rm -ri to ask before deleting files instead)
rm(1) deleted every file and directory in the current working directory because you asked it to.
To see roughly what happened, run this:
cd /etc ; ls -l | xargs echo
Pay careful attention to the output.
I strongly recommend using echo in place of rm -rf when constructing command lines. Only if the output looks fine should you then re-run the command with rm -rf. When in doubt, maybe just use rm -r so that you do not accidentally blow away too much. rm -ir if you are very skeptical of your command line. (I have been using Linux since 1994 and I still use this echo trick when constructing slightly complicated command lines to selectively delete a pile of files.)
Incidentally, I would avoid parsing ls(1) output in any fashion -- filenames can contain any character except ASCII NUL and / chars -- including newlines, tabs, and output that looks like ls -l output. Trying to parse this with tools such as xargs(1) can be dangerous.
Instead, use find(1) for these sorts of things. To delete all files in all directories named bark.*, I'd run a command like this:
find . -type d -name 'bark.*' -print0 | xargs -0 rm -r
Again, I'd use echo in place of rm -r for the first execution -- and if it looked fine, then I'd re-run with rm -r.
The ls -l command gave a list of all the subdirectories in your current present-working-directory (PWD).
The rm command can delete multiple files/directories if you pass them to it as a list.
eg: rm test1.txt test2.txt myApp will delete all three of the files with names:
test1.txt
test2.txt
myApp
Also, the flags for the rm command you used are common in many a folly.
rm -f - Force deletion of files without asking or confirming
rm -r - Recurse into all subdirectories and delete all their contents and subdirectories
So, let's say you are in /home/user, and the directory structure looks like so:
/home/user
|->dir1
|->dir2
`->file1.txt
the ls -l command will provide the list containing "dir1 dir2 file1.txt", and the result of the command ls -l | xargs rm -rf will look like this:
rm -rf dir1 dir2 file1.txt
If we expand your original question with the example above, the final command that gets passed to the system becomes:
rm -rf di1 dir2 file1.txt bark.*
So, everything in the current directory gets wiped out, so the bark.* is redundant (you effectively told the machine to destroy everything in the current directory anyway).
I think what you meant to do was delete all files in the current directory and all subdirectories (recurse) that start with bark. To do that, you just have to do:
find -iname bark.* | xargs rm
The command above means "find all files in this directory and subdirectories, ignoring UPPERCASE/lowercase/mIxEdCaSe, that start with the characters "bark.", and delete them". This could still be a bad command if you have a typo, so to be sure, you should always test before you do a batch-deletion like this.
In the future, first do the following to get a list of all the files you will be deleting first to confirm they are the ones you want deleted.
find -iname bark.* | xargs echo
Then if you are sure, delete them via
find -iname bark.* | xargs rm
Hope this helps.
As a humorous note, one of the most famous instances of "rm -rf" can be found here:
https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commit/a047be85247755cdbe0acce6f1dafc8beb84f2ac
An automated script runs something like rm -rf /usr/local/........., but due to accidentally inserting a space, the command became rm -rf /usr /local/......, so this effectively means "delete all root folders that start with usr or local", effectively destroying the system of anyone who uses it. I feel bad for that developer.
You can avoid these kinds of bugs by quoting your strings, ie:
rm -rf "/usr/ local/...." would have provided an error message and avoided this bug, because the quotes mean that everything between them is the full path, NOT a list of separate paths/files (ie: you are telling rm that the file/folder has a SPACE character in its name).
Let's say you have a first.sh file in a directory: "/home/userbob/scripts/foo/". Basically I would like to know how to loop through specific directories, each time going back up to a higher level directory and repeating.
The .sh file has something like this pseudocode:
#!/bin/bash
curdi={$PATH} #where the first.sh file sits on the server
FOLDERS="$curdi/waffles/inner/
$curdi/pancakes/inner/
$curdi/bagels/inner/"
for f in $FOLDERS
do
cd $f
cp innerofinner/* .
cd $curdi
done
The idea is to somehow copy all the contents of /home/userbob/scripts/foo/waffles/inner/innerofinner to /home/userbob/scripts/foo/waffles/inner/
(and basically repeating just with the path having pancakes, bagels.etc.)
Can't do it for all directories (*) under /home/userbob/scripts/foo/ because there are some that I don't want to copy.
This should do it:
for name in waffles pancakes bagels
do
cp "$curdi/$name/inner/innferofinner/"* "$curdi/waffles/inner"
done
Walking file trees? Sounds like a job for find!
#!/usr/local/bin/env bash
# only environment variables should be all-caps
dirs=({bagels,pancakes}/inner)
find "${dirs[#]}" -type d -maxdepth 1 -mindepth 1 -name innerofinner -execdir bash -c 'cp "$1"/* .' -- {} \;
I did a partial path and assumed a working directory of /home/userbob/scripts/foo. An absolute path would work, too, and would look like
dirs=(/home/userbob/scripts/foo/{bagels,pancakes}/inner)
This finds all directories exactly one level below the listed directory that are named "innerofinner" and, in their parent directories, executs bash and a simple cp script.
If you're wondering how this works, read below.
The dirs=() syntax creates an empty array named dirs. dirs+(a b) creates an array with a at index 0 and b at index 1. Any whitespace-delimited string will work, here. In a shell script {a,b,c} expands to a b c but A{a,b,c}B expands to AaB AbB AcB. So specifying {bagels,pancakes}/inner is just a way to say both bagels/inner and pancakes/inner without having to type as much.
A variable in bash can be expanded with $foo or with ${foo}; these are the same. An array in shell can be expanded to all of its elements with ${foo[#]} delimited by spaces (if you know perl or php this will make some sense) and quoting the expansion (always a good idea in shell!) prevents spaces innside the variable from being processed again by the shell. Thus, "${dir[#]}" becomes bagels/inner pancakes/inner.
Knowing this we see that the find command has become find bagels/inner pancakes/inner -maxdepth 1 -mindepth 1 -type d -name innerofinner and if you execute this it will return exactly two lines: both full paths to each innerofinner directory. All we want now is to do something for each one, which -execdir does nicely.
Use a recursive function or invoke the script recursively.
I am not sure if I understand your problem statement correctly. Your psuedo code seems good. But, I see a problem with the following line.
curdi={$PWD}
It does not give you the directory where the script resides but gives the directory you are in. If your script directory is in the path and you are running the script from your home directory then $curdi would point to your home directory and not the directory where your script resides. This will lead to undesired results.
Incidentally, if you really wanted to do it in the way that your pseudo-script attempts it, you'd do it like this
#!/usr/bin/env bash
for f in "$PWD"/{waffles,pancakes,bagels}/inner ; do
cd "$f"
cp innerofinner/* .
# if you know for sure that it's one level up
cd ..
done
Presuming that $PWD is a good enough indicator of "current" directory for you. Me, I'd pass it in to the script.
#!/usr/bin/env bash
base="${1-$PWD}"
for f in "$base"/{waffles,pancakes,bagels}/inner ; do
cd "$f"
cp innerofinner/* .
cd ..
done
at call it like
breakfast.sh /home/userbob/scripts/foo/
find . \( -iname '*waffles*innferofinner*' -o \
-iname '*pancakes*innferofinner*' -o \
-iname '*baggels*innferofinner*' \) \
-type f \
-exec cp {} "`echo {} | sed 's:\(.*\)/[^/]\+/[^/]\+:\1:'` \;
Should do. Finds every file in the desired subdirs, then copies it based on its name.
HTH