I'm trying to copy files by mask with preserving folder structure(using --parents), I can't use cp -r --parents or rsync directly because of argument list too long error.
ls folder1/folder2/ | head | xargs -I {} cp -r --parents folder1/folder2/{}/neutral* neutral_data/
but seems asterix symbols don't work here as expected, instead I get few errors like:
cp: cannot stat 'folder1/folder2/folder3/neutral*': No such file or directory
What is the proper way of using asterix symbol in this context or maybe any other method to solve this problem?
Update:
Based on this answer https://unix.stackexchange.com/a/5247/221416 I tried
ls folder1/folder2/ | head | xargs -I {} sh -c cp -r --parents folder1/folder2/{}/neutral* neutral_data/
but it gives error:
cp: missing file operand
Here is a solution:
ls folder1/folder2/ | xargs -I {} bash -c "cp -r --parents folder1/folder2/{}/neutral* neutral_data/"
Related
I am tring to use xargs -a to read the contents of a file that has a list of filenames in it.
My directory working in looks like:
backups
file1.bak
file2.bak
file3.bak
bakfiles.txt
File name with filenames in it: bakfiles.txt
bakfiles.txt contents:
file1.bak
file2.bak
file3.bak
So essentially I'm trying to copy file1.bak,file2.bak,file3.bak into the folder backups. But using the contents of bakfiles.txt to do so.
I tried:
xargs -a bakfiles.txt | cp {} backups
But I get the error:
cp: cannot stat `{}': No such file or directory
I should mention i also tried:
xargs -a bakfiles.txt cp {} backups
And get the error:
cp: target `file3.bak' is not a directory
This works for me on Windows 7 using mks toolkit version of 'xargs'
cat bakfiles.txt | xargs -I '{}' cp '{}' backups/'{}'
From the following link i figured it out:
https://superuser.com/questions/180251/copy-list-of-files
Heres the command:
xargs -a bakfiles.txt cp -t backups
I have a working grep command that selects files meeting a certain condition. How can I take the selected files from the grep command and pipe it into a cp command?
The following attempts have failed on the cp end:
grep -r "TWL" --exclude=*.csv* | cp ~/data/lidar/tmp-ajp2/
cp: missing destination file operand after
‘/home/ubuntu/data/lidar/tmp-ajp2/’ Try 'cp --help' for more
information.
cp `grep -r "TWL" --exclude=*.csv*` ~/data/lidar/tmp-ajp2/
cp: invalid option -- '7'
grep -l -r "TWL" --exclude=*.csv* | xargs cp -t ~/data/lidar/tmp-ajp2/
Explanation:
grep -l option to output file names only
xargs to convert file list from the standard input to command line arguments
cp -t option to specify target directory (and avoid using placeholders)
you need xargs with the placeholder option:
grep -r "TWL" --exclude=*.csv* | xargs -I '{}' cp '{}' ~/data/lidar/tmp-ajp2/
normally if you use xargs, it will put the output after the command, with the placeholder ('{}' in this case), you can choose the location where it is inserted, even multiple times.
This worked for me when searching for files with a specific date:
ls | grep '2018-08-22' | xargs -I '{}' cp '{}' ~/data/lidar/tmp-ajp2/
To copy files to grep found directories, use -printf to output directories and -i to place the command argument from xarg (after pipe)
find ./ -name 'filename.*' -print '%h\n' | xargs -i cp copyFile.txt {}
this copies copyFile.txt to all directories (in ./) containing "filename"
grep -rl '/directory/' -e 'pattern' | xargs cp -t /directory
I have some file like
love.txt
loveyou.txt
in directory useful; I want to copy this file to directory /tmp.
I use this command:
find ./useful/ -name "love*" | xargs cp /tmp/
but is doesn't work, just says:
cp: target `./useful/loveyou.txt' is not a directory
when I use this command:
find ./useful/ -name "love*" | xargs -i cp {} /tmp/
it works fine,
I want to know why the second works, and more about the usage of -i cp {}.
xargs puts the words coming from the standard input to the end of the argument list of the given command. The first form therefore creates
cp /tmp/ ./useful/love.txt ./useful/loveyou.txt
Which does not work, because there are more than 2 arguments and the last one is not a directory.
The -i option tells xargs to process one file at a time, though, replacing {} with its name, so it is equivalent to
cp ./useful/love.txt /tmp/
cp ./useful/loveyou.txt /tmp/
Which clearly works well.
When using the xargs -i command, {} is substituted with each element you find. So, in your case, for both "loveyou.txt" and "love.txt", the following command will be run:
cp ./useful/loveyou.txt /tmp/
cp ./useful/love.txt /tmp/
if you omit the {}, all the elements you find will automatically be inserted at the end of the command, so, you will execute the nonsensical command:
cp /tmp/ ./useful/loveyou.txt ./useful/love.txt
xargs appends the values fed in as a stream to the end of the command - it does not run the command once per input value. If you want the same command run multiple times - that is what the -i cp {} syntax is for.
This works well for commands which accept a list of arguments at the end (e.g. grep) - unfortunately cp is not one of those - it considers the arguments you pass in as directories to copy to, which explains the 'is not a directory' error.
The first example will do this:
cp /tmp/ love.txt loveyou.txt
Which can't be done, since they attempt to copy the directory /tmp and the file love.txt to the file loveyou.txt.
In the second example, -i tells xargs to replace every instance of {} with the argument, so it will do:
cp love.txt /tmp/
cp loveyou.txt /tmp/
find ./useful/ -name "love*" | xargs cp -t /tmp/
You might avoid xargs that way:
find ./useful/ -name "love*" -exec sh -c 'cp "$#" /tmp' sh {} +
I don't expected that this will be a problem. Because I thought the coreutils support these things and then, that a dirty combination of cp ls and rm would be enough.
However, this was not the case and I would be really much appreciated if you now explain me why my approuch is failing and further, how I should do this in a proper way.
Code
function CheckoutFolder {
local dir=$1
mkdir "$dir/.CheckoutFolderTmp"
(
cd "$dir" \
&& cp -R $(ls -Q -A "$dir" --ignore=".CheckoutFolderTmp") "$dir/.CheckoutFolderTmp" \
&& rm -Rf $(ls -Q -A "$dir" --ignore=".CheckoutFolderTmp")
)
mv "$dir/.CheckoutFolderTmp" "$dir/src"
mkdir -p "$dir/"{build,log}
}
Sample output
++ CheckoutFolder /home/tobias/Develop/src/thelegacy/RCMeta
++ local dir=/home/tobias/Develop/src/thelegacy/RCMeta
++ mkdir /home/tobias/Develop/src/thelegacy/RCMeta/.CheckoutFolderTmp
++ cd /home/tobias/Develop/src/thelegacy/RCMeta
+++ ls -Q -A /home/tobias/Develop/src/thelegacy/RCMeta --ignore=.CheckoutFolderTmp
++ cp -R '"build"' '"buildmythli.sh"' '"CMakeLists.txt"' '".directory"' '".libbuildmythli.sh"' '"log"' '"RCMeta"' '"RCMetaTest"' '"src"' /home/tobias/Develop/src/thelegacy/RC
cp: cannot stat `"build"': No such file or directory
cp: cannot stat `"buildmythli.sh"': No such file or directory
cp: cannot stat `"CMakeLists.txt"': No such file or directory
cp: cannot stat `".directory"': No such file or directory
cp: cannot stat `".libbuildmythli.sh"': No such file or directory
cp: cannot stat `"log"': No such file or directory
cp: cannot stat `"RCMeta"': No such file or directory
cp: cannot stat `"RCMetaTest"': No such file or directory
cp: cannot stat `"src"': No such file or directory
++ mv /home/tobias/Develop/src/thelegacy/RCMeta/.CheckoutFolderTmp /home/tobias/Develop/src/thelegacy/RCMeta/src
++ mkdir -p /home/tobias/Develop/src/thelegacy/RCMeta/build /home/tobias/Develop/src/thelegacy/RCMeta/log
++ return 0
Mythli
As Les says, ls -Q is putting quotation-marks around the filenames, and those quotation-marks are getting passed in the arguments to cp and rm. (The use of quotation-marks to quote and delimit arguments is an aspect of the Bash command-line, when you actually type in a command; it doesn't work when you're passing the output of one command into another.) In general, parsing the output of ls is not generally a good idea.
Here is an alternative approach:
function CheckoutFolder() (
cd "$1"
mkdir .CheckoutFolderTmp
find -mindepth 1 -maxdepth 1 -not -name .CheckoutFolderTmp \
-exec mv {} .CheckoutFolderTmp/{} \;
mv .CheckoutFolderTmp src
mkdir build log
)
(Note that I surrounded the function body with parentheses (...) rather than curly-brackets {...}. This causes the whole function to be run in a subshell.)
The $(ls ...) command is putting unwanted quotes around the names. Consider using xargs and back-quotes instead. For example...
(cd "$dir" && cp -R `ls -Q -A "$dir" --ignore=".CheckoutFolderTmp"` "$dir/.CheckoutFolderTmp" && ls -Q -A "$dir" --ignore=".CheckoutFolderTmp" | xargs rm -Rf )
The cp output is not too friendly, but it does give the information you need.
cp: cannot stat '"build"': No such file or directory
Skip to the final statement "no such file or directory". The "cannot stat" cryptic, but it means that "cp" used "stat" to get some information about the file or directory it was trying to copy. "stat" failed. It failed with the errno for "no such file or directory" on a file (or directory) named '"build"'. That's because, the actual argument internal to cp is "build" (notice) the quotes), while the file name you want is build (notice no quotes).
The $(ls... ) is called with -Q to put quotes on (presumably to handle file names with spaces and commas and other offending characters). But the $(ls...) already puts quotes on for you. xargs can also handle funky filenames if you use -0.
I have txt file with content like this
/home/username/Desktop/folder/folder3333/IMAGw488.jpg
/home/username/Desktop/folder/folder3333/IMAG04f88.jpg
/home/username/Desktop/folder/folder3333/IMAGe0488.jpg
/home/username/Desktop/folder/folder3333/IMAG0r88.jpg
/home/username/Desktop/folder/folder3333/
/home/username/Desktop/folder/
/home/username/Desktop/folder/IMAG0488.jpg
/home/username/Desktop/folder/fff/fff/feqw/123.jpg
/home/username/Desktop/folder/fffa/asd.png
....
these are filenames paths but also paths of folders.
The problem I want to solve is to create all folders that doesn't exist.
I want to call mkdir command for every folder that does not exist
How can I do this on easy way ?
Thanks
This can be done in native bash syntax without any calls to external binaries:
while read line; do mkdir -p "${line%/*}"; done < infile
Or perhaps with a just a single call to mkdir if you have bash 4.x
mapfile -t arr < infile; mkdir -p "${arr[#]%/*}"
How about...
for p in $(xargs < somefile.txt);
do
mkdir -p $(dirname ${p})
done
xargs -n 1 dirname <somefile.txt | xargs mkdir -p
It can be done without loop also (provided input file not huge):
mkdir -p $(perl -pe 's#/(?!.*/).*$##' file.txt)
If you have file "file1" with filenames you could try this oneliner:
cat file1 |xargs -I {} dirname "{}"| sort -u | xargs -I{} mkdir -p "{}"
Use of:
xargs -I{} mkdir -p "{}"
ensures that even path names with spaces will be created
Using a perl one-liner and File::Path qw(make_path):
perl -MFile::Path=make_path -lne 'make_path $_' dirlist.txt