I have a working grep command that selects files meeting a certain condition. How can I take the selected files from the grep command and pipe it into a cp command?
The following attempts have failed on the cp end:
grep -r "TWL" --exclude=*.csv* | cp ~/data/lidar/tmp-ajp2/
cp: missing destination file operand after
‘/home/ubuntu/data/lidar/tmp-ajp2/’ Try 'cp --help' for more
information.
cp `grep -r "TWL" --exclude=*.csv*` ~/data/lidar/tmp-ajp2/
cp: invalid option -- '7'
grep -l -r "TWL" --exclude=*.csv* | xargs cp -t ~/data/lidar/tmp-ajp2/
Explanation:
grep -l option to output file names only
xargs to convert file list from the standard input to command line arguments
cp -t option to specify target directory (and avoid using placeholders)
you need xargs with the placeholder option:
grep -r "TWL" --exclude=*.csv* | xargs -I '{}' cp '{}' ~/data/lidar/tmp-ajp2/
normally if you use xargs, it will put the output after the command, with the placeholder ('{}' in this case), you can choose the location where it is inserted, even multiple times.
This worked for me when searching for files with a specific date:
ls | grep '2018-08-22' | xargs -I '{}' cp '{}' ~/data/lidar/tmp-ajp2/
To copy files to grep found directories, use -printf to output directories and -i to place the command argument from xarg (after pipe)
find ./ -name 'filename.*' -print '%h\n' | xargs -i cp copyFile.txt {}
this copies copyFile.txt to all directories (in ./) containing "filename"
grep -rl '/directory/' -e 'pattern' | xargs cp -t /directory
Related
I'm trying to copy files by mask with preserving folder structure(using --parents), I can't use cp -r --parents or rsync directly because of argument list too long error.
ls folder1/folder2/ | head | xargs -I {} cp -r --parents folder1/folder2/{}/neutral* neutral_data/
but seems asterix symbols don't work here as expected, instead I get few errors like:
cp: cannot stat 'folder1/folder2/folder3/neutral*': No such file or directory
What is the proper way of using asterix symbol in this context or maybe any other method to solve this problem?
Update:
Based on this answer https://unix.stackexchange.com/a/5247/221416 I tried
ls folder1/folder2/ | head | xargs -I {} sh -c cp -r --parents folder1/folder2/{}/neutral* neutral_data/
but it gives error:
cp: missing file operand
Here is a solution:
ls folder1/folder2/ | xargs -I {} bash -c "cp -r --parents folder1/folder2/{}/neutral* neutral_data/"
I have list of files a.xxx a.yyy. a.zzz
I need copy all files by select by extension
for example
ls *.xxx | xargs cp a.* dir
I write such code
ls mysql/db/*.MYD | xargs -n1 basename | sed 's/\.MYD//g' | xargs -i cp mysql/db/{}.* new_folder
but get error
cp: cannot stat 'mysql/db/ps_opc_social_customer.*'
The problem here is that the * in the last command gets expanded by the shell at the very instant when you press return, so the copy command does not get an expanded string, but the literal <file>.* string.
You need to get all the files you need in one go or to use a new shell to do the glob expansion for you:
ls mysql/db/*.MYD | xargs -n1 basename | sed 's/\.MYD//g' | xargs -i bash -c "cp mysql/db/{}.* new_folder"
I've been trying various combinations of xargs and piping but I just can't get the right result. Previous questions don't quite cover exactly what I want to do:
I have a source directory somewhere, lets say /foo/source, with a mix of different files
I want to copy just the csv files found in source to a different destination, say /foo/dest
But I ALSO at the same time need to remove 232 header rows (eg using tail)
I've figured out that I need to pipe the results of find into xargs, which can then run commands on each find result. But I'm struggling to tail then copy. If I pipe tail into cp, cp does not seem to receive the file (missing file operand). Here's some examples of what I've tried so far:
find /foo/source -name "*.csv" | xargs -I '{}' sh -c 'tail -n +232 | cp -t /foo/dest'
cp: missing file operand
find /foo/source -name "*.csv" | xargs -I '{}' sh -c 'tail -n +232 {} | cp -t /foo/dest'
Result:
cp: failed to access '/foo/dest': No such file or directory ...
find /foo/source -name "*.csv" | xargs -I '{}' sh -c 'tail -n +232 {} > /foo/dest/{}'
sh: /foo/dest/foo/source/0001.csv: No such file or directory ...
Any pointers would be really appreciated!
Thanks
Just use find with exec and copy the file name in a variable:
find your_dir -name "*.csv" -exec sh -c 'f="$1"; tail -n +5 "$f" > dest_dir/$(basename "$f")' -- {} \;
See f={} makes $f hold the name of the file, with the full path. Then, it is a matter of redirecting the output of tail into the file, stripping the path from it.
Or, based on Random832's suggestion below in comments (thanks!):
find your_dir -name "*.csv" -exec sh -c 'tail -n +5 "$1" > dest_dir/$(basename "$1")' -- {} \;
Your last command is close, but the problem is that {} is replaced with the full pathname, not just the filename. Use the basename command to extract the filename from it.
find /foo/source -name "*.csv" | xargs -I '{}' sh -c 'tail -n +232 {} > /foo/dest/$(basename {})'
As an alternative to find and xargs you could use a for loop, and as an alternative to tail you could use sed, consider this:
source=/foo/source
dest=/foo/dest
for csv in $source/*.csv; do sed '232,$ !d' $csv > $dest/$(basename $csv); done
Using GNU Parallel you would do:
find /foo/source -name "*.csv" | parallel tail -n +232 {} '>' /foo/dest/{/}
I am trying to delete erroneous emails based on finding the email address in the file via Linux CLI.
I can get the files with
find . | xargs grep -l email#example.com
But I cannot figure out how to delete them from there as the following code doesn't work.
rm -f | xargs find . | xargs grep -l email#example.com
Solution for your command:
grep -l email#example.com * | xargs rm
Or
for file in $(grep -l email#example.com *); do
rm -i $file;
# ^ prompt for delete
done
For safety I normally pipe the output from find to something like awk and create a batch file with each line being "rm filename"
That way you can check it before actually running it and manually fix any odd edge cases that are difficult to do with a regex
find . | xargs grep -l email#example.com | awk '{print "rm "$1}' > doit.sh
vi doit.sh // check for murphy and his law
source doit.sh
You can use find's -exec and -delete, it will only delete the file if the grep command succeeds. Using grep -q so it wouldn't print anything, you can replace the -q with -l to see which files had the string in them.
find . -exec grep -q 'email#example.com' '{}' \; -delete
I liked Martin Beckett's solution but found that file names with spaces could trip it up (like who uses spaces in file names, pfft :D). Also I wanted to review what was matched so I move the matched files to a local folder instead of just deleting them with the 'rm' command:
# Make a folder in the current directory to put the matched files
$ mkdir -p './matched-files'
# Create a script to move files that match the grep
# NOTE: Remove "-name '*.txt'" to allow all file extensions to be searched.
# NOTE: Edit the grep argument 'something' to what you want to search for.
$ find . -name '*.txt' -print0 | xargs -0 grep -al 'something' | awk -F '\n' '{ print "mv \""$0"\" ./matched-files" }' > doit.sh
Or because its possible (in Linux, idk about other OS's) to have newlines in a file name you can use this longer, untested if works better (who puts newlines in filenames? pfft :D), version:
$ find . -name '*.txt' -print0 | xargs -0 grep -alZ 'something' | awk -F '\0' '{ for (x=1; x<NF; x++) print "mv \""$x"\" ./matched-files" }' > doit.sh
# Evaluate the file following the 'source' command as a list of commands executed in the current context:
$ source doit.sh
NOTE: I had issues where grep could not match inside files that had utf-16 encoding.
See here for a workaround. In case that website disappears what you do is use grep's -a flag which makes grep treat files as text and use a regex pattern that matches any first-byte in each extended character. For example to match Entité do this:
grep -a 'Entit.e'
and if that doesn't work then try this:
grep -a 'E.n.t.i.t.e'
Despite Martin's safe answer, if you've got certainty of what you want to delete, such as in writing a script, I've used this with greater success than any other one-liner suggested before around here:
$ find . | grep -l email#example.com | xargs -I {} rm -rf {}
But I rather find by name:
$ find . -iname *something* | xargs -I {} echo {}
rm -f `find . | xargs grep -li email#example.com`
does the job better. Use `...` to run the command to offer the file names containing email.#example.com (grep -l lists them, -i ignores case) to remove them with rm (-f forcibly / -i interactively).
find . | xargs grep -l email#example.com
how to remove:
rm -f 'find . | xargs grep -l email#example.com'
Quick and efficent. Replace find_files_having_this_text with the text you want to search.
grep -Ril 'find_files_having_this_text' . | xargs rm
I'm using grep to generate a list of files I need to move:
grep -L -r 'Subject: \[SPAM\]' .
How can I pass this list to the mv command and move the files somewhere else?
If you want to find and move files that do not match your pattern (move files that don't contain 'Subject \[SPAM\]' in this example) use:
grep -L -Z -r 'Subject: \[SPAM\]' . | xargs -0 -I{} mv {} DIR
The -Z means output with zeros (\0) after the filenames (so spaces are not used as delimeters).
xargs -0
means interpret \0 to be delimiters.
The -L means find files that do not match the pattern. Replace -L with -l if you want to move files that match your pattern.
Then
-I{} mv {} DIR
means replace {} with the filenames, so you get mv filenames DIR.
This alternative works where xargs is not availabe:
grep -L -r 'Subject: \[SPAM\]' . | while read f; do mv "$f" out; done
This is what I use in Fedora Core 12:
grep -l 'Subject: \[SPAM\]' | xargs -I '{}' mv '{}' DIR
This is what helped me:
grep -lir 'spam' ./ | xargs mv -t ../spam
Of course, I was already in required folder (that's why ./) and moved them to neighboring folder. But you can change them to any paths.
I don't know why accepted answer didn't work. Also I didn't have spaces and special characters in filenames - maybe this will not work.
Stolen here: Grep command to find files containing text string and move them
mv `grep -L -r 'Subject: \[SPAM\]' .` <directory_path>
Assuming that the grep you wrote returns the files paths you're expecting.
Maybe this will work:
mv $(grep -l 'Subject: \[SPAM\]' | awk -F ':' '{print $1}') your_file
There are several ways but here is a slow but failsafe one :
IFS=$'\n'; # set the field separator to line break
for $mail in $(grep -L -r 'Subject: \[SPAM\]' .); do mv "$mail" your_dir; done;
IFS=' '; # restore FS
Work perfect fo me :
move files who contain the text withe the word MYSTRINGTOSEARCH to directory MYDIR.
find . -type f -exec grep -il 'MYSTRINGTOSEARCH' {} \; -exec mv {} MYDIR/ \;
I hope this helps
You can pass the result to the next command by using
grep ... | xargs mv {} destination
Check man xargs for more info.