I want to mimic copying a directory structure recursively (as in cp -r or rsync -a), but only touch the copied files, i.e. make all the copied files empty.
The specific use case is for a Snakemake pipeline; Snakemake looks for existing files in order to decide whether to re-run a pipeline step, and I want to make it believe the steps have already been run while avoiding fully downloading all the files.
This is a little kludgy, but you could pipe the output of find or rsync -nv into a little bash loop with mkdir -p and touch:
find /some/dir -type f | while read FILE; do
mkdir -p $(dirname $FILE)
touch $FILE
done
Related
Update
Okay, I've got inotify going, but I'm not sure how to pipe it into cp.
I can run inotifywait -rm source/ and it will print to the terminal when changes happen to the file or new files are created and so forth.
Now I need to listen to when any change happens and just run cp -u source/* destination/
How do I do this in bash in one line?
Old stuff below here...
I'm using the cp command to copy files for my static site builder. I'm trying to also create a watch feature.
Can I use cp to somehow watch a file or folder full of files and other nested folders of files, and upon any saved changes to those files and folders copy them to a destination folder, effectively syncing one directory to another?
Or rsync or another command line tool?
I think this should work for you:
inotifywait -rm -e CLOSE_WRITE --format "%w" source | stdbuf -o0 sed 's#/$##' | xargs -n1 -I{} rsync -Rva {} destination
Using inotify to watch for close_write events (rather than all or modify), only outputting the directory in which the change occurred, trimming the trailing slash and feeding that to rsync (telling it to preserver the full path and not use compression).
In makefile, I am doing something like this -
#while read -r file; do \
if [ ! -z "$$file" ]; then \
cp -R path/to/someplace/$$file path/to/someplace/else/$$file \
fi \
done <filename.txt
filename.txt contains files and folders like -
abc/*.txt
folder1/
folder2/
textfile.txt
I am able to copy files and folders but in the case of abc/*.txt it shows an error:
cp: target `/path/to/someplace/else/*.txt' is not a directory.
Is there some way possible to copy these files with wildcard characters?
Actually the problem is not in the pattern expansion. That bit would appear to work as expected. The error is actually due to the resulting cp command. You end up with for instance:
cp abc/a.txt abc/b.txt other/path/abc/*.txt
However, for multiple source files cp expect destination to be a directory, which other/place/abc/* is not (it could be if you created it, but is unlikely what you wanted).
That said. You could for instance create the target directory by calling mkdir -p other/path/`dirname $$file` (escaped $ for make) and with cp use `dirname $$file` as well for destination.
There are few caveats. For instance if absolute paths were to be encountered or globing characters where also used on directories leading up to the files themselves (dir?/*.txt, some/*/file). It would perhaps be safer to use for instance tar or cpio:
tar cf - -C /source/path src/*.txt | tar xf - -C /target/path/
I have a directory that I want to copy all of it, but to a directory with a different name.
Example:
/Home/user/DirA-Web
copy its contents to (but it needs to be created)
/Home/user/version1/DirB-Img
/Home/user/version2/DirB-Img
I could always copy it and the rename it, I suppose.
Edit: I currently rsync the directories to the desired location and them mv in a for loop to rename them. I am looking for something cleaner.
If the directory
/Home/user/version1/
exists, a simple cp will do:
cp -r /Home/user/DirA-Web /Home/user/version1/DirB-Img
If not, you need to use mkdir beforehand, because cp has no option to
create your target directory recursively:
mkdir -p /Home/user/version1/DirB-Img && cp -r /Home/user/DirA-Web /Home/user/version1/DirB-Img
Okay so essentially what I'm doing, is I'm taking all the directories inside of the /servers/ folder, and moving them to a secondary hard drive mounted at /media/backupdrive/. This script is ran once a day, so it makes the directory with the name of the date, and should copy the folders directly over there (The reason I have to do it this way is because my client has limited disk space on his main hard drive and his worlds are upwards of 6-7gb each). Anyway, I can get them to copy the folders to /media/backupdrive/currentdate, but then when I try to compress it, it says it can't compress an empty directory or something along the lines of that.
Here's the code:
#!bin/bash
folderName=$(date +"%m-%d-%y")
mkdir "/media/backupdrive/$folderName"
for i in servers/*; do
cp -rf $i /media/backupdrive/$folderName/
cd /media/backupdrive/$folderName/
tar -C ${i:8} -czvf "${i:8}.tar.gz"
cd /root/multicraft/
done
Sorry for the image, it was on a virtual machine and I had to re-type it, because I couldn't copy and paste.
It looks to me like your tar command is missing its input (e.g., a final "."), and therefore says, "tar: Cowardly refusing to create an empty archive".
Your script appears to work for me with this tar command:
tar -C ${i#servers/} -czvf "${i#servers/}.tar.gz" .
I'd try a slightly different approach. tar by itself doesn't use temporary files, so you could tar the sources directly to the destination and compress them wizh gzip in a second step.
#!bin/bash
dst="/media/backupdrive/$(date +"%m-%d-%y")"
for d in servers/*; do
tarfile="$dst/${d#servers/}.tar"
tar -C "$d" -cvf "$tarfile" .
gzip -9 "$tarfile"
done
This question already has answers here:
Linux: copy and create destination dir if it does not exist
(27 answers)
Closed 7 years ago.
When copying a file using cp to a folder that may or may not exist, how do I get cp to create the folder if necessary? Here is what I have tried:
[root#file nutch-0.9]# cp -f urls-resume /nosuchdirectory/hi.txt
cp: cannot create regular file `/nosuchdirectory/hi.txt': No such file or directory
To expand upon Christian's answer, the only reliable way to do this would be to combine mkdir and cp:
mkdir -p /foo/bar && cp myfile "$_"
As an aside, when you only need to create a single directory in an existing hierarchy, rsync can do it in one operation. I'm quite a fan of rsync as a much more versatile cp replacement, in fact:
rsync -a myfile /foo/bar/ # works if /foo exists but /foo/bar doesn't. bar is created.
I didn't know you could do that with cp.
You can do it with mkdir ..
mkdir -p /var/path/to/your/dir
EDIT
See lhunath's answer for incorporating cp.
One can also use the command find:
find ./ -depth -print | cpio -pvd newdirpathname
mkdir -p `dirname /nosuchdirectory/hi.txt` && cp -r urls-resume /nosuchdirectory/hi.txt
There is no such option. What you can do is to run mkdir -p before copying the file
I made a very cool script you can use to copy files in locations that doesn't exist
#!/bin/bash
if [ ! -d "$2" ]; then
mkdir -p "$2"
fi
cp -R "$1" "$2"
Now just save it, give it permissions and run it using
./cp-improved SOURCE DEST
I put -R option but it's just a draft, I know it can be and you will improve it in many ways. Hope it helps you
rsync is work!
#file:
rsync -aqz _vimrc ~/.vimrc
#directory:
rsync -aqz _vim/ ~/.vim
cp -Rvn /source/path/* /destination/path/
cp: /destination/path/any.zip: No such file or directory
It will create no existing paths in destination, if path have a source file inside.
This dont create empty directories.
A moment ago i've seen xxxxxxxx: No such file or directory, because i run out of free space. without error message.
with ditto:
ditto -V /source/path/* /destination/path
ditto: /destination/path/any.zip: No space left on device
once freed space cp -Rvn /source/path/* /destination/path/ works as expected