I'm looking to run a Linux command that will recursively compare two directories and output only the file names of what is different. This includes anything that is present in one directory and not the other or vice versa, and text differences.
From the diff man page:
-q Report only whether the files differ, not the details of the differences.
-r When comparing directories, recursively compare any subdirectories found.
Example command:
diff -qr dir1 dir2
Example output (depends on locale):
$ ls dir1 dir2
dir1:
same-file different only-1
dir2:
same-file different only-2
$ diff -qr dir1 dir2
Files dir1/different and dir2/different differ
Only in dir1: only-1
Only in dir2: only-2
You can also use rsync
rsync -rv --size-only --dry-run /my/source/ /my/dest/ > diff.out
If you want to get a list of files that are only in one directory and not their sub directories and only their file names:
diff -q /dir1 /dir2 | grep /dir1 | grep -E "^Only in*" | sed -n 's/[^:]*: //p'
If you want to recursively list all the files and directories that are different with their full paths:
diff -rq /dir1 /dir2 | grep -E "^Only in /dir1*" | sed -n 's/://p' | awk '{print $3"/"$4}'
This way you can apply different commands to all the files.
For example I could remove all the files and directories that are in dir1 but not dir2:
diff -rq /dir1 /dir2 | grep -E "^Only in /dir1*" | sed -n 's/://p' | awk '{print $3"/"$4}' xargs -I {} rm -r {}
The approach of running diff -qr old/ new/ has one major drawback: it may miss files in newly created directories. E.g. in the example below the file data/pages/playground/playground.txt is not in the output of diff -qr old/ new/ whereas the directory data/pages/playground/ is (search for playground.txt in your browser to quickly compare). I also posted the following solution on Unix & Linux Stack Exchange, but I'll copy it here as well:
To create a list of new or modified files programmatically the best solution I could come up with is using rsync, sort, and uniq:
(rsync -rcn --out-format="%n" old/ new/ && rsync -rcn --out-format="%n" new/ old/) | sort | uniq
Let me explain with this example: we want to compare two dokuwiki releases to see which files were changed and which ones were newly created.
We fetch the tars with wget and extract them into the directories old/ and new/:
wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-2014-09-29d.tgz
wget http://download.dokuwiki.org/src/dokuwiki/dokuwiki-2014-09-29.tgz
mkdir old && tar xzf dokuwiki-2014-09-29.tgz -C old --strip-components=1
mkdir new && tar xzf dokuwiki-2014-09-29d.tgz -C new --strip-components=1
Running rsync one way might miss newly created files as the comparison of rsync and diff shows here:
rsync -rcn --out-format="%n" old/ new/
yields the following output:
VERSION
doku.php
conf/mime.conf
inc/auth.php
inc/lang/no/lang.php
lib/plugins/acl/remote.php
lib/plugins/authplain/auth.php
lib/plugins/usermanager/admin.php
Running rsync only in one direction misses the newly created files and the other way round would miss deleted files, compare the output of diff:
diff -qr old/ new/
yields the following output:
Files old/VERSION and new/VERSION differ
Files old/conf/mime.conf and new/conf/mime.conf differ
Only in new/data/pages: playground
Files old/doku.php and new/doku.php differ
Files old/inc/auth.php and new/inc/auth.php differ
Files old/inc/lang/no/lang.php and new/inc/lang/no/lang.php differ
Files old/lib/plugins/acl/remote.php and new/lib/plugins/acl/remote.php differ
Files old/lib/plugins/authplain/auth.php and new/lib/plugins/authplain/auth.php differ
Files old/lib/plugins/usermanager/admin.php and new/lib/plugins/usermanager/admin.php differ
Running rsync both ways and sorting the output to remove duplicates reveals that the directory data/pages/playground/ and the file data/pages/playground/playground.txt were missed initially:
(rsync -rcn --out-format="%n" old/ new/ && rsync -rcn --out-format="%n" new/ old/) | sort | uniq
yields the following output:
VERSION
conf/mime.conf
data/pages/playground/
data/pages/playground/playground.txt
doku.php
inc/auth.php
inc/lang/no/lang.php
lib/plugins/acl/remote.php
lib/plugins/authplain/auth.php
lib/plugins/usermanager/admin.php
rsync is run with theses arguments:
-r to "recurse into directories",
-c to also compare files of identical size and only "skip based on checksum, not mod-time & size",
-n to "perform a trial run with no changes made", and
--out-format="%n" to "output updates using the specified FORMAT", which is "%n" here for the file name only
The output (list of files) of rsync in both directions is combined and sorted using sort, and this sorted list is then condensed by removing all duplicates with uniq
On my linux system to get just the filenames
diff -q /dir1 /dir2|cut -f2 -d' '
I have a directory.
$ tree dir1
dir1
├── a
│ └── 1.txt
├── b
│ └── 2.txt
└── c
├── 3.txt
├── 4.txt
└── d
└── 5.txt
4 directories, 5 files
I have another directory.
$ tree dir2
dir2
├── a
│ └── 1.txt
├── b
└── c
├── 3.txt
├── 5.txt
└── d
└── 5.txt
4 directories, 4 files
I can diff two directories.
$ diff <(cd dir1; find . -type f | sort) <(cd dir2; find . -type f| sort)
--- /dev/fd/11 2022-01-21 20:27:15.000000000 +0900
+++ /dev/fd/12 2022-01-21 20:27:15.000000000 +0900
## -1,5 +1,4 ##
./a/1.txt
-./b/2.txt
./c/3.txt
-./c/4.txt
+./c/5.txt
./c/d/5.txt
rsync -rvc --delete --size-only --dry-run source dir target dir
Related
I guess that this question has been asked here, but I didn't find it.
I am at directory var/ and I have a folder var/A that have some files inside, what I want is to move this files inside A to var/. so what i want to do is the following:
from:
var
├── A
| ├── file1.txt
| └── file2.txt
└── file3.txt
to:
var
├── file1.txt
├── file2.txt
└── file3.txt
In /var I have tryed the following commands
sudo mv A /
sudo mv A/ ./
sudo mv A/ .
And no one had worked. Thank you in advance
I have found the answer:
mv A/* .
I hope it could help another one
You have to specify the right path or match
if you want to move the entire files or directory you have to something
like this using * (wildcards)
user#akuma:~/var$ mv ./A/* ./
You can type mv --help in your terminal for more information to see more option like below
user#akuma:~/WORKSPACE/Trash$ mv --help
Usage: mv [OPTION]... [-T] SOURCE DEST
or: mv [OPTION]... SOURCE... DIRECTORY
or: mv [OPTION]... -t DIRECTORY SOURCE...
Rename SOURCE to DEST, or move SOURCE(s) to DIRECTORY.
Mandatory arguments to long options are mandatory for short options too.
--backup[=CONTROL] make a backup of each existing destination file
-b like --backup but does not accept an argument
-f, --force do not prompt before overwriting
-i, --interactive prompt before overwrite
-n, --no-clobber do not overwrite an existing file
If you specify more than one of -i, -f, -n, only the final one takes effect.
--strip-trailing-slashes remove any trailing slashes from each SOURCE
argument
-S, --suffix=SUFFIX override the usual backup suffix
-t, --target-directory=DIRECTORY move all SOURCE arguments into DIRECTORY
-T, --no-target-directory treat DEST as a normal file
-u, --update move only when the SOURCE file is newer
than the destination file or when the
destination file is missing
-v, --verbose explain what is being done
-Z, --context set SELinux security context of destination
file to default type
--help display this help and exit
--version output version information and exit
The backup suffix is '~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX.
The version control method may be selected via the --backup option or through
the VERSION_CONTROL environment variable. Here are the values:
none, off never make backups (even if --backup is given)
numbered, t make numbered backups
existing, nil numbered if numbered backups exist, simple otherwise
simple, never always make simple backups
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Full documentation at: <http://www.gnu.org/software/coreutils/mv>
or available locally via: info '(coreutils) mv invocation'
I managed to replicate your situation.
.
└── var
├── A
│ ├── file1.txt
│ └── file2.txt
└── file3.txt
2 directories, 3 files
user#host:~/soQ$ mv var/A/*.txt var/ && rm -rf var/A/
user#host:~/soQ$ tree
.
└── var
├── file1.txt
├── file2.txt
└── file3.txt
Hope this helps!
If you want to do this from var just do mv A/*.txt . && rm -rf A/
Also, let's assume you have the following structure in your directory.
.
├── A
│ ├── file3.txt
│ └── folder1
│ ├── file1.txt
│ └── file2.txt
└── file4.txt
If you want to copy just the *.txt file in var you need to use find like this:
find ./A/* -type f -exec mv {} ${PWD} \; && rm A
This looks at all the files and folders and returns the path for each file (each .txt for example) and then executes mv on each line and moves the files into the current directory (${PWD}).
I have two folders, each with sub folders, and I want to see if there are any sub folders in one file that does not exist the other folder. I have tried this command:
diff -r file1 file2
but it does not provide the results that I want.
For example if file1 contains three folders A, B, and C and file 2 contains 1 folder B, then the output should be folders A and C.
diff -r dir1 dir2 | grep dir1 | awk '{print $4}' > difference1.txt
Explanation:
diff -r dir1 dir2 shows which files are only in dir1 and those
only in dir2 and also the changes of the files present in both directories if any.
diff -r dir1 dir2 | grep dir1 shows which files are only in dir1
awk to print only filename.
I have a buch of logs with names in "filename.logdate month year" (for example, filename.log25 Aug 2015, note there are space between the date/month/year) and I'd like to change them to "filename.logmonthdateyear" (for example filename.logOct052015, with no space).
These files are in a bunch of sub folders which makes it more challenging.
Parent Folder
--- sub folder1
file1
file2
--- sub folder2
file3
file4
etc.
Can anyone suggest a bash script that can do this?
Thank you!
find and rename should do the trick
strawman example:
to go from
...
├── foo/
│ ├── file name with spaces
│ └── bar/
│ └── another file with spaces
...
you can use
find foo/ -type f -exec rename 's/ //g' '{}' \;
to get
...
├── foo/
│ ├── filenamewithspaces
│ └── bar/
│ └── anotherfilewithspaces
...
in your case:
in your case, it would be something like
find path/to/files/ -type f -exec rename 's/ //g' '{}' \;
but you can use fancier filters in your find command like
find path/to/files/ -type f -name *.log* -exec rename 's/ //g' '{}' \;
to select only .log files in case there are other file names with spaces you don't want to touch
heads up:
as pointed out in the comments there's the potential to overwrite files if their names only differ by space placement (e.g., a bc.log and ab c.log if carelessly renamed would end up with a single abc.log).
for your case, you have two things on your side:
rename will give you a heads up as long as you're not using it's --force option
and will give you a helpful message like ./ab c.log not renamed: ./abc.log already exists
your files are named programatically, and you're stripping the spaces in dates, so, assuming that's all you have in there, you shouldn't have any problems
regardless, it's good to be mindful of this sort of thing
This is a way to do it with just Bash (4+) and 'mv':
# Prevent breakages when nothing matches patterns
shopt -s nullglob
# Enable '**' matches (requires Bash 4)
shopt -s globstar
topdir=$PWD
for folder in **/ ; do
# Work in the directory to avoid problems if its path has spaces
cd -- "$folder"
for file in *' '*' '* ; do
# Use the '-i' option to prevent silent clobbering
mv -i -- "$file" "${file// /}"
done
cd -- "$topdir"
done
If there is just one level of subfolders (as stated in the question), the requirement for Bash 4+ can be dropped: remove the shopts -s globstar, and change the first line of the outer loop to for folder in */ ; do.
How I can pack into tar-file only last directory from path?
For example, I have several paths
/usr/local/files/dir1/
file1.txt
file2.txt
file3.txt
/usr/local/files/dir2/
file3.txt
file4.txt
file5.txt
if I run command:
tar czf my_arch.tar.gz -C /usr/local/files/dir1 .
I gain only containment of dir1 catalog, without itself.
So I have - my_arch.tar.gz/file1.txt, file2.txt, file3.txt, But I need structure like that inside my archive -
my_arch.tar.gz/dir1/file1.txt, file2.txt, file3.txt
How I can do this?
Thank you.
try
cd /usr/local/files
tar -cvzf my_arch.tar.gz dir1
The -C directive will make you change into dir1 and thus not archive the folder, but its contents:
-C, --directory DIR
change to directory DIR
you cannot do this directly through tar.
here's my suggestion :
#!/bin/bash
mydir=/my_dir/whit/long_and/complicated_path/the_stuff_is_here
dirname=$(dirname $mydir )
basename=$(basename $mydir )
tar cvf /tmp/$basename.tar -C $dirname $basename
$ tar vczf tmp/export/files.tar.gz -C tmp/export src
structure for files.tar.gz
src
src/app
src/app/main.js
src/app/util
src/app/util/runtime.js
If I understand what you are asking correctly, you want your tar file to contain the directory.
Try it without the -C flag as in:
tar -czf my_arch.tar.gz /usr/local/files/dir1
If you specify -C then you directory path is ./. Probably the following works like you want:
$ touch asdf/foo/bar/{1,2,3}
$ tree asdf/
asdf/
└── foo
└── bar
├── 1
├── 2
└── 3
2 directories, 3 files
$ tar -cv -C asdf/foo/bar/ -f asdf.tar ./
./
./3
./2
./1
$ tar tf asdf.tar
./
./3
./2
./1
I am wondering - how can I move all the files in a directory except those files in a specific directory (as 'mv' does not have a '--exclude' option)?
Lets's assume the dir structure is like,
|parent
|--child1
|--child2
|--grandChild1
|--grandChild2
|--grandChild3
|--grandChild4
|--grandChild5
|--grandChild6
And we need to move files so that it would appear like,
|parent
|--child1
| |--grandChild1
| |--grandChild2
| |--grandChild3
| |--grandChild4
| |--grandChild5
| |--grandChild6
|--child2
In this case, you need to exclude two directories child1 and child2, and move rest of the directories in to child1 directory.
use,
mv !(child1|child2) child1
This will move all of rest of the directories into child1 directory.
Since find does have an exclude option, use find + xargs + mv:
find /source/directory -name ignore-directory-name -prune -print0 | xargs -0 mv --target-directory=/target/directory
Note that this is almost copied from the find man page (I think using mv --target-directory is better than cpio).
First get the names of files and folders and exclude whichever you want:
ls --ignore=file1 --ignore==folder1 --ignore==regular-expression1 ...
Then pass filtered names to mv as the first parameter and the second parameter will be the destination:
mv $(ls --ignore=file1 --ignore==folder1 --ignore==regular-expression1 ...) destination/
This isn't exactly what you asked for, but it might do the job:
mv the-folder-you-want-to-exclude somewhere-outside-of-the-main-tree
mv the-tree where-you-want-it
mv the-excluded-folder original-location
(Essentially, move the excluded folder out of the larger tree to be moved.)
So, if I have a/ and I want to exclude a/b/c/*:
mv a/b/c ../c
mv a final_destination
mkdir -p a/b
mv ../c a/b/c
Or something like that. Otherwise, you might be able to get find to help you.
This will move all files at or below the current directory not in the ./exclude/ directory to /wherever...
find -E . -not -type d -and -not -regex '\./exclude/.*' -exec echo mv {} /wherever \;
ls | grep -v exclude-dir | xargs -t -I '{}' mv {} exclude-dir
rename your directory to make it hidden so the wildcard does not see it:
mv specific_dir .specific_dir
mv * ../other_dir
#!/bin/bash
touch apple banana carrot dog cherry
mkdir fruit
F="apple banana carrot dog cherry"
mv ${F/dog/} fruit
# this removes 'dog' from the list F, so it remains in the
current directory and not moved to 'fruit'
Inspired by #user13747357 's answer.
First you can ls the file and filter them by:
ls | egrep -v '(dir_name|file_name.ext)'
Then you can run the following command to move the files except the specific ones:
mv $(ls | egrep -v '(dir_name|file_name.ext)') target_dir
* Note that I tested this inside a specific directory. Cross-directory operation should be more carefully executed :)
suppose you directory is
.
├── dir1
│ └── a.txt
├── dir2
│ ├── b.txt
│ └── hello.c
├── file1.txt
├── file2.txt
└── file3.txt
and you gonna put file1 file2 file3 into dir2.
you can use
mv $(ls -p | grep -v /) /dir2 to finish it, because
ls -p | grep -v / will print all files except directory in cwd.
For example, if I want to move all files/directories - except a specified file or directory - inside "var/www/html" to a sub-folder named "my_sub_domain", then I use "mv" with the command "!(what_to_exclude)":
$ cd /var/www/html
$ mv !(my_sub_domain) my_sub_domain
To exclude more I use "|" to seperate file/directory names:
$ mv !(my_sub_domain|test1.html) my_sub_domain
mv * exclude-dir
was the perfect solution for me