When I run mv * with no destination directory on a directory with say 10 files, I get an error as follows
root#tryit-apparent:~/test2# ls
file1.txt file10.txt file2.txt file3.txt file4.txt file5.txt file6.txt file7.txt file8.txt file9.txt
root#tryit-apparent:~/test2# mv *
mv: target 'file9.txt' is not a directory
When I run it on a directory with two files it overwrites the file with one just file.
root#tryit-apparent:~/test# ls
tempfile tempfile2
root#tryit-apparent:~/test# mv *
root#tryit-apparent:~/test# ls
tempfile2
I read the man pages but couldn't understand this behaviour. Would like to know what's causing this behavior and what's going on under the hood?
What other linux commands have such pitfalls and have destructive actions that are executed silently if the user is not aware of such behavior?
In Unix, unlike some other OSes, wildcards like * are expanded by the shell, before being passed to the command being run. So when you run mv * with tempfile and tempfile2 as the only files in the current directory, what the shell actually executes is mv tempfile tempfile2, which as normal will rename the first file over the second one, erasing the previous contents of tempfile2. The shell doesn't know or care that this command treats its last argument specially, and mv has no way of knowing that its two arguments came from a wildcard expansion. Hence the behavior you're seeing.
You can have similar issues even with more than two files. For instance, if you have files named tempfile1 through tempfile9 and a subdirectory named zyzzx, then mv * will move all your temp files into the zyzzx subdirectory.
Mostly, you just have to be aware that this is how wildcards work, and use caution with commands that treat one of their arguments specially (e.g. as a destination). cp is another one to watch out for, for the same reason. For interactive usage, you may want to get used to using the -i option to mv and cp, which asks for confirmation before overwriting files; or use an alias to make this the default.
Move is intented to move or rename a file or a directory, so you need a source and a destination.
If the path of the file is unchange then it becomes a rename operation.
If the path changes and the name remains the same it's a move.
You can do both by chaning the path and the name.
Man pages can be challenging to wrap your head around.
Googling can help: https://www.howtoforge.com/linux-mv-command/
Off the top of my head, you could do a cp operation followed by a rm to achieve similar results, but that's two steps, rather than one.
Related
Im trying to run mv command using shell script but it gives me
mv: cannot stat `/opt/logs/merchantportal/logger.log.20160501.*': No such file or directory
mv: cannot stat `/opt/logs/merchantapi/logger.log.20160501.*': No such file or directory
// THIS IS MY SHELL SCRIPT
#!/bin/bash
now="$(date +'%Y%m%d')"
merchantPortalLogsPath="/opt/logs/merchantportal"
merchantApiLogsPath="/opt/logs/merchantapi"
currentDate="$(date +%Y%m%d)"
olderDate="$(date "+%Y%m%d" -d "1 days ago")"
merchantPortalLogsPathBackup=$merchantPortalLogsPath"."$olderDate
merchantApiLogsPathBackup=$merchantApiLogsPath"."$olderDate
mkdir $merchantPortalLogsPathBackup
mkdir $merchantApiLogsPathBackup
echo $merchantPortalLogsPath"/logger.log."$olderDate".*" $merchantPortalLogsPathBackup"/"
echo $merchantApiLogsPath"/logger.log."$olderDate".*" $merchantApiLogsPathBackup"/"
mv $merchantPortalLogsPath"/logger.log."$olderDate".*" $merchantPortalLogsPathBackup"/"
mv $merchantApiLogsPath"/logger.log."$olderDate".*" $merchantApiLogsPathBackup"/"
// BUT DIRECTORY IS CREATED SUCCESSFULLY
".*"
Putting the * inside double quotes will prevent the shell from treating that as a wildcard and will instead take it as a literal * character. Instead, change your script so that it does not double quote the *. For example:
mv ${merchantPortalLogsPath}/logger.log.${olderDate}.* ${merchantPortalLogsPathBackup}/
mv ${merchantApiLogsPath}/logger.log.${olderDate}.* ${merchantApiLogsPathBackup}/
Note: Technically should actually double quote the variable expansions to handle paths with spaces and other special characters in them. But I have not shown that to focus just on the problem at hand.
The log directories exist, but the log files within those directories did not. Since mv cannot move from data a log file that doesn't first exist, it complains, rather vaguely:
No such file or directory
Note: IMHO vague error messages are documentation/interface bugs -- if the error had said only:
No such file
And not left the user wondering if there was a missing directory it would have seemed less puzzling, since that message would clearly imply that the directory where the file was supposed to exist did in fact exist.
But consider this egregious GNU mv example, where a directory /tmp/a/ does not exist:
mv /tmp/a/b/c/d /tmp/foo
Output to STDERR:
mv: cannot stat '/tmp/a/b/c/d': No such file or directory
Now, the directory /tmp/a/ doesn't exist, and also the directories /tmp/b/ and /tmp/a/b/c/, and the file /tmp/a/b/c/d, none of them exist. The user is given no indication of which of those is the problem, and it's even possible (unusual, but possible) that /tmp/ doesn't exist. Where as writing just a few lines of code could output an error message that said something so much more useful, like:
mv: cannot stat '/tmp/a/b/c/d': `/tmp/` exists, but not directory `/tmp/a/`
...which collectively would probably save years of user-time.
I have a directory containing a set of subdirectories and files. I need to recursively copy all the content of this directory to all the subdirectories of another directory, also recursively.
How do I achieve this, preferably without using a script and only with the cp command?
You can write this in a script but you don't have to. Just write it line by line in the terminal:
# $TARGET is the directory containing subdirectories where you want to STORE the copies
# $SOURCE is the directory containing the subdirectories you want to COPY
for dir in $(ls $TARGET); do
cp -r $SOURCE/* $TARGET/$dir
done
Only uses cp and runs on both bash and zsh.
You can't. cp can copy multiple sources but will only copy to a single destination. You need to arrange to invoke cp multiple times - once per destination - for what you want to do; using, as you say, a loop or some other tool.
The first part of the command before the pipe instruct tar to create an archive of everything in the current directory and write it to standard output (the – in place of a file-name frequently indicates stdout).
tar cf - * | ( cd /target; tar xfp -)
The commands within parentheses cause the shell to change directory to the target directory and untar data from standard input. Since the cd and tar commands are contained within parentheses, their actions are performed together.
The -p option in the tar extraction command directs tar to preserve permission and ownership information, if possible given the user executing the command. If you are running the command as superuser, this option is turned on by default and can be omitted.
Also you can use the following command, but it seems to be quite slower than tar;
cp -a * /target
I'm looking for a good shell one liner to move or rename a file inside a directory, where the target and destination parent directories are the same, and different than the current working directory. For example, the thing I don't want to write:
$ mv /usr/share/nginx/html/app.xml /usr/share/nginx/html/index.html
How can I do this same thing without typing '/usr/share/nginx/html/' twice or using multiple commands (to switch directory, pushd, etc)?
You can use braces expansion:
$ mv /usr/share/nginx/html/{app.xml,index.html}
You can use a subshell:
(cd /usr/share/nginx/html; mv app.xml index.html)
I am trying to do a grep and then a sed to search for specific strings inside files, which are inside multiple tars, all inside one master tar archive. Right now, I modify the files by
First extracting the master tar archive.
Then extracting all the tars inside it.
Then doing a recursive grep and then sed to replace a specific string in files.
Finally packaging everything again into tar archives, and all the archives inside the master archive.
Pretty tedious. How do I do this automatically using shell scripting?
There isn't going to be much option except automating the steps you outline, for the reasons demonstrated by the caveats in the answer by Kimvais.
tar modify operations
The tar command has some options to modify existing tar files. They are, however, not appropriate for your scenario for multiple reasons, one of them being that it is the nested tarballs that need editing rather than the master tarball. So, you will have to do the work longhand.
Assumptions
Are all the archives in the master archive extracted into the current directory or into a named/created sub-directory? That is, when you run tar -tf master.tar.gz, do you see:
subdir-1.23/tarball1.tar
subdir-1.23/tarball2.tar
...
or do you see:
tarball1.tar
tarball2.tar
(Note that nested tars should not themselves be gzipped if they are to be embedded in a bigger compressed tarball.)
master_repackager
Assuming you have the subdirectory notation, then you can do:
for master in "$#"
do
tmp=$(pwd)/xyz.$$
trap "rm -fr $tmp; exit 1" 0 1 2 3 13 15
cat $master |
(
mkdir $tmp
cd $tmp
tar -xf -
cd * # There is only one directory in the newly created one!
process_tarballs *
cd ..
tar -czf - * # There is only one directory down here
) > new.$master
rm -fr $tmp
trap 0
done
If you're working in a malicious environment, use something other than tmp.$$ for the directory name. However, this sort of repackaging is usually not done in a malicious environment, and the chosen name based on process ID is sufficient to give everything a unique name. The use of tar -f - for input and output allows you to switch directories but still handle relative pathnames on the command line. There are likely other ways to handle that if you want. I also used cat to feed the input to the sub-shell so that the top-to-bottom flow is clear; technically, I could improve things by using ) > new.$master < $master at the end, but that hides some crucial information multiple lines later.
The trap commands make sure that (a) if the script is interrupted (signals HUP, INT, QUIT, PIPE or TERM), the temporary directory is removed and the exit status is 1 (not success) and (b) once the subdirectory is removed, the process can exit with a zero status.
You might need to check whether new.$master exists before overwriting it. You might need to check that the extract operation actually extracted stuff. You might need to check whether the sub-tarball processing actually worked. If the master tarball extracts into multiple sub-directories, you need to convert the 'cd *' line into some loop that iterates over the sub-directories it creates.
All these issues can be skipped if you know enough about the contents and nothing goes wrong.
process_tarballs
The second script is process_tarballs; it processes each of the tarballs on its command line in turn, extracting the file, making the substitutions, repackaging the result, etc. One advantage of using two scripts is that you can test the tarball processing separately from the bigger task of dealing with a tarball containing multiple tarballs. Again, life will be much easier if each of the sub-tarballs extracts into its own sub-directory; if any of them extracts into the current directory, make sure you create a new sub-directory for it.
for tarball in "$#"
do
# Extract $tarball into sub-directory
tar -xf $tarball
# Locate appropriate sub-directory.
(
cd $subdirectory
find . -type f -print0 | xargs -0 sed -i 's/name/alternative-name/g'
)
mv $tarball old.$tarball
tar -cf $tarball $subdirectory
rm -f old.$tarball
done
You should add traps to clean up here, too, so the script can be run in isolation from the master script above and still not leave any intermediate directories around. In the context of the outer script, you might not need to be so careful to preserve the old tarball before the new is created (so rm -f $tarbal instead of the move and remove command), but treated in its own right, the script should be careful not to damage anything.
Summary
What you're attempting is not trivial.
Debuggability splits the job into two scripts that can be tested independently.
Handling the corner cases is much easier when you know what is really in the files.
You probably can sed the actual tar as tar itself does not do compression itself.
e.g.
zcat archive.tar.gz|sed -e 's/foo/bar/g'|gzip > archive2.tar.gz
However, beware that this will also replace foo with bar also in filenames, usernames and group names and ONLY works if foo and bar are of equal length
I'm trying to use something along the lines of
unexpand -t 4 *.php
but am unsure how to write this command to do what I want.
Weirdly,
unexpand -t 4 file.php > file.php
gives me an empty file. (i.e. overwriting file.php with nothing)
I can specify multiple files okay, but don't know how to then overwrite each file.
I could use my IDE, but there are ~67000 instances of to be replaced over 200 files, and this will take a while.
I expect that the answers to my question(s) will be standard unix fare, but I'm still learning...
You can very seldom use output redirection to replace the input. Replacing works with commands that support it internally (since they then do the basic steps themselves). From the shell level, it's far better to work in two steps, like so:
Do the operation on foo, creating foo.tmp
Move (rename) foo.tmp to foo, overwriting the original
This will be fast. It will require a bit more disk space, but if you do both steps before continuing to the next file, you will only need as much extra space as the largest single file, this should not be a problem.
Sketch script:
for a in *.php
do
unexpand -t 4 $a >$a-notab
mv $a-notab $a
done
You could do better (error-checking, and so on), but that is the basic outline.
Here's the command I used:
for p in $(find . -iname "*.js")
do
unexpand -t 4 $(dirname $p)/"$(basename $p)" > $(dirname $p)/"$(basename $p)-tab"
mv $(dirname $p)/"$(basename $p)-tab" $(dirname $p)/"$(basename $p)"
done
This version changes all files within the directory hierarchy rooted at the current working directory.
In my case, I only wanted to make this change to .js files; you can omit the iname clause from find if you wish, or use different args to cast your net differently.
My version wraps filenames in quotes, but it doesn't use quotes around 'interesting' directory names that appear in the paths of matching files.
To get it all on one line, add a semi after lines 1, 3, & 4.
This is potentially dangerous, so make a backup or use git before running the command. If you're using git, you can verify that only whitespace was changed with git diff -w.