Can someone please explain me how to use ">" and "|" in linux commands and convert me these three lines into one line of code please?
mysqldump --user=*** --password=*** $db --single-transaction -R > ${db}-$(date +%m-%d-%y).sql
tar -cf ${db}-$(date +%m-%d-%y).sql.tar ${db}-$(date +%m-%d-%y).sql
gzip ${db}-$(date +%m-%d-%y).sql.tar
rm ${db}-$(date +%m-%d-%y).sql (after conversion I guess this line will be useless)
The GNU tar program can itself do the compression normally done by gzip. You can use the -z flag to enable this. So the tar and gzip could be combined into:
tar -zcf ${db}-$(date +%m-%d-%y).sql.tar.gz ${db}-$(date +%m-%d-%y).sql
Getting tar to read from standard input for archiving is not a simple task but I would question its necessity in this particular case.
The intent of tar is to be able to package up a multitude of files into a single archive file but, since it's only one file you're processing (the output stream from mysqldump), you don't need to tar it up, you can just pipe it straight into gzip itself:
mysqldump blah blah | gzip > ${db}-$(date +%m-%d-%y).sql.gz
That's because gzip will compress standard input to standard output if you don't give it any file names.
This removes the need for any (possibly very large) temporary files during the compression process.
You can use next script:
#!/bin/sh
USER="***"
PASS="***"
DB="***"
mysqldump --user=$USER --password=$PASS $DB --single-transaction -R | gzip > ${DB}-$(date +%m-%d-%y).sql.gz
You can learn more about "|" here - http://en.wikipedia.org/wiki/Pipeline_(Unix). I can say that this construction moves output of mysqldump command to the standard input of gzip command, so that is like you connect output of one command with input of other via pipeline.
I dont see the point in using tar: You just have one file, and for compression you call gzip explicit. Tar is used to archive/pack multiple files into one.
You cammandline should be (the dump command is shorted, but I guess you will get it):
mysqldump .... | gzip > filename.sql.gz
To append the commands together in one line, I'd put && between them. That way if one fails, it stops executing them. You could also use a semicolon after each command, in which case each will run regardless if the prior command fails or not.
You should also know that tar will do the gzip for you with a "z" option, so you don't need the extra command.
Paxdiablo makes a good point that you can just pipe mysqldump directly into gzip.
Related
I need to copy a file from multiple locations to the BACK UP directory by retaining its directory structure. For example, I have a file "a.txt" at the following locations /a/b/a.txt /a/c/a.txt a/d/a.txt a/e/a.txt, I now need to copy this file from multiple locations to the backup directory /tmp/backup. The end result should be:
when i list /tmp/backup/a --> it should contain /b/a.txt /c/a.txt /d/a.txt & /e/a.txt.
For this, I had used the command: echo /a/*/a.txt | xargs -I {} -n 1 sudo cp --parent -vp {} /tmp/backup. This is throwing the error "cp: cannot stat '/a/b/a.txt /a/c/a.txt a/d/a.txt a/e/a.txt': No such file or directory"
-I option is taking the complete input from echo instead of individual values (like -n 1 does). If someone can help debug this issue that would be very helpful instead of providing an alternative command.
Use rsync with the --relative (-R) option to keep (parts of) the source paths.
I've used a wildcard for the source to match your example command rather than the explicit list of directories mentioned in your question.
rsync -avR /a/*/a.txt /tmp/backup/
Do the backups need to be exactly the same as the originals? In most cases, I'd prefer a little compression. [tar](https://man7.org/linux/man-pages/man1/tar.1.html) does a great job of bundling things including the directory structure.
tar cvzf /path/to/backup/tarball.tgz /source/path/
tar can't update compressed archives, so you can skip the compression
tar uf /path/to/backup/tarball.tar /source/path/
This gives you versioning of a sort, as if only updates changed files, but keeps the before and after versions, both.
If you have time and cycles and still want the compression, you can decompress before and recompress after.
i used the following:
gzip -9 -c -r <some_directory> > directory.gz
how do i decompress this directory ?
I have tried
gunzip directory.gz
i am just left with a single file and not a directory structure.
As others have already mentioned, gzip is a file compression tool and not an archival tool. It cannot work with directories. When you run it with -r, it will find all files in a directory hierarchy and compress them, i.e. replacing path/to/file with path/to/file.gz. When you pass -c the gzip output is written to stdout instead of creating files. You have effectively created one big file which contains several gzip-compressed files.
Now, you could look for the gzip file header/magic number, which is 1f8b and then reconstruct your files manually.
The sensible thing to do now is to create backups (if you haven't already). Backups always help (especially with problems such as yours). Create a backup of your directory.gz file now. Then read on.
Fortunately, there's an easier way than manually reconstructing all files: using binwalk, a forensics utility which can be used to extract files from within other files. I tried it with a test file, which was created the same way as yours. Running binwalk -e file.gz will create a folder with all extracted files. It even manages to reconstruct the original file names. The hierarchy of the directories is probably lost. But at least you have your file contents and their names back. Good luck!
Remember: backups are essential.
(For completeness' sake: What you probably intended to run: tar czf directory.tar.gz directory and then tar xf directory.tar.gz)
gzip will compress 1+ files, though not meant to function like an archive utility. The posted cmd-line would yield N compressed file images concatenated to stdout, redirected to the named output file; unfortunately stuff like filenames and any dirs would not be recorded. A pair like this should work:
(create)
tar -czvf dir.tar.gz <some-dir>
(extract)
tar -xzvf dir.tar.gz
I am attempting to make a script that will make a backup file of a video file in the same directory with the time stamp at the end of the tar file. The script is for demonstration purposes only, that is why I do not intend on sending the file to a different directory. Below is how far I have come with it.
#!/bin/bash
cd Compression_Play/
echo Me.$(date +%d_%b_%Y-%k:%M:%S).tar.xz
tar -I "pixz -9t" -cvf Me.$(date +%d_%b_%Y-%k:%M:%S).tar.xz 2017-03-23_01-13-02.avi
My problem is whenever I try to execute the script it gives me this:
Me.29_Mar_2017-22:03:49.tar.xz
tar: -9: (PROGRAM ERROR) Option should have been recognized!?
Try 'tar --help' or 'tar --usage' for more information.
As best as I can tell the problem is with the quotes in my tar command. Is there a way to make the script so I can keep the quotes or substitute them?
Given your goal to make an archive, presumably preserving owner, file modes, file flags and ACLs, if available, then this should do what you need:
#!/bin/bash
archive_name="Me.$(date +%d_%b_%Y-%k:%M:%S).tar.xz"
$(cd Compression_Play/
tar -cvp 2017-03-23_01-13-02.avi |
pixz -9t > $archive_name
)
Based on the GitHub page for tarball compress/decompress for pixz, you will need to reverse the pipe to get your data out:
pixz -x9T < $archive_name | tar xvpf -
Greg gave a good answer and it works very well. I also figured out another way after nocking my head on it for several hours and taking the things from the help I got.
#!/bin/bash
cd /home/apowell/Compression_Play/Test_Backup
tar -I 'pixz -9t' -cvf Me.$(date +%d_%b_%Y-%H_%M_%S).tar.xz /home/apowell/Compression_Play/Linux_Programs
and this one compresses a little more by replacing pixz with just plain xz.
#!/bin/bash
cd /home/apowell/Compression_Play/Test_Backup
tar -I 'xz -9ve -T8 -k' -cvf Me.$(date +%d_%b_%Y-%H_%M_%S).tar.xz /home/apowell/Compression_Play/Linux_Programs
Is there any possible way to compress a directory with GZip, BZip, BZip2, xz format. I'm building a command line tool(using bash) which I need these options to be included.
A command like
tar czf output.tar.gz yourdir/
should work.
c means that tar will create an archive
z means that the output will be compressed (using gzip)
the output filename is after f
at the end, you can specify any number of directories/files (space-separated)
To answer the "why" part of your question, it is because of the Unix philosophy of having many small tools that do their job well that you can string together, as opposed to one big tool that doesn't do anything well and is hard to make better. Your examples are a perfect illustration of this philosophy, where you have several compression tools to choose from, and it is easy to add a new compression tool to your tool box. The archiving part, turning a directory of files into a byte stream, is a different task that is its own tool that can be combined with any of those or any future compression tools.
The body of your question then asks "how". You use a pipe with tar, cpio, or pax. tar is the most common. You then name the file accordingly so the consumer of the file can tell what it is from the name. E.g. ending with .tar.gz. Like this:
tar cf - somedirectory | gzip > somedirectory.tar.gz
or
tar cf - somedirectory | xz > somedirectory.tar.xz
These tar up the directory into a byte stream, which is then piped to a compressor. The output of the compressor is then written to the file containing the compressed directory contents.
To decompress:
gzip -dc somedirectory.tar.gz | tar xf -
Here it is done in the reverse order to first decompress the file and feed the output of that to tar to extract the files and recreate the directory structure. The - means to put the archive to stdout or get the archive from stdin.
Having said all that stuff about how much better it is to have small tools that do their job well, this application of tar is so incredibly common that it is built into the tar options. So you can instead:
tar czf somedirectory.tar.gz somedirectory
tar cJf somedirectory.tar.xz somedirectory
tar will run the gzip or xz executables and pipe the data through them itself.
(J is a recent gnutar addition, so your tar may not have it.)
I am currently running the following script to make an automatic backup of my Music:
#!/bin/bash
while :; do
rsync -ruv /mnt/hdd1/Music/ /mnt/hdd2/Music/
done
Whenever a new file is added to my music folder, it is detected by rsync and it is copied to my other disk. This script runs fine, but I would also like to convert the detected file to an ogg opus file for putting on my phone.
My question is: How do I run a command on a new file found by rsync -u?
I will also accept answers which work totally differently, but have the same result.
rsync -ruv /mnt/hdd1/Music /mnt/hdd2/ | sed -n 's|^Music/||p' >~/filelist.tmp
while IFS= read filename
do
[ -f "$filename" ] || continue
# do something with file
echo "Now processing '$filename'"
done <~/filelist.tmp
With the -v option, rsync prints the names of files it copies to stdout. I use sed to capture just those filenames, excluding the informational messages, to a file. The filenames in that file can be processed later as you like.
The approach with sed above depends on rsync displaying filenames starting with the final part of the source directory, e.g. "Music/" in my example above, which is then removed assuming that you don't need it. Alternately, one could try an explicit approach for excluding noise messages.