I have all my Apache access log files as access.log, access.log.1 access.log.1.gz etc... What I want is to zcat all files in and not in gzip format and pipe them into an X program.
I know I can do: zcat /var/log/apache2/access.log.*.gz | someapp... but that will just work for *.gz and not the first two logs.
Any ideas will be appreciate it
use zcat -f, it will copy uncompressed files as is
For the specific use case of HTTP log server files, consider the zmergelog command (from the mergelog package). It additionally sorts the result of the merge chronologically.
Related
I need to create a bunch of .gz files for testing something. Is it possible (in unix based shell) to make an empty file (e.g. by touch file1) and zip (e.g. file1.gz) using a simple one-liner (along: touch file1 | zip ). How can I redirect the file into zip?
gzip < /dev/null > file.gz
I.e. feed gzip empty input and redirect the output to file.gz. No temporary empty file needed. 😉
This assumes that there's a /dev/null available (i.e. a POSIX(-ish) system). Alternatives would be:
: | gzip […]
true | gzip […]
printf '' | gzip […]
echo -n | gzip […] # Won't work in strictly POSIX environments
…
I.e. anything that's available and prints nothing to STDOUT.
You mentioned zip, but since you said you needed .gz files I assume you want gzip.
For example:
touch file1 && zip file1.zip file1
Can you give an example for compressing files with lzma2?
I searched on Google but I didn't find any examples that I could understand.
You can also use the lzma binary or the xz with the --format=lzma argument.,
If you want to compress a set of files, I would recommend using tar to glue them together and then lzma, as for example:
$ tar --lzma -cf foo.tar.lzma file1 file2 ...
$ file foo.tar.lzma
foo.tar.lzma: LZMA compressed data, streamed
The XZ Utils package is installed in most Linux distributions and is becoming the de facto standard compression format. The tool compresses with LZMA2. Usage is as simple as it gets,
$ xz <file>
To produce file.xz.
This worked:
sudo tar --lzma -cf newname.tar.lzma file1 file2 file3
Replace the newname with the new name you want to give to the file
and replace file1 file2 file3... with the names of the files you want to zip and compress.
I have a bunch of files with filename.bz2.gz which I want to convert to filename.gz.
any help ?
thanks
Having your filename *.bz2.gz I assume the file had been created using the following order of compressions:
echo test | bzip2 | gzip -f > file.bz2.gz
Meaning it is a gzipped bzip2 file (for whatever reason). If my assumption is correct you can change it's compression to gzip-only, using the following commands:
gunzip < file.bz2.gz | bunzip2 | gzip > file.gz
If you just want to rename then do this.
for i in `ls|awk -F. '{print $1}'`
do
mv "$i".bz2.gz "$i".gz
done
I would refine Ajit's solution in this way:
for i in *.bz2.gz; do
i=${i%.bz2.gz}
mv "$i.bz2.gz" "$i.gz"
done
Using a glob rather than command subsitution avoids problems with word-splitting for filenames with whitespace. It also avoids the extra ls process, which is marginally more efficient, particularly on platforms like Cygwin with slow process forking. For the same reason, the awk command can be replaced with the ${parameter%[word]} parameter expansion syntax. (Quoting style of "$i".gz vs "$i.gz" makes no difference and is just personal preference.)
I want to search for a particular string from a .gz file containing a text file without extracting in linux terminal. I know how to search for a string from a text file using grep "text to search" ./myfile.txt. But how to make it work for .gz files?
You can use zgrep. Usage is similar to grep.
zgrep "pattern" file.gz
From the man page's description:
Zgrep invokes grep on compressed or gzipped files. All options
specified are passed directly to grep. If no file is specified, then
the standard input is decompressed if necessary and fed to grep.
Otherwise the given files are uncompressed if necessary and fed to
grep.
gunzip -c mygzfile.gz | grep "string to be searched"
But this would only work if the .gz file contains text file which is true in your case.
Already it have been answered, but it could be really helpful if you want to search in multiple .gz files.
For searching in all the .gz files in a specific folder you can use
zgrep "yourString" *
Currently I am in this directory-
/data/real/test
When I do ls -lt at the command prompt. I get like below something-
REALTIME_235000.dat.gz
REALTIME_234800.dat.gz
REALTIME_234600.dat.gz
REALTIME_234400.dat.gz
REALTIME_234200.dat.gz
How can I consolidate the above five dat.gz files into one dat.gz file in Unix without any data loss. I am new to Unix and I am not sure on this. Can anyone help me on this?
Update:-
I am not sure which is the best way whether I should unzip each of the five file then combine into one? Or
combine all those five dat.gz into one dat.gz?
If it's OK to concatenate files content in random order, then following command will do the trick:
zcat REALTIME*.dat.gz | gzip > out.dat.gz
Update
This should solve order problem:
zcat $(ls -t REALTIME*.dat.gz) | gzip > out.dat.gz
What do you want to happen when you gunzip the result? If you want the five files to reappear, then you need to use something other than the gzip (.gz) format. You would need to either use tar (.tar.gz) or zip (.zip).
If you want the result of the gunzip to be the concatenation of the gunzip of the original files, then you can simply cat (not zcat or gzcat) the files together. gunzip will then decompress them to a single file.
cat [files in whatever order you like] > combined.gz
Then:
gunzip combined.gz
will produce an output that is the concatenation of the gunzip of the original files.
The suggestion to decompress them all and then recompress them as one stream is completely unnecessary.