I need to create a bunch of .gz files for testing something. Is it possible (in unix based shell) to make an empty file (e.g. by touch file1) and zip (e.g. file1.gz) using a simple one-liner (along: touch file1 | zip ). How can I redirect the file into zip?
gzip < /dev/null > file.gz
I.e. feed gzip empty input and redirect the output to file.gz. No temporary empty file needed. 😉
This assumes that there's a /dev/null available (i.e. a POSIX(-ish) system). Alternatives would be:
: | gzip […]
true | gzip […]
printf '' | gzip […]
echo -n | gzip […] # Won't work in strictly POSIX environments
…
I.e. anything that's available and prints nothing to STDOUT.
You mentioned zip, but since you said you needed .gz files I assume you want gzip.
For example:
touch file1 && zip file1.zip file1
Related
I have two directories.
In directory_1, I have many .txt files
Content of these files (for example file1.txt) are a list of characters
file1.txt
--
rer_098
dfrkk9
In directory_2, I have many files, two of them are ‘rer_098’ and ‘dfrkk9’.
Content of these files are as follows:
rer_098
--
>123_nbd
sasert
>456_nbd
ffjko
dfrkk9
--
>789_nbd
figyi
>012_nbd
jjjygk
Now in a separate output directory (directory_3), for this above example, I want output files like:
file1.txt
--
>123_nbd
sasert
>456_nbd
ffjko
>789_nbd
figyi
>012_nbd
jjjygk
and so on for file2.txt
Thanks!
This might work for you (GNU parallel):
parallel 'cat {} | parallel -I## cat dir_2/## > dir_3/{/}' ::: dir_1/*.txt
Use two invocations of parallel, the first traverses dir_1 and pipes its output in a second parallel. This cats the input files and outputs the result dir_3 keeping the original name from the first parallel invocation.
N.B. The use of the -I option to rename the parameter delimiters from the default {} to ##.
Pretty easy to do with just shell. Something like
for fullname in directory_1/*.txt; do
file=$(basename "$fullname")
while read -r line; do
cat "directory_2/$line"
done <"$fullname" >"directory_3/$file"
done
for file in directory_1/*.txt; do
awk 'NR==FNR{ARGV[ARGC++]="directory_2/"$0; next} 1' "$file" > "directory_3/${file##%/}"
done
I want to generate a text file with the list of files present in the folder
ls | xargs echo > text.txt
I want to prepend the IP address to each file so that I can run parallel wget as per this post : Parallel wget in Bash
So my text.txt file content will have these lines :
123.123.123.123/file1
123.123.123.123/file2
123.123.123.123/file3
How can I append a string as the ls feeds xargs? (and also add line break at the end.)
Thank you
Simply printf and globbing to get the filenames:
printf '123.123.123.123/%s\n' * >file.txt
Or longer approach, leverage a for construct with help from globbing:
for f in *; do echo "123.123.123.123/$f"; done >file.txt
Assuming no filename with newline exists.
I have several .vcf.gz files:
subset_file1.vcf.vcf.gz
subset_file2.vcf.vcf.gz
subset_file3.vcf.vcf.gz
I want to gunzip these file and rename them (remove subset_ and redudant .vcf extension in one go and get these files:
file1.vcf
file2.vcf
file3.vcf
This is the script I have tried:
iFILES=/file/path/*.gz
for i in $iFILES;
do gunzip -k $i > /get/in/this/dir/"${i##*/}"
done
Since you have to three operation at your output path name
1.remove the directory part
2.remove prefix subset_
3.remove redudant extension .vcf
It's hard to accomplish with only one command.
Following is a modification version. Be CAREFUL to try it. I didn't test it thorough in my computer.
for i in /file/path/*.gz;
do
# get the output file name
o=$(echo ${i##*/} | sed 's/.*_\(.*\)\(\.[a-z]\{3\}\)\{2\}.*/\1\2/g')
gunzip -k $i > /get/in/this/dir/$o
done
I have a bunch of files with filename.bz2.gz which I want to convert to filename.gz.
any help ?
thanks
Having your filename *.bz2.gz I assume the file had been created using the following order of compressions:
echo test | bzip2 | gzip -f > file.bz2.gz
Meaning it is a gzipped bzip2 file (for whatever reason). If my assumption is correct you can change it's compression to gzip-only, using the following commands:
gunzip < file.bz2.gz | bunzip2 | gzip > file.gz
If you just want to rename then do this.
for i in `ls|awk -F. '{print $1}'`
do
mv "$i".bz2.gz "$i".gz
done
I would refine Ajit's solution in this way:
for i in *.bz2.gz; do
i=${i%.bz2.gz}
mv "$i.bz2.gz" "$i.gz"
done
Using a glob rather than command subsitution avoids problems with word-splitting for filenames with whitespace. It also avoids the extra ls process, which is marginally more efficient, particularly on platforms like Cygwin with slow process forking. For the same reason, the awk command can be replaced with the ${parameter%[word]} parameter expansion syntax. (Quoting style of "$i".gz vs "$i.gz" makes no difference and is just personal preference.)
Currently I am in this directory-
/data/real/test
When I do ls -lt at the command prompt. I get like below something-
REALTIME_235000.dat.gz
REALTIME_234800.dat.gz
REALTIME_234600.dat.gz
REALTIME_234400.dat.gz
REALTIME_234200.dat.gz
How can I consolidate the above five dat.gz files into one dat.gz file in Unix without any data loss. I am new to Unix and I am not sure on this. Can anyone help me on this?
Update:-
I am not sure which is the best way whether I should unzip each of the five file then combine into one? Or
combine all those five dat.gz into one dat.gz?
If it's OK to concatenate files content in random order, then following command will do the trick:
zcat REALTIME*.dat.gz | gzip > out.dat.gz
Update
This should solve order problem:
zcat $(ls -t REALTIME*.dat.gz) | gzip > out.dat.gz
What do you want to happen when you gunzip the result? If you want the five files to reappear, then you need to use something other than the gzip (.gz) format. You would need to either use tar (.tar.gz) or zip (.zip).
If you want the result of the gunzip to be the concatenation of the gunzip of the original files, then you can simply cat (not zcat or gzcat) the files together. gunzip will then decompress them to a single file.
cat [files in whatever order you like] > combined.gz
Then:
gunzip combined.gz
will produce an output that is the concatenation of the gunzip of the original files.
The suggestion to decompress them all and then recompress them as one stream is completely unnecessary.