I'm trying to create a zip file from file contents which are being piped in, e.g.
mysql [params and query] | zip -q output.zip -
This writes the zip correctly, but when you open the zip, the file within it is called "-". Is there any way of specifying what the filename of the piped in data should be within the zip?
You can do this.
ls | zip test.zip -#
this is done from the notion that i have 3 files in the dir.
-rw-rw-r-- 1 xxx domain users 6 Jan 7 11:41 test1.txt
-rw-rw-r-- 1 xxx domain users 6 Jan 7 11:41 test2.txt
-rw-rw-r-- 1 xxx domain users 6 Jan 7 11:41 test3.txt
and the file itself, the result is then
Archive: test.zip
Length Date Time Name
-------- ---- ---- ----
6 01-07-10 11:41 test1.txt
6 01-07-10 11:41 test2.txt
6 01-07-10 11:41 test3.txt
-------- -------
18 3 files
From the Linux Zip Man page
If the file list is specified as -#,
zip takes the list of input files
from standard input.
You can use a named pipe, and send the request output to it, while zipping from it.
mkfifo output.txt ; mysql [params and query] > output.txt & zip output.zip -FI output.txt ; rm output.txt
I couldn't manage with the PHP answer (out of memory on bigger mysql dumps), and the FIFO was not working as I wanted, so my solution is to rename the file inside the ZIP archive after running the dump, using zipnote (which is included with the zip package on Debian).
mysql [params and query] | zip -q output.zip -
echo -e "# -\n#=newname.sql" | zipnote -w output.zip
From what i can gather you cannot do both with the zip command, i mean you cannot both specify the filenames and pipe the content. You can either pipe the contents and the resulting file is - or you can pipe the filenames with -#.
That does not mean that doing so is impossible using other techniques. I outline one of those below. It does mean that you have to have PHP installed and the zip extension loaded.
There could be a whole bunch of other ways to do it. But this is the easiest that I know of. Oh and it's a one-liner too.
This is a working example using PHP
echo contents | php -r '$z = new ZipArchive();$z->open($argv[1],ZipArchive::CREATE);$z->addFromString($argv[2],file_get_contents("php://stdin"));$z->close();' test.zip testfile
To run on windows just swap single and double quotes. Or just place the script in a file.
"test.zip" is the resulting Zip file, "testfile" is the name of the contents that are being piped into the command.
Result from unzip -l test.zip
Archive: test.zip
Length Date Time Name
--------- ---------- ----- ----
6 01-07-2010 12:56 testfile
--------- -------
6 1 file
And here is a working example using python
echo contents | python -c "import sys
import zipfile
z = zipfile.ZipFile(sys.argv[1],'w')
z.writestr(sys.argv[2],sys.stdin.read())
z.close()
" test5.zip testfile2
Result of unzip -l
Archive: test5.zip
Length Date Time Name
-------- ---- ---- ----
9 01-07-10 13:43 testfile2
-------- -------
9 1 file
Result of unzip -p
contents
mysql [params and query] | python3 -c "import sys as s,os,zipfile as m;z=m.ZipFile(os.fdopen(s.stdout.fileno(),'wb'),'w');z.writestr(s.argv[1],s.stdin.read());z.close()" 'filename.sql' > output.zip
Python 3.5 added support for writing to unseekable streams.
It's not pretty but works.
Just stick with the - as filename, as zip --help suggests:
The default action is to add or replace zipfile entries from list,
which can include the special name - to compress standard input.
but rename it after. You can use 7zip rn for this purpose or any other compression tool.
Here an example reading from curl a plain text, compressing it with best compression and then renaming the minus file after archive is ready.
curl -q 'https://ws-export.wmcloud.org/?lang=en&page=In+Event+of+Moon+Disaster&format=txt&fonts=' | zip -9 moon.zip -
7z rn moon.zip -- - moondisaster.txt
Why double --? To pass - as filename instead of command option.
Result is a zip named moon.zip with a single moondisaster.txt file inside.
You can try:
mysql [params and query] | zip -q output.zip -#
Related
I want to get the content of the file1.txt of my archive.zip without extrancting the file.
For all these commands I obtain caution: filename not matched: file1.txt
unzip -p ~/archive.zip ~/archive.zip/file1.txt | less
unzip -p ~/archive.zip ~/archive/file1.txt | less
unzip -p ~/archive.zip file1.txt | less
the archive.zip is at the home directory, and the respective names are correct.
Hardcoding the path, produces the same undesired outcome:
unzip -p /home/pi/archive.zip ~/archive.zip/file1.txt | less
unzip -p /home/pi/archive.zip ~/archive/file1.txt | less
unzip -p /home/pi/archive.zip file1.txt | less
I am trying to do this in a raspberry-pi.
The expected output is the content of the file1.txt.
It's possible that the zip extracts into a directory and the file is present there, eg file.zip creates a someproject-name top level directory and the contents are under that. So you can do something like this:
unzip -p /home/pi/archive.zip '*/file1.txt' So it would look at the top level directory aswell due to the glob.
I have a .txt file containing a column of IDs and their ages (as integers). I've already created separate folders in my directory for each age category (ranging from 20-86). For every ID in my .txt file I would like to move their image (which is currently stored in the folder "data") to the appropriate folder, based on their age category listed in column two of my .txt file.
Any help on how to do this in Linux would be really appreciated!
Updated example with files ending in different suffixes.
Current working directory:
data/ 20/ 21/ 22/ 23/ 24/ ...
text file:
ID001 21
ID002 23
ID003 20
ID004 22
ID005 21
ls data/
ID001-XXX-2125.jpg
ID002-YYY-2370.jpg
ID003-XXX-2125.jpg
ID004-YYY-2370.jpg
ID005-XXX-2125.jpg
Desired output:
20/
ID003-XXX-2125.jpg
21/
ID001-XXX-2125.jpg
ID005-XXX-2370.jpg
22/
ID004-YYY-2370.jpg
23/
ID002-YYY-2370.jpg
As you suggest, awk can do this kind of task (though see Ed Morton's remark below). You may try the following, which is tested with GNU awk. From your working directory you can do:
awk '{system("mv data/"$1"*.jpg " $2)}' inputfile
Explanation: Here the system() function is used. The system() function allows you to execute a command supplied as an expression. In this case:
We use the mv (move) command and use the first field $1 in the input file to address the JPG file in the data directory.
Then we use the second field $2 of the input file for the destination directory.
The system() function is modeled after the standard C library function. Further reading in The GNU Awk User’s Guide
while read fil id
do
mv -f "data/"*"$fil"*".jpg" "$id/"
done < file
Read the two fields from the file (called file in this case) in a loop and use the variables to construct and execute the mv command.
Consider having your .txt file is in your current working directory. Can you try this written and tested script?
#!/bin/sh
DIR_CWD="/path/to/current_working_directory"
cd "$DIR_CWD/data"
for x in *; do
ID_number=`echo $x | awk -F"-" '{print $1}'`
DIR_age=`cat "$DIR_CWD/file.txt" | grep $ID_number | awk '{print $2}'`
mv -- "$DIR_CWD/data/$x" "$DIR_CWD/$DIR_age"
done
Note that DIR_CWD must be stated as the path of your current working directory.
We have a folder in our embedded board "statuslogs", this folder contains logs which are of the format : daily_status_date_time.log.
We need to get all the files of a particular year into a single file, for fetching from the server.
We did the following in our script
gzip -c statuslogs/daily_status_2017*.log > status_2017.gz
gzip -c statuslogs/daily_status_2018*.log > status_2018.gz
gzip -c statuslogs/daily_status_2019*.log > status_2019.gz
gzip -c statuslogs/daily_status_2020*.log > status_2020.gz
gzip -c statuslogs/daily_status_2021*.log > status_2021.gz
The problem with this logic is that it will still create status_*.gz file for the years 2019,2020,2021.
I tried writing the following logic
if [ - f statuslogs/daily_status_2017*.log ] but it fails due to regex may be. And I am not using bash, the interpreter is ash.
Can you please help me to optimize the script
Thanks for your time
You have a syntax error. It's -f, not - f. Example:
if [ -f statuslogs/daily_status_2017*.log ]; then
gzip -c statuslogs/daily_status_2017*.log > status_2017.gz
fi
However, with this you will probably run into a "too many arguments" error, which will happen if you have more than one matching file. So this would work better:
if find statuslogs/daily_status_2017*.log -mindepth 0 -maxdepth 0|head -n1; then
gzip -c statuslogs/daily_status_2017*.log > status_2017.gz
fi
It would be better to instead stop the loop when you reach the current year. For example,
for year in $(seq 2017 $(date +%Y)); do
Gzip only works on single files. If you want the separate files you need to do one of the following:
Combine the files using tar:
tar cf status_2017.tar.gz statuslogs/daily_status_2017*.log
OR use zip which supports multiple files directly
zip status_2017.zip statuslogs/daily_status_2017*.log
Now, if the problem is just that you want one archive for every year, but only for the years for which files exist, you can handle all the years using a for loop:
for year in `ls statuslogs/daily_status_* | cut -d _ -f 3 | sort | uniq`; do
tar cf status_$year.tar.gz statuslogs/daily_status_$year*.log;
done
If your shell doesn't support that format of calling, you can try this instead
ls statuslogs/daily_status_* | cut -d _ -f 3 | sort | uniq > years
cat years | while read year; do
tar cf status_$year.tar.gz statuslogs/daily_status_$year*.log;
done
If you just want one file for all the logs, you can just forget about the year part completely
tar cf statuslogs.tar.gz statuslogs/daily_status*.log
I found the slimier post from STO but those does not filter files with extension. So writing again.
I an writing a shell script to keep last (most latest) 3 .txt files in directory and wants to remove all other .txt files.
For Example... In Directory "Home" I have following files.
test.txt
my.txt
image.jpg
test.avi
sample.txt
country.txt
study.txt
When I run linux script, output should be like as below....
Keep File (keep only last 3 .txt files only)
test.txt
my.txt
image.jpg
test.avi
sample.txt
Delete File
country.txt
study.txt
Thanks
List entries by ctime (newest first), skip the first three items, delete the rest:
ls -c *.txt | tail -n +4 | xargs rm
If I choose a zip file and right click "extract here" a folder with the zip filename is created and the entire content of the zip file is extracted into it.
However, I would like to convert several zip files via shell.
But when I do
unzip filename.zip
the folder "filename" is not created but all the files are extracted into the current directory.
I have looked at the parameters but there is no such parameter.
I also tried
for zipfile in \*.zip; do mkdir $zipfile; unzip $zipfile -d $zipfile/; done
but the .zip extension of the 2. $zipfile and 4. $zipfile have to be removed with sed.
If I do
for zipfile in \*.zip; do mkdir sed 's/\.zip//i' $zipfile; unzip $zipfile -d sed 's/\.zip//i' $zipfile/; done
it is not working.
How do I replace the .zip extension of $zipfile properly?
Is there an easier way than a shell script?
unzip file.zip -d xxx will extract files to directory xxx, and xxx will be created if it is not there. You can check the man page for details.
The awk line below should do the job:
ls *.zip|awk -F'.zip' '{print "unzip "$0" -d "$1}'|sh
See the test below,
note that I removed |sh at the end, since my zips are fake archives; I just want to show the generated command line here.
kent$ ls -l
total 0
-rw-r--r-- 1 kent kent 0 Nov 12 23:10 001.zip
-rw-r--r-- 1 kent kent 0 Nov 12 23:10 002.zip
-rw-r--r-- 1 kent kent 0 Nov 12 23:10 003.zip
-rw-r--r-- 1 kent kent 0 Nov 12 23:10 004.zip
-rw-r--r-- 1 kent kent 0 Nov 12 23:10 005.zip
-rw-r--r-- 1 kent kent 0 Nov 12 23:10 006.zip
-rw-r--r-- 1 kent kent 0 Nov 12 23:10 007.zip
kent$ ls *.zip|awk -F'.zip' '{print "unzip "$0" -d "$1}'
unzip 001.zip -d 001
unzip 002.zip -d 002
unzip 003.zip -d 003
unzip 004.zip -d 004
unzip 005.zip -d 005
unzip 006.zip -d 006
unzip 007.zip -d 007
"extract here" is merely a feature of whatever unzip wrapper you are using. unzip will only extract what actually is in the archive. There is probably no simpler way than a shell script. But sed, awk etc. are not needed for this if you have a POSIX-compliant shell:
for f in *.zip; do unzip -d "${f%*.zip}" "$f"; done
(You MUST NOT escape the * or pathname expansion will not take place.) Be aware that if the ZIP archive contains a directory, such as with Eclipse archives (which always contain eclipse/), you would end up with ./eclipse*/eclipse/eclipse.ini in any case. Add echo before unzip for a dry run.
p7zip, the command line version of 7zip does the job
7z x '*.zip' -o'*'
On Debian and Ubuntu, you have to install p7zip-full.
Add the folder name and slash after the switch, example:
unzip -d newfolder/ zipfile.zip
zip will create the folder 'newfolder' and extract the archive into it.
Note that the trailing slash is not required but I guess it's just from old habit (I use Debian and Ubuntu).
aunpack from atool will do this by default for all sorts of archives.
In Termux bash I ended up with this, tested for spaces and dots in filenames. It only removes the last file extension and dot for the folder name. This assumes the zip file has a .zip extension otherwise it won't work. Went nuts trying to \ escape the quotes or use single quotes before realizing they just needed to be there as plain quote marks. Drop an echo in front of the zip command, or add a -l to audit the command. For some reason the position of the -d seems important for this implementation.
for f in *.zip; do unzip "$f" \-d "./${f%.*}/"; done; echo -end-
Open the terminal and locate the 00* files
cat filename.zip.00* > filename.zip
wait until the process ends, it depends on the file size.
The file joining process finished, now you can run the output file
unzip filename.zip