I would like to know if there is any way to compress a set of .txt files in a folder using scripting when the number of files get more than a set limit.
The txt files are automatically generated by another script.
You can use array size to detect the number of files:
limit=100
files=(*.txt)
if (( ${#files[#]} > limit )) ; then
zip archive.zip *.txt
fi
It sounds like you want logrotate with a custom (non-/etc) configuration file with rules for compressing/removing by size.
Related
I have a directory where there are files -:
abc_002.txt
abc_003.txt
abc_004.txt
abc_005.txt
xyz_001.txt
for_ex.sh
abc_001.txt
I want to gzip only files starting with names abc, like abc*.
for file in abc*.txt
do
gzip $file
done
I have large amount of wav files (over 50 000) and I need to split every wav file to 10 second long parts. It's nearly impossible to do it one by one, so my question is: is there any way to do it in ffmpeg or for example in sox? I'm an amateur, so I need exact instructions. Please, if you can, write it like you would for a dummy :)... (I'm a Windows 7 user)
Here is my try with sox
Thank you!
I had exactly the same task few days ago. I prefer to do it on Linux OS machines, but I had to do some demonstration on Windows machines so I have written this batch file using SoX:
#ECHO OFF
rem Author: PetarF, 2017-07-01, v0.1
rem
rem Example of how to do batch processing with SoX on MS-Windows.
rem In this example, for every .wav file in folder the splited .wav file ends up in a folder called as original file without extension
rem In my case, this file has to be run as administrator
rem
rem USAGE:
rem Place this file in the same folder as sox.exe (& rename it as appropriate).
rem Place .wav files that has to be splited in same folder as sox.exe
ECHO Split started
cd %~dp0
for %%A in (*.wav) do (
mkdir %%~nA
sox %%A %%~nA/%%~nA_.wav trim 0 10 : newfile : restart
echo Created folder %%~nA with splited files
)
echo SoX has splited files...
pause
I came across a problem in Bash when I would try to only open images based upon the information stored in .txt files about them. I am trying to sort a number of images by size or height, and display an image with them in the sorted order, but if there exists a .jpg in the folder without a .txt file with the same name, it should not process it.
I have the sorting piece of my situation done, and am trying to figure out how I would go about opening only the images that have a .jpg extension WITH a .txt file.
I figured a solution would look like me putting every .jpg's name (without extension) in a list and then process through the list and run something like:
[if -f $filename.txt ]; then ~~~
but I came across the problem of iterating through without a for-loop, or else all the pictures would open multiple times. My attempt was:
for i in *jpg; do
y=$y ${i.jpg}
done
if[ -f $y.txt ] then
(sorting parts)
This only looked at the last filename in y, as it should, but I am trying to figure out a way to look at each separate filename and see if there exists that textfile, in order to include it in the sorting.
Thanks so much for your help!
Collecting a list of file names in a single variable is an antipattern. You want to collect them in an array instead.
a=()
for f in *.jpg; do
if [ -e "${f%.jpg}".txt ]; then
continue
fi
a+=("$f")
done
# now do things with "${a[#]}"
Frequently, you don't really need to collect the files in an array -- just do everything you were doing inside the for loop to each individual file as you traverse the files.
(And actually y=$y ${i%.jpg} doesn't append to y -- it sets y to itself for the duration of attempting to execute a file named i sans the .jpg extension, which would most likely fail in the vast majority of cases.)
I would do the file check first such that find just reports files that have a corresponding text file. The following snippet will just display jpg files that have a corresponding txt file:
find . -name "*.jpg" -maxdepth 1 -exec /bin/bash -c '[ -e "${0%.*}.txt" ] && echo "$0";' {} \;
I want to download all episodes of a podcast that is poorly organized. Every episode is placed in a separate subfolder on the server, and they all have the file name "file.mp3" - I'd like to download them, sequentially downloading, then renaming, and then moving on to the next file. Using something like wget causes each file to overwrite the previous file, given they have the same file name.
wget normally doesn't overwrite files, it adds a number as a suffix:
file.mp3
file.mp3.1
file.mp3.2
...
But you can prevent that by calling it in a loop and using its -O option to specify the name:
count=0
urls=( http://example.com/folderA/file.mp3
http://example.com/folderB/file.mp3
http://example.com/folderC/file.mp3
)
for url in "${urls[#]}" ; do
wget -O file-$count.mp3 "$url"
(( count++ ))
done
I have list of gzip files:
file1.gz
file2.gz
file3.gz
Is there a way to concatenate or gzipping these files into one gzip file
without having to decompress them?
In practice we will use this in a web database (CGI). Where the web will receive
a query from user and list out all the files based on the query and present them
in a batch file back to the user.
With gzip files, you can simply concatenate the files together, like so:
cat file1.gz file2.gz file3.gz > allfiles.gz
Per the gzip RFC,
A gzip file consists of a series of "members" (compressed data sets). [...] The members simply appear one after another in the file, with no additional information before, between, or after them.
Note that this is not exactly the same as building a single gzip file of the concatenated data; among other things, all of the original filenames are preserved. However, gunzip seems to handle it as equivalent to a concatenation.
Since existing tools generally ignore the filename headers for the additional members, it's not easily possible to extract individual files from the result. If you want this to be possible, build a ZIP file instead. ZIP and GZIP both use the DEFLATE algorithm for the actual compression (ZIP supports some other compression algorithms as well as an option - method 8 is the one that corresponds to GZIP's compression); the difference is in the metadata format. Since the metadata is uncompressed, it's simple enough to strip off the gzip headers and tack on ZIP file headers and a central directory record instead. Refer to the gzip format specification and the ZIP format specification.
Here is what man 1 gzip says about your requirement.
Multiple compressed files can be concatenated. In this case, gunzip will extract all members at once. For example:
gzip -c file1 > foo.gz
gzip -c file2 >> foo.gz
Then
gunzip -c foo
is equivalent to
cat file1 file2
Needless to say, file1 can be replaced by file1.gz.
You must notice this:
gunzip will extract all members at once
So to get all members individually, you will have to use something additional or write, if you wish to do so.
However, this is also addressed in man page.
If you wish to create a single archive file with multiple members so that members can later be extracted independently, use an archiver such as tar or zip. GNU tar supports the -z option to invoke gzip transparently. gzip is designed as a complement to tar, not as a replacement.
Just use cat. It is very fast (0.2 seconds for 500 MB for me)
cat *gz > final
mv final final.gz
You can then read the output with zcat to make sure it's pretty:
zcat final.gz
I tried the other answer of 'gz -c' but I ended up with garbage when using already gzipped files as input (I guess it double compressed them).
PV:
Better yet, if you have it, 'pv' instead of cat:
pv *gz > final
mv final final.gz
This gives you a progress bar as it works, but does the same thing as cat.
You can create a tar file of these files and then gzip the tar file to create the new gzip file
tar -cvf newcombined.tar file1.gz file2.gz file3.gz
gzip newcombined.tar