I use linux, bash.
I have a bunch of image files in several directories. All have specific names that must be saved (filenames AND directories). But I need to convert all images to different size and ppi. I used the
convert -units PixelsPerInch 75 -resize 200x200 ~/filepath/*.jpg ~/filepath.*jpg
but it replaced the wrong file. Also I've tried
"{}" "{}"%03d.jpg
to add a number to the new filename but it gives only 001.jpg etc.
Solutions like
Bash - get last dirname/filename in a file path argument
or
Extract filename and extension in Bash
with ${filename%.} and other parts of scripts
or
http://bytebar.blogspot.com/2008/08/bash-filename-extraction.html
do not work or I have no such knowledge to understand the code...
How can I convert those files keeping their names and directories?
ls ~/filepath.*jpg | xargs -n 1 -I{} convert -units PixelsPerInch 75 -resize 200x200 {} {}
"xargs -n 1" calls the following function with provided arguments one at a time.
The -I{} argument of xargs, tells xargs to replace each occurence of {} with the given argument.
Related
I want to change multiple different strings across all files in a folder to one new string.
When the string in the text files (within a same directory) is like this:
file1.json: "url/1.png"
file2.json: "url/2.png"
file3.json: "url/3.png"
etc.
I would need to point them all to a single PNG, i.e., "url/static.png", so all three files have the same URL inside pointing to the same PNG.
How can I do that?
you can use the command find and sed for this. make sure you are in the folder that you want to replace files.
find . -name '*.*' -print|xargs sed -i "s/\"url\/1.png\"/\"url\/static.png\"/g"
Suggesting bash script:
#!/bin/bash
# for each file with extension .json in current directory
for currFile in *.json; do
# extract files ordinal from from current filename
filesOrdinal=$(echo "#currFile"| grep -o "[[:digit:]]\+")
# use files ordinal to identify string and replace it in current file
sed -i -r 's|url/'"$filesOrdinal".png'|url/static.png|' $currFile
done
This question already has answers here:
Rename multiple files based on pattern in Unix
(24 answers)
Closed 5 years ago.
Write a simple script that will automatically rename a number of files. As an example we want the file *001.jpg renamed to user defined string + 001.jpg (ex: MyVacation20110725_001.jpg) The usage for this script is to get the digital camera photos to have file names that make some sense.
I need to write a shell script for this. Can someone suggest how to begin?
An example to help you get off the ground.
for f in *.jpg; do mv "$f" "$(echo "$f" | sed s/IMG/VACATION/)"; done
In this example, I am assuming that all your image files contain the string IMG and you want to replace IMG with VACATION.
The shell automatically evaluates *.jpg to all the matching files.
The second argument of mv (the new name of the file) is the output of the sed command that replaces IMG with VACATION.
If your filenames include whitespace pay careful attention to the "$f" notation. You need the double-quotes to preserve the whitespace.
You can use rename utility to rename multiple files by a pattern. For example following command will prepend string MyVacation2011_ to all the files with jpg extension.
rename 's/^/MyVacation2011_/g' *.jpg
or
rename <pattern> <replacement> <file-list>
this example, I am assuming that all your image files begin with "IMG" and you want to replace "IMG" with "VACATION"
solution : first identified all jpg files and then replace keyword
find . -name '*jpg' -exec bash -c 'echo mv $0 ${0/IMG/VACATION}' {} \;
for file in *.jpg ; do mv $file ${file//IMG/myVacation} ; done
Again assuming that all your image files have the string "IMG" and you want to replace "IMG" with "myVacation".
With bash you can directly convert the string with parameter expansion.
Example: if the file is IMG_327.jpg, the mv command will be executed as if you do mv IMG_327.jpg myVacation_327.jpg. And this will be done for each file found in the directory matching *.jpg.
IMG_001.jpg -> myVacation_001.jpg
IMG_002.jpg -> myVacation_002.jpg
IMG_1023.jpg -> myVacation_1023.jpg
etcetera...
find . -type f |
sed -n "s/\(.*\)factory\.py$/& \1service\.py/p" |
xargs -p -n 2 mv
eg will rename all files in the cwd with names ending in "factory.py" to be replaced with names ending in "service.py"
explanation:
In the sed cmd, the -n flag will suppress normal behavior of echoing input to output after the s/// command is applied, and the p option on s/// will force writing to output if a substitution is made. Since a sub will only be made on match, sed will only have output for files ending in "factory.py"
In the s/// replacement string, we use "& " to interpolate the entire matching string, followed by a space character, into the replacement. Because of this, it's vital that our RE matches the entire filename. after the space char, we use "\1service.py" to interpolate the string we gulped before "factory.py", followed by "service.py", replacing it. So for more complex transformations youll have to change the args to s/// (with an re still matching the entire filename)
Example output:
foo_factory.py foo_service.py
bar_factory.py bar_service.py
We use xargs with -n 2 to consume the output of sed 2 delimited strings at a time, passing these to mv (i also put the -p option in there so you can feel safe when running this). voila.
NOTE: If you are facing more complicated file and folder scenarios, this post explains find (and some alternatives) in greater detail.
Another option is:
for i in *001.jpg
do
echo "mv $i yourstring${i#*001.jpg}"
done
remove echo after you have it right.
Parameter substitution with # will keep only the last part, so you can change its name.
Can't comment on Susam Pal's answer but if you're dealing with spaces, I'd surround with quotes:
for f in *.jpg; do mv "$f" "`echo $f | sed s/\ /\-/g`"; done;
You can try this:
for file in *.jpg;
do
mv $file $somestring_${file:((-7))}
done
You can see "parameter expansion" in man bash to understand the above better.
I have a text file - accessions.txt (below is a subset of this file):
KRO94967.1
KRO95967.1
KRO96427.1
KRO94221.1
KRO94121.1
KRO94145.1
WP_088442850.1
WP_088252850.1
WP_088643726.1
WP_088739685.1
WP_088283155.1
WP_088939404.1
And I have a directory with multiple files (*.align).
I want to find the filenames (*.align) which content matches any line within my accessions.txt text file.
I know that find . -exec grep -H 'STRING' {} + works to find specific strings (e.g replacing STRING with WP_088939404.1 returns every filename where the string WP_088939404.1 is present).
Is there a way to replace STRING with "all strings inside my text file" ?
Or
Is there another (better) way to do this?
I was trying to avoid writing a loop that reads the content of all my files as there are too many of them.
Many thanks!
You're looking for grep's -f option.
find . -name '*.align' -exec grep -Fxqf accessions.txt {} \; -print
grep can take a list of patterns to match with -f.
grep -lFf accessions.txt directory/*.align
-F tells grep to interpret the lines as fixed strings, not regex patterns.
Sometimes, -w is also needed to prevent matching inside words, e.g.
abcd
might match not only abcd, but also xabcd or abcdy. Sometimes, preprocessing the input list is needed to prevent unwanted matching if the rules are more complex.
I have a folder which has around 300k files of each file contains 2-3mb
Now I want to run a command to find the count of char { in shell
My command:
nohup cat *20200119*| grep "{" | wc -l > /mpt_sftp/mpt_cdr_ocs/file.txt
This works fine with small number of files
When i run in files location where I have all the files (300k files) it showing
Argument too long
Would you please try the following:
find . -maxdepth 1 -type f -name "*20200119*" -print0 | xargs -0 grep -F -o "{" | wc -l > /mpt_sftp/mpt_cdr_ocs/file.txt
I have actually tested with 300,000 files of 10-character-long filenames and it is working well.
xargs automatically adjusts the length of argument list fed to grep and we don't need to worry about it. (You can see how the grep command is executed by putting -t option to xargs.)
The -F option drastically speeds-up the execution of grep to search for a fixed string, not a regex.
The -o option will be needed if the character { appears multiple times in a line and you want to count them individually.
The maximum size of the argument list varies, but it is usually something like 128 KiB or 256 KiB. That means you have an awful lot of files if the *20200119* part is overflowing the maximum argument list. But you say "around 3 lakhs files", which is around 300,000 — each file has at least the 8-character date string in it, plus enough other characters to make the name unique, so the list of file names will be far too long for even the largest plausible 'maximum argument list size'.
Note that the nohup cat part of your command is not sensible (see UUoC: Useless Use of Cat); you should be using grep '{' *20200119* to save transferring all that data down a pipe unnecessarily. However, that too would run into problems with the argument list being too long.
You will probably have to use a variant of the following command to get the desired result without overflowing your command line:
find . -depth 1 -name '*20200119*' -exec grep '{' {} + | wc -l
This uses the feature of POSIX find that groups as many arguments as will fit on the command line without overflowing to run grep on large (but not too large) numbers of files, and then pass the output of the grep commands to wc. If you're worried about the file names appearing in the output, suppress them with the grep -h.
Or you might use:
find . -depth 1 -name '*20200119*' -exec grep -c -h '{' {} + |
awk '{sum += $1} END {print sum}'
The grep -c -h on macOS produces a simple number (the count of the number of lines containing at least one {) on its standard output for each file listed in its argument list; so too does GNU grep. The awk script adds up those numbers and prints the result.
Using -depth 1 is supported by find on macOS; so too is -maxdepth 1 — they are equivalent. GNU find does not appear to support -depth 1. It would be better to use -maxdepth 1. POSIX find only supports -depth with no number. You'd probably get a better error message from using -maxdepth 1 with a find that only supports POSIX's rather minimal set of options than you would when using -depth 1.
How can I create a number of text files, which include the filenames of the files in a specific directory, up to a maximum of 999 rows per text file?
I started from this:
find ./J0902-405/*.evt -maxdepth 1 -type f -fprintf files_xselect.list %f\\n
And it writes the filenames properly in the textfile.
But afterwords, I need to put the 999 rows limit, and after that limit, create another text file with the following 999 names, and so on, until all the *.evt files are listed.
find ./J0902-405/*.evt -maxdepth 1 -type f | split -l999 -
From the manual page:
NAME
split - split a file into pieces
SYNOPSIS
split [OPTION]... [INPUT [PREFIX]]
DESCRIPTION
Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default size is 1000
lines, and default PREFIX is `x'. With no INPUT, or when INPUT is -, read standard
input.
-l, --lines=NUMBER
put NUMBER lines per output file
#DopeGhoti's answer is the right approach, but let me flesh it out a bit, for those new to split (like me):
find ./J0902-405 -maxdepth 1 -name '*.evt' -type f -printf '%f\n' | \
split -l 999 -d - files_xselect.list.
-name ... with a quoted filename pattern lets find do the pathname expansion (as opposed to the shell - no point in letting both the shell and find do the work).
-printf '%f\n' ensures that only filenames (no path components) are output, as in the OP.
-l 999 specifies the split size in lines; default is 1000.
-d causes numerical suffixes to be used for the output files (00, 01, ...) rather than the default letters (aa, ab, ...) [note: won't work on OSX]; default suffix length is 2; to control the number of digits/chars. in the suffix, use -a {length}.
- causes split to read from stdin - in this case, the output from find.
files_xselect.list. is the output-file prefix; thus, we get files files_xselect.list.00, files_xselect.list.01, ...
If you want more control over the output filename - e.g., to move the suffix data to a different part of the filename - you can use the --filter option (note: won't work on OS X), which accepts a shell command to which the output data for each file is piped, along with variable $FILE containing the name of the respective output filename; this gives you the chance to modify the output filename based on it:
For instance, to create output files named files_xselect.00.list, ... - i.e., to place the suffix data before the filename extension, you'd use:
... | split -l 999 -d --filter='> ${FILE}.list' - 'files_xselect.'
Something like
#!/bin/bash
for file in ./J0902-405/*.evt; do
[[ $i > 999 ]] && i=0 && j=$((j+1))
[[ -f $file ]] && i=$((i+1)) && echo "${file##*/}" >> "fileofnames$j.txt"
done