Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Closed 8 years ago.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Improve this question
I'm trying to do three things with the mv command, but not sure it's possible? Probably need a script. not sure how to write it. All files are in same folder.
1) Files ending with v9.zip should just be .zip (the v9 removed)
2) Files containing _ should be -
3) Files with Uppercase letter next to a lowercase letter (or lowercase next to an Uppercase) should have a space between them. So MoveOverNow would be Move Over Now and ruNaway would be ruN away
[A-Z][a-z] or [a-z][A-Z] becomes [A-Z] [a-z] and [a-z] [A-Z]
There's a rename command provided with most Debian/Ubuntu based distros which was written by Robin Barker based on Larry Wall's original code from around 1998(!).
Here's an excerpt from the documentation:
"rename" renames the filenames supplied according to the rule specified as the first argument. The perlexpr argument is a Perl expression which is expected to modify the $_ string in Perl for at least some of the filenames
specified. If a given filename is not modified by the expression, it will not be renamed. If no filenames are given on the command line, filenames will be read via standard input.
For example, to rename all files matching "*.bak" to strip the extension, you might say
rename 's/\.bak$//' *.bak
To translate uppercase names to lower, you'd use
rename 'y/A-Z/a-z/' *
It uses perl so you can use perl expressions to match the pattern, in fact I believe it works much like tchrist's scripts.
One other really useful set of tools for bulk file renaming is the renameutils collection by Oskar Liljeblad. The source code is hosted by the Free Software Foundation. Additionally many distros (especially Debian/Ubuntu based distros) have a renameutils package with these tools.
On one of those distros you can install it with:
$ sudo apt-get install renameutils
And then to rename files just run this command:
$ qmv
It will pop open a text editor with the list of files, and you can manipulate them with your editor's search and replace function.
I haven't tested these, so I put echo at the front of the commands so you can try them before removing the echo to run them for real.
for f in *v9.zip; do echo mv "${f}" "${f%v9.zip}.zip"; done
for f in *_*; do echo mv "${f}" "${f//_/-}"; done
As for your third problem I'm sure it can be done too but maybe a more sophisticated approach than raw shell one-liners will help, as #tchrist mentioned.
My favorite solution is my own rename script. The simplest example that maps to your problems are these:
% rename 's/_/-/g' *
% rename 's/(\p{Lower})(\p{Upper})/$1 $2/g' *
Although I really hate whitespace in my filenames, especially vertical whitespace:
% rename 's/\s//g' *
% rename 's/\v//g' *
et cetera. It’s based on a script by The Larry Wall, but extended with options, as in:
usage: /home/tchrist/scripts/rename [-ifqI0vnml] [-F file] perlexpr [files]
-i ask about clobbering existent files
-f force clobbers without inquiring
-q quietly skip clobbers without inquiring
-I ask about all changes
-0 read null-terminated filenames
-v verbosely says what its doing
-V verbosely says what its doing but with newlines between old and new filenames
-n don't really do it
-m to always rename
-l to always symlink
-F path read filelist to change from magic path(s)
As you see, it can change not just the names of files, but where symbolic links are pointing to using the same pattern. You don’t have to use a s/// pattern, although often one does.
The other tools in that directory are mostly for Unicode work, of which there are some super-useful ones.
The above answers apply to Debian, Ubuntu etc
For RHEL and co: rename from_pattern to_pattern files
I think the link is broken and I couldn't find the page in the webarchive to the rename script in tchrist's post, so here is another one in Perl.
#!/usr/bin/perl
# -w switch is off bc HERE docs cause erroneous messages to be displayed under
# Cygwin
#From the Perl Cookbook, Ch. 9.9
# rename - Larry's filename fixer
$help = <<EOF;
Usage: rename expr [files]
This script's first argument is Perl code that alters the filename
(stored in \$_ ) to reflect how you want the file renamed. It can do
this because it uses an eval to do the hard work. It also skips rename
calls when the filename is untouched. This lets you simply use
wildcards like rename EXPR * instead of making long lists of filenames.
Here are five examples of calling the rename program from your shell:
% rename 's/\.orig$//' *.orig
% rename 'tr/A-Z/a-z/ unless /^Make/' *
% rename '$_ .= ".bad"' *.f
% rename 'print "$_: "; s/foo/bar/ if <STDIN> =~ /^y/i' *
% find /tmp -name '*~' -print | rename 's/^(.+)~$/.#$1/'
The first shell command removes a trailing ".orig" from each filename.
The second converts uppercase to lowercase. Because a translation is
used rather than the lc function, this conversion won't be locale-
aware. To fix that, you'd have to write:
% rename 'use locale; $_ = lc($_) unless /^Make/' *
The third appends ".bad" to each Fortran file ending in ".f", something
a lot of us have wanted to do for a long time.
The fourth prompts the user for the change. Each file's name is printed
to standard output and a response is read from standard input. If the
user types something starting with a "y" or "Y", any "foo" in the
filename is changed to "bar".
The fifth uses find to locate files in /tmp that end with a tilde. It
renames these so that instead of ending with a tilde, they start with
a dot and a pound sign. In effect, this switches between two common
conventions for backup files
EOF
$op = shift or die $help;
chomp(#ARGV = <STDIN>) unless #ARGV;
for (#ARGV) {
$was = $_;
eval $op;
die $# if $#;
rename($was,$_) unless $was eq $_;
}
Related
I have a directory containing files following the following naming convention:
Label_0000_AA.gz
Label_0001_BB.gz
Label_0002_CC.gz
...
All I want to do is to rename these files so that the _#### number pattern is removed, resulting in:
Label_AA.gz
Label_BB.gz
Label_CC.gz
...
but only up to a certain number. E.g.: I may have 10000 files but might only want to remove the pattern in the first 3000. Would this be possible using something like bash?
If you don't have prename or rename -
(assuming the names are consistent)
for f in Label_[0-9][0-9][0-9][0-9]_[A-Z][A-Z].gz
do mv "$f" "${f//_[0-9][0-9][0-9][0-9]/}"
done
To just do a certain range -
for n in {0000..2999}
do for f in Label_${n}_??.gz
do mv $f ${f//_$n/}
done
done
You're sure there are not collisions?
If you can name the pattern you want to change/remove in a regex you can use the command prename:
prename 's/_[0-3][[:digit:]]{3}_/_/g' Label_*.gz
This regex would only remove numbers 0000-3999.
Using the flag -n does a "dry-run" and shows what it would do.
Edit: Thanks #KamilCuk to remind me about two renames. I made it clear and changed the name to prename.
I'm trying to display the output of an AWS lambda that is being captured in a temporary text file, and I want to remove that file as I display its contents. Right now I'm doing:
... && cat output.json && rm output.json
Is there a clever way to combine those last two commands into one command? My goal is to make the full combined command string as short as possible.
For cases where
it is possible to control the name of the temporary text file.
If file is not used by other code
Possible to pass "/dev/stdout" as the.name of the output
Regarding portability: see stack exchange how portable ... /dev/stdout
POSIX 7 says they are extensions.
Base Definitions,
Section 2.1.1 Requirements:
The system may provide non-standard extensions. These are features not required by POSIX.1-2008 and may include, but are not limited to:
[...]
• Additional character special files with special properties (for example, /dev/stdin, /dev/stdout, and /dev/stderr)
Using the mandatory supported /dev/tty will force output into “current” terminal, making it impossible to pipe the output of the whole command into different program (or log file), or to use the program when there is no connected terminals (cron job, or other automation tools)
No, you cannot easily remove the lines of a file while displaying them. It would be highly inefficient as it would require removing characters from the beginning of a file each time you read a line. Current filesystems are pretty good at truncating lines at the end of a file, but not at the beginning.
A simple but extremely slow method would look like this:
while [ -s output.json ]
do
head -1 output.json
sed -i 1d output.json
done
While this algorithm is plain and simple, you should know that each time you remove the first line with sed -i 1d it will copy the whole content of the file but the first line into a temporary file, resulting in approximately 0.5*n² lines written in total (where n is the number of lines in your file).
In theory you could avoid this by do something like that:
while [ -s output.json ]
do
line=$(head -1 output.json)
printf -- '%s\n' "$line"
fallocate -c -o 0 -l $((${#len}+1)) output.json
done
But this does not account for variable newline characters (namely DOS-formatted newlines) and fallocate does not always work on xfs, among other issues.
Since you are trying to consume a file alongside its creation without leaving a trace of its existence on disk, you are essentially asking for a pipe functionality. In my opinion you should look into how your output.json file is produced and hopefully you can pipe it to a script of your own.
This question already has answers here:
Read user input inside a loop
(6 answers)
Closed 5 years ago.
First post here! I really need help on this one, I looked the issue on google, but can't manage to find an useful answer for me. So here's the problem.
I'm having fun coding some like of a framework in bash. Everyone can create their own module and add it to the framework. BUT. To know what arguments the script require, I created an "args.conf" file that must be in every module, that kinda looks like this:
LHOST;true;The IP the remote payload will connect to.
LPORT;true;The port the remote payload will connect to.
The first column is the argument name, the second defines if it's required or not, the third is the description. Anyway, long story short, the framework is supposed to read the args.conf file line by line to ask the user a value for every argument. Here's the piece of code:
info "Reading module $name argument list..."
while read line; do
echo $line > line.tmp
arg=`cut -d ";" -f 1 line.tmp`
requ=`cut -d ";" -f 2 line.tmp`
if [ $requ = "true" ]; then
echo "[This argument is required]"
else
echo "[This argument isn't required, leave a blank space if you don't wan't to use it]"
fi
read -p " $arg=" answer
echo $answer >> arglist.tmp
done < modules/$name/args.conf
tr '\n' ' ' < arglist.tmp > argline.tmp
argline=`cat argline.tmp`
info "Launching module $name..."
cd modules/$name
$interpreter $file $argline
cd ../..
rm arglist.tmp
rm argline.tmp
rm line.tmp
succes "Module $name execution completed."
As you can see, it's supposed to ask the user a value for every argument... But:
1) The read command seems to not be executing. It just skips it, and the argument has no value
2) Despite the fact that the args.conf file contains 3 lines, the loops seems to be executing just a single time. All I see on the screen is "[This argument is required]" just one time, and the module justs launch (and crashes because it has not the required arguments...).
Really don't know what to do, here... I hope someone here have an answer ^^'.
Thanks in advance!
(and sorry for eventual mistakes, I'm french)
Alpha.
As #that other guy pointed out in a comment, the problem is that all of the read commands in the loop are reading from the args.conf file, not the user. The way I'd handle this is by redirecting the conf file over a different file descriptor than stdin (fd #0); I like to use fd #3 for this:
while read -u3 line; do
...
done 3< modules/$name/args.conf
(Note: if your shell's read command doesn't understand the -u option, use read line <&3 instead.)
There are a number of other things in this script I'd recommend against:
Variable references without double-quotes around them, e.g. echo $line instead of echo "$line", and < modules/$name/args.conf instead of < "modules/$name/args.conf". Unquoted variable references get split into words (if they contain whitespace) and any wildcards that happen to match filenames will get replaced by a list of matching files. This can cause really weird and intermittent bugs. Unfortunately, your use of $argline depends on word splitting to separate multiple arguments; if you're using bash (not a generic POSIX shell) you can use arrays instead; I'll get to that.
You're using relative file paths everywhere, and cding in the script. This tends to be fragile and confusing, since file paths are different at different places in the script, and any relative paths passed in by the user will become invalid the first time the script cds somewhere else. Worse, you aren't checking for errors when you cd, so if any cd fails for any reason, then entire rest of the script will run in the wrong place and fail bizarrely. You'd be far better off figuring out where your system's root directory is (as an absolute path), then referencing everything from it (e.g. < "$module_root/modules/$name/args.conf").
Actually, you're not checking for errors anywhere. It's generally a good idea, when writing any sort of program, to try to think of what can go wrong and how your program should respond (and also to expect that things you didn't think of will also go wrong). Some people like to use set -e to make their scripts exit if any simple command fails, but this doesn't always do what you'd expect. I prefer to explicitly test the exit status of the commands in my script, with something like:
command1 || {
echo 'command1 failed!' >&2
exit 1
}
if command2; then
echo 'command2 succeeded!' >&2
else
echo 'command2 failed!' >&2
exit 1
fi
You're creating temp files in the current directory, which risks random conflicts (with other runs of the script at the same time, any files that happen to have names you're using, etc). It's better to create a temp directory at the beginning, then store everything in it (again, by absolute path):
module_tmp="$(mktemp -dt module-system)" || {
echo "Error creating temp directory" >&2
exit 1
}
...
echo "$answer" >> "$module_tmp/arglist.tmp"
(BTW, note that I'm using $() instead of backticks. They're easier to read, and don't have some subtle syntactic oddities that backticks have. I recommend switching.)
Speaking of which, you're overusing temp files; a lot of what you're doing with can be done just fine with shell variables and built-in shell features. For example, rather than reading line from the config file, then storing them in a temp file and using cut to split them into fields, you can simply echo to cut:
arg="$(echo "$line" | cut -d ";" -f 1)"
...or better yet, use read's built-in ability to split fields based on whatever IFS is set to:
while IFS=";" read -u3 arg requ description; do
(Note that since the assignment to IFS is a prefix to the read command, it only affects that one command; changing IFS globally can have weird effects, and should be avoided whenever possible.)
Similarly, storing the argument list in a file, converting newlines to spaces into another file, then reading that file... you can skip any or all of these steps. If you're using bash, store the arg list in an array:
arglist=()
while ...
arglist+=("$answer") # or ("#arg=$answer")? Not sure of your syntax.
done ...
"$module_root/modules/$name/$interpreter" "$file" "${arglist[#]}"
(That messy syntax, with the double-quotes, curly braces, square brackets, and at-sign, is the generally correct way to expand an array in bash).
If you can't count on bash extensions like arrays, you can at least do it the old messy way with a plain variable:
arglist=""
while ...
arglist="$arglist $answer" # or "$arglist $arg=$answer"? Not sure of your syntax.
done ...
"$module_root/modules/$name/$interpreter" "$file" $arglist
... but this runs the risk of arguments being word-split and/or expanded to lists of files.
I have separated some tracks from mp3 mixes using mp3splt.
BASH: (mp3splt -c('**!!***use .cue file***!!**') [cuefile.cue] [nonstopmix.mp3] ~for anyone interested, is in the Ubu repos~)
And I ended up with these filenames: "Antares" - 01 - "Xibalba".mp3 which is not a format I prefer, now I've made it a little project to change them with a shell script but its more difficult than I anticipated.
I want to change the filename from:
"Antares" - 01 - "Xibalba".mp
to:
01-Antares_-_Xibalba.mp3
so far I've used :
for var in *.mp3; do mv $var {var/"/}; done
and I could repeat that until I'm through, delete the 0x number and add one but I'd like to do it more efficient.
Could anyone give me a pointer (!not a script!) ?
I'd still like to write it myself but there's so much options that I'm a bit lost.
so far I thought to use this program flow:
read all the filenames containing .mp3 and declare as variable $var
strip $var from quotes
select 0x number, append delimiter _ (0x_)
move 0x_ to the beginning of the string
select remaining ' - - ' and change to '-'
done
which bash programs to use? especially changing the 0x puzzles me cuz I need a loop which increments this number and test if it is present in the filename variable and then it has to be changed.
It is easy to do in python 2.x. You can use this logic in any language you want.
import string
a=raw_input('Enter the name of song')
a=a.replace('"', "")
a=a.replace('.mp', ' .mp3')
words = a.split()
print words[2]+'-'+words[0]+'_-_'+words[4]+words[5]
Logic:
I removed ", then make .mp to .mp3, then splitted the string, which created a list ( array ) and then printed the elements according to need.
Try doing this :
rename -n 's/"(\w+)"\s+-\s*(\d+)\s*-\s*"(\w+)"\.mp/$2-$1_-_$3.mp3/' *mp
from the shell prompt. It's very useful, you can put some perl tricks like I does in a substitution.
You can remove the -n (dry-run mode switch) when your tests become valids.
There are other tools with the same name which may or may not be able to do this, so be careful.
If you run the following command (linux)
$ file $(readlink -f $(type -p rename))
and you have a result like
.../rename: Perl script, ASCII text executable
then this seems to be the right tool =)
If not, to make it the default (usually already the case) on Debian and derivative like Ubuntu :
$ sudo update-alternatives --set rename /path/to/rename
Last but not least, this tool was originally written by Larry Wall, the Perl's dad.
I prefer to edit in one large file rather than many independent files, but due to limitations in languages, source control, and the preference of team mates I need to output to many files.
What I'm looking for would recurse through all the files in a source directory and generate a single file to edit in VIM, with special file seperator markers. On save it would save the the changes to the correct file(s) ideally in a smart manner, based only on changes made.
Does something like this exist?
shar
Well, you could use shar(1), but it puts an X in front of each line that you will probably find annoying. (Shar came with my Mac but on my Linux systems you need to add a package.)
Shar is just, itself, a short shell script, so you could modify it easily enough to work without the X.
You might try copying /usr/bin/shar to /tmp and applying this diff with patch(1).
--- /usr/bin/shar 2009-07-13 22:26:18.000000000 -0700
+++ /tmp/shar2 2010-12-24 19:05:34.000000000 -0800
## -65,8 +65,8 ##
echo "mkdir -p $i > /dev/null 2>&1"
else
echo "echo x - $i"
- echo "sed 's/^X//' >$i << 'END-of-$i'"
- sed 's/^/X/' $i
+ echo "cat >$i << 'END-of-$i'"
+ cat $i
echo "END-of-$i"
fi
done
It reminds me of vimballs format. However, it's meant to expand files into the user runtimepath directory.
In other words, you can list all the files you want join and apply :MkVimBall (here is an example).
Then, for the extraction, you will have to momentarily (i.e. save and restore its value after the extraction) set &runtimepath to the root directory of your project before extracting with :so %.
You'll also have to play with various options like the &filetype, etc.
It's a dirty hack, but well ... it shall do the job.
Instead of dumping several files into one, processing this one and then separating stuff apart again, you could use bufdo or windo to repeat a command on all opened buffers: open the buffers to be processed, then cast the bufdo command and it will work on every opened file: http://vimdoc.sourceforge.net/htmldoc/windows.html#list-repeat