I am using an application that creates a text file on a Linux server. I then have the ability to execute a shell script (BASH 3.2.57) in which I need to convert the text file from Unix line endings to DOS and also change the extension of the file from .txt to .log.
I currently have a sed based command to do this. This command is rewritten by the application at run time to point to the specific folder and file name, in this example where you see ABC (all capital 3 letters in all my examples are a variable that can be any 3 letters).
pushd /rootfolder/parentfolder/ABC/
sed 's/$/\r/' prABC.txt > prABC.log
popd
The problem with this is that if a user runs the application for 2 different groups, say ABC and DEF at nearly the same time, the script will get overwritten with the DEF variables before ABC had a chance to fire off and do its thing with the file. Additionally the .txt is left in the folder regardless and I would like that to be removed.
A friend of mine came up with the following code that seems to work if its determined to be our best solution, but I would think and hope we have a cleaner more dynamic way to do this. Also this current method requires that when my user decides to add a GHI directory and file I now have to update the code, which i can program my application to do for me but i don't want this script to have to be rewritten every time the application wants to use it.
pushd /rootfolder/parentfolder/ABC
if [[ -f prABC.txt ]]
then
sed 's/$/\r/' prABC.txt > prABC.log
rm prABC.txt
fi
popd
pushd /rootfolder/parentfolder/DEF
if [[ -f prABC.txt ]]
then
sed 's/$/\r/' prABC.txt > prABC.log
rm prABC.txt
fi
popd
I would like to call this script at anytime from my application and it find any file named pr*.txt below the /rootfolder/parentfolder/ directory (if that has to include the parentfolder in its search that won't be a problem) and convert the line endings from LF to CRLF and change the extension of the file from .txt to .log.
I've done a ton of searching and have found near solutions for this but not exactly what I need and I want to be sure it's as safe as possible (issues with using "find with for". I don't know what utilities are installed on this build so i would like to keep it as basic/supportable as possible Thanks in advance :)
You should almost never need pushd and popd in scripts. In fact, you rarely need cd, either.
#!/bin/bash
for d in /rootfolder/parentfolder/ABC /rootfolder/parentfolder/DEF
do
if [[ -f "$d/prABC.txt" ]]
then
sed 's/$/\r/' "$d/prABC.txt" > "$d/prABC.log" &&
rm "$d/prABC.txt"
fi
done
Recall that a && b is shorthand for
if a; then
b
fi
In other words, if sed fails (because the source file can't be read, or the destination can't be written) we don't rm the source file. There should be an error message already so we don't add another one.
Not only is this more succinct, it is also easier to change if you decide that the old file should be renamed instead of removed, or you want to filter out all lines which contain "beef" in the sed script. Generally you should avoid repeated code; see also the DRY principle on Wikipedia.
Something is seriously wrong somewhere if you require DOS line endings in your files on Unix.
Related
How to simply recreate copy/paste functionality like in gui environments?
My typical scenario for copying file/directory in Linux console is:
cp source_path target_path
Sometimes paths are relative, sometimes absolute, but I need to provide them both. It works, but there are situations where I would like to recreate scenario from gui which is:
1. go to source directory
2. copy file/directory
3. go to target directory
4. paste file/directory
I imagine something like
cd source_directory_path
copy_to_stash source_name
cd target_directory_path
paste_from_stash [optional_new_target_name]
I know that there is a xclip app, but a documentation says that it copies content of a file, not a file handle. Also, I can use $OLDPWD variable and expand it when I copy file, but this is not a solution without some cumbersome.
Is there some simple, general, keyboard only, not awkward to use equivalent?
I've also asked the same question on superuser and answer that I've received is good enough for me.
In short: two additional scripts and temporary variable to hold intermediate value.
Below is a code and link to original answer.
#!/bin/bash
# source me with one of:
# source [file]
# . [file]
# Initialize
sa_file=
sa(){
# Fuction to save a file in the current PWD
if [[ -e "$PWD/$1" ]]; then
sa_file=$PWD/$1
echo "Saved for later: $sa_file"
else
echo "Error: file $PWD/$1 does not exist"
fi
}
pa(){
# Paste if file exists, to $1 if exists
if [[ -e "$sa_file" ]]; then
if [[ $1 ]]; then
cp -v "$sa_file" "$1"
else
cp -v "$sa_file" .
fi
else
echo "Error: file $sa_file does not exist, could not copy"
fi
}
https://superuser.com/a/1405953/614464
The way I see it your only option is to write a script to do all of those steps. You could easily implement the clipboard functionality by copying the file to the /tmp directory before copying again from it.
This should work as intended.
Usage: script [from] [to]
filename=$(basename "$0")
tmpfile=/tmp/$filename.$RANDOM
cd $(dirname "$0")
cp $tmpfile $filename
cd $(dirname "$1")
cp $tmpfile $(basename "$1")
One option: you can either copy-paste the filename using mouse, using copy-paste feature from your terminal emulator (e.g. Konsole or GNOME Terminal), but this: 1) requires a GUI since the terminal emulator software run in GUI; 2) well, requires a mouse.
Another option: utilize shell tab completion. You still need to type the filename, but not all of it.
Third option, and this is closer to how you work in a GUI file explorer: use a TUI-based file explorer, e.g. the dual-pane style Midnight Commander. You can use arrow keys (if you turn on the Lynx-like motion setting, which is very recommended) to quickly navigate the directory tree. Then select files using the Insert, +, -, or * keys, then copy/move files from one pane to another. It's very convenient. In fact half of the time I spend in CLI, I spend in MC.
There are multiple files in /opt/dir/ABC/ named allfile_123-abc allfile_123-def allfile_123-ghi allfile_123-xxx.
I need the files to be named new_name-abc.pgp new_name-def.pgp new_name-ghi.pgp new_name-xxx.pgp and then moved to /usr/tst/output
for file in /opt/dir/ABC/allfile_123* ;
do mv $file /usr/tst/output/"$file.pgp";
rename allfile_123 new_name /usr/tst/output/*.pgp ; done
I know the above doesn't work because $file = /opt/dir/ABC/allfile_123*. Is it possible to make this work, or is it a different command instead of 'for loop'?
This is for the Autosys application in which the jil contains a command to pass to the command line of a linux server running bash.
I could only find versions of each part of my question but not altogether and I was hoping to keep it on the command line of this jil. Unless a script is absolutely necessary.
No need for the loop, you can do this with just rename and mv:
rename -v 's/$/.pgp/' /opt/dir/ABC/allfile_123*
rename -v s/allfile_123/new_name/ /opt/dir/ABC/allfile_123*
mv /opt/dir/ABC/new_name* /usr/tst/output/
But I'm not sure the rename you are using is the same as mine.
However,
since the replacement you want to perform is fairly simple,
it's easy to do in pure Bash:
for file in /opt/dir/ABC/allfile_123*; do
newname=new_name${file##*allfile_123}.gpg
mv "$file" /usr/tst/output/"$newname"
done
If you want to write it on a single line:
for file in /opt/dir/ABC/allfile_123*; do newname=new_name${file##*allfile_123}.gpg; mv "$file" /usr/tst/output/"$newname"; done
I am looking to write a script to move files from a directory:
/home/mydir/
To another directory based on tokens in the file name. I have a bunch of files named as such:
red_office_mike_2015_montreal_546968.ext
or
$color_$location_$name_$year_$city_$numbers.extension (files will be various movie files: mov, mp4, mkv, etc.)
I would like the script to move the files to the following location:
/dir/work/$color/$name
Then verify the file has successfully copied, and delete the original file once it has.
I would also love it if the script would create the to directory if it does not already exist.
So in summary, I need a script to move files based on underscore separated tokens, create the to directory if it doesn't already exist, verify the successful copy (maybe with a size check), then delete the original file.
I am working on linux, and would prefer a bash script. The variables I have given are generic, and I will incorporate some other things to the script, I'm just looking for help on building the skeleton.
Thanks in advance for any help!
It's not a bash script, but perl is much better at this kind of thing and is installed on all Linux systems
while(<>) {
chomp;
$file = $_;
($colour, $location, $name, $year, $city, $numbers) = split(/_/,$file);
$dest0 = "/dir/work/$colour";
$dest1 = "$dest0/$name";
mkdir ($dest0) unless (-d $dest0);
mkdir ($dest1) unless (-d $dest1);
rename ($file, "$dest1/$file");
}
The script splits your input file on the underscore character, creates all the directories to the destination and then renames the file to the new filename. Rename takes care of all the copying and deleting for you. In fact it just changes the directory entries without any copying at all.
UPDATE
The above version takes its input from a file containing a list of filenames to process. For an alternative version which processes all files in the current directory, replace the while line with
while(glob("*")) {
I was able to fumble around online and come up with a for loop to do this task. I used cut and it made things simple. Here is what worked for me:
#!/bin/sh
cd "${1:-.}"
for f in *.*; do
color=`echo "$f" | cut -d'_' -f1`
name=`echo "$f" | cut -d'_' -f3`
todir="/dir/work/$color/$name"
mkdir -p "$todir"
mv "$f" "$todir"
done
This worked perfectly and I hope it can help others who might need to create directories based on portions of filenames.
The first line under the shebang made it so that it will either look at the current working directory or a directory you pass it as an argument.
Thanks to those who chimed in on the original post. I'm new with scripting so it take me a while to figure this stuff out. I love this site though, it is super helpful!
I have a RedHat linux box and I had written a script in the past to move files from one location to another with a specific text in the body of the file.
I typically only write scripts once a year so every year I forget more and more... That being said,
Last year I wrote this script and used it and it worked.
For some reason, I can not get it to work today and I know it's a simple issue and I shouldn't even be asking for help but for some reason I'm just not looking at it correctly today.
Here is the script.
ls -1 /var/text.old | while read file
do
grep -q "to.move" $file && mv $file /var/text.old/TBD
done
I'm listing all the files inside the /var/text.old directory.
I'm reading each file
then I'm grep'ing for "to.move" and holing the results
then I'm moving the resulting found files to the folder /var/text.old/TBD
I am an admin and I have rights to the above files and folders.
I can see the data in each file
I can mv them manually
I have use pwd to grab the correct spelling of the directory.
If anyone can just help me to see what the heck I'm missing here that would really make my day.
Thanks in advance.
UPDATE:
The files I need to move do not have Whitespaces.
The Error I'm getting is as follows:
grep: 9829563.msg: No such file or directory
NOTE: the file "982953.msg" is one of the files I need to move.
Also note: I'm getting this error for every file in the directory that I'm listing.
You didn't post any error, but I'm gonna take a guess and say that you have a filename with a space or special shell character.
Let's say you have 3 files, and ls -1 gives us:
hello
world
hey there
Now, while splits on the value of the special $IFS variable, which is set to <space><tab><newline> by default.
So instead of looping of 3 values like you expect (hello, world, and hey there), you loop over 4 values (hello, world, hey, and there).
To fix this, we can do 2 things:
Set IFS to only a newline:
IFS="
"
ls -1 /var/text.old | while read file
...
In general, I like setting IFS to a newline at the start of the script, since I consider this to be slightly "safer", but opinions on this probably vary.
But much better is to not parse the output of ls, and use for:
for file in /var/text.old/*`; do
This won't fork any external processes (piping to ls to while starts 2), and behaves "less surprising" in other ways. See here for some examples.
The second problem is that you're not quoting $file. You should always quote pathnames with double quoted: "$file" for the same reasons. If $file has a space (or a special shell character, such as *, the meaning of your command changes:
file=hey\ *
mv $file /var/text.old/TBD
Becomes:
mv hey * /var/text.old/TBD
Which is obviously very different from what you intended! What you intended was:
mv "hey *" /var/text.old/TBD
I'm trying to use something along the lines of
unexpand -t 4 *.php
but am unsure how to write this command to do what I want.
Weirdly,
unexpand -t 4 file.php > file.php
gives me an empty file. (i.e. overwriting file.php with nothing)
I can specify multiple files okay, but don't know how to then overwrite each file.
I could use my IDE, but there are ~67000 instances of to be replaced over 200 files, and this will take a while.
I expect that the answers to my question(s) will be standard unix fare, but I'm still learning...
You can very seldom use output redirection to replace the input. Replacing works with commands that support it internally (since they then do the basic steps themselves). From the shell level, it's far better to work in two steps, like so:
Do the operation on foo, creating foo.tmp
Move (rename) foo.tmp to foo, overwriting the original
This will be fast. It will require a bit more disk space, but if you do both steps before continuing to the next file, you will only need as much extra space as the largest single file, this should not be a problem.
Sketch script:
for a in *.php
do
unexpand -t 4 $a >$a-notab
mv $a-notab $a
done
You could do better (error-checking, and so on), but that is the basic outline.
Here's the command I used:
for p in $(find . -iname "*.js")
do
unexpand -t 4 $(dirname $p)/"$(basename $p)" > $(dirname $p)/"$(basename $p)-tab"
mv $(dirname $p)/"$(basename $p)-tab" $(dirname $p)/"$(basename $p)"
done
This version changes all files within the directory hierarchy rooted at the current working directory.
In my case, I only wanted to make this change to .js files; you can omit the iname clause from find if you wish, or use different args to cast your net differently.
My version wraps filenames in quotes, but it doesn't use quotes around 'interesting' directory names that appear in the paths of matching files.
To get it all on one line, add a semi after lines 1, 3, & 4.
This is potentially dangerous, so make a backup or use git before running the command. If you're using git, you can verify that only whitespace was changed with git diff -w.