I need to extract an attachment that I receive every day via email, on a linux server.
I'm using ripMIME for this task and have a script like this:
theFile=$(ls -t * | head -n 1)
ripmime -i $theFile -d /home/myDirectory/
First line assigns the name of the newest file (email) to the variable "theFile"
Second line should extract it's attachments to the /home/myDirectory/ path, however it doesn't extracts anything.
However, if I execute this line: (including the file name instead of the variable)
ripmime -i 1536138112.M623890P26484.myDomain.com,S\=1345977,W\=1363482:2,S -d /home/myDirectory/
...then the files are successfully extracted and copied to the specified directory.
I need to use a variable since I can't possibly know the name of the file, I just need to extract the files from the newest email using a script.
Also, I don't get any output when the instruction fails, so I'm in the dark here.
The ripMIME tool documentation can be found here
Any help will be appreciated.
When I included those lines inside a script file (.sh) and executed it, then everything worked like a charm. That didn't happen when I was trying to execute it directly from command line.
Related
I have a generic script that uses wget to download the file (passed as parameter to the script) from FTP server. The script always downloads the files into the same local folder. The problem I am running into is that .listing file created by wget gets deleted by default so if the script is called in parallel for different files, whichever process gets to delete the .listing file succeeds and the rest fail.
So I tried to use --no-remove-listing along with wget command, but then I get the error:
File ".listing" already there; not retrieving.
I looked at another post but as mentioned in the comments by original poster, the question hasn't been answered even though it is marked so.
One option I was thinking about is to change the script to create subdirectory with filename and download the file there. But since it is a large script, I was trying to see if there is an easier option to just change wget command.
I'm new to Bash scripting. My script intended role is to access a provided path and then apply some software (RTG - Real time Genomics) commands on the data provided in that path. However, when i try to execute the bash from CLI, it gives me following error
ERROR:There were invalid input file paths
The path I have provided in the script is accurate. That is, In the original directory, where the program 'RTG' resides, I have made folders accordingly like /data/reads/NA19240 and placed both *_1.fastq and *_2.fastq files inside NA19240.
Here is the script:
#!/bin/bash
for left_fastq in /data/reads/NA19240/*_1.fastq; do
right_fastq=${left_fastq/_1.fastq/_2.fastq}
lane_id=$(basename ${left_fastq/_1.fastq})
rtg format -f fastq -q sanger -o ${lane_id} -l ${left_fastq} -r ${right_fastq} --sam-rg "#RG\tID:${lane_id}\tSM:NA19240\tPL:ILLUMINA"
done
I have tried many workarounds but still not being able to bypass this error. I will be really grateful if you guys can help me fixing this problem. Thanks
After adding set -aux in bash script for debugging purpose, I'm getting following output now
adnan#adnan-VirtualBox[Linux] ./format.sh
+ for left_fastq in '/data/reads/NA19240/*_1.fastq'
+ right_fastq='/data/reads/NA19240/*_2.fastq'
++ basename '/data/reads/NA19240/*'
+ lane_id='*'
+ ./rtg format -f fastq -q sanger -o '*' -l '/data/reads/NA19240/*_1.fastq' -r '/data/reads/NA19240/*_2.fastq' --sam-rg '#RG\tID:*\tSM:NA19240\tPL:ILLUMINA'
Error: File not found: "/data/reads/NA19240/*_1.fastq"
Error: File not found: "/data/reads/NA19240/*_2.fastq"
Error: There were 2 invalid input file paths
You need to set the nullglob option in the script, like so:
shopt -s nullglob
By default, non-matching globs are expanded to themselves. The output you got by setting set -aux indicates that the file glob /data/reads/NA19240/*_1.fastq is getting interpreted literally. The only way this would happen is if there were no files found, and nullglob was disabled.
In the original directory, where the program 'RTG' resides, I have
made folders accordingly like /data/reads/NA19240 and placed both
*_1.fastq and *_2.fastq files inside NA19240.
So you say, your data folders are in the original directory (whatever that may be), but in the script you wrongly specify them to be in the root directory (by the leading /).
Since you start the script in the original directory, just drop the leading / and use a relative path:
for left_fastq in data/reads/NA19240/*_1.fastq
I used logrotate to create a log file that gets rolled over every day at 03:00; so the file that is created has the following format:
userlog.%Y-%m-%d
the script then zips the file as well so the end result is userlog.%Y-%m-%d.gz .... an actual file name is userlog.2015-09-09.gz.
I am writing a shell script that will copy the file created every day and will then unzip it, run a search on the log, extract results, format them, and then email them, and then delete the copied zip file.
I can get everything to work smoothly but I cannot copy the file using the method that it was created in.
I am trying to run the following:
sudo cp userlog.%Y-%d-%m.gz /home/local/temp
but the command does not execute. My understanding is that the current date, month, and year should be substituted into the variable fields in the file name.
If you could please correct my approach and understanding of this concept. Or please let me know if you feel this should work. Or if there is an alternate approach to call a file created in this fashion then kindly advise.
Presently I am having to write the filename into the script manually every morning and I would very much like to escape this hardship.
You need a simple command substitution.
sudo cp userlog.$(date +%Y-%d-%m).gz /home/local/temp
%Y by itself is just a static string; but when passed as an argument to the date command, it specifies the four-digit year for the given date. (Default is today's date.)
Regardless of your problems with this particular command, you should certainly not need to edit the script file every day. A very fundamental feature of scripts is the parametrization of arguments. If your current script looks like
#!/bin/bash
gzgrep -c "failure" /path/to/userlog.2015-09-11.gz
sudo cp /path/to/userlog.2015-09-11.gz /home/local/temp
then it can easily be refactored to read the date as a command-line parameter:
#!/bin/bash
: ${1?"Syntax: $0 yyyy-mm-dd"}
gzgrep -c "failure" /path/to/userlog."$1".gz
sudo cp /path/to/userlog."$1".gz /home/local/temp
If you saved this as userlog somewhere in your PATH, the solution to your original problem would now be simply
userlog $(date +%Y-%m-%d)
which of course could be saved as a script, too, or as a shell function in your personal .bash_profile or similar. But if this is mandatory daily routine, you'll probably prefer to add it as a cron job to run every morning (even if you are away). Cron will send you an email with any output. (Do notice that cron does not necessarily have the same PATH etc as your interactive login shell, though.)
I need to write a shell script to run as a cron task, or preferably on creation of a file in a certain folder.
I have an incoming and an outgoing folder (they will be used to log mail). There will be files created with codes as follows...
bmo-001-012-dfd-11 for outgoing and 012-dfd-003-11 for incoming. I need to filter the project/client code (012-dfd) and then place it in a folder in the specific project folder.
Project folders are located in /projects and follow the format 012-dfd. I need to create symbolic links inside the incoming or outgoing folders of the projects, that leads to the correct file in the general incoming and outgoing folders.
/incoming/012-dfd-003-11.pdf -> /projects/012-dfd/incoming/012-dfd-003-11.pdf
/outgoing/bmo-001-012-dfd-11.pdf -> /projects/012-dfd/outgoing/bmo-001-012-dfd-11.pdf
So my questions
How would I make my script run when a file is added to either incoming or outgoing folder
Additionally, is there any associated disadvantages with running upon file modification compared with running as cron task every 5 mins
How would I get the filename of recent (since script last run) files
How would I extract the code from the filename
How would I use the code to create a symlink in the desired folder
EDIT: What I ended up doing...
while inotifywait outgoing; do find -L . -type l -delete; ls outgoing | php -R '
if(
preg_match("/^\w{3}-\d{3}-(\d{3}-\w{3})-\d{2}(.+)$/", $argn, $m)
&& $m[1] && (file_exists("projects/$m[1]/outgoing/$argn") != TRUE)
){
`ln -s $(pwd)/outgoing/$argn projects/$m[1]/outgoing/$argn;`;
}
'; done;
This works quite well - cleaning up deleted symlinks also (with find -L . -type l -delete) but I would prefer to do it without the overhead of calling php. I just don't know bash well enough yet.
Some near-answers for your task breakdown:
On linux, use inotify, possibly through one of its command-line tools, or script language bindings.
See above
Assuming the project name can be extracted thinking positionally from your examples (meaning not only does the project name follows a strict 7-character format, but what precedes it in the outgoing file also does):
echo `basename /incoming/012-dfd-003-11.pdf` | cut -c 1-7
012-dfd
echo `basename /outgoing/bmo-001-012-dfd-11.pdf`| cut -c 9-15
012-dfd
mkdir -p /projects/$i/incoming/ creates directory /projects/012-dfd/incoming/ if i = 012-dfd,
ln -s /incoming/foo /projects/$i/incoming/foo creates a symbolic link from the latter argument, to the preexisting, former file /incoming/foo.
How would I make my script run when a file is added to either incoming or outgoing folder
Additionally, is there any associated disadvantages with running upon file modification compared with running as cron task
every 5 mins
If a 5 minutes delay isn't an issue, I would go for the cron job (it's easier and -IMHO- more flexible)
How would I get the filename of recent (since script last run) files
If your script runs every 5 minutes, then you can tell that all the files created in between now (and now - 5 minutes) are newso, using the command ls or find you can list those files.
How would I extract the code from the filename
You can use the sed command
How would I use the code to create a symlink in the desired folder
Once you have the desired file names, you can usen ln -s command to create the symbolic link
Well, I'm a linux newbie, and I'm having an issue with a simple bash script.
I've got a program that adds to a log file while it's running. Over time that log file gets huge. I'd like to create a startup script which will rename and move the log file before each run, effectively creating separate log files for each run of the program. Here's what I've got so far:
pastebin
DATE=$(date +"%Y%m%d%H%M")
mv server.log logs/$DATE.log
echo program
When run, I see this:
: command not found
program
When I cd to the logs directory and run dir, I see this:
201111211437\r.log\r
What's going on? I'm assuming there's some syntax issue I'm missing, but I can't seem to figure it out.
UPDATE: Thanks to shellter's comment below, I've found the problem to be due to the fact that I'm editing the .sh file in Notepad++ in windows, and then sending via ftp to the server, where I run the file via ssh. After running dos2unix on the file, it works.
New question: How can I save the file correctly in the first place, to avoid having to perform this fix every time I resend the file?
mv server.log logs/$(date -d "today" +"%Y%m%d%H%M").log
The few lines you posted from your script look okay to me. It's probably something a bit deeper.
You need to find which line is giving you this error. Add set -xv to the top of your script. This will print out the line number and the command that's being executed to STDERR. This will help you identify where in your script you're getting this particular error.
BTW, do you have a shebang at the top of your script? When I see something like this, I normally expect its an issue with the Shebang. For example, if you had #! /bin/bash on top, but your bash interpreter is located in /usr/bin/bash, you'll see this error.
EDIT
New question: How can I save the file correctly in the first place, to avoid having to perform this fix every time I resend the file?
Two ways:
Select the Edit->EOL Conversion->Unix Format menu item when you edit a file. Once it has the correct line endings, Notepad++ will keep them.
To make sure all new files have the correct line endings, go to the Settings->Preferences menu item, and pull up the Preferences dialog box. Select the New Document/Default Directory tab. Under New Document and Format, select the Unix radio button. Click the Close button.
A single line method within bash works like this.
[some out put] >$(date "+%Y.%m.%d-%H.%M.%S").ver
will create a file with a timestamp name with ver extension.
A working file listing snap shot to a date stamp file name as follows can show it working.
find . -type f -exec ls -la {} \; | cut -d ' ' -f 6- >$(date "+%Y.%m.%d-%H.%M.%S").ver
Of course
cat somefile.log > $(date "+%Y.%m.%d-%H.%M.%S").ver
or even simpler
ls > $(date "+%Y.%m.%d-%H.%M.%S").ver
I use this command for simple rotate a file:
mv output.log `date +%F`-output.log
In local folder I have 2019-09-25-output.log
Well, it's not a direct answer to your question, but there's a tool in GNU/Linux whose job is to rotate log files on regular basis, keeping old ones zipped up to a certain limit. It's logrotate
You can write your scripts in notepad but just make sure you convert them
using this ->
$ sed -i 's/\r$//' yourscripthere
I use it all they time when I'm working in cygwin and it works. Hope this helps
First, thanks for the answers above! They lead to my solution.
I added this alias to my .bashrc file:
alias now='date +%Y-%m-%d-%H.%M.%S'
Now when I want to put a time stamp on a file such as a build log I can do this:
mvn clean install | tee build-$(now).log
and I get a file name like:
build-2021-02-04-03.12.12.log