When I was checking processes of perl using ps aux | grep perl the list of perl pid's got listed where its waiting in queue. so i started investigating how this processes got initiated /usr/local/bin/processLogs. I wanted to know what does -w meant in this process execution /usr/bin/perl -w /usr/local/bin/processLogs.
[root#test]# ps aux | grep perl
root 4223 1.6 0.0 132560 4576 ? R 03:11 1:06 /usr/bin/perl -w /usr/local/bin/processLogs
root 4233 1.3 0.0 132560 4552 ? R 03:11 0:55 /usr/bin/perl -w /usr/local/bin/processLogs
root 4246 1.1 0.0 132560 4552 ? R 03:11 0:49 /usr/bin/perl -w /usr/local/bin/processLogs
root 4259 1.0 0.0 132560 4548 ? R 03:11 0:44 /usr/bin/perl -w /usr/local/bin/processLogs
root 4272 0.8 0.0 132560 4544 ? R 03:11 0:33 /usr/bin/perl -w /usr/local/bin/processLogs
root 4288 0.6 0.0 132560 4580 ? R 03:11 0:28 /usr/bin/perl -w /usr/local/bin/processLogs
If you run perl -h you will get the help menu, where it says for -w:
-w enable many useful warnings
This simply enables warnings. If this program is run in an automatic process, the warnings may be redirected to a log where you can read them. Assuming there are any warnings. You will need to refer to the program file processLogs to see what the program does.
Typically, an experienced user will not use -w, but prefer to use the lexically scoped use warnings instead.
The full output:
$ perl -h
Usage: perl [switches] [--] [programfile] [arguments]
-0[octal] specify record separator (\0, if no argument)
-a autosplit mode with -n or -p (splits $_ into #F)
-C[number/list] enables the listed Unicode features
-c check syntax only (runs BEGIN and CHECK blocks)
-d[:debugger] run program under debugger
-D[number/list] set debugging flags (argument is a bit mask or alphabets)
-e program one line of program (several -e's allowed, omit programfile)
-E program like -e, but enables all optional features
-f don't do $sitelib/sitecustomize.pl at startup
-F/pattern/ split() pattern for -a switch (//'s are optional)
-i[extension] edit <> files in place (makes backup if extension supplied)
-Idirectory specify #INC/#include directory (several -I's allowed)
-l[octal] enable line ending processing, specifies line terminator
-[mM][-]module execute "use/no module..." before executing program
-n assume "while (<>) { ... }" loop around program
-p assume loop like -n but print line also, like sed
-s enable rudimentary parsing for switches after programfile
-S look for programfile using PATH environment variable
-t enable tainting warnings
-T enable tainting checks
-u dump core after parsing program
-U allow unsafe operations
-v print version, patchlevel and license
-V[:variable] print configuration summary (or a single Config.pm variable)
-w enable many useful warnings
-W enable all warnings
-x[directory] ignore text before #!perl line (optionally cd to directory)
-X disable all warnings
Run 'perldoc perl' for more help with Perl.
Related
the server where I have hosted my app usually restarts due to maintenance and when it does a function that I kept open in the background stops and I have to manually turn it on.
Here are the commands that I do in ssh
ssh -p19199 -i <my ssh key file name> <my username>#server.net
source /home/myapp/virtualenv/app/3.8/bin/activate
cd /home/myapp/app
celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach
I need to start this celery app whenever there is no 'celery' function in the command ps axuww. If it is running already then it will show:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
myapp 8792 0.1 0.2 1435172 82252 ? Sl Jun27 1:27 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 8898 0.0 0.2 1115340 92420 ? S Jun27 0:32 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 8899 0.0 0.2 1098900 76028 ? S Jun27 0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 8900 0.0 0.2 1098904 76028 ? S Jun27 0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 8901 0.0 0.2 1098908 76040 ? S Jun27 0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 28216 0.0 0.0 10060 2928 pts/1 Ss 15:57 0:00 -bash
myapp 28345 0.0 0.0 49964 3444 pts/1 R+ 15:57 0:00 ps axuww
I need the cron job to check every 15 minutes.
You have everything figured out already, just put it inside a IF block in a shell (Bash) script, save it wherever you feel like (eg, $HOME). After the script, we'll set Cron.
I'm gonna call this script "start_celery.sh":
#!/usr/bin/env bash -ue
#
# Start celery only if it is not running yet
#
if [[ `ps auxww | grep "[c]elery"` ]]; then
source /home/myapp/virtualenv/app/3.8/bin/activate
cd /home/myapp/app
celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach
fi
Remember to make it executable!
$ chmod +x start_celery.sh
A few notes:
the output from ps is being filtered by grep "[c]elery" because we don't one the grep line that would come out if we simply did grep "celery" (reference: https://www.baeldung.com/linux/grep-exclude-ps-results).
Using grep "celery" | grep -v "grep" would not work in our case because we want to use the lack of celery commands (exit code "1") to trigger the IF block.
setting "-ue" in the script fails the whole thing/script in case something goes wrong (this is a precaution I always use, and I live it here as a shared knowledge/hint. Even if we are not using variables (-u) I like to use it). Feel free to remove it. (reference https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html)
Now we have the script -- ~/start_celery.sh in our home directory, we just have to include it in our (user's) crontab jobs. To do that, we can use some help from some websites of our (awesome) geek community:
The first one is https://crontab.guru/ . If nothing else, it puts the cronjobs syntax in very simple, visual, straight words. They also offer a payed service in case you want to monitor your jobs.
In the same spirit of online monitoring (maybe of your interest, I live it here for the records and because it's free), you may find interesting https://cron-job.org/ .
Whatever the resources we use to figure out the crontab line we need, "every 15 minutes" is: */15 * * * * .
To set that up, we open our (user's) cronjob table (crontab) and set our job accordingly:
Open crontab jobs:
$ crontab -e
Set the (celery) job (inside crontab):
*/15 * * * * /home/myapp/start_celery.sh
This should work, but I didn't test the whole thing. Let me know how that goes.
I think what you need is a corn job running your script at startup as long as you don't experience any function crashes.
Start by opening a terminal window and run the command below. Don't forget to <add your ssh key> and <username>. This will generate the script.sh file for you and save it in $HOME
echo -e "ssh -p19199 -i <my ssh key file name> <my username>#server.net\n\nsource /home/myapp/virtualenv/app/3.8/bin/activate\n\ncd /home/myapp/app\n\ncelery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach" >> $HOME/startup.sh
Make it executable using the following command:
sudo chmod +x $HOME/startup.sh
Now create the cron job to run the script when you restart your server. In the terminal create a new cron job.
crontab -e
This will open a new editor screen for you to write the cron job. Press the letter i on your keyboard to start the insert mode and type the following
#reboot sh $HOME/startup.sh
Exit the editor and save your changes by pressing esc key then type :wq and hit Enter
Restart your server and the cron job should run the script for you.
If you want to manage the tasks that should run when a system reboots, shutdowns are alike, you should not make use of cron. These tasks are managed by the Power Management Utilities (pm-utils). Have a look at man pm-action and this question for a somewhat detailed explanation.
Obviously, as a user it is not always possible to configure these hooks for pm-util and alternative options are considered. Having a cron-job running every x minutes is an obvious solution that would allow the user to automatically start a job.
The command you would run in cron would look something like:
pgrep -f celery 2>/dev/null || /path/to/celery_startup
Where /path/to/celery_startup is a simple bash script that contains the commands to be executed (the ones mentioned in the OP).
I know that grouping commands(command-list) creates a subshell environment, and each listed command is executed in that subshell. But if I execute a simple command in the grouping command, (use the ps command to output the processes), then no subshell process is output. But if I tried to execute a list of commands (compound command) in the grouping command, then a subshell process is output. Why does it produce such a result?
A test of executing a simple command (only a ps command) in a grouping command:
[root#localhost ~]# (ps -f)
with the following output:
UID PID PPID C STIME TTY TIME CMD
root 1625 1623 0 13:49 pts/0 00:00:00 -bash
root 1670 1625 0 15:05 pts/0 00:00:00 ps -f
Another test of executing a compound command(a list of commands) in a grouping command:
[root#localhost ~]# (ps -f;cd)
with the following output:
UID PID PPID C STIME TTY TIME CMD
root 1625 1623 0 13:49 pts/0 00:00:00 -bash
root 1671 1625 0 15:05 pts/0 00:00:00 -bash
root 1672 1671 0 15:05 pts/0 00:00:00 ps -f
I tested a lot of other commands (compound commands and simple commands), but the results are the same. I guess even if I execute a simple command in a grouping command, bash should fork a subshell process, otherwise it can't execute the command. But why can't I see it?
Bash optimizes the execution. It detects that only one command is inside the ( ) group and calls fork + exec instead of fork + fork + exec. That's why you see one bash process less in the list of processes. It is easier to detect when using command that take more time ( sleep 5 ) to eliminate timing. Also, you may want to read this thread on unix.stackexchange.
I think the optimization is done somewhere inside execute_cmd.c in execute_in_subshell() function (arrows > added by me):
/* If this is a simple command, tell execute_disk_command that it
might be able to get away without forking and simply exec.
>>>> This means things like ( sleep 10 ) will only cause one fork
If we're timing the command or inverting its return value, however,
we cannot do this optimization. */
and in execute_disk_command() function we can also read:
/* If we can get away without forking and there are no pipes to deal with,
don't bother to fork, just directly exec the command. */
It looks like an optimization and dash appears to be doing it too:
Running
bash -c '( sleep 3)' & sleep 0.2 && ps #or with dash
as does, more robustly:
strace -f -e trace=clone dash -c '(/bin/sleep)' 2>&1 |grep clone # 1 clone
shows that the subshell is skipped, but if there's post work to be done in the subshell after the child, the subshell is created:
strace -f -e trace=clone dash -c '(/bin/sleep; echo done)' 2>&1 |grep clone #2 clones
Zsh and ksh are taking it even one step further and for (when they see it's the last command in the script):
strace -f -e trace=clone ksh -c '(/bin/sleep; echo done)' 2>&1 |grep clone # 0 clones
they don't fork (=clone) at all, execing directly in the shell process.
When I run the ps command i get a warning saying "Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ"
How do i suppress this warning? Is there some system setting that i must do for this. The command that i fire is :
[root#localhost home]# ps -aux | grep "curl -s -o Test"
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
root 4856 0.0 0.0 4044 672 pts/0 S+ 07:20 0:00 grep curl -s -o Test
[root#localhost home]#
Note that I have to fire the exact same command as above. That is why i am looking for some system setting that will suppress the warning.
From the FAQ:
Why does "ps -aux" complain about a bogus '-'?
According to the POSIX and UNIX standards, the above command asks to display
all processes with a TTY (generally the commands users are running) plus all
processes owned by a user named "x". If that user doesn't exist, then ps will
assume you really meant "ps aux". The warning is given to gently break you of a
habit that will cause you trouble if a user named "x" were created.
On my system, where a user x does not exist, I get no warning message. Therefore one can surmise that, on your system, a user named x exists.
If you can remove user x you can probably get the warning to go away. If not, you are stuck with the warning message.
Try:
ps -aux | grep "curl -s -o Test" 2> /dev/null
Or a variant of that.
Let's say I have written a Perl script called "foo.pl" that takes in a password argument via the -p switch.
However, while it is running, anyone can do a ps and see the entire command-line string, including the password:
$ ps a |grep 'foo\.pl'
32310 pts/4 S+ 0:00 /usr/bin/perl -w ./foo.pl -p password
32313 pts/5 S+ 0:00 grep foo.pl
What is the easiest/simplest way to hide the password and replace it with something like xxxxxx?
Ask for the password from inside the script, so you don't have to pass it as an argument.
Update
Apparently this work for me, simulating a mysql behaviour:
#!/usr/bin/perl
($0 = "$0 #ARGV") =~ s/--password=\K\S+/x/;
<STDIN>;
$ ./s --user=me --password=secret
^Z
$ ps
PID TTY TIME CMD
1637 ttys000 0:00.12 -bash
2013 ttys000 0:00.00 ./s --user=me --password=x
Under MacOS 10.6
Passing passwords on the command line is not really a good idea, as already mentioned.
But: you can usually (it is OS-dependent) change the name that is shown by ps by assigning to $0.
e.g. (tested on Linux)
$ cat secret.pl
#!/usr/bin/perl
$0 = "my secret perl script";
sleep 15;
$ ./secret.pl -p foobar &
[2] 426
$ ps a | grep perl
426 pts/0 S 0:00 my secret perl script
428 pts/0 S+ 0:00 grep perl
See the section on $0 in the perlvar manpage for details.
There are a couple of ways to go. The most immediate is to (like sidyll says) prompt for the password in the actual script. Don't put in on the command line, and you won't have to hide it.
Another option is a private password file. This file can be read through shell interpolation, but it's still kind of a kludge.
You could add a bit more flexibility to the private password file by wrapping your script in a "launcher" script. Essentially, you write a script whose sole purpose is to "set up" the password file, and then launch your real script.
I have some scripts that ought to have stopped running but hang around forever. Is there some way I can figure out what they're writing to STDOUT and STDERR in a readable way?
I tried, for example, to do:
$ tail -f /proc/(pid)/fd/1
but that doesn't really work. It was a long shot anyway.
Any other ideas?
strace on its own is quite verbose and unreadable for seeing this.
Note: I am only interested in their output, not in anything else. I'm capable of figuring out the other things on my own; this question is only focused on getting access to stdout and stderr of the running process after starting it.
Since I'm not allowed to edit Jauco's answer, I'll give the full answer that worked for me (Russell's page relies on un-guaranteed behaviour that, if you close file descriptor 1 for STDOUT, the next creat call will open FD 1.
So, run a simple endless script like this:
import time
while True:
print 'test'
time.sleep(1)
Save it to test.py, run with
$ python test.py
Get the PID:
$ ps auxw | grep test.py
Now, attach gdb:
$ gdb -p (pid)
and do the fd magic:
(gdb) call creat("/tmp/stdout", 0600)
$1 = 3
(gdb) call dup2(3, 1)
$2 = 1
Now you can tail /tmp/stdout and see the output that used to go to STDOUT.
There's several new utilities that wrap up the "gdb method" and add some extra touches. The one I use now is called "reptyr" ("Re-PTY-er"). In addition to grabbing STDERR/STDOUT, it will actually change the controlling terminal of a process (even if it wasn't previously attached to a terminal).
The best use of this is to start up a screen session, and use it to reattach a running process to the terminal within screen so you can safely detach from it and come back later.
It's packaged on popular distros (Ex: 'apt-get install reptyr').
http://onethingwell.org/post/2924103615/reptyr
GDB method seems better, but you can do this with strace, too:
$ strace -p <PID> -e write=1 -s 1024 -o file
Via the man page for strace:
-e write=set
Perform a full hexadecimal and ASCII dump of all the
data written to file descriptors listed in the spec-
ified set. For example, to see all output activity
on file descriptors 3 and 5 use -e write=3,5. Note
that this is independent from the normal tracing of
the write(2) system call which is controlled by the
option -e trace=write.
This prints out somewhat more than you need (the hexadecimal part), but you can sed that out easily.
I'm not sure if it will work for you, but I read a page a while back describing a method that uses gdb
I used strace and de-coded the hex output to clear text:
PID=some_process_id
sudo strace -f -e trace=write -e verbose=none -e write=1,2 -q -p $PID -o "| grep '^ |' | cut -c11-60 | sed -e 's/ //g' | xxd -r -p"
I combined this command from other answers.
strace outputs a lot less with just -ewrite (and not the =1 suffix). And it's a bit simpler than the GDB method, IMO.
I used it to see the progress of an existing MythTV encoding job (sudo because I don't own the encoding process):
$ ps -aef | grep -i handbrake
mythtv 25089 25085 99 16:01 ? 00:53:43 /usr/bin/HandBrakeCLI -i /var/lib/mythtv/recordings/1061_20111230122900.mpg -o /var/lib/mythtv/recordings/1061_20111230122900.mp4 -e x264 -b 1500 -E faac -B 256 -R 48 -w 720
jward 25293 20229 0 16:30 pts/1 00:00:00 grep --color=auto -i handbr
$ sudo strace -ewrite -p 25089
Process 25089 attached - interrupt to quit
write(1, "\rEncoding: task 1 of 1, 70.75 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.76 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.77 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.78 % "..., 73) = 73^C
You can use reredirect (https://github.com/jerome-pouiller/reredirect/).
Type
reredirect -m FILE PID
and outputs (standard and error) will be written in FILE.
reredirect README also explains how to restore original state of process, how to redirect to another command or to redirect only stdout or stderr.
You don't state your operating system, but I'm going to take a stab and say "Linux".
Seeing what is being written to stderr and stdout is probably not going to help. If it is useful, you could use tee(1) before you start the script to take a copy of stderr and stdout.
You can use ps(1) to look for wchan. This tells you what the process is waiting for. If you look at the strace output, you can ignore the bulk of the output and identify the last (blocked) system call. If it is an operation on a file handle, you can go backwards in the output and identify the underlying object (file, socket, pipe, etc.) From there the answer is likely to be clear.
You can also send the process a signal that causes it to dump core, and then use the debugger and the core file to get a stack trace.