I'm trying to use parallels with the file transfer program bbcp but there seems to be some parsing issue that I'm unable to figure out. I run the following command and get the following output -
Command -
cat testfiles.log | parallel --will-cite -j 5 bbcp -4 c -r -z -Vv -P 2 -S "ssh -x -a -oFallBackToRsh=no %I -l %U %H module load bbcp; bbcp" {} ./testing_parallel/
Output -
bbcp: /source/path is a directory.
bbcp: Network is unreachable unable to connect to port [::ffff:IP]:PortNumber
bbcp: Unable to allocate more than 0 of 4 data streams.
bbcp: Accept timed out on port PortNumber
bbcp: Unable to allocate more than 0 of 4 data streams.
Usage: bbcp [Options] [Inspec] Outspec
Options: [-a [dir]] [-A] [-b [+]bf] [-B bsz] [-c [lvl]] [-C cfn] [-D] [-d path]
[-e] [-E csa] [-f] [-F] [-g] [-h] [-i idfn] [-I slfn] [-k] [-K]
[-L opts[#logurl]] [-l logf] [-m mode] [-n] [-N nio] [-o] [-O] [-p]
[-P sec] [-r] [-R [args]] [-q qos] [-s snum] [-S srcxeq] [-T trgxeq]
[-t sec] [-v] [-V] [-u loc] [-U wsz] [-w [=]wsz] [-x rate] [-y] [-z]
[-Z pnf[:pnl]] [-4 [loc]] [-~] [-# {copy|follow|ignore}] [-$] [-#] [--]
I/Ospec: [user#][host:]file
**bbcp: Value not specified for '-x'.**
If I use the command on its own without parallel, it does work and doesnt give the 'value not specified' error. Any idea how to get past this?
Edit 1 - anyone??
Maybe try changing the order of the commands.
Instead of:
"ssh -x -a -oFallBackToRsh=no %I -l %U %H module load bbcp; bbcp"
try:
"module load bbcp; ssh -x -a -oFallBackToRsh=no %I -l %U %H bbcp"
Some versions of the bbcp are a bit buggy, and difficult to work with. Also I'm not sure if you can run more than one instances of bbcp in parallel (multiple instances of it would try to connect on the same ports; so you may need to specify port range for each instance)
In your case, I would suggest using an alternative like lftp, without the parallel command, but with the lftp's built-in --parallel[=N] option, which achieves the same thing that you are trying to get with GNU parallel.
Related
In Linux Shell script, I am trying to trace each command status (either Success or Failure) in a log file.
Below is my code snippet to log the entries in a log file.
scp $user#$host:$from $to 2>error.txt
command_run_status=$?
if [[ $command_run_status == 0 ]]; then log_message="secure copy on ("${host}") with ("${user}") is successful"; else log_message="error copying files "`cat error.txt`; fi
./logging.sh "$CURRENT_PID" "$log_message" "D"
Log file is created with an entry as below:
DEBUG 2019-06-21 10:05:35,347 BST [pid1234] [autouser]: error copying files usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file] [-l limit] [-o ssh_option] [-P port] [-S program] [[user#]host1:]file1 [...] [[user#]host2:]file2
However, I am expecting the log entry as below - error.txt file content with new line character.
DEBUG 2019-06-21 10:05:35,347 BST [pid1234] [autouser]: error copying files
usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]
[-l limit] [-o ssh_option] [-P port] [-S program]
[[user#]host1:]file1 [...] [[user#]host2:]file2
Content of error.txt is as below:
usage: scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]
[-l limit] [-o ssh_option] [-P port] [-S program]
[[user#]host1:]file1 [...] [[user#]host2:]file2
Can someone comment on the cause of displaying multi-line file content are displaying in a single line in the log file?
What changes are required on the command to print the error file content (with newline character) in the log file?
`cat error.txt`
Bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
To prevent word splitting, include the command substitution in double quotes.
echo $(printf 'a\nb\nc')
prints
a b c
while
echo "$(printf 'a\nb\nc')"
prints
a
b
c
(Prefer $(command) to old style `command`).
String="This is a string of words."
read -r -a Words <<< "$String"
I use the man read command to check, but there is no option at all. I also try to search pattern /-r. It comes up with no result. How can I find what does -r actually do?
read is a shell builtin command and you can find information about it from man bash:
read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...]
...
-r Backslash does not act as an escape character. The backslash is considered to be part of the line. In particular, a backslash-newline pair may not be used as a line continuation.
I am running into an odd situation. I am on RHEL 6.3. I am trying to use the built-in 'read' command to read the lines in the file in an array and to process it.
In my script, I am using
while IFS=" " read -r -a appInfo
do
echo "4 ${appInfo[0]}"
echo "4 ${appInfo[1]}"
echo "4 ${appInfo[2]}"
echo "4 line *{appInfo[#]}"
currentApp="${appInfo[*]}"
.....
done < details.txt
It is working fine, when I execute the script from the command line. The read command shows the below info
read: usage: read [-ers] [-a array] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...]
If I do read --version on command line, I get
Commandline version read (AT&T Research) 2006-12-19
But when I try to execute the exact same script from a Bamboo project, the read command generates an error
read: -a: unknown option
Usage: read [-ACprsv] [-d delim] [-u fd] [-t timeout] [-n nchar] [-N nchar]
[var?prompt] [var ...]
Bamboo : version read (AT&T Research) 2006-12-19
So, if the version shows the same, why do the flags show differently?
What could be the problem. How can I solve the issue, so that I can read the line into an array even when running it from a Bamboo plan?
Thanks to #John1024, I was able to fix the problem by changing the #! line to #!/bin/bash from #!/bin/sh ( which was running the default ksh shell ).
The credit of solution goes to #John1024, just putting this as an answer so that it can be marked as a solution to question.
I am attempting to execute a sqlplus command within a file. The file should be able to sudo switch into the oracle user and run the commands needed. The oracle user will need to first source a file with the parameters for the database, then be able to call a sql file from sqlplus.
The script I have is
#!/bin/bash
sudo -iu user oracle -c `
cd /tmp;
. .sourceme;
sqlplus user/password << EOF;
SPOOL logfile.lst;
#runme.sql;
spool off;
exit;
EOF;`
Whether I run the scrupt with sudo or not, it won't give an error. The only thing it will return is the usage for sudo
usage: sudo -h | -K | -k | -L | -V
usage: sudo -v [-AknS] [-p prompt]
usage: sudo -l[l] [-AknS] [-g groupname|#gid] [-p prompt] [-U username] [-u username|#uid] [-g groupname|#gid] [command]
usage: sudo [-AbEHknPS] [-r role] [-t type] [-C fd] [-g groupname|#gid] [-p prompt] [-u username|#uid] [-g groupname|#gid] [VAR=value] [-i|-s] [<command>]
usage: sudo -e [-AknS] [-r role] [-t type] [-C fd] [-g groupname|#gid] [-p prompt] [-u username|#uid] file ...
I figure that this means that my syntax is probably incorrect. However, without putting the command entirely on one line, I'm not sure how to fix the syntax. I've tried using double quotes around the command, removing the semicolon, and changing the grave symbol to use parenthesis instead. None of these produce any results. Trying it as a one liner and removing the parenthesis and grave marks gives me the usage information as well as a bunch of
./background: line 2: spool: command not found
Try using a heredoc:
#!/bin/bash
sudo -iu user oracle <<XXX
cd /tmp;
. .sourceme;
sqlplus user/password << EOF;
SPOOL logfile.lst;
#runme.sql;
spool off;
exit;
EOF;
XXX
I am trying to get processes attached with a port 7085 on SunOS. i tried following commands.
netstat -ntlp | grep 7085 didn't return anything
netstat -anop | grep 7085 tried this one also. This switches are not valid in SunOs
I am getting the following output.
#netstat -anop
netstat: illegal option -- o
usage: netstat [-anv] [-f address_family]
netstat [-n] [-f address_family] [-P protocol] [-g | -p | -s [interval [count]]]
netstat -m [-v] [interval [count]]
netstat -i [-I interface] [-an] [-f address_family] [interval [count]]
netstat -r [-anv] [-f address_family|filter]
netstat -M [-ns] [-f address_family]
netstat -D [-I interface] [-f address_family]
The version of SunOS is SunOS 5.10. I believe netstat is the only command can do this.
What is the exact switches for netstat which will give me the process id attached with port?
pfiles /proc/* 2>/dev/null | nawk '
/^[0-9]*:/ { pid=$0 }
/port: 7085$/ { printf("%s %s\n",pid,$0);}'
pfiles /proc/* is retrieving all processes file descriptors details
2>/dev/null is dropping out errors due to transient processes died in the meantime
each line starting with a number followed by a colon reports the process id and details, it is stored in the awk pid variable
when a line ends with the string port: <portnumber> (here is 7085), the corresponding pid variable is displayed.
Note: you need the required privilege(s) to get port information from processes you do not own (root has all privileges).
Have a look on lsof http://linux.about.com/library/cmd/blcmdl8_lsof.htm command.
This command describes which processes are using which file descriptors. Remember that anything on port 7085 will have its own file descriptor which you can use to trace back to the process using it.
I would try something like:
$ lsof -i :7085
Hope it can help.
I got his script from HERE . Log into solaris system. Open vi editor. Go into insert mode. Copy and paste this script. save the file and give the name PCP. Give execute permission. Run this script with -p or -P swithc. It will give an output with the PID, PROCESS Name and Port.
Make sure you need to be in ksh shell to execute it.
PCP is a script that enables administrators to see what open TCP ports are in use on a Solaris system. It maps ports to PIDs and vice versa. It accepts wildcards and will also show at a glance all open ports and their corresponding
PIDs. It is nice script gives a very fine out put. Just try it.
Example:
#pcp -p PORT_NUMBER or #pcp -P PROCESS_ID
#!/usr/bin/ksh
#
# # PCP (PID con Port)
# v1.10 08/10/2010 Sam Nelson sam # unix.ms
#
# If you have a Solaris 8, 9 or 10 box and you can't
# install lsof, try this. It maps PIDS to ports and vice versa.
# It also shows you which peers are connected on which port.
# Wildcards are accepted for -p and -P options.
#
# Many thanks Daniel Trinkle trinkle # cs.purdue.edu
# for the help, much appreciated.
#
i=0
while getopts :p:P:a opt
do
case "${opt}" in
p ) port="${OPTARG}";i=3;;
P ) pid="${OPTARG}";i=3;;
a ) all=all;i=2;;
esac
done
if [ $OPTIND != $i ]
then
echo >&2 "usage: $0 [-p PORT] [-P PID] [-a] (Wildcards OK) "
exit 1
fi
shift `expr $OPTIND - 1`
if [ "$port" ]
then
# Enter the port number, get the PID
#
port=${OPTARG}
echo "PID\tProcess Name and Port"
echo "_________________________________________________________"
for proc in `ptree -a | awk '/ptree/ {next} {print $1};'`
do
result=`pfiles $proc 2> /dev/null| egrep "port: $port$"`
if [ ! -z "$result" ]
then
program=`ps -fo comm= -p $proc`
echo "$proc\t$program\t$port\n$result"
echo "_________________________________________________________"
fi
done
elif [ "$pid" ]
then
# Enter the PID, get the port
#
pid=$OPTARG
# Print out the information
echo "PID\tProcess Name and Port"
echo "_________________________________________________________"
for proc in `ptree -a | awk '/ptree/ {next} $1 ~ /^'"$pid"'$/ {print $1};'`
do
result=`pfiles $proc 2> /dev/null| egrep port:`
if [ ! -z "$result" ]
then
program=`ps -fo comm= -p $proc`
echo "$proc\t$program\n$result"
echo "_________________________________________________________"
fi
done
elif [ $all ]
then
# Show all PIDs, Ports and Peers
#
echo "PID\tProcess Name and Port"
echo "_________________________________________________________"
for proc in `ptree -a | sort -n | awk '/ptree/ {next} {print $1};'`
do
out=`pfiles $proc 2>/dev/null| egrep "port:"`
if [ ! -z "$out" ]
then
name=`ps -fo comm= -p $proc`
echo "$proc\t$name\n$out"
echo "_________________________________________________________"
fi
done
fi
exit 0