I try to execute command:
# service jail start myjail
I debug the /etc/rc.d/jail and dump that really command is:
/usr/sbin/jail -l -U root -i -f /var/run/jail.myjail.conf -c myjail
The output is:
usage: jail [-dhilqv] [-J jid_file] [-u username] [-U username]
-[cmr] param=value ... [command=command ...]
jail [-dqv] [-f file] -[cmr] [jail]
jail [-qv] [-f file] -[rR] ['*' | jail ...]
jail [-dhilqv] [-J jid_file] [-u username] [-U username]
[-n jailname] [-s securelevel]
path hostname [ip[,...]] command ...
The file /var/run/jail.myjail.conf is autogenrated by rc jail script based on variables of previously worked jail from rc.conf
The content is:
myjail {
host.hostname = "myjail.example.com";
path = "/var/jail/myjail.root";
ip4.addr += "192.168.0.150/32";
allow.raw_sockets = 0;
exec.clean;
exec.system_user = "root";
exec.jail_user = "root";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_myjail_console.log";
mount.devfs;
allow.set_hostname = 0;
allow.sysvipc = 0;
}
What is wrong?
The problem solved by replace old style configuration variables in rc.conf by one line:
jail_myjail_conf="/var/run/jail.myjail.conf"
Related
Im running a bash command with subprocess.popen in python:
cmd = "bwa-mem2/bwa-mem2 mem -R \'#RG\\tID:2064-01\\tSM:2064-01\\tLB:2064-01\\tPL:ILLUMINA\\tPU:2064-01\' reference_genome/human_g1k_v37.fasta BHYHT7CCXY.RJ-1967-987-02.2_1.fastq BHYHT7CCXY.RJ-1967-987-02.2_2.fastq -t 14 | samtools view -bS -o dna_seq/aligned/2064-01/2064-01.6.bam -"
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
The problem is that I get returncode 0 even if the first command fails.
I have googled and found out about pipefail and it seems that this is what I should use.
However, I don't understand where to write it. I have tried:
"set -o pipefail && bwa-mem2/bwa-mem2 mem -R \'#RG\\tID:2064-01\\tSM:2064-01\\tLB:2064-01\\tPL:ILLUMINA\\tPU:2064-01\' reference_genome/human_g1k_v37.fasta BHYHT7CCXY.RJ-1967-987-02.2_1.fastq BHYHT7CCXY.RJ-1967-987-02.2_2.fastq -t 14 | samtools view -bS -o dna_seq/aligned/2064-01/2064-01.6.bam -"
which gives: /bin/sh: 1: set: Illegal option -o pipefail
any ideas how I should incorporate this?
Edit:
I'm not sure if it is correct to edit my answer when responding to an answer? there was not enough characters to respond in a comment:/
Anyway,
I tried your second approach without shell=True #Charles Duffy.
(cmd_1 and cmd_2 are equal to what you wrote in your solution)
This is the code I use:
try:
p1 = Popen(shlex.split(cmd_1), stdout=PIPE)
p2 = Popen(shlex.split(cmd_2), stdin=p1.stdout, stdout=PIPE, stderr=STDOUT, text=True)
p1.stdout.close()
output, error = p2.communicate()
p1.wait()
rc_1 = p1.poll()
rc_2 = p2.poll()
print("rc_1:", rc_1)
print("rc_2:", rc_2)
if rc_1 == 0 and rc_2 == 0:
self.log_to_file("DEBUG", "# Process ended with returncode = 0")
if text: self.log_to_file("INFO", f"{text} succesfully
else:
print("Raise exception")
raise Exception(f"stdout: {output} stderr: {error}")
except Exception as e:
print(f"Error: {e} in misc.run_command()")
self.log_to_file("ERROR", f"# Process ended with returncode != 0, {e}")
this is the result i get when deliberately causing an error by renaming one file:
[E::main_mem] failed to open file `/home/jonas/BASE/dna_seq/reads/2064-01/test_BHYHT7CCXY.RJ-1967-987-02.2_2.fastq.gz'.
free(): double free detected in tcache 2
rc_1: -6
rc_2: 0
Raise exception
Error: stdout: stderr: None in misc.run_command()
ERROR: # Process ended with returncode != 0, stdout: stderr: None
It seems to capture the faulty returncode.
But why is stdout empty and stderr= None?
How can I capture the output to have it logged to a logger both when the process is successful and when it fails?
First, With A Shell
Instead of letting shell=True specify sh by default, specify bash explicitly to ensure that pipefail is an available feature:
shell_script = r'''
set -o pipefail || exit
bwa-mem2/bwa-mem2 mem \
-R '#RG\tID:2064-01\tSM:2064-01\tLB:2064-01\tPL:ILLUMINA\tPU:2064-01' \
reference_genome/human_g1k_v37.fasta \
BHYHT7CCXY.RJ-1967-987-02.2_1.fastq \
BHYHT7CCXY.RJ-1967-987-02.2_2.fastq \
-t 14 \
| samtools view -bS \
-o dna_seq/aligned/2064-01/2064-01.6.bam -
'''
process = subprocess.Popen(["bash", "-c", shell_script],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
This works, but it's not the best available option.
Second, With No Shell At All
p1 = subprocess.Popen(
['bwa-mem2/bwa-mem2', 'mem',
'-R', r'#RG\tID:2064-01\tSM:2064-01\tLB:2064-01\tPL:ILLUMINA\tPU:2064-01',
'reference_genome/human_g1k_v37.fasta',
'BHYHT7CCXY.RJ-1967-987-02.2_1.fastq',
'BHYHT7CCXY.RJ-1967-987-02.2_2.fastq', '-t', '14'],
stdout=subprocess.PIPE)
p2 = subprocess.Popen(
['samtools', 'view', '-bS',
'-o', 'dna_seq/aligned/2064-01/2064-01.6.bam', '-'],
stdin=p1.stdout,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
p1.stdout.close()
output, _ = p2.communicate() # let p2 finish running
p1.wait() # ensure p1 has properly exited
print(f'bwa-mem2 exited with status {p1.returncode}')
print(f'samtools exited with status {p2.returncode}')
...which lets you check p1.returncode and p2.returncode separately.
How do I print the value of foo to the console?
nicholas#mordor:~/chimp$
nicholas#mordor:~/chimp$ VARNAME="foo"
nicholas#mordor:~/chimp$ export foo="bar"
nicholas#mordor:~/chimp$
nicholas#mordor:~/chimp$ echo $foo
bar
nicholas#mordor:~/chimp$
nicholas#mordor:~/chimp$ pwsh
PowerShell 7.1.0
Copyright (c) Microsoft Corporation.
https://aka.ms/powershell
Type 'help' to get help.
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> Get-Variable HOME -valueOnly
/home/nicholas
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> Get-Variable foo -valueOnly
Get-Variable: Cannot find a variable with the name 'foo'.
PS /home/nicholas/chimp>
see also:
https://askubuntu.com/q/58814/847449
https://opensource.com/article/19/9/environment-variables-powershell
Certainly foo is listed below:
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> Get-ChildItem env:
Name Value
---- -----
_ /snap/bin/pwsh
CLR_ICU_VERSION_OVERRIDE 60.2
COLORTERM truecolor
DBUS_SESSION_BUS_ADDRESS unix:path=/run/user/1000/bus
DESKTOP_SESSION ubuntu
DISPLAY :1
foo bar
GDMSESSION ubuntu
GJS_DEBUG_OUTPUT stderr
GJS_DEBUG_TOPICS JS ERROR;JS LOG
GNOME_DESKTOP_SESSION_ID this-is-deprecated
GNOME_SHELL_SESSION_MODE ubuntu
GNOME_TERMINAL_SCREEN /org/gnome/Terminal/screen/e541ff8a_1fb3_477b_a685_074b316b8bcc
GNOME_TERMINAL_SERVICE :1.78
GPG_AGENT_INFO /run/user/1000/gnupg/S.gpg-agent:0:1
GRADLE_HOME /home/nicholas/.sdkman/candidates/gradle/current
GTK_MODULES gail:atk-bridge
HOME /home/nicholas
IM_CONFIG_PHASE 1
INVOCATION_ID 8c2492a5c0464f27a979cad923c16937
JAVA_HOME /home/nicholas/.sdkman/candidates/java/current
JOURNAL_STREAM 9:260051
LANG en_US.utf8
LANGUAGE en_CA:en
LC_ALL en_US.UTF-8
LC_CTYPE en_US.UTF-8
LESSCLOSE /usr/bin/lesspipe %s %s
LESSOPEN | /usr/bin/lesspipe %s
LOGNAME nicholas
LS_COLORS rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:s…
MANAGERPID 14143
OLDPWD /home/nicholas
PATH /snap/powershell/149/opt/powershell:/home/nicholas/.sdkman/candidates/java/current/bin:/home/nichol…
POWERSHELL_DISTRIBUTION_CHANN… PSSnap
PSModulePath /home/nicholas/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/snap/powershell…
PWD /home/nicholas/chimp
QT_ACCESSIBILITY 1
QT_IM_MODULE ibus
SDKMAN_CANDIDATES_API https://api.sdkman.io/2
SDKMAN_CANDIDATES_DIR /home/nicholas/.sdkman/candidates
SDKMAN_DIR /home/nicholas/.sdkman
SDKMAN_PLATFORM Linux64
SDKMAN_VERSION 5.9.1+575
SESSION_MANAGER local/mordor:#/tmp/.ICE-unix/14354,unix/mordor:/tmp/.ICE-unix/14354
SHELL /bin/bash
SHLVL 1
SNAP /snap/powershell/149
SNAP_ARCH amd64
SNAP_COMMON /var/snap/powershell/common
SNAP_CONTEXT q-DbUBQMK4V-sXTO62RcZMXvkKfoVBRGFLw7wD8IiowIRR2JVsv3
SNAP_COOKIE q-DbUBQMK4V-sXTO62RcZMXvkKfoVBRGFLw7wD8IiowIRR2JVsv3
SNAP_DATA /var/snap/powershell/149
SNAP_INSTANCE_KEY
SNAP_INSTANCE_NAME powershell
SNAP_LIBRARY_PATH /var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void
SNAP_NAME powershell
SNAP_REAL_HOME /home/nicholas
SNAP_REEXEC
SNAP_REVISION 149
SNAP_USER_COMMON /home/nicholas/snap/powershell/common
SNAP_USER_DATA /home/nicholas/snap/powershell/149
SNAP_VERSION 7.1.0
SSH_AGENT_PID 14316
SSH_AUTH_SOCK /run/user/1000/keyring/ssh
TERM xterm-256color
USER nicholas
USERNAME nicholas
VTE_VERSION 6003
WINDOWPATH 4
XAUTHORITY /run/user/1000/gdm/Xauthority
XDG_CONFIG_DIRS /etc/xdg/xdg-ubuntu:/etc/xdg
XDG_CURRENT_DESKTOP ubuntu:GNOME
XDG_DATA_DIRS /usr/share/ubuntu:/home/nicholas/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:…
XDG_MENU_PREFIX gnome-
XDG_RUNTIME_DIR /run/user/1000
XDG_SESSION_CLASS user
XDG_SESSION_DESKTOP ubuntu
XDG_SESSION_TYPE x11
XMODIFIERS #im=ibus
PS /home/nicholas/chimp>
just not sure how to get that specific value:
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> Get-ChildItem Env:foo
Name Value
---- -----
foo bar
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> Get-ChildItem Env:foo -valueOnly
Get-ChildItem: A parameter cannot be found that matches parameter name 'valueOnly'.
PS /home/nicholas/chimp>
not sure why there's no valueOnly:
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> Get-Variable HOME -valueOnly
/home/nicholas
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> Get-Variable env:HOME -valueOnly
Get-Variable: Cannot find a variable with the name 'env:HOME'.
PS /home/nicholas/chimp>
where HOME has similar behavior.
works:
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> $f=Get-ChildItem Env:foo
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> $f.Name
foo
PS /home/nicholas/chimp>
PS /home/nicholas/chimp> $f.Value
bar
PS /home/nicholas/chimp>
but perhaps a bit awkward
Set Environment Variable Like So:
$env:TESTENVVAR = 'TEST VALUE HERE'
List all Environment Variables Like So:
Get-ChildItem env:
Value of a specific environment variable can be retrieved like so:
(Get-ChildItem Env:TESTENVVAR).Value
This question already has answers here:
Python Subprocess: Unable to Escape Quotes
(2 answers)
Closed last year.
I need to quote part of the rsync line that subprocess.run uses that contains the ssh parameters, unfortunately nothing I have tried has worked so far.
Can someone please advise me on the correct way to quote the ssh parameters, so that it will run under rsync.
At first I had a list of lists that got passed to subprocess.run, that fails with:
Traceback (most recent call last):
File "./tmp.py", line 20, in <module>
process = subprocess.run(rsync_cmd, stderr=subprocess.PIPE)
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not list
Flatten it to an ordinary list:
Unexpected remote arg: example.com:/var/log/maillog
rsync error: syntax or usage error (code 1) at main.c(1361) [sender=3.1.2]
Which makes sense, as part of the command line for rsync needs to be quoted.
So I try to quote it:
rsync: Failed to exec /usr/bin/ssh -F /home/rspencer/.ssh/config -o PreferredAuthentications=publickey -o StrictHostKeyChecking=accept-new -o TCPKeepAlive=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=24 -o ConnectTimeout=30 -o ExitOnForwardFailure=yes -o ControlMaster=autoask -o ControlPath=/run/user/1000/foo-ssh-master-%C -l root -p 234 -o Compression=yes: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(85) [Receiver=3.1.2]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in IPC code (code 14) at io.c(235) [Receiver=3.1.2]
Which is due, I expect, to it being a string instead of a list. Although I'm guessing and that does not make complete sense to me.
Summarized code of my last attempt:
#!/usr/bin/python3
import subprocess
ssh_args = [
"-F",
"/home/rspencer/.ssh/config",
"-o",
"PreferredAuthentications=publickey",
"-o",
"StrictHostKeyChecking=accept-new",
"-o",
"TCPKeepAlive=yes",
"-o",
"ServerAliveInterval=5",
"-o",
"ServerAliveCountMax=24",
"-o",
"ConnectTimeout=30",
"-o",
"ExitOnForwardFailure=yes",
"-o",
"ControlMaster=autoask",
"-o",
"ControlPath=/run/user/1000/foo-ssh-master-%C",
"-l",
"root",
"-p",
"234",
]
rsync_params = []
src = "example.com:/var/log/maillog"
dest = "."
# Build SSH command
ssh_cmd = ["/usr/bin/ssh"] + ssh_args
# Use basic compression
ssh_cmd.extend(["-o", "Compression=yes"])
ssh_cmd = " ".join(ssh_cmd)
ssh_cmd = f'"{ssh_cmd}"'
# Build rsync command
rsync_cmd = ["/usr/bin/rsync", "-vP", "-e", ssh_cmd] + rsync_params + [src, dest]
# Run rsync
process = subprocess.run(rsync_cmd, stderr=subprocess.PIPE)
if process.returncode != 0:
print(process.stderr.decode("UTF-8").strip())
What the correct command would look like on the command line:
/usr/bin/rsync -vP -e "/usr/bin/ssh -F /home/rspencer/.ssh/config -o \
PreferredAuthentications=publickey -o StrictHostKeyChecking=accept-new -o \
TCPKeepAlive=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=24 -o \
ConnectTimeout=30 -o ExitOnForwardFailure=yes -o ControlMaster=autoask \
-o ControlPath=/run/user/1000/foo-ssh-master-%C -l root -p 234 -o \
Compression=yes" example.com:/var/log/maillog .
Turns out the trick is to not try to quote it.
I removed the following line and it worked without further modification:
ssh_cmd = f'"{ssh_cmd}"'
I've read so much documentation and missed it until asking the question. Murphy.
Rereading the post "How not to quote argument in subprocess?" and finally understanding what Greg Hewgill was saying helped me. I blame lack of sleep.
"If you use quotes on the shell command line, then put the whole contents in one element of args (without the quotes). ..." - Greg Hewgill
Below is my code about using pexpect module achieve SSH logon function.
#!/usr/bin/env python
import pexpect
import sys
#use ssh to logon server
user="inteuser" #username
host="146.11.85.xxx" #host ip
password="xxxx" #password
command="ls -l" #list file on home/user directory
child = pexpect.spawn('ssh -l %s %s %s'%(user, host, command))
child.expect('password:')
child.sendline(password)
childlog = open('prompt.log',"ab") # restore prompt log to file prompt.log
__console__ = sys.stdout # make a backup of system output to console
sys.stdout = childlog # print the system output to childlog
child.expect(pexpect.EOF)
childlog.close()
sys.stdout = __console__ # back to the original state of system output
print(child.before) # print the contents before match expect function
after I execute my script
[~/Liaohaifeng]$ python3 ssh_test.py
b' \r\ntotal 69636\r\n-rw-rw-r-- 1 inteuser inteuser 949 Nov 28 02:01
01_eITK_trtest01_CrNwid.log\r\n
[~/Liaohaifeng]$ cat prompt.log
total 69412
-rw-rw-r-- 1 inteuser inteuser 949 Nov 28 02:01 01_eITK_trtest01_CrNwid.log
I think this result is not my expected. when I remove the code child.expect(pexpect.EOF) in my script, the output about print(child.before) can be correct(it should print the content before matching password)
Below is the output after I remove child.expect(pexpect.EOF)
[~/Liaohaifeng]$ python3 ssh_test.py
b"\r\n-------------------------------------------------------------------------------\r\n...
These computer resources are provided for authorized users only. For legal,
\r\n
security and cost reasons, utilization and access of resources are sxx, in\r\n
accordance with approved internal procedures, at any time if IF YOU ARE NOT AN AUTHORIZED USER; PLEASE EXIT IMMEDIATELY...\r\n "
my purpose is print out all the output to a file after executing the script,but the log file still only contains the output of listing directory. So why this happen? could you please help update my script? thank you very much.
You can use the spawn().logfile_read.
[STEP 101] # cat example.py
import pexpect, sys
child = pexpect.spawn('bash --norc')
if sys.version_info[0] <= 2:
# python2
child.logfile_read = open('/tmp/pexpect.log', 'w')
else:
# python3
fp = open('/tmp/pexpect.log', 'w')
child.logfile_read = fp.buffer
child.expect('bash-[.0-9]+[$#] ')
child.sendline('echo hello world')
child.expect('bash-[.0-9]+[$#] ')
child.sendline('exit')
child.expect(pexpect.EOF)
child.logfile_read.close()
[STEP 102] # python3 example.py
[STEP 103] # cat /tmp/pexpect.log
bash-4.4# echo hello world
hello world
bash-4.4# exit
exit
[STEP 104] #
It is a simple question, just adjust code order is OK.
#!/usr/bin/env python
import pexpect
import sys
#use ssh to logon server
user="inteuser" #username
host="146.11.85.xxx" #host ip
password="xxxx" #password
command="ls -l" #list file on home/user directory
child = pexpect.spawn('ssh -l %s %s %s'%(user, host, command))
childlog = open('prompt.log',"ab")
child.logfile = childlog
child.expect('password:')
child.sendline(password)
child.expect(pexpect.EOF)
childlog.close()
I am walking around this problem a long time - cgroups just don't want to work when reloading config file again(hangs on mount), have to reboot each time to changes take effect.
This are my steps:
(1.)Fresh start of OS.
(2.)cgsnapshot -s
# Configuration file generated by cgsnapshot
mount {
cpuset = /sys/fs/cgroup/cpuset;
cpu = /sys/fs/cgroup/cpu;
cpuacct = /sys/fs/cgroup/cpuacct;
memory = /sys/fs/cgroup/memory;
devices = /sys/fs/cgroup/devices;
freezer = /sys/fs/cgroup/freezer;
net_cls = /sys/fs/cgroup/net_cls;
blkio = /sys/fs/cgroup/blkio;
perf_event = /sys/fs/cgroup/perf_event;
}
(3.)cgclear(4.)cgsnapshot -s
# Configuration file generated by cgsnapshot
(5.)cgconfigparser -l /etc/cgconfig.conf (6.)cgsnapshot -s
mount {
cpu = /cgroup/cpu_mem_blkio;
cpuacct = /cgroup/cpu_mem_blkio;
memory = /cgroup/cpu_mem_blkio;
blkio = /cgroup/cpu_mem_blkio;
}
group hello1 {
...
group hello2 {
...
(7.)bash script /etc/rc.d/rc.cgred start
Now everything is working, but when i do this (the same config):
(8.)cgclear
(9.)cgconfigparser -l /etc/cgconfig.conf
It hangs forever, when i use strace it stops at:
mount("cgroup", "/cgroup/cpu_mem_blkio", "cgroup", 0,
"cpu,cpuacct,blkio,memory") = ? ERESTARTNOINTR (To be restarted)
Could someone point me whats wrong?
How i can i add new group, without rebooting?
Is this normal behavior of cgroups?
I even tried adding this patch from here:
--- a/kernel/cgroup.c
+++ b/kernel/cgroup.c
<at> <at> -1909,7 +1909,7 <at> <at> static void cgroup_kill_sb(struct super_block *sb)
*
* And don't kill the default root.
*/
- if (css_has_online_children(&root->cgrp.self) ||
+ if (!list_empty(&root->cgrp.self.children) ||
root == &cgrp_dfl_root)
cgroup_put(&root->cgrp);
else
still testing, but looks the same.
Looks like the right way of doing it, is setting everything from the command line.
mount -t cgroup -o cpu,memory,blkio,cpuacct cpu_mem_blkio /cgroup/cpu_mem_blkio
mkdir /cgroup/cpu_mem_blkio/hello1
mkdir /cgroup/cpu_mem_blkio/hello2
echo 200 > /cgroup/cpu_mem_blkio/hello1/cpu.shares
echo 200M > /cgroup/cpu_mem_blkio/hello1/memory.limit_in_bytes
echo 400M > /cgroup/cpu_mem_blkio/hello1/memory.memsw.limit_in_bytes
echo 100 > /cgroup/cpu_mem_blkio/hello1/blkio.weight
...