ParallelSSHClient - Python - Handle authentication errors - python-3.x

I have a problem which I didn't found the solution here.
I am working with SSHClient for connecting to multiple servers.
But, if there is 1 server in the list that I cannot access with my username and password (SSH), it's throwing me an exception.
I've tried to work with try and except but it didn't work.
Here is my original code:
from pssh.clients import ParallelSSHClient
import configparser
res = []
servers = ['test1', 'test2', 'test3', 'test4']
client = ParallelSSHClient(servers, user='test', password='test')
output = client.run_command('service ntpd status')
client.join(output)
for host_out in output:
for line in host_out.stdout:
if 'running' or 'Running' in line:
continue
else:
res.append(host_out.host + ' is not running')
if res:
return res
else:
return "All servers are running"
server test2 isn't accessible with my username so the script is throwing me an exception and failing the script:
pssh.exceptions.AuthenticationError: No authentication methods succeeded
How can I continue the script without the server test2 (if it is not accessible)

try something like this:
from pssh.clients import ParallelSSHClient
from pssh.config import HostConfig
host_config = [
HostConfig(user='user',
password='pwd'),
HostConfig(user='user',
password='pwd'),
HostConfig(user='user',
password='pwd')
]
servers = ['10.97.180.90', '10.97.180.99', "10.97.180.88"]
client = ParallelSSHClient(servers, host_config=host_config, num_retries=1, timeout=3)
output = client.run_command('uname -a', stop_on_errors=False, timeout=3)
for host_output in output:
try:
for line in host_output.stdout:
print(line)
exit_code = host_output.exit_code
except TypeError:
print("timeOut...")
And output looks like this:
Linux hostName-001 3.10.0-957.21.3.el7.x86_64 #1 SMP Fri Jun 14 02:54:29 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
Linux hostName-003 3.10.0-1160.15.2.el7.x86_64 #1 SMP Thu Jan 21 16:15:07 EST 2021 x86_64 x86_64 x86_64 GNU/Linux
timeOut...
More info can be found here on stackoverflow:
Python parallel-ssh run_command does not timeout when using pssh.clients

Related

How to see python script errors being run from rsyslog action

This is my action in rsyslog.conf:
module(load="omprog")
if( $msg contains "UPDOWN") then {
action(type="omprog" binary="/etc/rsyslog.d/netmiko.py" template="RSYSLOG_TraditionalFileFormat")
}
This is the python script I am working on:
pattern = re.compile('GigabitEthernet0\/\d{1,2}')
def process_line(line):
state = ''
if 'to up' in line:
state = f'UP\n'
elif 'to down' in line:
state = f'DOWN\n'
file = open("/home/blinky/python.log","a")
result = re.findall(pattern, line)
if len(result) > 0:
file.write(f'{result} - {state}')
file.close()
try:
msg = sys.stdin.readline()
file = open("/home/blinky/python.log","a")
file.write(line)
file.close()
process_line(msg)
except Exception as e:
file = open("/etc/rsyslog.d/python_error.log","a")
file.write(e)
file.close()
So the issue I have is trying to debug the python script, I can not see any of the errors it produces, as you can see I am trying to output the exception to a file but I get nothing there either. I have looked in the log file and this is the response I get from doing a shut no shut on the switch port:
Nov 20 21:50:39 10.0.0.254 1281: Nov 20 21:50:38.013: %LINK-5-CHANGED: Interface GigabitEthernet0/14, changed state to administratively down
Nov 20 21:50:39 10.0.0.254 1282: Nov 20 21:50:39.013: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/14, changed state to down
Nov 20 21:50:39 repperio rsyslogd: omprog: program '/etc/rsyslog.d/netmiko.py' (pid 2006160) terminated; will be restarted [v8.2112.0 try https://www.rsyslog.com/e/2119 ]
Nov 20 21:50:39 repperio rsyslogd: action 'action-1-omprog' suspended (module 'omprog'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]
Nov 20 21:50:40 repperio rsyslogd: action 'action-1-omprog' resumed (module 'omprog') [v8.2112.0 try https://www.rsyslog.com/e/2359 ]
Nov 20 21:50:43 10.0.0.254 1283: Nov 20 21:50:42.756: %LINK-3-UPDOWN: Interface GigabitEthernet0/14, changed state to up
Nov 20 21:50:43 10.0.0.254 1284: Nov 20 21:50:43.756: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/14, changed state to up
Nov 20 21:50:43 repperio rsyslogd: child process (pid 2006317) exited with status 1 [v8.2112.0]
Nov 20 21:50:43 repperio rsyslogd: omprog: program '/etc/rsyslog.d/netmiko.py' (pid 2006317) terminated; will be restarted [v8.2112.0 try https://www.rsyslog.com/e/2119 ]
Nov 20 21:50:43 repperio rsyslogd: action 'action-1-omprog' suspended (module 'omprog'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]
Nov 20 21:50:44 repperio rsyslogd: action 'action-1-omprog' resumed (module 'omprog') [v8.2112.0 try https://www.rsyslog.com/e/2359 ]
The script monitors the Cisco switch for interfaces going up and down and triggers the python script, this in turn will alter the configuration of the switch port using Netmiko. Without the ability to debug the python script I am scuppered, any ideas?

No MESSAGE-ID and get imap_tools work for imap.mail.yahoo.com

The question is twofold, about getting MESSAGE-ID, and using imap_tools. For an email client ("handmade") in Python I need to lessen the data amount read from the server (presently it takes 2 min to read the whole mbox folder of ~170 msg for yahoo), I believe that having MESSAGE-ID will help me.
imap_tools has IDLE command which is essential to keep the yahoo server connection alive and other features which I believe will simplify the code.
To learn about MESSAGE-ID I started with the following code (file fetch_ssl.py):
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import imaplib
import email
import os
import ssl
import conf
# Why UID==1 has no MESSAGE-ID ?
if __name__ == '__main__':
args = conf.parser.parse_args()
host, port, env_var = conf.config[args.host]
if 0 < args.verbose:
print(host, port, env_var)
with imaplib.IMAP4_SSL(host, port,
ssl_context=ssl.create_default_context()) as mbox:
user, pass_ = os.getenv('USER_NAME_EMAIL'), os.getenv(env_var)
mbox.login(user, pass_)
mbox.select()
typ, data = mbox.search(None, 'ALL')
for num in data[0].split():
typ, data = mbox.fetch(num, '(RFC822)')
msg = email.message_from_bytes(data[0][1])
print(f'num={int(num)}, MESSAGE-ID={msg["MESSAGE-ID"]}')
ans = input('Continue[Y/n]? ')
if ans.upper() in ('', 'Y'):
continue
else:
break
Where conf.py is:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import argparse
HOST = 'imap.mail.yahoo.com'
PORT = 993
config = {'gmail': ('imap.gmail.com', PORT, 'GMAIL_APP_PWD'),
'yahoo': ('imap.mail.yahoo.com', PORT, 'YAHOO_APP_PWD')}
parser = argparse.ArgumentParser(description="""\
Fetch MESSAGE-ID from imap server""")
parser.add_argument('host', choices=config)
parser.add_argument('-verbose', '-v', action='count', default=0)
fetch_ssl.py outputs:
$ python fetch_ssl.py yahoo
num=1, MESSAGE-ID=None
Continue[Y/n]?
num=2, MESSAGE-ID=<83895140.288751#communications.yahoo.com>
Continue[Y/n]? n
I'd like to understand why the message with UID == 1 has no MESSAGE-ID? Does that happen from time to time (I mean there are messages with no MESSAGE-ID)? How to handle these cases? I haven't found such cases for gmail.
Then I attempted to do similar with imap_tools (Version: 0.56.0), (file fetch_tools.py):
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os
import ssl
from imap_tools import MailBoxTls
import conf
# https://github.com/ikvk/imap_tools/blob/master/examples/tls.py
# advices
# ctx.load_cert_chain(certfile="./one.crt", keyfile="./one.key")
if __name__ == '__main__':
args = conf.parser.parse_args()
host, port, env_var = conf.config[args.host]
if 0 < args.verbose:
print(host, port, env_var)
user, pass_ = os.getenv('USER_NAME_EMAIL'), os.getenv(env_var)
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ctx.options &= ~ssl.OP_NO_SSLv3
# imaplib.abort: socket error: EOF
with MailBoxTls(host=host, port=port, ssl_context=ctx) as mbox:
mbox.login(user, pass_, 'INBOX')
for msg in mbox.fetch():
print(msg.subject, msg.date_str)
Command
$python fetch_tools.py yahoo
outputs:
Traceback (most recent call last):
File "/home/vlz/Documents/python-scripts/programming_python/Internet/Email/ymail/imap_tools_lab/fetch_tools.py", line 20, in <module>
with MailBoxTls(host=host, port=port, ssl_context=ctx) as mbox:
File "/home/vlz/Documents/.venv39/lib/python3.9/site-packages/imap_tools/mailbox.py", line 322, in __init__
super().__init__()
File "/home/vlz/Documents/.venv39/lib/python3.9/site-packages/imap_tools/mailbox.py", line 35, in __init__
self.client = self._get_mailbox_client()
File "/home/vlz/Documents/.venv39/lib/python3.9/site-packages/imap_tools/mailbox.py", line 328, in _get_mailbox_client
client = imaplib.IMAP4(self._host, self._port, self._timeout) # noqa
File "/usr/lib/python3.9/imaplib.py", line 205, in __init__
self._connect()
File "/usr/lib/python3.9/imaplib.py", line 247, in _connect
self.welcome = self._get_response()
File "/usr/lib/python3.9/imaplib.py", line 1075, in _get_response
resp = self._get_line()
File "/usr/lib/python3.9/imaplib.py", line 1185, in _get_line
raise self.abort('socket error: EOF')
imaplib.abort: socket error: EOF
Command
$ python fetch_tools.py gmail
Produces identical results. What are my mistakes?
Using Python 3.9.2, Debian GNU/Linux 11 (bullseye), imap_tools
(Version: 0.56.0)
EDIT
Headers from the message with no MESSAGE-ID
X-Apparently-To: vladimir.zolotykh#yahoo.com; Sun, 25 Oct 2015 20:54:21 +0000
Return-Path: <mail#product.communications.yahoo.com>
Received-SPF: fail (domain of product.communications.yahoo.com does not designate 216.39.62.96 as permitted sender)
...
X-Originating-IP: [216.39.62.96]
Authentication-Results: mta1029.mail.bf1.yahoo.com from=product.communications.yahoo.com; domainkeys=neutral (no sig); from=product.communications.yahoo.com; dkim=pass (ok)
Received: from 127.0.0.1 (EHLO n3-vm4.bullet.mail.gq1.yahoo.com) (216.39.62.96)
by mta1029.mail.bf1.yahoo.com with SMTPS; Sun, 25 Oct 2015 20:54:21 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=product.communications.yahoo.com; s=201402-std-mrk-prd; t=1445806460; bh=5PTgF8Jghm92xeMD5mSHp6A3eRVV70PWo1oQ15K7Tfk=; h=Date:From:Reply-To:To:Subject:From:Subject; b=D7ItgOiuLbiexJGHvORgbpRi22X+sYso6gwZKDXVca79DxMMy2R1dUtZTIg7tcft1lovVJUDw/7fC51orDltRidlfnpayeY8lT+94DRlSBwopuxgOqqR9oTTjTBZ0oEvdxUcXl/q54N2GxuBFvmg8UO0OZoCnFPpUVYo9x4arMjt/0TOW1Q5d/yjdmO7iwiued/rliP/Bsq0TaZYcb0oCAT7Q50tb1fB7wcXLYNSC1OCQ1l1LajbUqmU1LWWNse36mUUTBieO2sZT0ERFrHaCTaTNQSXKQG2AxYF7Dd/8i0Iq3xqdcS0bDpjmWE25uoKvCdtXtUbylsuQSChuLFMTw==
Received: from [216.39.60.185] by n3.bullet.mail.gq1.yahoo.com with NNFMP; 25 Oct 2015 20:54:20 -0000
Received: from [98.137.101.84] by t1.bullet.mail.gq1.yahoo.com with NNFMP; 25 Oct 2015 20:54:20 -0000
Date: 25 Oct 2015 20:54:20 +0000
Received: from [127.0.0.1] by nu-repl01.direct.gq1.yahoo.com with NNFMP; 25 Oct 2015 20:54:20 -0000
X-yahoo-newman-expires: 1445810060
From: "Yahoo Mail" <mail#product.communications.yahoo.com>
Reply-To: replies#communications.yahoo.com
To: <ME>#yahoo.com
Subject: Welcome to Yahoo! Vladimir
X-Yahoo-Newman-Property: ydirect
Content-Type: text/html
Content-Length: 25180
I skipped only X-YMailISG.
EDIT II
Of 167 messages 21 have no MESSAGE-ID header.
fetch_ssl.py takes 4m12.342s, and fetch_tools.py -- 3m41.965s
It looks simply like your email without a Message-ID legitimately does not have one; it appears the welcome email Yahoo sent you actually lacks it. Since it's a system generated email, that's not that unexpected. You'd just have to skip over it.
The second problem is that you need to use imap_tools.MailBox.
Looking at the documentation and source at the repo it appears that the relevant classes to use are:
MailBox - for a normal encrypted connection. This is what most email servers use these days, aka IMAPS (imap with SSL/TLS)
MailBoxTls - For a STARTTLS connection: this creates a plaintext connection then upgrades it later by using a STARTTLS command in the protocol. The internet has mostly gone to the "always encrypted" rather than "upgrade" paradigm, so this is not the class to use.
MailBoxUnencrypted - Standard IMAP without SSL/TLS. You should not use this on the public internet.
The naming is a bit confusing. MailBox corresponds to imaplib.IMAP4_SSL; MailBoxTls corresponds to imaplib.IMAP4, then using startls() on the resulting connection; and MailboxUnencrypted corresponds to imaplib.IMAP4 with no security applied. I imagine it's this way so the most common one (Mailbox) is a safe default.

sqlite3.OperationalError('near "(": syntax error') in Google Colab

Observing some odd behavior with SQLite 2.6, where the ROW_NUMBER() throws an error only in Google Colab (Python 3.6.9), whereas the code works fine in my local Python 3.6.9 and Python 3.9.1 instances. Can you help me debug this further?
Code
import sqlite3, sys
try:
print('Py.version : ' + (sys.version))
print('sqlite3.version : ' + (sqlite3.version))
print('sqlite3.sqlite_version : ' + (sqlite3.sqlite_version)+'\n')
conn = sqlite3.connect(':memory:')
conn.execute('''CREATE TABLE team_data(team text, total_goals integer);''')
conn.commit()
conn.execute("INSERT INTO team_data VALUES('Real Madrid', 53);")
conn.execute("INSERT INTO team_data VALUES('Barcelona', 47);")
conn.commit()
sql='''
SELECT
team,
ROW_NUMBER () OVER (
ORDER BY total_goals
) RowNum
FROM
team_data
'''
print('### DB Output ###')
cursor = conn.execute(sql)
for row in cursor:
print(row)
except Exception as e:
print('ERROR : ' + str(e))
finally:
conn.close()
Output
Google Colab (ROW_NUMBER() causes SQL to fail):
Py.version : 3.6.9 (default, Oct 8 2020, 12:12:24) [GCC 8.4.0]
sqlite3.version : 2.6.0
sqlite3.sqlite_version : 3.22.0
### DB Output ###
ERROR : near "(": syntax error
Local Python 3.6.9 (Succeeds):
Py.version : 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 14:00:49) [MSC v.1915 64 bit (AMD64)]
sqlite3.version : 2.6.0
sqlite3.sqlite_version : 3.33.0
### DB Output ###
('Barcelona', 1)
('Real Madrid', 2)
Local Python 3.9.1 (Succeeds):
Py.version : 3.9.1 (default, Dec 11 2020, 09:29:25) [MSC v.1916 64 bit (AMD64)]
sqlite3.version : 2.6.0
sqlite3.sqlite_version : 3.33.0
### DB Output ###
('Barcelona', 1)
('Real Madrid', 2)
Note: Above SQL and code is simplified for error reproduction purposes only
The query in question is a window function and support for that was added in version 3.25. You can check the library (opposed to package) version with sqlite3.sqlite_version or as #forpas shared with the query select sqlite_version().
You can upgrade your sqlite version. Use this code.
!add-apt-repository -y ppa:sergey-dryabzhinsky/packages
!apt update
!apt install sqlite3
# MENU: Runtime > Restart runtime
import sqlite3
sqlite3.sqlite_version # '3.33.0'

Python3 time.sleep() behaving erratically

L.S,
i am trying to learn python(3) and am baffled by the behavior of time.sleep()
i run raspbian buster on a rpi3b.
#!/usr/bin/python3
import bluetooth
import sys
import time
_shuttermac ="11:22:33:44:5D:6A"
while True:
# check if the device is switched on by mac address lookup
print("Checking " + time.strftime("%a, %d %b %Y %H:%M:%S - ", time.gmtime()),end ="")
result = bluetooth.lookup_name(_shuttermac, timeout=5)
#print(result)
if (result != None):
print("Device detected by bluetooth mac lookup")
else:
print("Device with MAC " + _shuttermac + " NOT detected")
time.sleep(1) # check every 5 secs if device comes online
sys.exit()
output
Checking Thu, 25 Jun 2020 11:21:02 - Device with MAC 11:22:33:44:5D:6A NOT detected
Checking Thu, 25 Jun 2020 11:21:08 - Device with MAC 11:22:33:44:5D:6A NOT detected
Checking Thu, 25 Jun 2020 11:21:14 - Device with MAC 11:22:33:44:5D:6A NOT detected
Checking Thu, 25 Jun 2020 11:21:20 - Device with MAC 11:22:33:44:5D:6A NOT detected
Checking Thu, 25 Jun 2020 11:21:26 - Device with MAC 11:22:33:44:5D:6A NOT detected
as you can see, the output statement is 6 seconds apart, not 1 as coded
where did i go wrong?
The six second interval comes because each loop of while True does two time-consuming things when the lookup fails:
bluetooth.lookup_name(_shuttermac, timeout=5) has a 5 second timeout. If it is found quickly, it should complete faster
time.sleep(1) pauses the whole Python program for 1 second
You could shorten the timeout on bluetooth.lookup_name and the sleep() duration. This may have unintended consequences in the accuracy of the lookup (I'm not very familiar with it).

Torque cannot communicate with host

I have been attempting to setup the torque scheduler for a small cluster. I followed the steps to setup the scheduler from http://docs.adaptivecomputing.com/torque/archive/3-0-2/1.2configuring_torque_on_server.php
However when i attempt
qterm -t quick
I get the following error
$ sudo qterm -t quick
Unable to communicate with Terra(192.168.1.25)
Cannot connect to specified server host 'Terra'.
qterm: could not connect to server '' (111) Connection refused
but the server starts just fine. However when I attempt to run a command that runs on multiple nodes such as
qsub -l nodes=2:ppn=4 /home/user/scripts/someScript
it prints out somethign like
7.Terra
where Terra is the name of the head node, but is also a node in the cluster. This isn't the problem. The problem is that it does not run. nor does it have any output anywhere :/
The torque server log: https://ptpb.pw/EaKo
The terra node log: https://ptpb.pw/9w5M
and the Marte log: https://ptpb.pw/o4PT
I can get it to run with a pbs script but only with one node....
#!/bin/bash
#PBS -l pmem=1gb,nodes=1:ppn=4
#PBS -m abe
cd Documents/
wc -l largeTest.csv
Here is the ouput of qstat after submitting a job
Job ID Name User Time Use S
Queue
------------------------- ---------------- --------------- -------- - -----
16.Terra testPerformance justin 0 R batch
the output of pbsnodes -a
Terra
state = free
power_state = Running
np = 4
properties = Tower
ntype = cluster
status = opsys=linux,uname=Linux Terra 4.17.14-arch1-1-ARCH #1 SMP PREEMPT Thu Aug 9 11:56:50 UTC 2018 x86_64,sessions=11525 22029,nsessions=2,nusers=1,idletime=57964,totmem=8111556kb,availmem=7539284kb,physmem=8111556kb,ncpus=4,loadave=0.00,gres=,netload=30570521372,state=free,varattr= ,cpuclock=Fixed,macaddr=e0:3f:49:44:72:20,version=6.1.1.1,rectime=1534937388,jobs=
mom_service_port = 15002
mom_manager_port = 15003
gpus = 1
Marte
state = free
power_state = Running
np = 4
properties = NFSServer
ntype = cluster
status = opsys=linux,uname=Linux Marte 4.18.1-arch1-1-ARCH #1 SMP PREEMPT Wed Aug 15 21:11:55 UTC 2018 x86_64,sessions=366 556 563,nsessions=3,nusers=2,idletime=58140,totmem=7043404kb,availmem=6703808kb,physmem=7043404kb,ncpus=4,loadave=0.02,gres=,netload=36500663511,state=free,varattr= ,cpuclock=Fixed,macaddr=c8:5b:76:4a:65:91,version=6.1.1.1,rectime=1534937359,jobs=
mom_service_port = 15002
mom_manager_port = 15003
and the /var/spool/torque/server_priv/nodes
Terra np=4 gpus=1 Tower
Marte np=4 NFSServer
Edit: Here are the most recent logs as well
Mom Log for Node: https://ptpb.pw/DhKi
Mom Log for head node: https://ptpb.pw/MTlD
and the server log: https://ptpb.pw/HPkE

Resources