So I'm encountering a strange encoding error in Python3.5, I'm reading a string consisting html-data, and I'm handling the string like this :
def parseHtml(self,url):
r = requests.get(self.makeUrl())
data = r.text.encode('utf-8').decode('ascii', 'ignore')
self.soup = BeautifulSoup(data,'lxml')
The error happens when I'm trying to print the following:
def extractTable(self):
table = self.soup.findAll("table", { "class" : "messageTable" })
print(table)
I have checked my locale, and tried various variations of encode / decode as stated in previous similar posts on SO. The strangest thing (for me) is that the script works flawlessly on a different machine and on my laptop. But on my Windows Machine (using cygwin to a remote server) and on my Ubuntu install it simply wont run and gives me:
UnicodeEncodeError: 'ascii' codec can't encode character '\xa0' in position 1273: ordinal not in range(128)
Okay, so I moved the file from the remote server to my local-machine and it executed perfectly. I then checked my sys.stdout.encoding :
>>> import sys
>>> sys.stdout.encoding
'ANSI_X3.4-1968'
Clearly something was wrong, so I ended up exporting :
export PYTHONIOENCODING=utf-8
And voìla!
Related
I am trying to determine vendor + version (using python NAPALM and parallel-ssh) of network switches (Huawei VRP5/8, Cisco Catalyst and Cisco SMB (SF/SG):
admin#server:~$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from napalm import get_network_driver
>>> driver = get_network_driver('ios')
>>> device = driver('ip', 'username', 'password')
>>> device.open()
>>> print(device.get_facts())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/altepro/.local/lib/python3.8/site-packages/napalm/ios/ios.py", line 811, in get_facts
show_ver = self._send_command('show version')
File "/home/altepro/.local/lib/python3.8/site-packages/napalm/ios/ios.py", line 165, in _send_command
output = self.device.send_command(command)
File "/home/altepro/.local/lib/python3.8/site-packages/netmiko/utilities.py", line 600, in wrapper_decorator
return func(self, *args, **kwargs)
File "/home/altepro/.local/lib/python3.8/site-packages/netmiko/base_connection.py", line 1694, in send_command
raise ReadTimeout(msg)
netmiko.exceptions.ReadTimeout:
Pattern not detected: '\x1b\\[Ksg300\\-ab\\-1\\#' in output.
Things you might try to fix this:
1. Explicitly set your pattern using the expect_string argument.
2. Increase the read_timeout to a larger value.
Where sg300-ab-1 is sysname of the switch (Cisco SMB - sg300 in this case, but i have tested this on several versions and types of the SMB lineup)
Things that i have tried:
Tried several version of netmiko, napalm (And its drivers including ios-350) and parallel-ssh. Tried several fresh linux servers with fresh install of napalm and parallel-ssh.
SSH is tested using the same server and credentials and it works without any problems.
When i use parallel-ssh the device doesnt even raise exception or timeout - it just goes stuck in the command:
output = client.run_command(cmd)
hosts = ['192.168.1.50']
client = ParallelSSHClient(hosts, user='my_user', password='my_pass')
cmd = 'show version'
output = client.run_command(cmd)
for host_out in output:
for line in host_out.stdout:
print(line)
Thanks for any kind of help !
It looks like the prompt isn't getting recognized properly. I'm not very familiar with either ParallelSSHClient or napalm, but I have worked with netmiko and that looks like where the error is. Here's some steps that can possible get you closer to figuring out what's happening. I suspect it's the prompt not being read correctly from the device.
Set up debugging and a netmiko session and run a simple command
import logging
import netmiko
logging.basicConfig(level=logging.DEBUG)
session = netmiko.ConnectHandler(
host='192.168.1.50',
username='my_user',
password='my_pass',
device_type='cisco_ios')
results = session.send_command('show version')
If this fails with the same error, then it's the prompt (possibly the \x1b escape character). Try again but with a simpler expect_string, like what's expected at the end of the prompt:
session.send_command('show version', expect_string="#")
If this gets you a result, then it's something about the how the prompt is being set for this device.
To see what's being found for the prompt:
session.find_prompt()
Edit:
Based on what you're reporting, the issue seems to be with the control code \x1b\[ being included in the prompt. It's possible this can be disabled on the device itself, but I'm unfamiliar with that platform. The napalm API doesn't expose netmiko's send_command method. It should still be fixable. This solution would be a hack to make things work, nothing that I'd recommend relying on.
Establish a class that will act as your fix. This will be instantiated with the netmiko session (device.device) and will be used to replace the send_command method.
class HackyFix:
def __init__(self, session):
self.session = session
self.original_send_command = session.send_command
def send_command(self, command):
original_prompt = self.session.find_prompt()
fixed_prompt = original_prompt.replace(r"\x1b[", "")
print(
f"send_command intercepted. {original_prompt} replaced with {fixed_prompt}"
)
return self.original_send_command(command, expect_string=fixed_prompt)
Then in your existing napalm code, add this right after device.open():
hackyfix = HackyFix(device.device)
device.device.send_command = hackyfix.send_command
Now all of napalm's calls to send_command will go through your custom fix that will find the prompt and modify it before passing it to expect_string.
Last edit.
It's an ANSI Escape Code that's being thrown in by the SG300. Specifically it's the one that clears from cursor to end of line. It's also a known issue with the SG300. The good news is that someone made a napalm driver to support it. One big difference between the SG300 driver and the IOS driver is the netmiko device_type is cisco_s300. When this device_type is used, strip_ansi_escape_codes is ran against the output.
Behavior of that escape code tested in bash:
$ printf "This gets cleared\r"; code="\x1b[K"; printf "${code}This is what you see\n"
This is what you see
You can validate that setting cisco_s300 as the device_type fixes the issue:
session = netmiko.ConnectHandler(
host='192.168.1.50',
username='my_user',
password='my_pass',
device_type='cisco_s300')
results = session.send_command('show version')
This should give a result with no modification to the expect_string value. If that works and you're looking to get results sooner or later, the following is a better fix than the hacky fix above.
from napalm.ios import IOSDriver
class QuickCiscoSG300Driver(IOSDriver):
def __init__(self, hostname, username, password, timeout=60, optional_args=None):
super().__init__(hostname, username, password, timeout, optional_args)
def open(self):
device_type = "cisco_s300"
self.device = self._netmiko_open(
device_type, netmiko_optional_args=self.netmiko_optional_args
)
device = QuickCiscoSG300Driver("192.168.1.50", "my_user", "my_pass")
device.open()
device.get_facts()
Or you can get the driver (better option, unless this happens to be the driver you already tried)
I'm running a python code where we read a fixed width file, which we extracted from ftp server. the code is working on windows without any issue. but when i am running the same code in the linux ec2 instance it's giving an error saying that "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x99 in position 1819: invalid start byte". but the same code running in windows without any error.since i am not aware about the encoding type of the source file i am passing the encoding type as None. And this also working fine in windows but when we running the code in linux its giving an error saying that "encoding type None is not recognize".
i am using the codecs library to read the file and python version that i am using is 3.7.3
with codecs.open("recode.dat",encoding=None,errors='replace') as open_src:
with open("target_file.dat", 'w+',encoding=None) as open_tgt:
for src_rec in open_src:
new_rec = ''
for f_length in data_type_length:
f_length = int(f_length)
field = '"' + src_rec[:f_length].strip() + '"|'
new_rec += (field)
src_rec = src_rec[f_length:]
open_tgt.write(new_rec[:-1] + '\n')
I write a Python script on Windows and work pretty well, now I just installed "elementary Os" that is a Ubuntu based distro, but some how when I start the script it just crashed... I dont really now how to fixed it.
I let u a part of the script making problem:
memos=open(str(os.getcwd())+'\\LOG\\tres.txt','w')
menf='3)PRESION LATERAL DEL SUELO DE RELLENO\n Ka = '+str(round(Ka,2))+'\nP = '+str(round(P,2))+'\nY = '+str(round(Y,2))+'''
\nHm = '''+str(round(Hm,2))+'\nPm = '+str(round(Pm,2))+'\nYm = '+str(round(Ym,2))
memos.write(menf)
memos.close()
So the deal should be...
memos=open(str(os.getcwd())+'\\LOG\\tres.txt','w')
Because show me an error...
UnicodeEncodeError: 'ascii' codec can't encode character '\xba' in position 199: ordinal not in range(128)
Now, when I change for...
memos=open(str(os.getcwd())+'/LOG/tres.txt','w')
It got me another error...
FileNotFoundError: [Errno 2] No such file or directory: '/home/jojaror/Documentos/Scripts de Python/LOG/tres.txt'
I tryed to solve at my own, but i can't... so, if anyway could help me on this it would be helpful.
I scraped a webpage (name changed in code here) as follows:
import requests
r = requests.get('https://www.samplewebpage.com')
Then I tried to write r.text to a file as follows:
f = open ('filename', 'w')
f.write(r.text)
f.close()
I get an error as:
UnicodeEncodeError: 'charmap' codec can't encode character '\u20b9' in position 158691: character maps to <undefined>
r.encoding shows UTF-8. How to resolve the above?
Have also tried the following:
- few other random webpages and am able to run the code without any error for most.
- instead of r.text used r.content.decode('utf-8', 'ignore') but same error as above
My environment/system specifications:
Python 3.6.4
Windows 8.1 Pro, 64 bit
Default IDLE as installed from https://www.python.org.
Tried with a script in Atom as well, but same error.
Suspecting console encoding mismatch as I read in another similar problem on this forum, I reconfirmed from that the Atom console is set to UTF-8, though I believe console encoding is not the problem here, as I want to write to a file.
Thanks
Try explicitly specifying the file's encoding:
f = open ('filename', 'w', encoding='utf8')
f.write(r.text)
f.close()
I've searched a lot for this and have not yet found a definitive solution. The closest thing I've found is this:
import shutil
from os.path import join
import os
import time
import sys
minecraft_dir = ('server diectory')
world_dir = ('server world driectory')
def server_command(cmd):
os.system('screen -S -X stuff "{}\015"'.format(cmd))
on = "1"
while True:
command=input()
command=command.lower()
if on == "1":
if command==("start"):
os.chdir(minecraft_dir)
os.system('"C:\Program Files\Java\jre1.8.0_111\bin\java.exe" -Xms4G -Xmx4G -jar craftbukkit-1.10.2.jar nogui java')
print("Server started.")
on = "0"
else:
server_command(command)
When I launch this program and type 'start' the CMD flashes up and closes instantly. Instead I want the CMD to stay open with the minecraft sever running from it. I'm not sure why this happens or what the problem is, any help would be greatly appreciated.
p.s. I have edited this to my needs (such as removing a backup script that was unnecessary) but it didn't work before. The original link is: https://github.com/tschuy/minecraft-server-control
os.system will simply run the command then return to your python script with no way to further communicate with it.
On the other hand using subprocess.Popen gives you access to the process while it runs, including writing to it's .stdin which is how you send data to the server:
def server_command(cmd):
process.stdin.write(cmd+"\n") #just write the command to the input stream
process = None
executable = '"C:\Program Files\Java\jre1.8.0_111\bin\java.exe" -Xms4G -Xmx4G -jar craftbukkit-1.10.2.jar nogui java'
while True:
command=input()
command=command.lower()
if process is not None:
if command==("start"):
os.chdir(minecraft_dir)
process = subprocess.Popen(executable, stdin=subprocess.PIPE)
print("Server started.")
else:
server_command(command)
you can also pass stdout=subprocess.PIPE so you can also read it's output and stderr=subprocess.PIPE to read from it's error stream (if any)
As well instead of process.stdin.write(cmd+"\n") you could also use the file optional parameter of the print function, so this:
print(cmd, file=process.stdin)
Will write the data to process.stdin formatted in the same way that print normally does, like ending with newline for you unless passing end= to override it etc.
Both of the above answers do not work in the environment I tried them in.
I think the best way is to use RCON, not sending keys to a window.
RCON is the protocol used by games to run commands.
Many python libraries support Minecraft RCON, and the default server.properties file has an option for RCON.
We will use the python module: MCRON.
Install it. It works for windows, mac, linux.
Type:
pip install mcrcon
Lets configure your server to allow RCON.
In server.properties, find the line 'enable-rcon' and make it look like this:
enable-rcon=true
Restart and stop your server.
Find the line 'rcon.password' and set it to any password you will remember.
You can leave the port default at 25575.
Now, open your terminal and type:
mcron localhost
Or your server ip.
You will be prompted to enter the password you set.
Then you can run commands and will get the result.
But we are doing this with python, not the PYPI MCRON scripts - so do this.
from mcrcon import MCRcon as r
with r('localhost', 'insertyourpasswordhere') as mcr:
resp = mcr.command('/list')
print(resp) #there are 0/20 players online: - This will be different for you.