I'm trying to read my nagios config data as follows:
pynag.Model.cfg_file = "path_to_nagios.cfg"
all_hosts = pynag.Model.Host.objects.all"
This returns an error
TypeError: endswith first arg must be bytes or a tuple of bytes
From what I've read so far, it seems that it's related to how files are opened in python3
Do you know how to correct this?
Thanks.
The fix was in the library code. The def parse_file() is opening files as 'rb'. The reason this is an error in Python 3 and not Python 2 is that Python 2 treats bytes as an alias or synonym for str. It doesn't make a distinction between byte strings and unicode strings as is done in Python 3.
In pynag/Parsers/init.py changed
lines = open(self.filename, 'rb').readlines()
to
lines = open(self.filename, 'r').readlines()
Related
I have a python script which calls a php function using subprocess. The output of the php script is captured as a variable for further use. When analysing the stored variable I have found that if I print the variable it has a different value than the variable itself and I can't understand why. I am bracing myself to be embarrassed so please enlighten me.
Yes the password in this fictional example is "password"
cmd = "/usr/bin/php -f /opt/hello_world.php {} {}".format(encrypted_password,of_secret)
decrypted_password = subprocess.getoutput(cmd)
print(decrypted_password)
pdb.set_trace()
in pdb here is the output
password
(Pdb) decrypted_password
'\x00p\x00a\x00s\x00s\x00w\x00o\x00r\x00d'
(Pdb) print(decrypted_password)
password
(Pdb) locals()
{'cmd': '/usr/bin/php -f /opt/hello_world.php f2e57ba074b3c3d8d4d010bcff13083dc5928107f8cfbfaa4a52fff0155eebe5 JqiRnKJBaSwEOCI', 'decrypted_password': '\x00p\x00a\x00s\x00s\x00w\x00o\x00r\x00d'}
subprocess.get_output() is returning a byte string, but Python is recognizing it as a string. So you need to first .encode() that and then decode() it:
>>> '\x00p\x00a\x00s\x00s\x00w\x00o\x00r\x00d'.encode().decode('utf_16_be')
'password'
It is encoded in UTF-16 BE, which is giving two bytes to each character hence the \x00 for a zero.
You may be able to adjust the locale or otherwise manipulate the encoding to recognize the string properly.
Python 2.7 on Ubuntu. I tried run small python script (file converter) for Python3, got error:
$ python uboot_mdb_to_image.py < input.txt > output.bin
Traceback (most recent call last):
File "uboot_mdb_to_image.py", line 29, in <module>
ascii_stdin = io.TextIOWrapper(sys.stdin.buffer, encoding='ascii', errors='strict')
AttributeError: 'file' object has no attribute 'buffer'
I suspect it's caused by syntax differences between python 3 and python 2, here is script itself:
#!/usr/bin/env python3
import sys
import io
BYTES_IN_LINE = 0x10 # Number of bytes to expect in each line
c_addr = None
hex_to_ch = {}
ascii_stdin = io.TextIOWrapper(sys.stdin.buffer, encoding='ascii', errors='strict')
for line in ascii_stdin:
line = line[:-1] # Strip the linefeed (we can't strip all white
# space here, think of a line of 0x20s)
data, ascii_data = line.split(" ", maxsplit = 1)
straddr, strdata = data.split(maxsplit = 1)
addr = int.from_bytes(bytes.fromhex(straddr[:-1]), byteorder = 'big')
if c_addr != addr - BYTES_IN_LINE:
if c_addr:
sys.exit("Unexpected c_addr in line: '%s'" % line)
c_addr = addr
data = bytes.fromhex(strdata)
if len(data) != BYTES_IN_LINE:
sys.exit("Unexpected number of bytes in line: '%s'" % line)
# Verify that the mapping from hex data to ASCII is consistent (sanity check for transmission errors)
for b, c in zip(data, ascii_data):
try:
if hex_to_ch[b] != c:
sys.exit("Inconsistency between hex data and ASCII data in line (or the lines before): '%s'" % line)
except KeyError:
hex_to_ch[b] = c
sys.stdout.buffer.write(data)
Can anyone advice how to fix this please?
It's an old question, but since I've run into a similar issue and it came up first when googling the error...
Yes, it's caused by a difference between Python 3 and 2. In Python 3, sys.stdin is wrapped in io.TextIOWrapper. In Python 2 it's a file object, which doesn't have a buffer attribute. The same goes for stderr and stdout.
In this case, the same functionality in Python 2 can be achieved using codecs standard library:
ascii_stdin = codecs.getreader("ascii")(sys.stdin, errors="strict")
However, this snippet provides an instance of codecs.StreamReader, not io.TextIOWrapper, so may be not suitable in other cases. And, unfortunately, wrapping Python 2 stdin in io.TextIOWrapper isn't trivial - see Wrap an open stream with io.TextIOWrapper for more discussion on that.
The script in question has more Python 2 incompabilities. Related to the issue in question, sys.stdout doesn't have a buffer attribute, so the last line should be
sys.stdout.write(data)
Other things I can spot:
str.split doesn't have maxsplit argument. Use line.split(" ")[:2] instead.
int doesn't have a from_bytes attribute. But int(straddr[:-1].encode('hex'), 16) seems to be equivalent.
bytes type is Python 3 only. In Python 2, it's an alias for str.
import os
def rename_files():
file_list = os.listdir(r"D:\360Downloads\test")
saved_path = os.getcwd()
os.chdir(r"D:\360Downloads\test")
for file_name in file_list:
os.rename(file_name, file_name.translate(None,"0123456789"))
rename_files()
the error message is TypeError: translate() takes exactly one argument (2 given). How can I format this so that translate() does not return an error?
Hope this helps!
os.rename(file_name,file_name.translate(str.maketrans('','','0123456789')))
or
os.rename(file_name,file_name.translate({ ord(i) : None for i in '0123456789' }))
Explanation:
I think you're using Python 3.x and syntax for Python 2.x. In Python 3.x translate() syntax is
str.translate(table)
which takes only one argument, not like Python 2.x in which translate() syntax is
str.translate(table[, deletechars])
which can takes more than one arguments.
We can make translation table easily using maketrans function.
In this case, In first two parameters, we're replacing nothing to nothing and in third parameter we're specifying which characters to be removed.
We can also make translation table manually using dictionary in which key contains ASCII of before and value contains ASCII of after character.If we want to remove some character it value must be None.
I.e. if we want to replace 'A' with 'a' and remove '1' in string then our dictionary looks like this
{65: 97, 49: None}
I am finding substring in compressed file using following python script. I am getting "TypeError: a bytes-like object is required, not 'str'". Please any one help me in fixing this.
from re import *
import re
import gzip
import sys
import io
import os
seq={}
with open(sys.argv[1],'r') as fh:
for line1 in fh:
a=line1.split("\t")
seq[a[0]]=a[1]
abcd="AGATCGGAAGAGCTCGTATGCCGTCTTCTGCTTG"
print(a[0],"\t",seq[a[0]])
count={}
with gzip.open(sys.argv[2]) as gz_file:
with io.BufferedReader(gz_file) as f:
for line in f:
for b in seq:
if abcd in line:
count[b] +=1
for c in count:
print(c,"\t",count[c])
fh.close()
gz_file.close()
f.close()
and input files are
TruSeq2_SE AGATCGGAAGAGCTCGTATGCCGTCTTCTGCTTG
the second file is compressed text file. The line "if abcd in line:" shows the error.
The "BufferedReader" class gives you bytestrings, not text strings - you can directly compare both objects in Python3 -
Since these strings just use a few ASCII characters and are not actually text, you can work all the way along with byte strings for your code.
So, whenever you "open" a file (not gzip.open), open it in binary mode (i.e.
open(sys.argv[1],'rb') instead of 'r' to open the file)
And also prefix your hardcoded string with a b so that Python uses a binary string inernally: abcd=b"AGATCGGAAGAGCTCGTATGCCGTCTTCTGCTTG" - this will avoid a similar error on your if abcd in line - though the error message should be different than the one you presented.
Alternativally, use everything as text - this can give you more methods to work with the strings (Python3's byte strigns are somewhat crippled) presentation of data when printing, and should not be much slower - in that case, instead of the changes suggested above, include an extra line to decode the line fetched from your data-file:
with io.BufferedReader(gz_file) as f:
for line in f:
line = line.decode("latin1")
for b in seq:
(Besides the error, your progam logic seens to be a bit faulty, as you don't actually use a variable string in your innermost comparison - just the fixed bcd value - but I suppose you can fix taht once you get rid of the errors)
This question already has answers here:
Convert bytes to a string
(22 answers)
Closed 2 years ago.
I have read in an XML email attachment with
bytes_string=part.get_payload(decode=False)
The payload comes in as a byte string, as my variable name suggests.
I am trying to use the recommended Python 3 approach to turn this string into a usable string that I can manipulate.
The example shows:
str(b'abc','utf-8')
How can I apply the b (bytes) keyword argument to my variable bytes_string and use the recommended approach?
The way I tried doesn't work:
str(bbytes_string, 'utf-8')
You had it nearly right in the last line. You want
str(bytes_string, 'utf-8')
because the type of bytes_string is bytes, the same as the type of b'abc'.
Call decode() on a bytes instance to get the text which it encodes.
str = bytes.decode()
How to filter (skip) non-UTF8 charachers from array?
To address this comment in #uname01's post and the OP, ignore the errors:
Code
>>> b'\x80abc'.decode("utf-8", errors="ignore")
'abc'
Details
From the docs, here are more examples using the same errors parameter:
>>> b'\x80abc'.decode("utf-8", "replace")
'\ufffdabc'
>>> b'\x80abc'.decode("utf-8", "backslashreplace")
'\\x80abc'
>>> b'\x80abc'.decode("utf-8", "strict")
Traceback (most recent call last):
...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0:
invalid start byte
The errors argument specifies the response when the input string can’t be converted according to the encoding’s rules. Legal values for this argument are 'strict' (raise a UnicodeDecodeError exception), 'replace' (use U+FFFD, REPLACEMENT CHARACTER), or 'ignore' (just leave the character out of the Unicode result).
UPDATED:
TO NOT HAVE ANY b and quotes at first and end
How to convert bytes as seen to strings, even in weird situations.
As your code may have unrecognizable characters to 'utf-8' encoding,
it's better to use just str without any additional parameters:
some_bad_bytes = b'\x02-\xdfI#)'
text = str( some_bad_bytes )[2:-1]
print(text)
Output: \x02-\xdfI
if you add 'utf-8' parameter, to these specific bytes, you should receive error.
As PYTHON 3 standard says, text would be in utf-8 now with no concern.