When trying to run stuff like
https://github.com/asciimoo/drawille/blob/master/examples/rotating_cube.py
Using my normal user I always get weird characters like
M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM-!~OM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM-
~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM- ~IM-bM-!~GM-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#
M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M-bM- ~#M
when running using root looks normal, my terminal prints unicode characters correctly when doing stuff like
python -c 'print unichr(0x28ff)'
or
echo -e '\u285a'
and $TERM is set to xterm in both user environments, any hint would be welcome.
Thanks
Googling finally fixed the issue by reinstalling the python 3.4.3 interpreter, looks like the python curses dynamic library
$ ll /home/snebel29/.pyenv/versions/*/lib/python*/lib-dynload/*cur*
-rwxr-xr-x 1 snebel29 snebel29 42620 Jan 28 2016 /home/snebel29/.pyenv/versions/2.6.6/lib/python2.6/lib-dynload/_curses_panel.so*
-rwxr-xr-x 1 snebel29 snebel29 254035 Jan 28 2016 /home/snebel29/.pyenv/versions/2.6.6/lib/python2.6/lib-dynload/_curses.so*
-rwxr-xr-x 1 snebel29 snebel29 42520 May 16 2015 /home/snebel29/.pyenv/versions/2.7.6/lib/python2.7/lib-dynload/_curses_panel.so*
-rwxr-xr-x 1 snebel29 snebel29 254661 May 16 2015 /home/snebel29/.pyenv/versions/2.7.6/lib/python2.7/lib-dynload/_curses.so*
-rwxr-xr-x 1 snebel29 snebel29 320512 Mar 15 11:50 /home/snebel29/.pyenv/versions/3.4.3/lib/python3.4/lib-dynload/_curses.cpython-34m.so*
-rwxr-xr-x 1 snebel29 snebel29 46244 Mar 15 11:50 /home/snebel29/.pyenv/versions/3.4.3/lib/python3.4/lib-dynload/_curses_panel.cpython-34m.so*
should point to libncursesw.so.5
$ ldd /home/snebel29/.pyenv/versions/3.4.3/lib/python3.4/lib-dynload/_curses.cpython-34m.so
linux-vdso.so.1 => (0x00007fffecad1000)
libncursesw.so.5 => /lib/x86_64-linux-gnu/libncursesw.so.5 (0x00007f868e9d0000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f868e7a7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f868e589000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f868e1c4000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f868dfc0000)
/lib64/ld-linux-x86-64.so.2 (0x00007f868ee1b000)
and somehow was pointing to libncurses.so.5, see here for my 2.7.6 interpreter where this is still faling
$ ldd /home/snebel29/.pyenv/versions/2.7.6/lib/python2.7/lib-dynload/_curses.so
linux-vdso.so.1 => (0x00007ffcb3f60000)
libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007feed55be000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007feed5395000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007feed5177000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007feed4db2000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007feed4bae000)
/lib64/ld-linux-x86-64.so.2 (0x00007feed59f5000)
However I still don't get why on library works and not the other and specially why running with root account make it works...
Your "normal user" doesn't have the locale set so that ncurses sees that it supports UTF-8 encoding. To see the difference, run the locale command as your normal user, and compare it with the output when running as root.
The M, by the way, means "meta", which denotes a character with the eighth (most-significant) bit set. Back in the day when 7-bit ASCII was common, and 8-bit characters relatively new, some people decided to treat that specially. If your locale is not set (i.e., POSIX), that's essentially 7-bit ASCII.
Some of that (including the use of M- as a prefix) is mentioned in the manual page for unctrl.
Regarding the followup (which relies upon information not in the question), that's an old problem with Python which I'd thought was fixed. As I pointed out in python+ncurses: I can't display accents, Python doesn't really distinguish between ncurses and ncursesw based on their interfaces, but only their name, and that some other components may load the wrong name before you get to the curses module:
It's more complicated than that: python's loading "ncurses"
dynamically.
It doesn't matter much (to python) which one it loads, but it's
specifying the library name explicitly. At the same time, some other
packages (such as readline) are separately loading the ncurses library
- and they also specify a library name. Rather than abstracting that
stuff out to another (more easily changed) level, it's embedded in the code.
If the initialization script is changed to load "ncursesw" then python
can use ncursesw without further change. There are a few patches to the
configuration that I've seen mentioned in the bug reports to enable
python to do this. Those are patches to python of course...
(this was a topic of discussion on this newsgroup about a year ago).
Related
The returned output from pexpect.run() includes \r\n at the end of every line. Printing to the terminal using print(returnVal.decode()) correctly prints one line for each line returned. When I examine the output I see that the byte string contains \r\n. When I log that to a file I get double returns to the log file. I'm on a Mac using Python 3.7. Is there a way to set the preferred new line when writing the output? I am using pythons logging class and using the info() method to write the string. Output looks like this:
total 80
-rw-r--r-- 1 xxxx admin 1048 Nov 12 00:41 Constants.py
-rw-r--r-- 1 xxxx admin 5830 Nov 12 13:33 file1.py
-rw-r--r-- 1 xxxx admin 2255 Nov 12 00:51 file2.py
When it should look like:
total 80
-rw-r--r-- 1 xxxx admin 1048 Nov 12 00:41 Constants.py
-rw-r--r-- 1 xxxx admin 5830 Nov 12 13:33 file1.py
-rw-r--r-- 1 xxxx admin 2255 Nov 12 00:51 file2.py
Here is a simplified version of my original Logger class:
class Logger():
def __init__( self, path ):
msgFormat = '%(asctime)s.%(msecs)d\t%(message)s'
dateFormat = '%m/%d/%Y %H:%M:%S'
logging.basicConfig( format=msgFormat, datefmt=dateFormat, filename=path, level=logging.INFO )
def Log ( self, theStr ):
logging.info( str( theStr ))
The string being returned from Pexpect looks something like:
Line1\r\nLine2
Depending on how you log the output, it's advisable to format the newlines before sending to logger. However, if you must override the logging module's newline parameter for FileHandler, and as an experiment, you can do so by monkey patching its _open method as the functionality isn't available by default.
I used source code for Python version 3.8 to get _open function's definition.
import logging
def custom_open(self):
"""
Monkey patched _open function of class logging.FileHandler (Python 3.8)
"""
return open(self.baseFilename, self.mode, encoding=self.encoding, newline='')
logging.FileHandler._open = custom_open
if __name__ == "__main__":
pexpect_return = "Output\nTest"
my_log = logging.getLogger("test_logger")
my_log.setLevel(logging.INFO)
my_log.addHandler(logging.FileHandler("test.log"))
my_log.info(pexpect_return)
How it works
Python's logging module has a class FileHandler, which uses a method _open to create a file handler object to write and append to log files on disk. Its default implementation as of version 3.8 does not have the newline parameter so it uses default newlines.
Monkey patching is when you replace or update a method/function in one of your imported classes, as the program is running. This line logging.FileHandler._open = custom_open tells python to replace the _open method of the FileHandler class, with my custom_open method. Then later when I use my_log.addHandler(logging.FileHandler("test.log")), the new custom_open method is used to open the file with newline paramater.
You can further confirm that the new method is used to open the file by adding a suffix to the file name like this:
return open(self.baseFilename+"__Monkey_Patched", self.mode, encoding=self.encoding, newline='')
If you will now run that demo code, the filename will be "test.log__Monkey_Patched".
This code, however, will not replace any newline characters which you pass to the logger as part of the string to log. You need to process that beforehand.
In my Python3 program, I take a bunch of paths and do things based on what they are. When I evaluate the following symlinks (snippet):
lrwxrwxrwx 1 513 513 5 Aug 19 10:56 console -> ttyS0
lrwxrwxrwx 1 513 513 11 Aug 19 10:56 core -> /proc/kcore
lrwxrwxrwx 1 513 513 13 Aug 19 10:56 fd -> /proc/self/fd
the results are:
symlink console -> ttyS0
file core -> /proc/kcore
symlink console -> ttyS0
It evaluates core as if it were a file (vs a symlink). What is the best way for me to evaluate it as a symlink vs a file? code below
#!/usr/bin/python3
import sys
import os
from pathlib import Path
def filetype(filein):
print(filein)
if Path(filein).is_file():
return "file"
if Path(filein).is_symlink():
return "symlink"
else:
return "doesn't match anything"
if __name__ == "__main__":
file = sys.argv[1]
print(str(file))
print(filetype(file))
The result of is_file is intended to answer the question "if I open this name, will I open a file". For a symlink, the answer is "yes" if the target is a file, hence the return value.
If you want to know if the name is a symlink, ask is_symlink.
An executable seemingly can't resolve a symbol in a linked library. The relevant output of LD_DEBUG=libs shows that the correct library is loaded:
6557: /usr/lib/libcharon.so.0: error: symbol lookup error: undefined symbol: auth_class_names (fatal)
/usr/libexec/ipsec/charon: symbol lookup error: /usr/lib/libcharon.so.0: undefined symbol: auth_class_names
nm -D shows that the symbol auth_class_names is defined:
nm -D /usr/lib/libcharon.so.0|grep auth_class_names
U auth_class_names
EDIT: Outputs of ldd added:
/usr/lib# ldd /usr/lib/libstrongswan.so
libpthread.so.0 => /lib/arm-linux-gnueabi/libpthread.so.0 (0xb6ecd000)
libdl.so.2 => /lib/arm-linux-gnueabi/libdl.so.2 (0xb6ec2000)
librt.so.1 => /lib/arm-linux-gnueabi/librt.so.1 (0xb6eb3000)
libc.so.6 => /lib/arm-linux-gnueabi/libc.so.6 (0xb6d78000)
/lib/ld-linux.so.3 (0xb6f25000)
/usr/lib# ldd /usr/lib/libcharon.so
libm.so.6 => /lib/arm-linux-gnueabi/libm.so.6 (0xb6ea6000)
libpthread.so.0 => /lib/arm-linux-gnueabi/libpthread.so.0 (0xb6e86000)
libdl.so.2 => /lib/arm-linux-gnueabi/libdl.so.2 (0xb6e7b000)
libcap.so.2 => /lib/arm-linux-gnueabi/libcap.so.2 (0xb6e70000)
libc.so.6 => /lib/arm-linux-gnueabi/libc.so.6 (0xb6d35000)
/lib/ld-linux.so.3 (0xb6fa6000)
libattr.so.1 => /lib/arm-linux-gnueabi/libattr.so.1 (0xb6d27000)
# nm -D /usr/lib/libstrongswan.so|grep auth_class
00036a50 D auth_class_names
nm -D shows that the symbol auth_class_names is defined
No: it shows that auth_class_names is undefined in libcharon.so.
libstrongswan provides the auth_class symbol, but libcharon doesn't reference it.
Wrong again: libcharon.so does reference the symbol.
ldd /usr/lib/libstrongswan.so
That's not what you want. You want ldd /usr/lib/libcharon.so.
Your problem is most likely that neigher libcharon.so, nor the main executable were linked against libstrongswan.so, so when you dynamically load libcharon.so, libstrongswan.so is nowhere to be found; hence the loading fails with undefined symbol.
There are several possible solutions, ordered from more correct to more hacky:
Link libcharon.so against libstrongswan.so. Loading libcharon.so will load all of its dependencies (which will now include libstrongswan.so, and the symbol will be found).
Link charon binary against libstrongswan.so.
Dynamically load libstrongswan.so before you load libcharon.so.
LD_PRELOAD=libstrongswan.so
Actually "U" means, that symbol is undefined. What does ldd show on your libcharon.so.0? libstrongswan.so.0 is where you should find auth_class_names.
I have a list of strings. Each string of the list has same format. I would like to convert each string into a class object (if that is the best option), so I can do some analysis of the list of class object.
As an example,
I have the following list
ls_list = ['-rw-r--r-- 1 ahmed None 0 Apr 21 17:10 bar1',
'-rw-r--r-- 1 ahmed None 0 Apr 21 17:10 bar2',
'-rw-r--r-- 1 ahmed None 0 Apr 21 17:10 foo1',
'-rw-r--r-- 1 ahmed None 0 Apr 21 17:10 foo2']
I would like to convert each of the above string into a class that has Nine members (perm, etc).
You don't want to do that. Use os.listdir() and os.stat() to get the information you want.
You do it something like this:
import os
data = {}
for file in os.listdir('.'):
data[file] = os.stat(file)
This gives you the information for all the files in the current directory, as objects, as you requested. These can then be inspected, and you can use other functions to figure out the username of the userid you get, etc.
Here's one way of doing what you ask, but as others have noted, don't use this for parsing the output of ls. Also, it assumes that there are only 8 areas of whitespace in your input string separating your data. If any of the data substrings also contain whitespace, this code will fail:
class FileAttribs(object):
ORDERED_ATTRIB_NAMES = ["permissions", "links", "owner",
"groups", "size", "month", "day", "time", "name"]
def __init__(self, lsString):
for (attrib, s) in zip(self.ORDERED_ATTRIB_NAMES, lsString.split()):
setattr(self, attrib, s)
if __name__ == '__main__':
ls_list = ['-rw-r--r-- 1 ahmed None 0 Apr 21 17:10 bar1',
'-rw-r--r-- 1 ahmed None 0 Apr 21 17:10 bar2',
'-rw-r--r-- 1 ahmed None 0 Apr 21 17:10 foo1',
'-rw-r--r-- 1 ahmed None 0 Apr 21 17:10 foo2']
files = []
for s in ls_list:
files.append(FileAttribs(s))
#do stuff with files, e.g.
for f in files:
print f.permissions, f.name
I have an application I'm building that's using the NetCDF C++ library, and NetCDF is pulling in the HDF-4 libary. However, it's pulling in the wrong HDF-4 library.
Here's how my app is linked:
/apps1/intel/bin/icpc -gxx-name=/apps1/gcc-4.5.0/bin/g++ -shared -o lib/libMyCustom.so
-Llib -L/apps1/boost-1.48.0/lib -Wl,-rpath=/apps1/boost-1.48.0/lib
-L/apps1/gdal-1.8.0-jasper/lib -Wl,-rpath=/apps1/gdal-1.8.0-jasper/lib
-L/new_apps1/hdf4/lib -Wl,-rpath=/new_apps1/hdf4/lib -L/new_apps1/netcdf/lib
-Wl,-rpath=/new_apps1/netcdf/lib -lboost_system -lboost_serialization
-lboost_date_time -lboost_thread -lgdal -ldf -lmfhdf -lnetcdf_c++
MyProj/obj/ProjUtility.o MyProj/obj/ProjMetadataException.o
MyProj/obj/ProjTimestampUtil.o
I have set my LD_LIBRARY_PATH very short:
LD_LIBRARY_PATH=/new_apps1/hdf4/lib:/new_apps1/hdf5/lib:
/apps1/intel/composerxe/lib/intel64:/apps1/gcc-4.5.0/lib64:/apps1/gcc-4.5.0/lib
And here is an excerpt from the ldd -v output:
libdf.so.0 => /new_apps1/hdf4/lib/libdf.so.0 (0x00002af5baabc000)
libmfhdf.so.0 => /new_apps1/hdf4/lib/libmfhdf.so.0 (0x00002af5bad61000)
libnetcdf_c++.so.5 => /new_apps1/netcdf/lib/libnetcdf_c++.so.5 (0x00002af5baf85000)
libhdf5.so.6 => /new_apps1/hdf5/lib/libhdf5.so.6 (0x00002af5bd1e7000)
libgif.so.4 => /usr/lib64/libgif.so.4 (0x0000003a6bc00000)
libpng12.so.0 => /usr/lib64/libpng12.so.0 (0x0000003a71000000)
libnetcdf.so.6 => /new_apps1/netcdf/lib/libnetcdf.so.6 (0x00002af5bd682000)
libhdf5_hl.so.6 => /new_apps1/hdf5/lib/libhdf5_hl.so.6 (0x00002af5be272000)
/new_apps1/hdf4/lib/libdf.so.0:
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/new_apps1/hdf4/lib/libmfhdf.so.0:
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/new_apps1/netcdf/lib/libnetcdf_c++.so.5:
libgcc_s.so.1 (GCC_3.0) => /apps1/gcc-4.5.0/lib64/libgcc_s.so.1
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libstdc++.so.6 (CXXABI_1.3) => /apps1/gcc-4.5.0/lib64/libstdc++.so.6
libstdc++.so.6 (GLIBCXX_3.4) => /apps1/gcc-4.5.0/lib64/libstdc++.so.6
/new_apps1/hdf5/lib/libhdf5.so.6:
libm.so.6 (GLIBC_2.2.5) => /lib64/libm.so.6
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/new_apps1/netcdf/lib/libnetcdf.so.6:
libm.so.6 (GLIBC_2.2.5) => /lib64/libm.so.6
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/new_apps1/hdf5/lib/libhdf5_hl.so.6:
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
So far, everything in the LD_LIBRARY_PATH, rpath, and ldd indicate that it's pointing to the HDF that I want to reference (/new_apps1/hdf4/lib/libmfhdf.so.0). But when I run, Valgrind is telling me that it's dying in the OLD HDF-4 library (which is probably why it's segfaulting), instead of the HDF-4 library I'm attempting to link against:
Invalid read of size 4
at 0x67CF765: NC_var_shape (in /apps1/hdf-4.2.6/lib/libmfhdf.so.0.0.0)
by 0x91327CA: nc_get_NC (v1hpg.c:1113)
by 0x91303C0: l3nc__open_mp (nc.c:1096)
by 0x915B279: nc3d__open_mp (dapdispatch3.c:336)
by 0x914A752: nc3d_open (ncdap3.c:94)
by 0x911F8A2: l4nc_open_file (nc4file.c:2338)
by 0x916A290: nc4d_open_file (ncdap4.c:122)
by 0x911CDDF: nc__open (nc4file.c:2407)
by 0x69E85F8: NcFile::NcFile(char const*, NcFile::FileMode, unsigned long*, unsigned long, NcFile::FileFormat) (netcdf.cpp:384)
by 0x710F0B8: getData(std::string const&) (ProjTimestampUtil.cc:593)
by 0x70E9BEA: (anonymous namespace)::parseOptions(int, char**) (ProjUtility.cc:190)
by 0x70EAAFB: main(int, char**) (ProjUtility.cc:243)
Address 0x1051 is not stack'd, malloc'd or (recently) free'd
Process terminating with default action of signal 11 (SIGSEGV)
Access not within mapped region at address 0x1051
at 0x67CF765: NC_var_shape (in /apps1/hdf-4.2.6/lib/libmfhdf.so.0.0.0)
by 0x91327CA: nc_get_NC (v1hpg.c:1113)
by 0x91303C0: l3nc__open_mp (nc.c:1096)
by 0x915B279: nc3d__open_mp (dapdispatch3.c:336)
by 0x914A752: nc3d_open (ncdap3.c:94)
by 0x911F8A2: l4nc_open_file (nc4file.c:2338)
by 0x916A290: nc4d_open_file (ncdap4.c:122)
by 0x911CDDF: nc__open (nc4file.c:2407)
by 0x69E85F8: NcFile::NcFile(char const*, NcFile::FileMode, unsigned long*, unsigned long, NcFile::FileFormat) (netcdf.cpp:384)
by 0x710F0B8: getData(std::string const&) (ProjTimestampUtil.cc:593)
by 0x70E9BEA: (anonymous namespace)::parseOptions(int, char**) (ProjUtility.cc:190)
by 0x70EAAFB: main(int, char**) (ProjUtility.cc:243)
Where else is my app getting path info when dynamically pulling in other libraries?
I'm not exactly sure of all the details of how -rpath and LD_LIBRARY_PATH work, and their precedence, but I did find some useful environment variables:
LD_DEBUG=all - This env variable turns on verbose dynamic linker debugging. Now doing an ldd on your app will spew output about the details of how all its dependencies find their dependencies.
LD_DEBUG_OUTPUT=<filename_prefix> - Used in conjunction with LD_DEBUG to specify output files to log the debugging info to.
The LD_DEBUG env variable helped me track down that /apps1/gdal-1.8.0-jasper/lib/libgdal.so.1 was compiled with an -rpath option that was pulling the old (wrong) versions of my libraries. It gave this helpful debug output:
search path=/pathXYZ/lib/tls/x86_64:/pathXYZ/lib/tls:/pathXYZ/lib/x86_64:
/pathABC/jasper/lib:/pathABC/hdf5/lib/tls/x86_64:/pathABC/hdf5/lib/tls:
/pathABC/hdf5/lib/x86_64:/pathABC/hdf5/lib:/pathABC/netcdf/lib/tls/x86_64:
/pathABC/netcdf/lib/tls:/pathABC/netcdf/lib/x86_64:/pathABC/netcdf/lib
(RPATH from file /apps1/gdal-1.8.0-jasper/lib/libgdal.so.1)
So the rpath of how the GDAL library was compiled seemed to be making an end-run around my LD_LIBRAR_PATH. Until I can get my lab team to rebuild libgdal correctly, I found this env var, which helped me load the "right" library versions that I wanted:
LD_PRELOAD=<path/to/libName.so> - Point this to the location of a library (or a space-separated list of libraries) that should be loaded before all others. See the ld.so man page.