I am creating a tool that gives an overview of hundredths of test results. This tool access a log file, checks for Pass and Fail verdicts. When it is a fail, I need to go back to previous lines of the log to capture the cause of failure.
The linecache.getline works in my workspace (Python Run via eclipse). But after I created a windows installer (.exe file) and installed the application in my computer, the linecache.getline returns nothing. Is there something I need to add into my setup.py file to fix this or is it my code issue?
Tool Code
precon:
from wx.FileDialog, access the log file
self.result_path = dlg.GetPath()
try:
with open(self.result_path, 'r') as file:
self.checkLog(self.result_path, file)
def checkLog(self, path, f):
line_no = 1
index = 0
for line in f:
n = re.search("FAIL", line, re.IGNORECASE) or re.search("PASS", line, re.IGNORECASE)
if n:
currentline = re.sub('\s+', ' ', line.rstrip())
finalresult = currentline
self.list_ctrl.InsertStringItem(index, finaltestname)
self.list_ctrl.SetStringItem(index, 1, finalresult)
if currentline == "FAIL":
fail_line1 = linecache.getline(path, int(line_no - 3)) #Get reason of failure
fail_line2 = linecache.getline(path, int(line_no - 2)) #Get reason of failure
cause = fail_line1.strip() + " " + fail_line2.strip()
self.list_ctrl.SetStringItem(index, 2, cause)
index += 1
line_no += 1
The issue was resolved by doing the get_line function from this link:
Python: linecache not working as expected?
Related
I am trying to donwload a huge zip file (~9Go zipped and ~130GO unzipped) from an FTP with python using the ftplib library but unfortunately when using the retrbinary method, it does create the file in my local diretory but it is not writing into the file. After a while the code runs, I get an timeout error. It used to work fine before, but when I tried to go deeper in the use of sockets by using this code it does not work anymore. Indeed, as the files I am trying to download are huge I want to have more control with the connection to prevent timeout error while downloading the files. I am not very familar with sockets so I may have misused it. I have been searching online but did not find any problems like this. (I tried with smaller files too for test but still have the same issues)
Here are the function that I tried but both have problems (method 1 is not writing to file, method 2 donwloads file but I can't unzip it)
import time
import socket
import ftplib
import threading
# To complete
filename = ''
local_folder = ''
ftp_folder = ''
host = ''
user = ''
mp = ''
# timeout error in method 1
def downloadFile_method_1(filename, local_folder, ftp_folder, host, user, mp):
try:
ftp = ftplib.FTP(host, user, mp, timeout=1600)
ftp.set_debuglevel(2)
except ftplib.error_perm as error:
print(error)
with open(local_folder + '/' + filename, "wb") as f:
ftp.retrbinary("RETR" + ftp_folder + '/' + filename, f.write)
# method 2 works to download zip file, but header error when unziping it
def downloadFile_method_2(filename, local_folder, ftp_folder, host, user, mp):
try:
ftp = ftplib.FTP(host, user, mp, timeout=1600)
ftp.set_debuglevel(2)
sock = ftp.transfercmd('RETR ' + ftp_folder + '/' + filename)
except ftplib.error_perm as error:
print(error)
def background():
f = open(local_folder + '/' + filename, 'wb')
while True:
block = sock.recv(1024*1024)
if not block:
break
f.write(block)
sock.close()
t = threading.Thread(target=background)
t.start()
while t.is_alive():
t.join(60)
ftp.voidcmd('NOOP')
def unzip_file(filename, local_folder):
local_filename = local_folder + '/' + filename
with ZipFile(local_filename, 'r') as zipObj:
zipObj.extractall(local_folder)
And the error I get for method 1:
ftplib.error_temp: 421 Timeout - try typing a little faster next time
And the error I get when I try to unzip after using method 2:
zipfile.BadZipFile: Bad magic number for file header
Alos, regarding this code If anyone could explain what this does concerning socketopt too would be helpful:
ftp.sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
ftp.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 75)
ftp.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60)
Thanks for your help.
When I run sqlplus in Python using subprocess I get no output when there are SQL errors, or update or insert statements returning number of rows updated or inserted. When I run select statements with no errors I do get the output.
Here is my code:
This creates a string with newlines that are then written to process.stdin.write()
def write_sql_string(process, args):
sql_commands = ''
sql_commands += "WHENEVER SQLERROR EXIT SQL.SQLCODE;\n"
sql_line = '#' + args.sql_file_name
if crs_debug:
print('DEBUG: ' + 'sys.argv', ' '.join(sys.argv))
if len(args.sql_args) > 0:
len_argv = len(args.sql_args)
sql_commands += "SET VERIFY OFF\n"
for i in range(0, len_argv):
sql_line += ' "' + args.sql_args[i] + '"'
sql_commands += sql_line + "\n"
# if prod_env:
sql_commands += "exit;\n"
if crs_debug:
print('DEBUG: ' + 'sql_line: ' + sql_line)
process.stdin.write(sql_commands)
This code executes the SQL commands
def execute_sql_file(username, dbpass, args):
db_conn_str = username + '/' + dbpass + '#' + args.dbname
# '-S', - Silent
sqlplus_cmd = ['sqlplus', '-S', '-L', db_conn_str]
if crs_debug:
print('DEBUG: ' + ' '.join(sqlplus_cmd))
process = subprocess.Popen(sqlplus_cmd,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
write_sql_string(process, args)
stdout, stderr = process.communicate()
# Get return code of sql query
stdout_lines = stdout.split("\n")
print('STDOUT')
for line in stdout_lines:
line = line.rstrip()
print(line)
stderr_lines = stderr.split("\n")
print('STDERR')
for line in stderr_lines:
line = line.rstrip()
print(line)
sqlplus_rc = process.poll()
# Check if sqlplus returned an error
if sqlplus_rc != 0:
print("FAILURE in " + script_name + " in connecting to Oracle exit code: " + str(sqlplus_rc))
print(stderr_data)
sys.exit(sqlplus_rc)
When I run I run my code for a SQL file that requires parameters, if I run it with missing parameters I get no output. If I run it with parameters I get the correct output.
Here is an example SQL file sel_dual.sql:
SELECT 'THIS IS TEXT &1 &2' FROM dual;
As an example command line:
run_sql_file.py dbname sql_file [arg1]...[argn]
If I run the script with
run_sql_file.py dbname sel_dual.py
I get no output, even though it should ask for a parameter and give other error output.
If I run the script with
run_sql_file.py dbname sel_dual.py Seth F
I get the proper output:
'THISISTEXTSETHF'
----------------------------------------------------------------------------
THIS IS TEXT Seth F
The args referred to is the result of processing args with the argparse module:
parser = argparse.ArgumentParser(description='Run a SQL file with ptional arguments using SQLPlus')
parser.add_argument('dbname', help='db (environment) name')
parser.add_argument('sql_file_name', help='sql file')
parser.add_argument('sql_args', nargs='*', help='arguments for sql file')
args = parser.parse_args()
Does anybody know what could be causing this? I've omitted the rest of the script since it basically gets command arguments and validates that the SQL file exists.
I am running sqlplus version Release 12.1.0.2.0 Production. I am running Python version 3.7.6. I am running on Linux (not sure what version). The kernel release is 4.1.12-124.28.5.el7uek.x86_64.
While trying to integrate sqlmap with my automation tool, when tried to run the command and save the output into a file, the line after which user's input is required is not getting printed on the console; It is getting printed after the arguments are passed. It is required that the console output to be printed on both the places (terminal and output file). As sqlmap requires the user's input during execution cannot use subprocess.check_output()
image
Code snippet:
[try:
cmd = cmd.rstrip()
process = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
while True:
out = process.stdout.readline()\[0:\]
if out == '' and process.poll() is not None:
break
if out:
output += out
print(out.strip())
except Exception as e:
exception_message = str(e)
output += exception_message
if 'exit status 1' not in exception_message:
self.utilities.print_color("\[!\] Error executing the command: " + cmd,'red')
output += '\r\n'
print(output)
return output][1]
Below is the sample data in input file. I need to process this file and turn it into a csv file. With some help, I was able to convert it to csv file. However not fully converted to csv since I am not able to handle \n, junk line(2nd line) and blank line(4th line). Also, i need help to filter transaction_type i.e., avoid "rewrite" transaction_type
{"transaction_type": "new", "policynum": 4994949}
44uu094u4
{"transaction_type": "renewal", "policynum": 3848848,"reason": "Impressed with \n the Service"}
{"transaction_type": "cancel", "policynum": 49494949, "cancel_table":[{"cancel_cd": "AU"}, {"cancel_cd": "AA"}]}
{"transaction_type": "rewrite", "policynum": 5634549}
Below is the code
import ast
import csv
with open('test_policy', 'r') as in_f, open('test_policy.csv', 'w') as out_f:
data = in_f.readlines()
writer = csv.DictWriter(
out_f,
fieldnames=[
'transaction_type', 'policynum', 'cancel_cd','reason'],lineterminator='\n',
extrasaction='ignore')
writer.writeheader()
for row in data:
dict_row = ast.literal_eval(row)
if 'cancel_table' in dict_row:
cancel_table = dict_row['cancel_table']
cancel_cd= []
for cancel_row in cancel_table:
cancel_cd.append(cancel_row['cancel_cd'])
dict_row['cancel_cd'] = ','.join(cancel_cd)
writer.writerow(dict_row)
Below is my output not considering the junk line,blank line and transaction type "rewrite".
transaction_type,policynum,cancel_cd,reason
new,4994949,,
renewal,3848848,,"Impressed with
the Service"
cancel,49494949,"AU,AA",
Expected output
transaction_type,policynum,cancel_cd,reason
new,4994949,,
renewal,3848848,,"Impressed with the Service"
cancel,49494949,"AU,AA",
Hmm I try to fix them but I do not know how CSV file work, but my small knoll age will suggest you to run this code before to convert the file.
txt = {"transaction_type": "renewal",
"policynum": 3848848,
"reason": "Impressed with \n the Service"}
newTxt = {}
for i,j in txt.items():
# local var (temporar)
lastX = ""
correctJ = ""
# check if in J is ascii white space "\n" and get it out
if "\n" in f"b'{j}'":
j = j.replace("\n", "")
# for grammar purpose check if
# J have at least one space
if " " in str(j):
# if yes check it closer (one by one)
for x in ([j[y:y+1] for y in range(0, len(j), 1)]):
# if 2 spaces are consecutive pass the last one
if x == " " and lastX == " ":
pass
# if not update correctJ with new values
else:
correctJ += x
# remember what was the last value checked
lastX = x
# at the end make J to be the correctJ (just in case J has not grammar errors)
j = correctJ
# add the corrections to a new dictionary
newTxt[i]=j
# show the resoult
print(f"txt = {txt}\nnewTxt = {newTxt}")
Termina:
txt = {'transaction_type': 'renewal', 'policynum': 3848848, 'reason': 'Impressed with \n the Service'}
newTxt = {'transaction_type': 'renewal', 'policynum': 3848848, 'reason': 'Impressed with the Service'}
Process finished with exit code 0
like the many other threads I've opened, I am trying to create a multi-feature instant replay system utilizing the blackmagic hyperdeck which operates over Telnet. The current feature I am trying to implement is an in-out replay which requires storing two timecode variables in the format of hh:mm:ss;ff where h=hours, m=minutes, s=seconds, and f=frames #30fps. the telnet command for this is transport info, and the response returns 9 lines of which I only want the timecode from the 7th. Any idea on how to do this, as it is way out of my league?
status: stopped
speed: 0
slot id: 1
clip id: 1
single clip: false
display timecode: 00:00:09;22
timecode: 00:00:09;22
video format: 1080i5994
loop: false
Here's ideally what I would like it to look like
import telnetlib
host = "192.168.1.13" #changes for each device
port = 9993 #specific for hyperdecks
timeout = 10
session = telnetlib.Telnet(host, port, timeout)
def In():
session.write(b"transport info \n")
line = session.read_until(b";00",.5)
print(line)
#code to take response and store given line as variable IOin
def out():
session.write(b"transport info \n")
line = session.read_until(b";00",.5)
print(line)
#code to take response and store given line as variable IOout
def IOplay():
IOtc = "playrange set: in: " + str(IOin) + " out: " + str(IOout) + " \n"
session.write( IOtc.encode() )
speed = "play: speed: " + str(Pspeed.get() ) + "\n"
session.write(speed.encode() )
For the most part here's what I got to at least partially work
TCi = 1
TCo = 1
def In():
global TCi
session.write(b"transport info \n")
by = session.read_until(b";00",.5)
print(by)
s = by.find(b"00:")
TCi = by[s:s+11]
def Out():
global TCo
session.write(b"transport info \n")
by = session.read_until(b";00",.5)
print(by)
s = by.find(b"00:")
TCo = by[s:s+11]
def IOplay():
IOtc = "playrange set: in: " + str(TCi) + " out: " + str(TCo) + " \n"
print(IOtc.encode() )
session.write(IOtc.encode() )
speed = "play: speed: 2 \n"
session.write(speed.encode() )
except that its encoding as
b"playrange set: in: b'00:00:01;11' out: b'00:00:03;10' \n"
rather than
"playrange set: in: 00:00:01;11 out: 00:00:03;10 \n"
I need to get rid of the apostrophe's and b prefix in front of the variables
Any ideas?
def get_timecode(text):
tc = ''
lines = text.split('\r\n')
for line in lines:
var, val = line.split(': ', maxsplit=1)
if var == 'timecode':
tc = val
return tc
You could choose to go directly to lines[6], without scanning,
but that would be more fragile if client got out of sync with server,
or if server's output formatting changed in a later release.
EDIT:
You wrote:
session.write(b"transport info \n")
#code to take response and store given line as variable IOin
You don't appear to be reading anything from the session.
I don't use telnetlib, but the docs suggest you'll
never obtain those nine lines of text if you don't do something like:
expect = b"foo" # some prompt string returned by server that you never described in your question
session.write(b"transport info\n")
bytes = session.read_until(expect, timeout)
text = bytes.decode()
print(text)
print('Timecode is', get_timecode(text))