I am writing a script in Fortinet device configuration, we are using in the thread but at the time we had can't save in the output file. we use 'a' and 'a+'.
Our Code
output = net_connect.send_config_from_file(config_file='Cmd_File')
print("{}: Printing output...".format(i))
Save_File = open("Config", 'a')
print('\n********************* '.format(i, ip) + '*' * 50)
Save_File.write(output)
time.sleep(3)
print(output)
Related
I am trying to create a new file recording every time this program runs and also convert those .wav files to .mp3. When I run this, it only creates a output.wav and output0.mp3 file and then when I run it again, no further files are created. Also the output0.mp3 that was converted is 0KB and cannot be played.
I do not get an error but it seems its not grabbing the output.wav properly that was originally created. I am running Python 3.7.
import os
import sounddevice as sd
from scipy.io.wavfile import write
from pydub import AudioSegment #for converting WAV to MP3
fs = 44100 # Sample rate
seconds = 3 # Duration of recording
myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)
sd.wait() # Wait until recording is finished
write('output.wav', fs, myrecording ) # Save as WAV file
#Increments file name by 1 so it writes a new file every time it runs
i = 0
while os.path.exists("output%s.wav" % i):
i += 1
# files for converting WAV to Mp3
src = ("output%s.wav" % i)
dst = ("output%s.mp3" % i)
# convert wav to mp3
sound = AudioSegment.from_mp3(src)
sound.export(dst, format="wav")
writefile = open("output%s.mp3" % i, "w")
EDIT:
Updated while loop to:
#Increments file name by 1 so it writes a new file every time it runs
i = 0
while os.path.exists("output%s.wav" % i):
# files for converting WAV to Mp3
src = ("output%s.wav" % i)
dst = ("output%s.mp3" % i)
# convert wav to mp3
sound = AudioSegment.from_mp3(src)
sound.export(dst, format="wav")
write("output%s.mp3" % i, "w")
i += 1
"create a new file recording every time this program runs " - To what I understand you just need to check for existing files and get a counter to reach +1 then the last file. Once you get that just create/convert file based on that.
I am not familiar with working of sound module, but in general below should be the code structure.
## This will create new recording file called output.wav
myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)
sd.wait() # Wait until recording is finished
write('output.wav', fs, myrecording ) # Save as WAV file
# Get the counter to reach the last file that has been created.
# For e.g. if last file generated was output5.wav then below loop will run 5 times
# and should change the value of i = 6.
i = 0
while os.path.exists("output%s.wav" % i):
i += 1
# Code for creating new file using value of 'i'
# Below code is outside of while loop and will run only once,
# as 1 file needs to be created per run of the program.
src = ("output.wav")
dst = ("output%s.mp3" % i)
# convert wav to mp3
sound = AudioSegment.from_mp3(src)
sound.export(dst, format="wav")
# Not sure if this is needed.
# Check working of sound module and what does sound.export do
writefile = open("output%s.mp3" % i, "w")
SOLUTION: Updated my while loop and changed the conversion method
i = 0
while not os.path.exists("output.wav"):
i += 1
fs = 44100 # Sample rate
seconds = 3 # Duration of recording
myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)
sd.wait() # Wait until recording is finished
write('output{0}.wav'.format(i), fs, myrecording ) # Save as WAV file
print("recording has finished")
datadir = str(Path(r"FilePathtoFolderWhereAudioFileIs"))
filetopen = datadir + "/" + 'output{0}.wav'.format(i)
sound = pydub.AudioSegment.from_wav(r"FilePathtoFolderWhereAudioFileIs""" + "\\output{0}.wav".format(i))
sound.export(r"FilePathtoFolderWhereAudioFileIs""" + "\\output{0}.mp3".format(i), format="mp3")
print("Converted wav to mp3")
time.sleep(3)
I'm using a multiprocessing pool with 80 processes on a 16GB machine. The flow is as follows:
Read objects in batches from an input file
Send the entire batch to a multiprocessing pool, and record the time taken by the pool to process the batch
Write the time recorded in step 2 above to an output file
To achieve the above, I wrote code in 2 ways:
Way 1:
with open('input_file', 'r') as input_file, open('output_file', 'a') as of:
batch = read_next_batch_of_lines()
start_time = time.time()
call_api_for_each_item_in_batch(batch)
end_time = time.time()
of.write('{}\n'.format(end_time-start_time))
Way 2:
with open('input_file', 'r') as input_file:
batch = read_next_batch_of_lines()
start_time = time.time()
call_api_for_each_item_in_batch(batch)
end_time = time.time()
with open('output_file', 'a') as of:
of.write('{}\n'.format(end_time-start_time))
In the first case, nothing is being appended to the output file despite batches being processed. I'm unable to figure out the reason for this.
Details of call_api_for_each_item_in_batch():
def call_api_for_each_item_in_batch(batch):
intervals = get_intervals(batch, pool_size) #this gives intervals. Ex. if batch size is 10 and pool size is 3, then intervals would be (0, 4, 7, 10)
pool = mp.Pool(pool_size)
arguments = list(zip(intervals, intervals[1:]))
pool.starmap(call_api, arguments)
pool.close()
def call_api(start, end):
for i in range(start, end):
item = batch[i]
call_external_api(item)
How is Way 1 different from Way 2 when a pool.close() is called in the call_api_for_each_item_in_batch itself?
Also, I used pool.close() followed by pool.join(), but faced the same issue.
I want to split the file into multiple files if file size of file_write is greater than 20MB.
In Random function, I am opening big_file.txt and removing noise using remove_noise() and writing clean line to outfile.
I am not sure how to split the file based on the size in my current implementation. Please find the code below:
(Apologies for not providing proper implementation with example because it is really complicated)
I have gone through the example at this link: Split large text file(around 50GB) into multiple files
import os
def parses(lines, my_date_list):
for line in reversed(list(lines)):
line = line.strip()
if not line:
continue
date_string = "2019-11-01" # assumption
yield date_string, line
def remove_noise(line):
""" dummy function"""
return line
def random_function(path, output, cutoff="2019-10-31"):
my_date_list = []
if os.path.exists(path):
with open(path) as f:
lines = parses(f, my_date_list)
for date, line in lines:
if cutoff <= date:
results = remove_noise(line)
output.write(results + '\n')
continue
else:
break
While writing lines to output, I need to check size. If size reached 20MB and I want to write it to second {may be output_2} and so on.
if __name__ == '__main__':
path = "./big_file.txt"
file_write = "./write_file.txt"
with open(file_write) as outfile:
random_function(path=path, output=outfile)
Question is how to properly process files in Python 3.7 multiprocessing in case when I'm crawling directories recursively.
My code is as following:
def f(directoryout,directoryoutfailed,datafile,filelist_failed,imagefile,rootpath,extension,debug):
[...] some processing
if __name__ == '__main__':
import csv
import os
debug = 0
timeout = 20
if debug == 0:
folder = '/home/debian/Desktop/environments/dedpul/files/fp/'
datafile = 'fpdata.csv' # file with results
directoryout = 'fp_out' # out directory for debugging
directoryoutfailed = 'fp_out_failed' # out directory for wrongly processed for debuggin mode
filelist = 'filelist.csv' # list of processed files
filelist_failed = 'filelist_failed.csv' # list of wrongly processed files
counter = 0
pool = Pool(processes=4)
for root, subFolders, files in os.walk(folder):
for imagefile in files:
rootpath = root+'/'
fullpath = root+'/'+imagefile
extension = os.path.splitext(imagefile)[1]
imagefilesplit = imagefile.split('.')[0]
counter += 1
print('\033[93m ## ',counter,' ## \033[0m',rootpath)
fileexist = 0
with open(filelist) as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
for row in csv_reader:
if row[0] == fullpath:
fileexist = 1
if fileexist == 1:
print(' File was processed, skipping...')
continue
with open(filelist, mode='a') as csv_file:
writer = csv.writer(csv_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
writer.writerow([fullpath])
# print(directoryout,directoryoutfailed,datafile,filelist_failed,imagefile,rootpath,extension,debug)
res = pool.apply(f, (directoryout,directoryoutfailed,datafile,filelist_failed,imagefile,rootpath,extension,debug))
pool.close()
pool.join()
1st, when I'm using pool.apply_async it uses all cores, however it doesn't process function f() correctly. With pool.apply() it works single-threading.
2nd, as you see, I'm crawling recursively list of files in folders in loop. If file was found as processed, this loop should continue. Should I do that in __ main __ function, or it should be moved to f() function? If yes, how to exchange what is during processing, which takes a few seconds per file?
3rd, function f() is independent, so if it will process image file and then it will add results to fpdata.csv file (or add name of not-well-processed file to filelist_failed.csv) and just close processing without any problems, so no real output is needed. I need just to start this function in multiprocessing.
What I am doing wrong? Should I use
with Pool(processes=4) as pool:
statement?
Before asking this query I've browsed tons of answers, but apparently it was extremely hard to find such file processing, in Python Manual as well.
Hello I'm looking for some help with a project I recently took up to help check pc connectivity in my workplace. I am fairly new to python and programming in general so a large portion of this may be wrong. I'm trying to create a simple IP logger that will ping the destination address and return a value to determine the connection state of the machine. It will take the data from a CSV file (excell.xlsx) and splice the IP address from the information provided in cell style format, then output the connection state with a simple "connected" or "disconnected" after relisting the input file. Here's what I've come up with so far:
import csv, platform, subprocess, os as sys
#Defines filepath locations (Adjustment may be required)
in_path = sys.os.path['C:\Users\User\Documents\IpLog\log1.xlsx']
out_path = sys.os.path['C:\Users\User\Documents\IpLog\ip_log.xlsx']
try:
ip = 0
reference = out_path
host = '?'
line_count = 0
with open(in_path, dialect='excel', delimeter=',', newline='') as csvfile:
for row in csvfile:
if line_count == 0:
ip_inp = csv.reader(in_path, dialect='excel') #adjust filename as needed.
ip = ip_inp[ip_inp.index('10.21')]
ip = ip[5::]
line_count += 1
for line in ip_inp:
def ping(ip):
"""
Returns True if host (str) responds to a ping request.
Remember that a host may not respond to a ping (ICMP) request even if the host name is valid.
"""
# Option for the number of packets as a function of
param = '-n' if platform.system().lower() == 'windows' else '-c'
# Building the command. Ex: "ping -c 1 google.com"
command = ['ping', param, '1', ip]
return subprocess.call(command) == 0
if subprocess.call(command) == 0:
status = subprocess.call(command)
else:
status = 1
if status == 0:
if line_count == 0:
net_state = 'Connected'
with open(out_path, delimiter=',') as csvfile2:
print(csvfile)
csv.writer(csvfile2, dialect='excel')
print(net_state)
else:
net_state = 'Disconnected'
with open(out_path, delimiter=',') as csvfile2:
print(csvfile)
csv.writer(csvfile2, dialect='excel')
print(net_state)
except IOError:
print('Error opening the input file please check file input name and file location and try again.')
finally:
raise SystemExit('Operation Completed. Results have been written to (%s)' % out_path)
Right now the error I keep running into is truncation in my global file path variables. Can anyone tell me why this may be happening?