I'm new to Python and am seeing something unexpected based on other languages
I've worked with.
This code writes to a log file.
import datetime
import time
date = datetime.datetime.today().strftime('%Y-%m-%d')
mtime = time.strftime("%H:%M:%S")
myfile = "log_file." + date + ".txt"
# fh = open(myfile, "a") # Read program won't read new records if the open
# is done outside the loop
for x in range(100):
fh = open(myfile, "a") # Read program works as expected if this open
mtime = time.strftime("%H:%M:%S")
msg = str (mtime + " This is entry number " + str(x+1) + "\n")
fh.write(msg)
time.sleep( 2 )
fh.close
This code prints out new records written to the log file
import datetime
import time
date = datetime.datetime.today().strftime('%Y-%m-%d')
myfile = "log_file." + date + ".txt"
# This reads through all the records currently in the file.
lastLine = None
with open(myfile,'r') as f:
while True:
line = f.readline()
if not line:
break
# print(line)
lastLine = line
# This prints out all of the new lines that are added to the file.
while True:
with open(myfile,'r') as f:
lines = f.readlines()
if lines[-1] != lastLine:
lastLine = lines[-1]
print(lines[-1])
time.sleep(1)
If I place the open() in the write code before the for loop the read code never
sees the new records.
If I place the open() in the write code inside the loop the read code prints out
the new lines added to the file as expected. Is this correct behavior? If so, why?
The file is running in buffered mode. The reason that it works inside of the loop is that they file is being opened repeatedly and the buffer is likely flushed. If you need writes to the file to be visible quickly in the reader, then you can disable buffering when you open the file with the buffering=0 keyword argument. This should make the new lines visible quickly in the reader. You can also explicitly call f.flush() in the writer. See the docs on open() for more details.
Related
Hi there and thank you in advance for your response! I'm very new to python so please keep that in mind as you read through this, thanks!
So I've been working on some code for a very basic game using python (just for practice) I've written out a function that opens another file, selects a variable from it and adjusts that variable by an amount or if it's a string changes it into another string, the funtion looks like this.
def ovr(file, target, change):
with open(file, "r+") as open_file:
opened = open_file.readlines()
open_file.close()
with open(file, "w+") as open_file:
position = []
for appended_list, element in enumerate(opened):
if target in element:
position.append(appended_list)
if type(change) == int:
opened[position[0]] = (str(target)) + (" = ") + (str(change)) + (str("\n"))
open_file.writelines(opened)
open_file.close()
else:
opened[position[0]] = (str(target)) + (" = ") + ("'") + (str(change)) + ("'") + (str("\n"))
open_file.writelines(opened)
open_file.close()
for loop in range(5):
ovr(file = "test.py", target = "gold", change = gold + 1)
At the end I have basic loop that should re-write my file 5 times, each time increasing the amount of gold by 1. If I write this ovr() funtion outside of the loop and just run the program over and over it works just fine increasing the number in the external file by 1 each time.
Edit: I should mention that as it stands if I run this loop the value of gold increases by 1. if I close the shell and rerun the loop it increases by 1 again becoming 2. If I change the loop to happen any number of times it only ever increases the value of gold by 1.
Edit 2: I found a truly horrific way of fixing this isssue, if anyone has a better way for the love of god please let me know, code below.
for loop in range(3):
ovr(file = "test.py", target = "gold", change = test.gold + 1)
reload(test)
sleep(1)
print(test.gold)
The sleep part is because it takes longer to rewrite the file then it does to run the full loop.
you can go for a workaround and write your new inforamtion into a file called: file1
So you can use ur working loop outside of the write file. Anfter using your Loop you can just change the content of your file by the following steps.
This is how you dont need to rewrite your loop and still can change your file content.
first step:
with open('file.text', 'r') as input_file, open('file1.txt', 'w') as output_file:
for line in input_file:
output_file.write(line)
second step:
with open('file1.tex', 'r') as input_file, open('file.tex', 'w') as output_file:
for line in input_file:
if line.strip() == '(text'+(string of old value of variable)+'text)':
output_file.write('text'+(string of new value of variable)+' ')
else:
output_file.write(line)
then you have updated your text file.
I've been learning Python and I wanted to write a script to count the number of characters in a text and calculate their relative frequencies. But first, I wanted to know the length of the file. My intention is that, while the script goes from line to line counting all the characters, it would print the current line and the total number of lines, so I could know how much it is going to take.
I executed a simple for loop to count the number of lines, and then another for loop to count the characters and put them in a dictionary. However, when I run the script with the first for loop, it stops early. It doesn't even go into the second for loop as far as I know. If I remove this loop, the rest of the code goes on fine. What is causing this?
Excuse my code. It's rudimentary, but I'm proud of it.
My code:
import string
fname = input ('Enter a file name: ')
try:
fhand = open(fname)
except:
print ('Cannot open file.')
quit()
#Problematic bit. If this part is present, the script ends abruptly.
#filelength = 0
#for lines in fhand:
# filelength = filelength + 1
counts = dict()
currentline = 1
for line in fhand:
if len(line) == 0: continue
line = line.translate(str.maketrans('','',string.punctuation))
line = line.translate(str.maketrans('','',string.digits))
line = line.translate(str.maketrans('','',string.whitespace))
line = line.translate(str.maketrans('','',""" '"’‘“” """))
line = line.lower()
index = 0
while index < len(line):
if line[index] not in counts:
counts[line[index]] = 1
else:
counts[line[index]] += 1
index += 1
print('Currently at line: ', currentline, 'of', filelength)
currentline += 1
listtosort = list()
totalcount = 0
for (char, number) in list(counts.items()):
listtosort.append((number,char))
totalcount = totalcount + number
listtosort.sort(reverse=True)
for (number, char) in listtosort:
frequency = number/totalcount*100
print ('Character: %s, count: %d, Frequency: %g' % (char, number, frequency))
It looks fine the way you are doing it, however to simulate your problem, I downloaded and saved a Guttenberg text book. It's a unicode issue. Two ways to resolve it. Open it as a binary file or add the encoding. As it's text, I'd go the utf-8 option.
I'd also suggest you code it differently, below is the basic structure that closes the file after opening it.
filename = "GutenbergBook.txt"
try:
#fhand = open(filename, 'rb')
#open read only and utf-8 encoding
fhand = open(filename, 'r', encoding = 'utf-8')
except IOError:
print("couldn't find the file")
else:
try:
for line in fhand:
#put your code here
print(line)
except:
print("Error reading the file")
finally:
fhand.close()
For the op, this is a specific occasion. However, for visitors, if your code below the for state does not execute, it is not a python built-in issue, most likely to be: an exception error handling in parent caller.
Your iteration is inside a function, which is called inside a try except block of caller, then if any error occur during the loop, it will get escaped.
This issue can be hard to find, especially when you dealing with intricate architecture.
I've read every thing I can find and tried about 20 examples from SO and google, and nothing seems to work.
This should be very simple, but I cannot get it to work. I just want to point to a folder, and replace every double quote in every file in the folder. That is it. (And I don't know Python well at all, hence my issues.) I have no doubt that some of the scripts I've tried to retask must work, but my lack of Python skill is getting in the way. This is as close as I've gotten, and I get errors. If I don't get errors it seems to do nothing. Thanks.
import glob
import csv
mypath = glob.glob('\\C:\\csv\\*.csv')
for fname in mypath:
with open(mypath, "r") as infile, open("output.csv", "w") as outfile:
reader = csv.reader(infile)
writer = csv.writer(outfile)
for row in reader:
writer.writerow(item.replace("""", "") for item in row)
You don't need to use csv-specific file opening and writing, I think that makes it more complex. How about this instead:
import os
mypath = r'\path\to\folder'
for file in os.listdir(mypath): # This will loop through every file in the folder
if '.csv' in file: # Check if it's a csv file
fpath = os.path.join(mypath, file)
fpath_out = fpath + '_output' # Create an output file with a similar name to the input file
with open(fpath) as infile
lines = infile.readlines() # Read all lines
with open(fpath_out, 'w') as outfile:
for line in lines: # One line at a time
outfile.write(line.replace('"', '')) # Remove each " and write the line
Let me know if this works, and respond with any error messages you may have.
I found the solution to this based on the original answer provided by u/Jeff. It was actually smart quotes (u'\u201d') to be exact, not straight quotes. That is why I could get nothing to work. That is a great way to spend like two days, now if you'll excuse me I have to go jump off the roof. But for posterity, here is what I used that worked. (And note - there is the left curving smart quote as well - that is u'\u201c'.
mypath = 'C:\\csv\\'
myoutputpath = 'C:\\csv\\output\\'
for file in os.listdir(mypath): # This will loop through every file in the folder
if '.csv' in file: # Check if it's a csv file
fpath = os.path.join(mypath, file)
fpath_out = os.path.join(myoutputpath, file) #+ '_output' # Create an output file with a similar name to the input file
with open(fpath) as infile:
lines = infile.readlines() # Read all lines
with open(fpath_out, 'w') as outfile:
for line in lines: # One line at a time
outfile.write(line.replace(u'\u201d', ''))# Remove each " and write the line
infile.close()
outfile.close()
I am using a microstacknode accelerometer and intend to save it into csv file.
while True:
numpy.loadtxt('foo.csv', delimiter=",")
raw = accelerometer.get_xyz(raw=True)
g = accelerometer.get_xyz()
ms = accelerometer.get_xyz_ms2()
a = numpy.asarray([[raw['x'],raw['y'],raw['z']]])
numpy.savetxt("foo.csv",a,delimiter=",",newline="\n")
However, the saving is only done on 1 line. Any help given? Still quite a noobie on python.
NumPy is not the best solution for this type of things.
This should do what you intend:
while True:
raw = accelerometer.get_xyz(raw=True)
fobj = open('foo.csv', 'a')
fobj.write('{},{},{}\n'.format(raw['x'], raw['y'], raw['z']))
fobj.close()
Here fobj = open('foo.csv', 'a') opens the file in append mode. So if the file already exists, the next writing will go to the end of file, keeping the data in the file.
Let's have look at your code. This line:
numpy.loadtxt('foo.csv', delimiter=",")
reads the whole file but doe not do anything with the at it read, because you don't assign to a variable. You would need to do something like this:
data = numpy.loadtxt('foo.csv', delimiter=",")
This line:
numpy.savetxt("foo.csv",a,delimiter=",",newline="\n")
Creates a new file with the name foo.csv overwriting the existing one. Therefore, you see only one line, the last one written.
This should do the same but dos not open and close the file all the time:
with open('foo.csv', 'a') as fobj:
while True:
raw = accelerometer.get_xyz(raw=True)
fobj.write('{},{},{}\n'.format(raw['x'], raw['y'], raw['z']))
The with open() opens the file with the promise to close it even in case of an exception. For example, if you break out of the while True loop with Ctrl-C.
Can you explain what is going on in this code? I don't seem to understand
how you can open the file and read it line by line instead of all of the sentences at the same time in a for loop. Thanks
Let's say I have these sentences in a document file:
cat:dog:mice
cat1:dog1:mice1
cat2:dog2:mice2
cat3:dog3:mice3
Here is the code:
from sys import argv
filename = input("Please enter the name of a file: ")
f = open(filename,'r')
d1ct = dict()
print("Number of times each animal visited each station:")
print("Animal Id Station 1 Station 2")
for line in f:
if '\n' == line[-1]:
line = line[:-1]
(AnimalId, Timestamp, StationId,) = line.split(':')
key = (AnimalId,StationId,)
if key not in d1ct:
d1ct[key] = 0
d1ct[key] += 1
The magic is at:
for line in f:
if '\n' == line[-1]:
line = line[:-1]
Python file objects are special in that they can be iterated over in a for loop. On each iteration, it retrieves the next line of the file. Because it includes the last character in the line, which could be a newline, it's often useful to check and remove the last character.
As Moshe wrote, open file objects can be iterated. Only, they are not of the file type in Python 3.x (as they were in Python 2.x). If the file object is opened in text mode, then the unit of iteration is one text line including the \n.
You can use line = line.rstrip() to remove the \n plus the trailing withespaces.
If you want to read the content of the file at once (into a multiline string), you can use content = f.read().
There is a minor bug in the code. The open file should always be closed. I means to use f.close() after the for loop. Or you can wrap the open to the newer with construct that will close the file for you -- I suggest to get used to the later approach.