while True:
with open('secrets.csv', 'a', encoding='utf-8', newline='') as sf:
secrets = ts_functions.get_secrets(driver)
new_secrets = []
writer_1 = csv.DictWriter(sf, keys)
not_first_run = exists('last_secret.csv')
# THIS IF STATEMENT RUNS FROM THE 2ND ITERATION ONWARDS, NOT IN THE 1ST ONE
if not_first_run:
with open('last_secret.csv', 'r', encoding='utf-8', newline='') as rlf:
reader = csv.DictReader(rlf)
for row in reader:
# THERE SHOULD ONLY BE 1 ROW BUT THE READER APPEARS EMPTY
only_secret = row
for secret in secrets:
if not_first_run:
if only_secret == secret:
break
new_secrets.append(secret)
new_secrets.reverse()
for secret in new_secrets:
writer_1.writerow(secret)
with open('last_secret.csv', 'w', encoding='utf-8', newline='') as wlf:
writer_2 = csv.DictWriter(wlf, keys)
# HERE I WRITE ONE ROW(JUST ONE) IN THE 'last_secret.csv' FILE BUT WHEN THE SECOND
# ITERATION CHECKS THE IF STATEMENT THERE IS AN ERROR BECAUSE THE READER IS EMPTY
writer_2.writerow(new_secrets[-1])
sleep(10)
My program is about web scraping but I got that part covered(I can post the related functions too if needed) but I think the problem most likely is just how I am writing the 'last_secret.csv' file. When I open the file myself in Visual Studio there is the line Ii am expecting but when the reader object reads it It appears as empty and that causes an error because I didn't declare the only_secret variable(it's None).
As I said before I was expecting the reader object to read 1 row(the only one I write onto the file) but instead It reads nothingg as if the file where empty, when I check the file for myself the line is there but the reader says otherwise. I tried changing the write mode to append but It changed nothing.
Thx in advance
Related
def main():
driver = ts_functions.start_selenium()
# A good practice to access the elements in the csv is using a dict
keys = ['age', 'numeration', 'popularity', 'country', 'num_of_comments', 'time', 'sex', 'text']
while True:
# For a file to change you gotta close it first maybe?
with open('secrets.csv', 'a', encoding='utf-8', newline='') as sf:
secrets = ts_functions.get_secrets(driver)
new_secrets = []
writer = csv.DictWriter(sf, keys)
not_first_run = exists('last_secret.csv')
if not_first_run:
with open('last_secret.csv', 'r', encoding='utf-8', newline='') as rlf:
reader = csv.DictReader(rlf)
for row in reader:
only_secret = row #There should be 1 element but it's empty
for secret in secrets:
if not_first_run:
if only_secret == secret: #<---- Error IS HERE
break
new_secrets.append(secret)
new_secrets.reverse()
for secret in new_secrets:
writer.writerow(secret)
# I use 'w' here because I only want the last message of the last while loop
# so it's ok if it erases the previous content
with open('last_secret.csv', 'w', encoding='utf-8', newline='') as wlf:
writer = csv.DictWriter(wlf, keys)
# I write the only line in the file here but then it appears empty even though
# I can see the file being created and line being written
writer.writerow(new_secrets[-1])
sleep(10)
This is my main function, the two helper functions do their job properly, but if neccessary I can share them, my program tries to scrape data from a website that uploads messages live(like twitch comments) and then store the messages(secrets) in one file(secrets.csv), then I want to use the other csv file(last_secret.csv) to identify wheter a message is new or has already been scraped(I store the newest message there and compare them to the next batch of messages, when it finds one that is the same, it stops).
The problem is that the in the second run when I try to read from the latter file the reader object is empty and an error occurs.
I checked the helper functions and they are fine, the problem is with the CSVs I changed the 'last_secret.csv' mode from 'w' to 'a' and it didn't solve anything. I checked the file's contents in the same run(the 1st one) just after writing the line in it and it also appeared empty.
This is my firt question srry for the bad formatting :p
user_input = ['email','fname','lname','password','gender','cohort','program','ID']
with open('users.csv','w') as inFile:
writer = csv.DictWriter(inFile, fieldnames=user_input)
writer.writerow({'email':email,'fname': fname ,'lname':lname , 'password':password, 'gender':gender, 'cohort':cohort, 'program':program,'ID':ID})
the issue with this is that, everytime a new user input is entered on the webpage, the former one is discarded
To append more data to a file to be opened, you should use 'a' mode for opening. In 'w' mode, the file is truncated first. See the open function documentation for more information.
# ...
user_input = ['email','fname','lname','password','gender','cohort','program','ID']
with open('users.csv', 'a') as f:
writer = csv.DictWriter(f, fieldnames=user_input)
writer.writerow({'email':email, 'fname':fname, 'lname':lname, 'password':password, 'gender':gender, 'cohort':cohort, 'program':program, 'ID':ID})
I have this script that reads a CSV and saves the second column to a list, I'm trying to get it to write the contents of the list to a new CSV. The problem is every entry should have its own row but the new file sets everything into the same row.
I've tried moving the second with open code to within the first with open and I've tried adding a for loop to the second with open but no matter what I try I don't get the right results.
Here is the code:
import csv
col_store=[]
with open('test-data.csv', 'r') as rf:
reader = csv.reader(rf)
for row in reader:
col_store.append(row[1])
with open('meow.csv', 'wt') as f:
csv_writer = csv.writer(f)
csv_writer.writerows([col_store])
In your case if you have a column of single letters/numbers then Y.R answer will work.
To have a code that works in all cases, use this.
with open('meow.csv', 'wt') as f:
csv_writer = csv.writer(f)
csv_writer.writerows(([_] for _ in col_store))
From here it is mentioned that writerows expect an an iterable of row objects. Every row object should be an iterable of strings or numbers for Writer objects
The problem is that you are using 'writerows' treating 'col_store' as a list with one item.
The simplest approach to fixing this is calling
csv_writer.writerows(col_store)
# instead of
csv_writer.writerows([col_store])
However, this will lead to a probably unwanted result - having blank lines between the lines.
To solve this, use:
with open('meow.csv', 'wt', newline='') as f:
csv_writer = csv.writer(f)
csv_writer.writerows(col_store)
For more about this, see CSV file written with Python has blank lines between each row
Note: writerows expects 'an iterable of row objects' and 'row objects must be an interable of strings or numbers'.
(https://docs.python.org/3/library/csv.html)
Therefore, in the generic case (trying to write integers for examlpe), you should use Sam's solution.
I have a csv which I'm creating from pandas data-frame.
But as soon as I append it, it throws: OSError: [Errno 95] Operation not supported
for single_date in [d for d in (start_date + timedelta(n) for n in range(day_count)) if d <= end_date]:
currentDate = datetime.strftime(single_date,"%Y-%m-%d")
#Send request for one day to the API and store it in a daily csv file
response = requests.get(endpoint+f"?startDate={currentDate}&endDate={currentDate}",headers=headers)
rawData = pd.read_csv(io.StringIO(response.content.decode('utf-8')))
outFileName = 'test1.csv'
outdir = '/dbfs/mnt/project/test2/'
if not os.path.exists(outdir):
os.mkdir(outdir)
fullname = os.path.join(outdir, outFileName)
pdf = pd.DataFrame(rawData)
if not os.path.isfile(fullname):
pdf.to_csv(fullname, header=True, index=False)
else: # else it exists so append without writing the header
with open(fullname, 'a') as f: #This part gives error... If i write 'w' as mode, its overwriting and working fine.
pdf.to_csv(f, header=False, index=False, mode='a')
I am guessing it because you opened the file in an append mode and then you are passing mode = 'a' again in your call to to_csv. Can you try simply do that?
pdf = pd.DataFrame(rawData)
if not os.path.isfile(fullname):
pdf.to_csv(fullname, header=True, index=False)
else: # else it exists so append without writing the header
pdf.to_csv(fullname, header=False, index=False, mode='a')
It didn't work out, with appending. So I created parque files and then read them as data frame.
I was having a similar issue and the root cause was Databrick Runtime > 6 does not support append or random write operation on the files which exist in DBFS. It was working fine for me until I updated my runtime from 5.5 to 6 as they suggested to do this because they were no longer supporting Runtime < 6 at that time.
I followed this workaround, read the file in code, appended the data, and overwritten it.
I have a map object x such that
list(x)
produces a list. I am presently trying:
def writethis(name, mapObject):
import csv
with open(name, 'wb') as f:
csv.writer(f, delimiter = ',').writerows(list(mapObject))
but get either get an error saying the list produced isn't a valid sequence, or, if I have another function pass writethis a map object or list in the interpreter, a blank CSV file.
I'm not very familiar with the csv module. What's going on?
First, note that you open files for CSV writing differently in Python 3. You want
with open(name, 'w', newline='') as f:
instead. Second, writerows expects a sequence of sequences, and it seems like you're only passing it a sequence. If you only have one row to write, then
def writethis(name, mapObject):
with open(name, 'w', newline='') as f:
csv.writer(f, delimiter = ',').writerow(list(mapObject))
should work.