I have to read this csv file into a list and I have declared type int for writing certain columns in list as integer type but this error has become a problem for me.
with open('new_toy_dataset.csv','r') as cf:
for row in cf:
toy_list.append([int(row[0]), row[1], row[2], int(row[3]), int(row[4]), row[5]])
Data Set
Error
The problem is that you are not skipping the header.
import csv
toy_list = []
ind = 0
with open('new_toy_dataset.csv','r') as cf:
reader = csv.reader(cf, delimiter=';') # whatever delimiter it is
for row in reader:
if ind == 0:
ind += 1
continue
toy_list.append([int(row[0]), row[1], row[2], int(row[3]), int(row[4]), row[5]])
Related
I have a code which iterates through the text, and tells me which is the maximum amount of times each dna STR is found. The only step missing to be able to match these values with the CSV file, is to store them into a list, BUT I AM NOT ABLE TO DO SO. When I run the code, the maximum values are printed independently for each STR sequence.
I have tried to "append" the values into a list, but I was not successful, thus, I cannot match it with the dna sequences of the CSV (large nor small).
Any help or advcise is greatly appreciated!
Here is my code, and the results I get with using "text 1" and "small csv":
`
import cs50
import sys
import csv
import os
if len(sys.argv) != 3:
print("Usage: python dna.py data.csv sequence.txt")
csv_db = sys.argv[1]
file_seq = sys.argv[2]
with open(csv_db, newline='') as csvfile: #with open(csv_db) as csv_file
csv_reader = csv.reader(csvfile, delimiter=',')
header = next(csv_reader)
i = 1
while i < len(header):
STR = header[i]
len_STR = len(STR)
with open(file_seq, 'r') as my_file:
file_reader = my_file.read()
counter = 0
a = 0
b = len_STR
list = []
for text in file_reader:
if file_reader[a:b] != STR:
a += 1
b += 1
else:
counter += 1
a += len_STR
b += len_STR
list.append(counter)
print(list)
i += 1
`
The problem is in place of variable "list" declaration. Every time you iterates through STRs in variable "header" you declares:
list = []
Thus, you create absolutely new variable, which stores only the length of current STR. To make a list with all STRs appended you need to declare variable "list" before the while loop and operator "print" after the while loop:
list = []
while i < len(header):
<your loop code>
print(list)
This should solve your problem.
P.S. Avoid to use "list" as a variable declaration. The "list" is a python built-in function and it is automatically declared. So, when you redeclare it, you will not be able to use list() function in your code.
I'm trying to load some data in Hana 2.0 database from Python 3.x Juypter Notebooks and getting an error.
Please advise.
the type of the column is INT as Pregnancies, the structure is as below
and in the CSV file, the data is a number
So above data should go to Pregnancies column.
Error I'm getting, while inserting data is below:-
hdbcli.dbapi.Error: (-10427, "Conversion of parameter/column (1) from
data type UCS2 (LE) to INT failed (invalid number: not a valid number
string 'Pregnancies')")
its big script..I'm getting issue in insert method
#staticmethod
def insert_data(connection,tablename,cols,inlist,data, batch_size):
sql = 'insert into ' + tablename + inlist
if len(data) > 0:
with connection.connection.cursor() as cur:
rows_inserted = cur.executemany(sql, data)
#staticmethod
def file_load(connection,table_descriptions,cols, inlist, filename,file_count,train_percentage,valid_percentage,test_percentage,batch_size):
with open(filename, 'r') as my_file:
reader = csv.reader(my_file, delimiter=',')
data = list()
data_list = list()
load_count = 0
for row in reader:
remain_count = file_count - load_count
if remain_count < batch_size:
batch_size = remain_count
if len(data) <= batch_size:
data.append(list(row))
if len(data) == batch_size:
DataSets.split_data_into_tables(connection,data,table_descriptions,train_percentage,valid_percentage,test_percentage, cols, inlist, batch_size, file_count)
load_count += len(data)
data = list()
print("Data Loaded:{}%".format(math.floor(load_count/file_count*100)))
whole code for this is in below link
https://github.com/SAP-samples/hana-ml-samples/blob/master/Python-API/pal/notebooks/data_load_utils.py
Based on the error message text, I’m rather sure that the problem is that the import code tries to read the first line of the CSV file (the line that contains the column names/headers) and importing these names to the target table is what fails.
To avoid this, just skip the first line of the CSV file.
the error says that the data in the column is of type 'UCSE' which is a character encoding standard in which characters are represented by a fixed-length 16 bits (2 bytes) and you have defined the columns as 'INT' type.
so change the type to 'INT' in the CSV file first and then load the data.
I have a csv file that I'm trying to clean up. I am trying to look at the first column and delete any rows that have anything other than chars for that row in the first column (I'm working on cleaning up rows where the first column has a ^ or . for now). It seems all my attempts either do nothing or nuke the whole csv file.
Interestingly enough, I have code that can identify the problem rows and it seems to work fine
def FindProblemRows():
with open('Data.csv') as csvDataFile:
ProblemRows = []
csvReader = csv.reader(csvDataFile)
data = [row for row in csv.reader(csvDataFile)]
length = len(data)
for i in range (0,length):
if data[i][0].find('^')!=-1 or data[i][0].find('.')!=-1:
ProblemRows.append(i)
return (ProblemRows)
Below I have my latest three failed attempts. Where am I going wrong and what should I change? Which of these comes closest?
'''
def Clean():
with open("Data.csv", "w", newline='') as f:
data = list(csv.reader(f))
writer = csv.writer(f)
Problems = FindProblemRows()
data = list(csv.reader(f))
length = len(data)
for row in data:
for i in Problems:
for j in range (0, length):
if row[j] == i:
writer.writerow(row)
Problems.remove(i)
def Clean():
Problems = FindProblemRows()
with open('Data.csv') as csvDataFile:
csvReader = csv.reader(csvDataFile)
data = [row for row in csv.reader(csvDataFile)]
length = len(data)
width = len(data[0])
with open("Data.csv","r") as csvFile:
csvReader = csv.reader( csvFile )
with open("CleansedData.csv","w") as csvResult:
csvWrite = csv.writer( csvResult )
for i in Problems:
for j in range (0, length):
if data[j] == i:
del data[j]
for j in range (0, length):
csvWrite.writerow(data[j])
'''
def Clean():
with open("Data.csv", 'r') as infile , open("CleansedData.csv", 'w') as outfile:
data = [row for row in infile]
for row in infile:
for column in row:
if "^" not in data[row][0]:
if "." not in data[row][0]:
outfile.write(data[row])
Update
Now I have:
def Clean():
df = pd.read_csv('Data.csv')
df = df['^' not in df.Symbol]
df = df['.' not in df.Symbol]
but I get KeyError: True
Shouldn't that work?
You should check whether the column Symbol contains any of the characters of interest. Method contains takes a regular expression:
bad_rows = df.Symbol.str.contains('[.^]')
df_clean = df[~bad_rows]
I have been working on code that takes rows from csv file and transfer them into the lists of integers for further mathematical operations. However, if a row turns out to be empty, it causes problems. Also, the user will not know which row is empty, so the solution should be general rather than pointing at a row and removing it. Here is the code:
import csv
import statistics as st
def RepresentsInt(i):
try:
int(i)
return True
except ValueError:
return False
l = []
with open('Test.csv', 'r') as f:
reader = csv.reader(f, delimiter=',')
for row in reader:
l.append([int(r) if RepresentsInt(r) else 0 for r in row])
for row in l:
row=[x for x in row if x!=0]
row.sort()
print(row)
I've tried l=[row for row in l if row!=[]] and ...
if row==[]:
l.remove(row)
... but both do nothing, and there is no error code for either. Here is my csv file:
1,2,5,4
2,3
43,65,34,56,7
0,5
7,8,9,6,5
33,45,65,4
If I run the code, I will get [] for row 4 and 6 (which are empty).
This worked on my machine:
import csv
def RepresentsInt(i):
try:
int(i)
return True
except ValueError:
return False
l = []
with open('Test.csv', 'r') as f:
reader = csv.reader(f, delimiter=',')
for row in reader:
l.append([int(r) for r in row if RepresentsInt(r)])
rows = [row for row in l if row]
for row in rows:
print(row)
It is unclear what you are doing with the statistics module, but the following program should you what you asked for. The pprint module is imported to make displaying the generated table easier to read. If this answer solves the problem presented in your question but you are having difficulty somewhere else, make sure you open another question targeted at the new problem.
#! /usr/bin/env python3
import csv
import pprint
def main():
table = []
# Add rows to table.
with open('Test.csv', newline='') as file:
table.extend(csv.reader(file))
# Convert table cells to numbers.
for index, row in enumerate(table):
table[index] = [int(value) if value.isdigit() else 0 for value in row]
# Remove zeros from the rows.
for index, row in enumerate(table):
table[index] = [value for value in row if value]
# Remove empty rows and display the table.
table = [row for row in table if row]
pprint.pprint(table)
if __name__ == '__main__':
main()
I am writing code which takes rows from a CSV file and transfers them into a lists of integers. However, if I leave some blank entries in the row, I get a "list index out of range" error. Here is the code:
import csv
with open('Test.csv', 'r') as f:
reader = csv.reader(f, delimiter=',')
rows = [[int(row[0]), int(row[1]),int(row[2]),int(row[3])] for row in reader]
for row in rows:
print(row)
I looked up some similar questions on this website and the best idea for the solution I got was:
rows = [[int(row[0]), int(row[1]),int(row[2]),int(row[3])] for row in reader if len(row)>1]
However, it resulted with the same error.
Thanks in advance!
The problem is that if you don't have an int or it is empty the cast will fail.
The below example inserts a zero '0' in case the value is not an int or is empty. Replace it by what you want.
You can optimize the code but this should work:
Edit: Shorter version
import csv
def RepresentsInt(s):
try:
int(s)
return True
except ValueError:
return False
l = []
with open('test.csv', 'r') as f:
reader = csv.reader(f, delimiter=',')
for row in reader:
l.append([int(r) if RepresentsInt(r) else 0 for r in row])
for row in l:
print(row)