I'm trying to replace my old.csv data that looks like this: 6004387,6219127,'12524449',10340
Into new.csv that should look like this: 6004387|6219127|12524449|10340
What I get now is "['6004387'| '6219127'| ""'12524449'""| '10340']"
How can I remove more than one symbol?
import csv
import string
input_file = open('old.csv', 'r')
output_file = open('new.csv', 'w')
data = csv.reader(input_file)
writer = csv.writer(output_file)
specials = ','
for row in data:
row = str(row)
new_row = str.replace(row,specials,'|')
writer.writerow(new_row.split(','))
input_file.close()
output_file.close()
If you want to remove quote characters from input file, specify quotechar="'" in csv.reader. Also, for | delimiter in output file, specify delimiter='|' in csv.writer:
import csv
input_file = open('old.csv', 'r')
output_file = open('new.csv', 'w')
data = csv.reader(input_file, quotechar="'")
writer = csv.writer(output_file, delimiter='|')
for row in data:
writer.writerow(row)
input_file.close()
output_file.close()
Creates new.csv:
6004387|6219127|12524449|10340
You can use the | as the delimeter and then recurively remove unwanted characters afterwards with a function such as str.strip("\"' ") which removes all characters from the string which match what's in the brackets.
Related
I am trying to read some data files '.txt' and some of them contain strange random characters and even extra columns in random rows, like in the following example, where the second row is an example of a right row:
CTD 10/07/30 05:17:14.41 CTD 24.7813, 0.15752, 1.168, 0.7954, 1497.¸ 23.4848, 0.63042, 1.047, 3.5468, 1496.542
CTD 10/07/30 05:17:14.47 CTD 23.4846, 0.62156, 1.063, 3.4935, 1496.482
I read the description of np.loadtxt and I have not found a solution for my problem. Is there a systematic way to skip rows like these?
The code that I use to read the files is:
#Function to read a datafile
def Read(filename):
#Change delimiters for spaces
s = open(filename).read().replace(':',' ')
s = s.replace(',',' ')
s = s.replace('/',' ')
#Take the columns that we need
data=np.loadtxt(StringIO(s),usecols=(4,5,6,8,9,10,11,12))
return data
This works without using csv like the other answer and just reads line by line checking if it is ascii
data = []
def isascii(s):
return len(s) == len(s.encode())
with open("test.txt", "r") as fil:
for line in fil:
res = map(isascii, line)
if all(res):
data.append(line)
print(data)
You could use the csv module to read the file one line at a time and apply your desired filter.
import csv
def isascii(s):
len(s) == len(s.encode())
with open('file.csv') as csvfile:
csvreader = csv.reader(csvfile)
for row in csvreader:
if len(row)==expected_length and all((isascii(x) for x in row)):
'write row onto numpy array'
I got the ascii check from this thread
How to check if a string in Python is in ASCII?
I have a csv file which contains data in two columns, as follows:
40500 38921
43782 32768
55136 49651
63451 60669
50550 36700
61651 34321
and so on...
I want to convert each data into it's hex equivalent, then concatenate them, and write them into a column in another csv file.
For example: hex(40500) = 9E34, and hex(38921) = 9809.
So, in output csv file, element A1 would be 9E349809
So, i am expecting column A in output csv file to be:
9E349809
AB068000
D760C1F3
F7DBECFD
C5768F5C
F0D38611
I referred a sample code which concatenates two columns, but am struggling with the converting them to hex and then concatenating them. Following is the code:-
import csv
inputFile = 'input.csv'
outputFile = 'output.csv'
with open(inputFile) as f:
reader = csv.reader(f)
with open(outputFile, 'w') as g:
writer = csv.writer(g)
for row in reader:
new_row = [''.join([row[0], row[1]])] + row[2:]
writer.writerow(new_row)
How can i convert data in each column to its hex equivalent, then concatenate them and write them in another file?
You could do this in 4 steps:
Read the lines from the input csv file
Use formatting options to get the hex values of each number
Perform string concatenation to get your result
Write to new csv file.
Sample Code:
with open (outputFile, 'w') as outfile:
with open (inputFile,'r') as infile:
for line in infile: # Iterate through each line
left, right = int(line.split()[0]), int(line.split()[1]) # split left and right blocks
newstr = '{:x}'.format(left)+'{:x}'.format(right) # create new string using hex values excluding '0x'
outfile.write(newstr) # write to output file
print ('Conversion completed')
print ('Closing outputfile')
Sample Output:
In[44] line = '40500 38921'
Out[50]: '9e349809'
ParvBanks solution is good (clear and functionnal), I would simplify it a little like that:
with open (inputFile,'r') as infile, open (outputFile, 'w+') as outfile:
for line in infile:
outfile.write("".join(["{:x}".format(int(v)) for v in line.split()]))
Background:
I have a CSV (csv_dump) file with data from a MySQL table. I want copy some of the lines that meet certain conditions (row[1] == condition_1 and row[2] == condition_2) into a temporary CSV file (csv_temp).
Code Snippet:
f_reader = open(csv_dump, 'r')
f_writer = open(csv_temp, 'w')
temp_file = csv.writer(f_writer)
lines_in_csv = csv.reader(f_reader, delimiter=',', skipinitialspace=False)
for row in lines_in_csv:
if row[1] == condition_1 and row[2] == condition_2:
temp_file.writerow(row)
f_reader.close()
f_writer.close()
Question:
How can I copy the line that is being read copy it "as is" into the temp file with Python3?
test.csv
data1,data2,data3
120,80,200
140,50,210
170,100,250
150,70,300
180,120,280
The code goes here
import csv
with open("test.csv", 'r') as incsvfile:
input_csv = csv.reader(incsvfile, delimiter=',', skipinitialspace=False)
with open('tempfile.csv', 'w', newline='') as outcsvfile:
spamwriter = csv.writer(outcsvfile, delimiter=',', quoting=csv.QUOTE_MINIMAL)
first_row = next(input_csv)
spamwriter.writerow(first_row)
for row in input_csv:
if int(row[1]) != 80 and int(row[2]) != 300:
spamwriter.writerow(row)
output tempfile.csv
data1,data2,data3
140,50,210
170,100,250
180,120,280
if you don't have title remove these two lines
first_row = next(input_csv)
spamwriter.writerow(first_row)
The following Python script seems to do the job... However, having said that, you should probably be using a MySQL query to do this work directly, instead of re-processing from an intermediate CSV file. But I guess there must be some good reason for wanting to do that?
mycsv.csv:
aa,1,2,5
bb,2,3,5
cc,ddd,3,3
hh,,3,1
as,hfd,3,3
readwrite.py:
import csv
with open('mycsv.csv', 'rb') as infile:
with open('out.csv', 'wb') as outfile:
inreader = csv.reader(infile, delimiter=',', quotechar='"')
outwriter = csv.writer(outfile)
for row in inreader:
if row[2]==row[3]:
outwriter.writerow(row)
out.csv:
cc,ddd,3,3
as,hfd,3,3
With a little more work, you could change the 'csv.writer' to make use of the same delimiters and escape/quote characters as the 'csv.reader'. It's not exactly the same as writing out the raw line from the file, but I think it will be practically as fast since the lines in question have already clearly been parsed without error if we have been able to check the value of specific fields.
Ive a csv file that I would like to get all the rows in one column. Ive tried importing into MS Excel or Formatting it with Notedpad++ . However with each try it considers a piece of data as a new row.
How can I format file with pythons csv module so that it removes a string "BRAS" and corrects the format. Each row is found between a quote " and delimiter is a pipe |.
Update:
"aa|bb|cc|dd|
ee|ff"
"ba|bc|bd|be|
bf"
"ca|cb|cd|
ce|cf"
The above is supposed to be 3 rows, however my editors see them as 5 rows or 6 and so forth.
import csv
import fileinput
with open('ventoya.csv') as f, open('ventoya2.csv', 'w') as w:
for line in f:
if 'BRAS' not in line:
w.write(line)
N.B I get a unicode error when trying to use in python.
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 18: character maps to <undefined>
This is a quick hack for small input files (the content is read to memory).
#!python2
fnameIn = 'ventoya.csv'
fnameOut = 'ventoya2.csv'
with open(fnameIn) as fin, open(fnameOut, 'w') as fout:
data = fin.read() # content of the input file
data = data.replace('\n', '') # make it one line
data = data.replace('""', '|') # split char instead of doubled ""
data = data.replace('"', '') # remove the first and last "
print data
for x in data.split('|'): # split by bar
fout.write(x + '\n') # write to separate lines
Or if the goal is only to fix the extra (unwanted) newline to form a single-column CSV file, the file can be fixed first, and then read through the csv module:
#!python2
import csv
fnameIn = 'ventoya.csv'
fnameFixed = 'ventoyaFixed.csv'
fnameOut = 'ventoya2.csv'
# Fix the input file.
with open(fnameIn) as fin, open(fnameFixed, 'w') as fout:
data = fin.read() # content of the file
data = data.replace('\n', '') # remove the newlines
data = data.replace('""', '"\n"') # add the newlines back between the cells
fout.write(data)
# It is an overkill, but now the fixed file can be read using
# the csv module.
with open(fnameFixed, 'rb') as fin, open(fnameOut, 'wb') as fout:
reader = csv.reader(fin)
writer = csv.writer(fout)
for row in reader:
writer.writerow(row)
For solving this you need not to go to even code.
1: Just open file in Notepad++
2: In first line select from | symble till next line
3: go to replace and replace the selected format with |
Search mode can be normal or extended :)
Well, since the line breaks are consistent, you could go in and do find/replace as suggested, but you could also do a quick conversion with your python script:
import csv
import fileinput
linecount = 0
with open('ventoya.csv') as f, open('ventoya2.csv', 'w') as w:
for line in f:
line = line.rstrip()
# remove unwanted breaks by concatenating pairs of rows
if linecount%2 == 0:
line1 = line
else:
full_line = line1 + line
full_line = full_line.replace(' ','')
# remove spaces from front of 2nd half of line
# if you want comma delimiters, uncomment next line:
# full_line = full_line.replace('|',',')
if 'BRAS' not in full_line:
w.write(full_line + '\n')
linecount += 1
This works for me with the test data, and if you want to change the delimiters while writing to file, you can. The nice thing about doing with code is: 1. you can do it with code (always fun) and 2. you can remove the line breaks and filter content to the written file at the same time.
2331,0,13:30:08,25.35,22.05,23.8,23.9,23.5,23.7,5455,350,23.65,132,23.6,268,23.55,235,23.5,625,23.45,459,23.7,83,23.75,360,23.8,291,23.85,186,23.9,331,0,1,25,1000,733580089,name,,,
I got a line like this and how could I cut it? I only need the first 9 variable like this:
2331,0,13:30:08,25.35,22.05,23.8,23.9,23.5,23.7,5455
the original data i save as txt.file, and could I rewrite the original one and save?
Use either csv or just to straight file io with string split function
For example:
import csv
with open('some.txt', 'rb') as f:
reader = csv.reader(f)
for row in reader:
print row[:9]
or if everything is on a single line and you don't want to use a csv interface
with open('some.txt', 'r') as f:
line = f.read()
print line.split(str=",")[:9]
If you have a file called "content.txt".
f = open("content.txt","r")
contentFile = f.read();
output = contentFile.split(",")[:9]
output = ",".join(output)
f.close()
f = open("content.txt","wb")
f.write(output)
If all your values are stored in an Array, you can slice like this:
arrayB = arrayA[:9]
To get your values into an array you could split your String at every ","
arrayA = inputString.split(str=",")