Text file to CSV conversion - python-3.x

I have a text file which have content like :
Name: Aar saa
Last Name: sh
DOB: 1997-03-22
Phone: 1212222
Graduation: B.Tech
Specialization: CSE
Graduation Pass Out: 2019
Graduation Percentage: 60
Higher Secondary Percentage: 65
Higher Secondary School Name: Guru Nanak Dev University,amritsar
City: hyd
Venue Details: CMR College of Engineering & Technology (CMRCET) Medchal Road, TS � 501401
Name: bfdg df
Last Name: df
DOB: 2005-12-16
Phone: 2222222
Graduation: B.Tech
Specialization: EEE
Graduation Pass Out: 2018
Graduation Percentage: 45
Higher Secondary Percentage: 45
Higher Secondary School Name: asddasd
City: vjd
Venue Details: Prasad V. Potluri Siddhartha Institute Of Technology, Kanuru, AP - 520007
Name: cc dd ee
Last Name: ee
DOB: 1995-07-28
Phone: 444444444
Graduation: B.Tech
Specialization: ECE
Graduation Pass Out: 2019
Graduation Percentage: 75
Higher Secondary Percentage: 93
Higher Secondary School Name: Sasi institute of technology and engineering
City: hyd
Venue Details: CMR College of Engineering & Technology (CMRCET) Medchal Road, TS � 501401
I want to convert it CSV file with headers as
['Name', 'Last Name','DOB', 'Phone', 'Graduation','Specialization','Graduation Pass Out','Higher Secondary School Name','City','Venue Details']
with value as all the value after ':'
I have done something like this:
writer = csv.writer(open('result.csv', 'a'))
writer.writerow(['Name', 'Last Name','DOB', 'Phone', 'Graduation','Specialization','Graduation Pass Out','Graduation Percentage','Higher Secondary Percentage','Higher Secondary School Name','City','Venue Details'])
with open('Name2.txt') as f:
text = f.read()
myarray = text.split("\n\n")
for text1 in myarray:
parselines(text1, writer)
def parselines(lines,writer):
data=[]
for line in lines.split('\n'):
Name = line.split(": ",1)[1]
data.append(Name)
writer.writerow(data)
It worked but any efficient way would be much appreciated.

This algorithm works (kind-of a state machine)
If blank line, make a new row
Otherwise: add to current row, collect all headers and fields
def parselines(lines):
header = []
csvrows = [{}]
for line in lines:
line = line.strip()
if not line:
csvrows.append({}) # new row, in dict form
else:
field, data = line.split(":", 1)
csvrows[-1][field] = data
if field not in header:
header.append(field)
# format CSV
print(",".join(header))
for row in csvrows:
print(",".join(row.get(h,"") for h in header))

Related

Best way to handle element of dict that has multiple key/value pairs inside it

[{'id': 2, 'Registered Address': 'Line 1: 1 Any Street Line 2: Any locale City: Any City Region / State: Any Region Postcode / Zip code: BA2 2SA Country: GB Jurisdiction: Any Jurisdiction'}]
I have the above read into a dataframe and that is the output so far. The issue is I need to break out the individual elements - due to names of places etc the values may or may not have spaces in them - looking at the above my keys are Line 1, Line 2, City, Region / State, Postcode / Zip, Country, Jurisdiction.
Output required for the "Registered Address"-'key'is the keys and values
"Line 1": "1 Any Street"
"Line 2": "Any locale"
"City": "Any City"
"Region / State": "Any Region"
"Postcode / Zip code": "BA2 2SA"
"Country": "GB"
"Jurisdiction": "Any Jurisdiction"
Just struggling to find a way to get to the end result.I have tried to pop out and use urllib.prse but fell short - is anypone able to point me in the best direction please?
Tried to write a code that generalizes your question, but there were some limitations, regarding your data format. Anyway I would do this:
def address_spliter(my_data, my_keys):
address_data = my_data[0]['Registered Address']
key_address = {}
for i,k in enumerate(keys):
print(k)
if k == 'Jurisdiction:':
key_address[k] = address_data.split('Jurisdiction:')[1].removeprefix(' ').removesuffix(' ')
else:
key_address[k] = address_data.split(k)[1].split(keys[i+1])[0].removeprefix(' ').removesuffix(' ')
return key_address
were you can call this function like this:
my_data = [{'id': 2, 'Registered Address': 'Line 1: 1 Any Street Line 2: Any locale City: Any City Region / State: Any Region Postcode / Zip code: BA2 2SA Country: GB Jurisdiction: Any Jurisdiction'}]
and
my_keys = ['Line 1:','Line 2:','City:', 'Region / State:', 'Postcode / Zip code:', 'Country:', 'Jurisdiction']
As you can see It'll work if only the sequence of keys is not changed. But anyway, you can work around this idea and change it base on your problem accordingly if it doesn't go as expected.

Create a string with an offset

I would like to create a String where the values have a fixed offset where to add values to a string.
Example
ID(0) Name(10) Lastname(20) City(30)
example
1 Chris Smith Paris
I have found
StringBuffer.putAt(IntRange range, Object value)
or similiar, but I don't want to have a range, but an index where to start.
StringBufferWriter.write(String text, int offset, int length)
I have found [StrindBufferWriter][1], but not sure if the package codehause is an offical package I can use.
Any suggestions what to use here?
You can use String.padRight to achieve this effect:
def users = [
[id: 1, name: 'Chris', lastname: 'Smith', city:'Paris'],
[id: 2, name: 'Tim', lastname: 'Yates', city:'Manchester'],
]
users.each { user ->
println "${user.id.toString().padRight(10)}${user.name.padRight(10)}${user.lastname.padRight(20)}$user.city"
}
Which prints:
1 Chris Smith Paris
2 Tim Yates Manchester

How to insert list of words inside particular regex

import re
text = """STAR PLUS LIMITED Unit B & C, 15/F, Casey Aberdeen House, 38 Heung Yip Road, Wong Chuk Hang, Hong Kong. Tel: (852)2511 0112 Fax: 2507 4300 Email: info#starplushk.com Ref No: LSM25781 SALES Sales Quote No: SP21-SQ10452 Buyer's Ref: LSM-021042-5 Messers JSC "Tander" Russian Federation 350002 Krasnodar"""
ref_no = re.findall(r"(?:(?<=Buyer's Ref: )|(?<=Ref No: ))[\w\d-]+",text)
print(ref_no)
Required solution: ['LSM25781', 'LSM-021042-5']
The script above outputs this, but I have man keywords, so I want to generate the regex dynamically. How can I do that?
Tried:
ref_keywords = ["Buyer's Ref:","Ref No:","Reference number:"]
b = r"(?:(?<=" + '|'.join(ref_keyword)+ r" ))[\w\d-]+"
ref_no = re.findall(b, text)
print(ref_no)
This results in the following error
Traceback (most recent call last):
File "/home/v/.config/JetBrains/PyCharm2021.3/scratches/scratch_2.py", line 7, in <module>
ref_no = re.findall(regex, text)
File "/home/v/.pyenv/versions/3.9.5/lib/python3.9/re.py", line 241, in findall
return _compile(pattern, flags).findall(string)
File "/home/v/.pyenv/versions/3.9.5/lib/python3.9/re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "/home/v/.pyenv/versions/3.9.5/lib/python3.9/sre_compile.py", line 768, in compile
code = _code(p, flags)
File "/home/v/.pyenv/versions/3.9.5/lib/python3.9/sre_compile.py", line 607, in _code
_compile(code, p.data, flags)
File "/home/v/.pyenv/versions/3.9.5/lib/python3.9/sre_compile.py", line 182, in _compile
raise error("look-behind requires fixed-width pattern")
re.error: look-behind requires fixed-width pattern
Process finished with exit code 1
Is there a solution to add list of keywords inside regex. I cannot use "|" because I have many list of keywords.
key_word = ['key1', 'key2', 'key2']
combined_word = ""
for key in key_word:
combined_word += "|"+key
import re
sentence = "I wana delete key1 and key2 but also key3."
re.split(combined_word, sentence)
You can do it the following way (you almost had it, you just need to create all the regex characters for each keyword:
import re
text = """STAR PLUS LIMITED Unit B & C, 15/F, Casey Aberdeen House, 38 Heung Yip Road, Wong Chuk Hang, Hong Kong. Tel: (852)2511 0112 Fax: 2507 4300 Email: info#starplushk.com Ref No: LSM25781 SALES Sales Quote No: SP21-SQ10452 Buyer's Ref: LSM-021042-5 Messers JSC "Tander" Russian Federation 350002 Krasnodar"""
ref_keywords = ["Buyer's Ref:", "Ref No:", "Reference number:"]
def keyword_to_regex(keyword: str) -> str:
# you missed creating these for each keyword
return f"(?<={keyword} )"
regex_for_all_keywords = r"(?:" + "|".join(map(keyword_to_regex, ref_keywords)) + r")[\w\d-]+"
ref_no = re.findall(regex_for_all_keywords, text)
print(ref_no) # ['LSM25781', 'LSM-021042-5']

How to capture words spread through multiple lines which have anywhite space(newline, space, tab)

import re
c = """
class_monitor std4:
Name: xyz
Roll number: 123
Age: 9
Badge: Blue
class_monitor std5:
Name: abc
Roll number: 456
Age: 10
Badge: Red
"""
I want to print Name, Roll number and age for std4 and Name, roll number and badge for std5.
pat = (class_monitor)(.*4:)(\n|\s|\t)*(Name:)(.*)(\s|\n|\t)*(Roll number:)(.*)(\s|\n|\t)*(Age:)(.*)(\s|\n|\t)*(Badge:)(.*)
it matches the respective std if I toggle the second group (.*4:) to (.*5:) in pythex.
However, in a script mode, it is not working. Am I missing something here?

Reformat csv file using python?

I have this csv file with only two entries. Here it is:
Meat One,['Abattoirs', 'Exporters', 'Food Delivery', 'Butchers Retail', 'Meat Dealers-Retail', 'Meat Freezer', 'Meat Packers']
First one is title and second is a business headings.
Problem lies with entry two.
Here is my code:
import csv
with open('phonebookCOMPK-Directory.csv', "rt") as textfile:
reader = csv.reader(textfile)
for row in reader:
row5 = row[5].replace("[", "").replace("]", "")
listt = [(''.join(row5))]
print (listt[0])
it prints:
'Abattoirs', 'Exporters', 'Food Delivery', 'Butchers Retail', 'Meat Dealers-Retail', 'Meat Freezer', 'Meat Packers'
What i need to do is that i want to create a list containing these words and then print them like this using for loop to print every item separately:
Abattoirs
Exporters
Food Delivery
Butchers Retail
Meat Dealers-Retail
Meat Freezer
Meat Packers
Actually I am trying to reformat my current csv file and clean it so it can be more precise and understandable.
Complete 1st line of csv is this:
Meat One,+92-21-111163281,Al Shaheer Corporation,Retailers,2008,"['Abattoirs', 'Exporters', 'Food Delivery', 'Butchers Retail', 'Meat Dealers-Retail', 'Meat Freezer', 'Meat Packers']","[[' Outlets Address : Shop No. Z-10, Station Shopping Complex, MES Market, Malir-Cantt, Karachi. Landmarks : MES Market, Station Shopping Complex City : Karachi UAN : +92-21-111163281 '], [' Outlets Address : Shop 13, Ground Floor, Plot 14-D, Sky Garden, Main Tipu Sultan Road, KDA Scheme No.1, Karachi. Landmarks : Nadra Chowrangi, Sky Garden, Tipu Sultan Road City : Karachi UAN : +92-21-111163281 '], ["" Outlets Address : Near Jan's Broast, Boat Basin, Khayaban-e-Roomi, Block 5, Clifton, Karachi. Landmarks : Boat Basin, Jans Broast, Khayaban-e-Roomi City : Karachi UAN : +92-21-111163281 View Map ""], [' Outlets Address : Gulistan-e-Johar, Karachi. Landmarks : Perfume Chowk City : Karachi UAN : +92-21-111163281 '], [' Outlets Address : Tee Emm Mart, Creek Vista Appartments, Khayaban-e-Shaheen, Phase VIII, DHA, Karachi. Landmarks : Creek Vista Appartments, Nueplex Cinema, Tee Emm Mart, The Place City : Karachi Mobile : 0302-8333666 '], [' Outlets Address : Y-Block, DHA, Lahore. Landmarks : Y-Block City : Lahore UAN : +92-42-111163281 '], [' Outlets Address : Adj. PSO, Main Bhittai Road, Jinnah Supermarket, F-7 Markaz, Islamabad. Landmarks : Bhittai Road, Jinnah Super Market, PSO Petrol Pump City : Islamabad UAN : +92-51-111163281 ']]","Agriculture, fishing & Forestry > Farming equipment & services > Abattoirs in Pakistan"
First column is Name
Second column is Number
Third column is Owner
Forth column is Business type
Fifth column is Y.O.E
Sixth column is Business Headings
Seventh column is Outlets (List of lists containing every branch address)
Eighth column is classification
There is no restriction of using csv.reader, I am open to any technique available to clean my file.
Think of it in terms of two separate tasks:
Collect some data items from a ‘dirty’ source (this CSV file)
Store that data somewhere so that it’s easy to access and manipulate programmatically (according to what you want to do with it)
Processing dirty CSV
One way to do this is to have a function deserialize_business() to distill structured business information from each incoming line in your CSV. This function can be complex because that’s the nature of the task, but still it’s advisable to split it into self-containing smaller functions (such as get_outlets(), get_headings(), and so on). This function can return a dictionary but depending on what you want it can be a [named] tuple, a custom object, etc.
This function would be an ‘adapter’ for this particular CSV data source.
Example of deserialization function:
def deserialize_business(csv_line):
"""
Distills structured business information from given raw CSV line.
Returns a dictionary like {name, phone, owner,
btype, yoe, headings[], outlets[], category}.
"""
pieces = [piece.strip("[[\"\']] ") for piece in line.strip().split(',')]
name = pieces[0]
phone = pieces[1]
owner = pieces[2]
btype = pieces[3]
yoe = pieces[4]
# after yoe headings begin, until substring Outlets Address
headings = pieces[4:pieces.index("Outlets Address")]
# outlets go from substring Outlets Address until category
outlet_pieces = pieces[pieces.index("Outlets Address"):-1]
# combine each individual outlet information into a string
# and let ``deserialize_outlet()`` deal with that
raw_outlets = ', '.join(outlet_pieces).split("Outlets Address")
outlets = [deserialize_outlet(outlet) for outlet in raw_outlets]
# category is the last piece
category = pieces[-1]
return {
'name': name,
'phone': phone,
'owner': owner,
'btype': btype,
'yoe': yoe,
'headings': headings,
'outlets': outlets,
'category': category,
}
Example of calling it:
with open("phonebookCOMPK-Directory.csv") as f:
lineno = 0
for line in f:
lineno += 1
try:
business = deserialize_business(line)
except:
# Bad line formatting?
log.exception(u"Failed to deserialize line #%s!", lineno)
else:
# All is well
store_business(business)
Storing the data
You’ll have the store_business() function take your data structure and write it somewhere. Maybe it’ll be another CSV that’s better structured, maybe multiple CSVs, a JSON file, or you can make use of SQLite relational database facilities since Python has it built-in.
It all depends on what you want to do later.
Relational example
In this case your data would be split across multiple tables. (I’m using the word “table” but it can be a CSV file, although you can as well make use of an SQLite DB since Python has that built-in.)
Table identifying all possible business headings:
business heading ID, name
1, Abattoirs
2, Exporters
3, Food Delivery
4, Butchers Retail
5, Meat Dealers-Retail
6, Meat Freezer
7, Meat Packers
Table identifying all possible categories:
category ID, parent category, name
1, NULL, "Agriculture, fishing & Forestry"
2, 1, "Farming equipment & services"
3, 2, "Abattoirs in Pakistan"
Table identifying businesses:
business ID, name, phone, owner, type, yoe, category
1, Meat One, +92-21-111163281, Al Shaheer Corporation, Retailers, 2008, 3
Table describing their outlets:
business ID, city, address, landmarks, phone
1, Karachi UAN, "Shop 13, Ground Floor, Plot 14-D, Sky Garden, Main Tipu Sultan Road, KDA Scheme No.1, Karachi", "Nadra Chowrangi, Sky Garden, Tipu Sultan Road", +92-21-111163281
1, Karachi UAN, "Near Jan's Broast, Boat Basin, Khayaban-e-Roomi, Block 5, Clifton, Karachi", "Boat Basin, Jans Broast, Khayaban-e-Roomi", +92-21-111163281
Table describing their headings:
business ID, business heading ID
1, 1
1, 2
1, 3
…
Handling all this would require a complex store_business() function. It may be worth looking into SQLite and some ORM framework, if going with relational way of keeping the data.
You can just replace the line :
print(listt[0])
with :
print(*listt[0], sep='\n')

Resources