lists and reading from file comma separated elements - python-3.x

Here is my code
#course registration
list_courses=[]
for line in open("courses.txt",'r').readlines():
list_courses.append(line.strip())
print ("Gathering course information from file: \n",list_courses)
close("courses.txt")
list_student=[]
for line in open("students.txt",'r').readlines():
list_student.append(line.strip())
print("Here is student info: \n",list_student)
close("students.txt")
this is giving me errors when I try to close the files. How do I close, I am basically reading contents of file and storing them in a list. Now later on want to close the open files.There I get error.
I edited the code as per suggestions below.
The new code is
list_courses=[]
with open("courses.txt",'r') as myfile1:
list_courses=myfile1.readlines()
list_courses=[x.strip() for x in list_courses]
print ("Gathering course information from file: \n",list_courses)
list_student=[]
with open("students.txt",'r') as myfile1:
list_student=myfile1.readlines()
list_student=[x.strip() for x in list_student]
print("Here is student info: \n",list_student)
The information in courses.txt is
cs101,C programming
cs102,Digital logic and design
cs103,Electrical engineering
cs231,IT networks
cs232,IT Workshop
cs233,IT programming
cs301,Compilers and automata
cs302,Operating Systems
cs303,Networks
cs401,Game Theory
cs402,Systems Programming
cs403,Automata
ec101,Digitization
ec102,Analog cicuit design
ec103,IP Telephony
ec201,Wireless Network
ec202,Microwave engineering
ec203,Antenna
ec301,Maths2
ec302,Theory of Circuits
ec303,PCB design
ec401,PLC programming
ec402,Scada
ec403,VLSI
When I run the code I get output
Gathering course information from file:
['cs101,C programming', 'cs102,Digital logic and design', 'cs103,Electrical engineering', 'cs231,IT networks', 'cs232,IT Workshop', 'cs233,IT programming', 'cs301,Compilers and automata', 'cs302,Operating Systems', 'cs303,Networks', 'cs401,Game Theory', 'cs402,Systems Programming', 'cs403,Automata', 'ec101,Digitization', 'ec102,Analog cicuit design', 'ec103,IP Telephony', 'ec201,Wireless Network', 'ec202,Microwave engineering', 'ec203,Antenna', 'ec301,Maths2', 'ec302,Theory of Circuits', 'ec303,PCB design', 'ec401,PLC programming', 'ec402,Scada', 'ec403,VLSI']
Instead of it what I want is the input cs101 from the first line of courses.txt to go in list_courses[0] and list_courses[1] to have c programming i.e.
list_courses[0]=cs101
list_courses[1]=C programming
So I tried methods where programe have taken a line and read the line stored that line as an element of list but there is a comma which separates two elements in courses.txt and comma separated values should be separate list elements.

This will work for you:
students=[]
subject=[]
with open("students.txt","r") as f:
for line in f.readlines():
eachl=line.split(",")
students.append(eachl[0])
subject.append(eachl[1][:-1])
you will get two lists containing student names and another one with subjects:
students list will look like:
['cs101', 'cs102', 'cs103', 'cs231', 'cs232', 'cs233', 'cs301', 'cs302', 'cs303', 'cs401', 'cs402', 'cs403', 'ec101', 'ec102', 'ec103', 'ec201', 'ec202', 'ec203', 'ec301', 'ec302', 'ec303', 'ec401', 'ec402', 'ec40`3]
subjects list will look like:
['C programming', 'Digital logic and design', 'Electrical engineering', 'IT networks', 'IT Workshop', 'IT programming', 'Compilers and automata', 'Operating Systems', 'Networks', 'Game Theory', 'Systems Programming', 'Automata', 'Digitization', 'Analog cicuit design', 'IP Telephony', 'Wireless Network', 'Microwave engineering', 'Antenna', 'Maths2', 'Theory of Circuits', 'PCB design', 'PLC programming', 'Scada', 'VLSI']

Why don’t you try
students = []
courses = []
open(“courses.txt”, “r”) as f
for line in f.readlines()
a, b = line.split(“,”)
students.append(a)
courses.append(b[:-1])
f.close()
This will produce two lists students and courses

Related

How to reconstruct original text from spaCy tokens, even in cases with complicated whitespacing and punctuation

' '.join(token_list) does not reconstruct the original text in cases with multiple whitespaces and punctuation in a row.
For example:
from spacy.tokenizer import Tokenizer
from spacy.lang.en import English
nlp = English()
# Create a blank Tokenizer with just the English vocab
tokenizerSpaCy = Tokenizer(nlp.vocab)
context_text = 'this is a test \n \n \t\t test for \n testing - ./l \t'
contextSpaCyToksSpaCyObj = tokenizerSpaCy(context_text)
spaCy_toks = [i.text for i in contextSpaCyToksSpaCyObj]
reconstruct = ' '.join(spaCy_toks)
reconstruct == context_text
>False
Is there an established way of reconstructing original text from spaCy tokens?
Established answer should work with this edge case text (you can directly get the source from clicking the 'improve this question' button)
" UNCLASSIFIED U.S. Department of State Case No. F-2014-20439 Doc No. C05795279 Date: 01/07/2016\n\n\n RELEASE IN PART\n B5, B6\n\n\n\n\nFrom: H <hrod17#clintonemail.com>\nSent: Monday, July 23, 2012 7:26 AM\nTo: 'millscd #state.gov'\nCc: 'DanielJJ#state.gov.; 'hanleymr#state.gov'\nSubject Re: S speech this morning\n\n\n\n Waiting to hear if Monica can come by and pick up at 8 to take to Josh. If I don't hear from her, can you send B5\nsomeone else?\n\n Original Message ----\nFrom: Mills, Cheryl D [MillsCD#state.gov]\nSent: Monday, July 23, 2012 07:23 AM\nTo: H\nCc: Daniel, Joshua J <Daniel1.1#state.gov>\nSubject: FW: S speech this morning\n\nSee below\n\n B5\n\ncdm\n\n Original Message\nFrom: Shah, Rajiv (AID/A) B6\nSent: Monday, July 23, 2012 7:19 AM\nTo: Mills, Cheryl D\nCc: Daniel, Joshua.'\nSubject: S speech this morning\n\nHi cheryl,\n\nI look fwd to attending the speech this morning.\n\nI had one last minute request - I understand that in the final version there is no reference to the child survival call to\naction, but their is a reference to family planning efforts. Could you and josh try to make sure there is some specific\nreference to the call to action?\n\nAlso, in terms of acknowledgements it would be good to note torn friedan's leadership as everyone is sensitive to our ghi\ntransition and we want to continue to send the usaid-pepfar-cdc working together public message. I don't know if he is\nthere, but wanted to flag.\n\nLook forward to it.\n\nRaj\n\n\n\n\n UNCLASSIFIED U.S. Department of State Case No. F-2014-20439 Doc No. C05795279 Date: 01/07/2016\n\x0c"
You can very easily accomplish this by changing two lines in your code:
spaCy_toks = [i.text + i.whitespace_ for i in contextSpaCyToksSpaCyObj]
reconstruct = ''.join(spaCy_toks)
Basically, each token in spaCy knows whether it is followed by whitespace or not. So you call token.whitespace_ instead of joining them on space by default.

Unable to print Dependent vowels

I am reading the text file consisting of bengali words. But I am unable to print the dependent vowels like KA,KI etc...
Here is my sample code and output
import unicodedata
bengali_phoneme_maplist={u'অ':'A',u'আ':'AA',u'ই':'I',u'ঈ':'II',u'উ':'U',u'ঊ ':'UU',u'ঋ ':'R',u'ঌ ':'L',u'এ ':'E',u'ঐ ':'AI',u'ও ':'O',u'ঔ ':'AU',u'ক':'KA',u'খ ':'KHA',u'গ ':'GA',u'ঘ':'GHA',u'ঙ ':'NGA',u'চ ':'CA',u'ছ':'CHA',u'জ ':'JA',u'ঝ':'JHA',u'ঞ':'NYA',u'ট ':'TTA',u'ঠ':'TTHA',u'ড ':'DDA',u'ঢ':'DDHA',u'ণ ':'NNA',u'ত ':'TA',u'ত ':'THA',u'দ':'DA',u'ধ':'DHA',u'ন':'NA',u'প':'PA',u'ফ':'PHA',u'ব':'BA',u'ভ':'BHA',u'ম ':'MA',u'য ':'YA',u'র':'RA',u'ল ':'LA',u'শ ':'SHA',u'ষ':'SSA',u'স ':'SA',u'হ':'ha',u' া ':'AAV',u' ি':'IV',u'ী':'IIV',u'ু':'UV',u'ূ':'UUV',u'ৃ':'RRV',u'ৄ ':'RR',u'ৄ':'EV',u' ৈ':'EV',u'়':'NUKTHA',u'ঽ':'AVAGRAHA'}
bengali_phoneme_maplist_normalise={unicodedata.normalize('NFKD',k):v
for k,v in bengali_phoneme_maplist.items()}
with open('bengali.txt','r')as infile:
lines=infile.readlines()
for index,line in enumerate(lines):
print('Phonemes in line{0}.total{1} symbols'.format(index,len(line)))
unknown=[]
words=line.split()
for word in words:
print(word,':',sep=' ', end='')
for character in word:
c=unicodedata.normalize('NFKD',character).casefold()
try:
print(bengali_phoneme_maplist_normalise[c],sep='',end='')
except KeyError:
print('_',sep='',end='')
if c not in unknown:
unknown.append(c)
print()
if unknown:
print('Unrecognised symbols:{0},total {1} symbols'.format(','.join(unknown),len(unknown)))
Sample input:
শিল্পাঞ্চলে ঢোকার মুখে, স্ন্যাক্সবারে খাবার কিনছিলেন, বহুজাতিক তথ্যপ্রযুক্তি সংস্থার কর্মী, শুভময় বন্দ্যোপাধ্যায়
Sample output:
Phonemes in line0.total129 symbols
text_000002 :___________
"শিল্পাঞ্চলে :_____PA_NYA____
ঢোকার :DDHA_KA_RA
মুখে, :_UV___
স্ন্যাক্সবারে :__NA___KA__BA_RA_
খাবার :__BA_RA
কিনছিলেন, :KA_NACHA___NA_
Unrecognisedsymbols:t,e,x,_,0,2,",শ,ি,ল,্,া,চ,ে,ো,ম,খ,,,স,য,জ,ত,থ,ং,য়,),
(Note that I know nohting about Bengali. :)
There are a few problems in your code:
There are many extra SPACE chars in the bengali_phoneme_maplist definition. For example, u'ঊ ' should be u'ঊ'. And it seems like it's not easy to input chars like u'া' in an text editor so I suggest you directly use unicode in the code, like '\u09be':'AAV'. (Actually I'd suggest you use '\uxxxx' for all chars and write the real chars in comments.)
u'ত':'TA',u'ত':'THA' should change to u'ত':'TA',u'থ':'THA'.
The chars in bengali_phoneme_maplist are not complete. For example there's no ো , ৌ , ্ and ং
After fixing these errors you will get the correct result.

Pull random results from a database?

I have been coding in Python for a 2 months or so, but I mostly ask for help from a more experienced friend when I run in to these kinds of issues. I should also, before I begin, specify that I use Python solely for a personal project; any questions I ask will relate to each other through that.
With those two things out of the way, I have a database of weaponry items that I created using the following script, made in Python 3.X:
#Start by making a list of every material, weapontype, and upgrade.
Materials=("Unobtanium","IvorySilk","BoneLeather","CottonWood","Tin","Copper","Bronze","Gold","Cobalt","Tungsten")
WeaponTypes=("Knife","Sword","Greatsword","Polearm","Battlestaff","Claw","Cane","Wand","Talis","Slicer","Rod","Bow","Crossbow","Handbow","Pistol","Mechgun","Rifle","Shotgun")
Upgrades=("0","1","2","3","4","5","6","7","8","9","10")
ForgeWInputs=[]
#Go through every material...
for m in Materials:
#And in each material, go through every weapontype...
for w in WeaponTypes:
#And in every weapontype, go through each upgrade...
for u in Upgrades:
ForgeWInputs.append((m,w,u))
#We now have a list "ForgeWInputs", which contains the 3-element list needed to
#Forge any weapon. For example...
MAT={}
MAT["UnobtaniumPD"]=0
MAT["UnobtaniumMD"]=0
MAT["UnobtaniumAC"]=0
MAT["UnobtaniumPR"]=0
MAT["UnobtaniumMR"]=0
MAT["UnobtaniumWT"]=0
MAT["UnobtaniumBuy"]=0
MAT["UnobtaniumSell"]=0
MAT["IvorySilkPD"]=0
MAT["IvorySilkMD"]=12
MAT["IvorySilkAC"]=3
MAT["IvorySilkPR"]=0
MAT["IvorySilkMR"]=3
MAT["IvorySilkWT"]=6
MAT["IvorySilkBuy"]=10
MAT["IvorySilkSell"]=5
MAT["CottonWoodPD"]=8
MAT["CottonWoodMD"]=8
MAT["CottonWoodAC"]=5
MAT["CottonWoodPR"]=0
MAT["CottonWoodMR"]=3
MAT["CottonWoodWT"]=6
MAT["CottonWoodBuy"]=14
MAT["CottonWoodSell"]=7
MAT["BoneLeatherPD"]=12
MAT["BoneLeatherMD"]=0
MAT["BoneLeatherAC"]=3
MAT["BoneLeatherPR"]=3
MAT["BoneLeatherMR"]=0
MAT["BoneLeatherWT"]=6
MAT["BoneLeatherBuy"]=10
MAT["BoneLeatherSell"]=5
MAT["TinPD"]=18
MAT["TinMD"]=6
MAT["TinAC"]=3
MAT["TinPR"]=5
MAT["TinMR"]=2
MAT["TinWT"]=12
MAT["TinBuy"]=20
MAT["TinSell"]=10
MAT["CopperPD"]=6
MAT["CopperMD"]=18
MAT["CopperAC"]=3
MAT["CopperPR"]=2
MAT["CopperMR"]=5
MAT["CopperWT"]=12
MAT["CopperBuy"]=20
MAT["CopperSell"]=10
MAT["BronzePD"]=10
MAT["BronzeMD"]=10
MAT["BronzeAC"]=5
MAT["BronzePR"]=3
MAT["BronzeMR"]=3
MAT["BronzeWT"]=15
MAT["BronzeBuy"]=30
MAT["BronzeSell"]=15
MAT["GoldPD"]=10
MAT["GoldMD"]=30
MAT["GoldAC"]=0
MAT["GoldPR"]=5
MAT["GoldMR"]=10
MAT["GoldWT"]=25
MAT["GoldBuy"]=50
MAT["GoldSell"]=25
MAT["CobaltPD"]=30
MAT["CobaltMD"]=10
MAT["CobaltAC"]=0
MAT["CobaltPR"]=10
MAT["CobaltMR"]=0
MAT["CobaltWT"]=25
MAT["CobaltBuy"]=50
MAT["CobaltSell"]=25
MAT["TungstenPD"]=20
MAT["TungstenMD"]=20
MAT["TungstenAC"]=0
MAT["TungstenPR"]=7
MAT["TungstenMR"]=7
MAT["TungstenWT"]=20
MAT["TungstenBuy"]=70
MAT["TungstenSell"]=35
WEP={}
WEP["KnifePD"]=0.5
WEP["KnifeMD"]=0.5
WEP["KnifeAC"]=1.25
WEP["SwordPD"]=1.0
WEP["SwordMD"]=1.0
WEP["SwordAC"]=1.0
WEP["GreatswordPD"]=1.67
WEP["GreatswordMD"]=0.67
WEP["GreatswordAC"]=0.5
WEP["PolearmPD"]=1.15
WEP["PolearmMD"]=1.15
WEP["PolearmAC"]=1.15
WEP["CanePD"]=1.15
WEP["CaneMD"]=1.15
WEP["CaneAC"]=0.7
WEP["ClawPD"]=1.1
WEP["ClawMD"]=1.1
WEP["ClawAC"]=0.8
WEP["BattlestaffPD"]=1.15
WEP["BattlestaffMD"]=1
WEP["BattlestaffAC"]=1.25
WEP["TalisPD"]=1.15
WEP["TalisMD"]=0.7
WEP["TalisAC"]=1.15
WEP["WandPD"]=0.0
WEP["WandMD"]=1
WEP["WandAC"]=1.33
WEP["RodPD"]=0.0
WEP["RodMD"]=1.67
WEP["RodAC"]=0.67
WEP["SlicerPD"]=0.67
WEP["SlicerMD"]=0.67
WEP["SlicerAC"]=0.67
WEP["BowPD"]=1.15
WEP["BowMD"]=1.15
WEP["BowAC"]=0.85
WEP["CrossbowPD"]=1.4
WEP["CrossbowMD"]=1.4
WEP["CrossbowAC"]=1
WEP["PistolPD"]=0.65
WEP["PistolMD"]=0.65
WEP["PistolAC"]=1.15
WEP["MechgunPD"]=0.2
WEP["MechgunMD"]=0.2
WEP["MechgunAC"]=1.5
WEP["ShotgunPD"]=1.3
WEP["ShotgunMD"]=1.3
WEP["ShotgunAC"]=0.4
WEP["RiflePD"]=0.75
WEP["RifleMD"]=0.75
WEP["RifleAC"]=1.75
WEP["HandbowPD"]=0.8
WEP["HandbowMD"]=0.8
WEP["HandbowAC"]=1.2
UP={}
UP["0PD"]=1.0
UP["1PD"]=1.1
UP["2PD"]=1.2
UP["3PD"]=1.3
UP["4PD"]=1.4
UP["5PD"]=1.5
UP["6PD"]=1.6
UP["7PD"]=1.7
UP["8PD"]=1.8
UP["9PD"]=1.9
UP["10PD"]=2.0
UP["0MD"]=1.0
UP["1MD"]=1.1
UP["2MD"]=1.2
UP["3MD"]=1.3
UP["4MD"]=1.4
UP["5MD"]=1.5
UP["6MD"]=1.6
UP["7MD"]=1.7
UP["8MD"]=1.8
UP["9MD"]=1.9
UP["10MD"]=2.0
UP["0AC"]=1.0
UP["1AC"]=1.1
UP["2AC"]=1.2
UP["3AC"]=1.3
UP["4AC"]=1.4
UP["5AC"]=1.5
UP["6AC"]=1.6
UP["7AC"]=1.7
UP["8AC"]=1.8
UP["9AC"]=1.9
UP["10AC"]=2.0
UP["0PR"]=1.0
UP["1PR"]=1.1
UP["2PR"]=1.2
UP["3PR"]=1.3
UP["4PR"]=1.4
UP["5PR"]=1.5
UP["6PR"]=1.6
UP["7PR"]=1.7
UP["8PR"]=1.8
UP["9PR"]=1.9
UP["10PR"]=2.0
UP["0MR"]=1.0
UP["1MR"]=1.1
UP["2MR"]=1.2
UP["3MR"]=1.3
UP["4MR"]=1.4
UP["5MR"]=1.5
UP["6MR"]=1.6
UP["7MR"]=1.7
UP["8MR"]=1.8
UP["9MR"]=1.9
UP["10MR"]=2.0
UP["0WT"]=1.0
UP["1WT"]=0.95
UP["2WT"]=0.9
UP["3WT"]=0.85
UP["4WT"]=0.8
UP["5WT"]=0.75
UP["6WT"]=0.7
UP["7WT"]=0.65
UP["8WT"]=0.6
UP["9WT"]=0.55
UP["10WT"]=0.5
def ForgeW(Material,WeaponType,UpgradeLevel):
"""The ForgeW function Forges a Weapon from its base components into a lethal tool."""
#Get the appropriate material stats...
OrePD=MAT[Material+"PD"]
OreMD=MAT[Material+"MD"]
OreAC=MAT[Material+"AC"]
#And weapon type stats...
SmithPD=WEP[WeaponType+"PD"]
SmithMD=WEP[WeaponType+"MD"]
SmithAC=WEP[WeaponType+"AC"]
#And apply the upgrade...
UpgradePD=UP[UpgradeLevel+"PD"]
UpgradeMD=UP[UpgradeLevel+"MD"]
UpgradeAC=UP[UpgradeLevel+"AC"]
#Then, add them all together.
ProductPD=(OrePD*SmithPD)*UpgradePD
ProductMD=(OreMD*SmithMD)*UpgradeMD
ProductAC=(OreAC*SmithAC)*UpgradeAC
return(ProductPD,ProductMD,ProductAC)
#Recall that ForgeW simply needs its three inputs, which we have a list of. So, let's make our
#database of weapon information.
OmniWeapData={}
#Go through every set of inputs we have...
for Inputs in ForgeWInputs:
#And create a key in the dictionary by combining their three names. Then, set that
#key equal to whatever ForgeW returns when those three inputs are put in.
OmniWeapData[Inputs[0]+Inputs[1]+Inputs[2]] = ForgeW(Inputs[0],Inputs[1],Inputs[2])
I would like to refer to the database created by this code and pull out weapons at random, and frankly I have no idea how. As an example of what I would like to do...
Well, hum. The code in question should spit out a certain number of results based on the complete products of the ForgeW function - if I specify, either within the code or through an input, that I would like 3 outputs, it might output a GoldKnife0, a TinPolearm5, and a CobaltGreatsword10. If I were to run the code again, it should dispense new equipment - not the same three every time.
I apologize if this is too much or too little data - it's my first time asking a question here.
"Take this... it may help you on your quest."
There is a library called random with a method called choice().
e.g.
import random
random.choice([1,2,3])
>>> 2
It sounds like you need one item from Materials, one item from WeaponTypes, and one from Upgrades.
Also, rarely is there ever a need for a triple nested FOR statement. This should get you started.

Reformat csv file using python?

I have this csv file with only two entries. Here it is:
Meat One,['Abattoirs', 'Exporters', 'Food Delivery', 'Butchers Retail', 'Meat Dealers-Retail', 'Meat Freezer', 'Meat Packers']
First one is title and second is a business headings.
Problem lies with entry two.
Here is my code:
import csv
with open('phonebookCOMPK-Directory.csv', "rt") as textfile:
reader = csv.reader(textfile)
for row in reader:
row5 = row[5].replace("[", "").replace("]", "")
listt = [(''.join(row5))]
print (listt[0])
it prints:
'Abattoirs', 'Exporters', 'Food Delivery', 'Butchers Retail', 'Meat Dealers-Retail', 'Meat Freezer', 'Meat Packers'
What i need to do is that i want to create a list containing these words and then print them like this using for loop to print every item separately:
Abattoirs
Exporters
Food Delivery
Butchers Retail
Meat Dealers-Retail
Meat Freezer
Meat Packers
Actually I am trying to reformat my current csv file and clean it so it can be more precise and understandable.
Complete 1st line of csv is this:
Meat One,+92-21-111163281,Al Shaheer Corporation,Retailers,2008,"['Abattoirs', 'Exporters', 'Food Delivery', 'Butchers Retail', 'Meat Dealers-Retail', 'Meat Freezer', 'Meat Packers']","[[' Outlets Address : Shop No. Z-10, Station Shopping Complex, MES Market, Malir-Cantt, Karachi. Landmarks : MES Market, Station Shopping Complex City : Karachi UAN : +92-21-111163281 '], [' Outlets Address : Shop 13, Ground Floor, Plot 14-D, Sky Garden, Main Tipu Sultan Road, KDA Scheme No.1, Karachi. Landmarks : Nadra Chowrangi, Sky Garden, Tipu Sultan Road City : Karachi UAN : +92-21-111163281 '], ["" Outlets Address : Near Jan's Broast, Boat Basin, Khayaban-e-Roomi, Block 5, Clifton, Karachi. Landmarks : Boat Basin, Jans Broast, Khayaban-e-Roomi City : Karachi UAN : +92-21-111163281 View Map ""], [' Outlets Address : Gulistan-e-Johar, Karachi. Landmarks : Perfume Chowk City : Karachi UAN : +92-21-111163281 '], [' Outlets Address : Tee Emm Mart, Creek Vista Appartments, Khayaban-e-Shaheen, Phase VIII, DHA, Karachi. Landmarks : Creek Vista Appartments, Nueplex Cinema, Tee Emm Mart, The Place City : Karachi Mobile : 0302-8333666 '], [' Outlets Address : Y-Block, DHA, Lahore. Landmarks : Y-Block City : Lahore UAN : +92-42-111163281 '], [' Outlets Address : Adj. PSO, Main Bhittai Road, Jinnah Supermarket, F-7 Markaz, Islamabad. Landmarks : Bhittai Road, Jinnah Super Market, PSO Petrol Pump City : Islamabad UAN : +92-51-111163281 ']]","Agriculture, fishing & Forestry > Farming equipment & services > Abattoirs in Pakistan"
First column is Name
Second column is Number
Third column is Owner
Forth column is Business type
Fifth column is Y.O.E
Sixth column is Business Headings
Seventh column is Outlets (List of lists containing every branch address)
Eighth column is classification
There is no restriction of using csv.reader, I am open to any technique available to clean my file.
Think of it in terms of two separate tasks:
Collect some data items from a ‘dirty’ source (this CSV file)
Store that data somewhere so that it’s easy to access and manipulate programmatically (according to what you want to do with it)
Processing dirty CSV
One way to do this is to have a function deserialize_business() to distill structured business information from each incoming line in your CSV. This function can be complex because that’s the nature of the task, but still it’s advisable to split it into self-containing smaller functions (such as get_outlets(), get_headings(), and so on). This function can return a dictionary but depending on what you want it can be a [named] tuple, a custom object, etc.
This function would be an ‘adapter’ for this particular CSV data source.
Example of deserialization function:
def deserialize_business(csv_line):
"""
Distills structured business information from given raw CSV line.
Returns a dictionary like {name, phone, owner,
btype, yoe, headings[], outlets[], category}.
"""
pieces = [piece.strip("[[\"\']] ") for piece in line.strip().split(',')]
name = pieces[0]
phone = pieces[1]
owner = pieces[2]
btype = pieces[3]
yoe = pieces[4]
# after yoe headings begin, until substring Outlets Address
headings = pieces[4:pieces.index("Outlets Address")]
# outlets go from substring Outlets Address until category
outlet_pieces = pieces[pieces.index("Outlets Address"):-1]
# combine each individual outlet information into a string
# and let ``deserialize_outlet()`` deal with that
raw_outlets = ', '.join(outlet_pieces).split("Outlets Address")
outlets = [deserialize_outlet(outlet) for outlet in raw_outlets]
# category is the last piece
category = pieces[-1]
return {
'name': name,
'phone': phone,
'owner': owner,
'btype': btype,
'yoe': yoe,
'headings': headings,
'outlets': outlets,
'category': category,
}
Example of calling it:
with open("phonebookCOMPK-Directory.csv") as f:
lineno = 0
for line in f:
lineno += 1
try:
business = deserialize_business(line)
except:
# Bad line formatting?
log.exception(u"Failed to deserialize line #%s!", lineno)
else:
# All is well
store_business(business)
Storing the data
You’ll have the store_business() function take your data structure and write it somewhere. Maybe it’ll be another CSV that’s better structured, maybe multiple CSVs, a JSON file, or you can make use of SQLite relational database facilities since Python has it built-in.
It all depends on what you want to do later.
Relational example
In this case your data would be split across multiple tables. (I’m using the word “table” but it can be a CSV file, although you can as well make use of an SQLite DB since Python has that built-in.)
Table identifying all possible business headings:
business heading ID, name
1, Abattoirs
2, Exporters
3, Food Delivery
4, Butchers Retail
5, Meat Dealers-Retail
6, Meat Freezer
7, Meat Packers
Table identifying all possible categories:
category ID, parent category, name
1, NULL, "Agriculture, fishing & Forestry"
2, 1, "Farming equipment & services"
3, 2, "Abattoirs in Pakistan"
Table identifying businesses:
business ID, name, phone, owner, type, yoe, category
1, Meat One, +92-21-111163281, Al Shaheer Corporation, Retailers, 2008, 3
Table describing their outlets:
business ID, city, address, landmarks, phone
1, Karachi UAN, "Shop 13, Ground Floor, Plot 14-D, Sky Garden, Main Tipu Sultan Road, KDA Scheme No.1, Karachi", "Nadra Chowrangi, Sky Garden, Tipu Sultan Road", +92-21-111163281
1, Karachi UAN, "Near Jan's Broast, Boat Basin, Khayaban-e-Roomi, Block 5, Clifton, Karachi", "Boat Basin, Jans Broast, Khayaban-e-Roomi", +92-21-111163281
Table describing their headings:
business ID, business heading ID
1, 1
1, 2
1, 3
…
Handling all this would require a complex store_business() function. It may be worth looking into SQLite and some ORM framework, if going with relational way of keeping the data.
You can just replace the line :
print(listt[0])
with :
print(*listt[0], sep='\n')

Search in directory of files based on keywords from another file

Perl Newbie here and looking for some help.
I have a directory of files and a "keywords" file which has the attributes to search for and the attribute type.
For example:
Keywords.txt
Attribute1 boolean
Attribute2 boolean
Attribute3 search_and_extract
Attribute4 chunk
For each file in the directory, I have to:
lookup the keywords.txt
search based on Attribute type
something like the below.
IF attribute_type = boolean THEN
search for attribute;
set found = Y if attribute found;
ELSIF attribute_type = search_and_extract THEN
extract string where attribute is Found
ELSIF attribute_type = chunk THEN
extract the complete chunk of paragraph where attribute is found.
This is what I have so far and I'm sure there is a more efficient way to do this.
I'm hoping someone can guide me in the right direction to do the above.
Thanks & regards,
SiMa
# Reads attributes from config file
# First set boolean attributes. IF keyword is found in text,
# variable flag is set to Y else N
# End Code: For each text file in directory loop.
# Run the below for each document.
use strict;
use warnings;
# open Doc
open(DOC_FILE,'Final_CLP.txt');
while(<DOC_FILE>) {
chomp;
# open the file
open(FILE,'attribute_config.txt');
while (<FILE>) {
chomp;
($attribute,$attribute_type) = split("\t");
$is_boolean = ($attribute_type eq "boolean") ? "N" : "Y";
# For each boolean attribute, check if the keyword exists
# in the file and return Y or N
if ($is_boolean eq "Y") {
print "Yes\n";
# search for keyword in doc and assign values
}
print "Attribute: $attribute\n";
print "Attribute_Type: $attribute_type\n";
print "is_boolean: $is_boolean\n";
print "-----------\n";
}
close(FILE);
}
close(DOC_FILE);
exit;
It is a good idea to start your specs/question with a story ("I have a ..."). But
such a story - whether true or made up, because you can't disclose the truth -
should give
a vivid picture of the situation/problem/task
the reason(s) why all the work must be done
definitions for uncommon(ly used)terms
So I'd start with: I'm working in a prison and have to scan the emails
of the inmates for
names (like "Al Capone") mentioned anywhere in the text; the director
wants to read those mails in toto
order lines (like "weapon: AK 4711 quantity: 14"); the ordnance
officer wants those info to calculate the amount of ammunition and
rack space needed
paragraphs containing 'family'-keywords like "wife", "child", ...;
the parson wants to prepare her sermons efficiently
Taken for itself, each of the terms "keyword" (~running text) and
"attribute" (~structured text) of may be 'clear', but if both are applied
to "the X I have to search for", things get mushy. Instead of general ("chunk")
and technical ("string") terms, you should use 'real-world' (line) and
specific (paragraph) words. Samples of your input:
From: Robin Hood
To: Scarface
Hi Scarface,
tell Al Capone to send a car to the prison gate on sunday.
For the riot we need:
weapon: AK 4711 quantity: 14
knife: Bowie quantity: 8
Tell my wife in Folsom to send some money to my son in
Alcatraz.
Regards
Robin
and your expected output:
--- Robin.txt ----
keywords:
Al Capone: Yes
Billy the Kid: No
Scarface: Yes
order lines:
knife:
knife: Bowie quantity: 8
machine gun:
stinger rocket:
weapon:
weapon: AK 4711 quantity: 14
social relations paragaphs:
Tell my wife in Folsom to send some money to my son in
Alcatraz.
Pseudo code should begin at the top level. If you start with
for each file in folder
load search list
process current file('s content) using search list
it's obvious that
load search list
for each file in folder
process current file using search list
would be much better.
Based on this story, examples, and top level plan, I would try to come
up with proof of concept code for a simplified version of the "process
current file('s content) using search list" task:
given file/text to search in and list of keywords/attributes
print file name
print "keywords:"
for each boolean item
print boolean item text
if found anywhere in whole text
print "Yes"
else
print "No"
print "order line:"
for each line item
print line item text
if found anywhere in whole text
print whole line
print "social relations paragaphs:"
for each paragraph
for each social relation item
if found
print paragraph
no need to check for other items
first implementation attempt:
use Modern::Perl;
#use English qw(-no_match_vars);
use English;
exit step_00();
sub step_00 {
# given file/text to search in
my $whole_text = <<"EOT";
From: Robin Hood
To: Scarface
Hi Scarface,
tell Al Capone to send a car to the prison gate on sunday.
For the riot we need:
weapon: AK 4711 quantity: 14
knife: Bowie quantity: 8
Tell my wife in Folsom to send some money to my son in
Alcatraz.
Regards
Robin
EOT
# print file name
say "--- Robin.txt ---";
# print "keywords:"
say "keywords:";
# for each boolean item
for my $bi ("Al Capone", "Billy the Kid", "Scarface") {
# print boolean item text
printf " %s: ", $bi;
# if found anywhere in whole text
if ($whole_text =~ /$bi/) {
# print "Yes"
say "Yes";
# else
} else {
# print "No"
say "No";
}
}
# print "order line:"
say "order lines:";
# for each line item
for my $li ("knife", "machine gun", "stinger rocket", "weapon") {
# print line item text
# if found anywhere in whole text
if ($whole_text =~ /^$li.*$/m) {
# print whole line
say " ", $MATCH;
}
}
# print "social relations paragaphs:"
say "social relations paragaphs:";
# for each paragraph
for my $para (split /\n\n/, $whole_text) {
# for each social relation item
for my $sr ("wife", "son", "husband") {
# if found
if ($para =~ /$sr/) {
## if ($para =~ /\b$sr\b/) {
# print paragraph
say $para;
# no need to check for other items
last;
}
}
}
return 0;
}
output:
perl 16953439.pl
--- Robin.txt ---
keywords:
Al Capone: Yes
Billy the Kid: No
Scarface: Yes
order lines:
knife: Bowie quantity: 8
weapon: AK 4711 quantity: 14
social relations paragaphs:
tell Al Capone to send a car to the prison gate on sunday.
Tell my wife in Folsom to send some money to my son in
Alcatraz.
Such (premature) code helps you to
clarify your specs (Should not-found keywords go into the output?
Is your search list really flat or should it be structured/grouped?)
check your assumptions about how to do things (Should the order line
search be done on the array of lines of thw whole text?)
identify topics for further research/rtfm (eg. regex (prison!))
plan your next steps (folder loop, read input file)
(in addition, people in the know will point out all my bad practices,
so you can avoid them from the start)
Good luck!

Resources