I am reading the text file consisting of bengali words. But I am unable to print the dependent vowels like KA,KI etc...
Here is my sample code and output
import unicodedata
bengali_phoneme_maplist={u'অ':'A',u'আ':'AA',u'ই':'I',u'ঈ':'II',u'উ':'U',u'ঊ ':'UU',u'ঋ ':'R',u'ঌ ':'L',u'এ ':'E',u'ঐ ':'AI',u'ও ':'O',u'ঔ ':'AU',u'ক':'KA',u'খ ':'KHA',u'গ ':'GA',u'ঘ':'GHA',u'ঙ ':'NGA',u'চ ':'CA',u'ছ':'CHA',u'জ ':'JA',u'ঝ':'JHA',u'ঞ':'NYA',u'ট ':'TTA',u'ঠ':'TTHA',u'ড ':'DDA',u'ঢ':'DDHA',u'ণ ':'NNA',u'ত ':'TA',u'ত ':'THA',u'দ':'DA',u'ধ':'DHA',u'ন':'NA',u'প':'PA',u'ফ':'PHA',u'ব':'BA',u'ভ':'BHA',u'ম ':'MA',u'য ':'YA',u'র':'RA',u'ল ':'LA',u'শ ':'SHA',u'ষ':'SSA',u'স ':'SA',u'হ':'ha',u' া ':'AAV',u' ি':'IV',u'ী':'IIV',u'ু':'UV',u'ূ':'UUV',u'ৃ':'RRV',u'ৄ ':'RR',u'ৄ':'EV',u' ৈ':'EV',u'়':'NUKTHA',u'ঽ':'AVAGRAHA'}
bengali_phoneme_maplist_normalise={unicodedata.normalize('NFKD',k):v
for k,v in bengali_phoneme_maplist.items()}
with open('bengali.txt','r')as infile:
lines=infile.readlines()
for index,line in enumerate(lines):
print('Phonemes in line{0}.total{1} symbols'.format(index,len(line)))
unknown=[]
words=line.split()
for word in words:
print(word,':',sep=' ', end='')
for character in word:
c=unicodedata.normalize('NFKD',character).casefold()
try:
print(bengali_phoneme_maplist_normalise[c],sep='',end='')
except KeyError:
print('_',sep='',end='')
if c not in unknown:
unknown.append(c)
print()
if unknown:
print('Unrecognised symbols:{0},total {1} symbols'.format(','.join(unknown),len(unknown)))
Sample input:
শিল্পাঞ্চলে ঢোকার মুখে, স্ন্যাক্সবারে খাবার কিনছিলেন, বহুজাতিক তথ্যপ্রযুক্তি সংস্থার কর্মী, শুভময় বন্দ্যোপাধ্যায়
Sample output:
Phonemes in line0.total129 symbols
text_000002 :___________
"শিল্পাঞ্চলে :_____PA_NYA____
ঢোকার :DDHA_KA_RA
মুখে, :_UV___
স্ন্যাক্সবারে :__NA___KA__BA_RA_
খাবার :__BA_RA
কিনছিলেন, :KA_NACHA___NA_
Unrecognisedsymbols:t,e,x,_,0,2,",শ,ি,ল,্,া,চ,ে,ো,ম,খ,,,স,য,জ,ত,থ,ং,য়,),
(Note that I know nohting about Bengali. :)
There are a few problems in your code:
There are many extra SPACE chars in the bengali_phoneme_maplist definition. For example, u'ঊ ' should be u'ঊ'. And it seems like it's not easy to input chars like u'া' in an text editor so I suggest you directly use unicode in the code, like '\u09be':'AAV'. (Actually I'd suggest you use '\uxxxx' for all chars and write the real chars in comments.)
u'ত':'TA',u'ত':'THA' should change to u'ত':'TA',u'থ':'THA'.
The chars in bengali_phoneme_maplist are not complete. For example there's no ো , ৌ , ্ and ং
After fixing these errors you will get the correct result.
Related
I have a large text file looking like:
....
sdsdsd
..........
asdfhjgjksdfk dfkaskk sdkfk skddkf skdf sdk ssaaa akskdf sdksdfsdf ksdf sd kkkkallwow.
sdsdllla lsldlsd lsldlalllLlsdd asdd. sdlsllall asdsdlallOEFOOASllsdl lsdlla.
slldlllasdlsd.ss;sdsdasdas.
......
ddss
................
asdfhjgjksdfk ddjafjijjjj.dfsdfsdfsdfsi dfodoof ooosdfow oaosofoodf aosolflldlfl , dskdkkfkdsa asddf;akkdfkdkk . sdlsllall asdsdlallOEFOOASllsdl lsdlla.
slldlllasdlsd.ss;sdsdasdas.
.....
xxxx
.......
asdfghjkl
I want to split the text files into multiple small text files and save them as .txt in my system on occurences of ..... [multiple period markers] saved like
group1_sdsdsd.txt
....
sdsdsd
..........
asdfhjgjksdfk dfkaskk sdkfk skddkf skdf sdk ssaaa akskdf sdksdfsdf ksdf sd kkkkallwow.
sdsdllla lsldlsd lsldlalllLlsdd asdd. sdlsllall asdsdlallOEFOOASllsdl lsdlla.
slldlllasdlsd.ss;sdsdasdas.
group1_ddss.txt
ddss
................
asdfhjgjksdfk ddjafjijjjj.dfsdfsdfsdfsi dfodoof ooosdfow oaosofoodf aosolflldlfl , dskdkkfkdsa asddf;akkdfkdkk . sdlsllall asdsdlallOEFOOASllsdl lsdlla.
slldlllasdlsd.ss;sdsdasdas.
and
group1_xxxx.txt
.....
xxxx
.......
asdfghjkl
I have figured that by usinf regex of sort of following can be done
txt =re.sub(r'(([^\w\s])\2+)', r' ', txt).strip() #for letters more than 2 times
but not able to figure out completely.
The saved text files should be named as group1_sdsdsd.txt , group1_ddss.txt and group1_xxxx.txt [group1 being identifier for the specific big text file as I have multiple bigger text files and need to do same on all to know which big text file i am splitting.
If you want to get the parts with multiple dots only on the same line, you can use and get the separate parts, you might use a pattern like:
^\.{3,}\n(\S+)\n\.{3,}(?:\n(?!\.{3,}\n\S+\n\.{3,}).*)*
Explanation
^ Start of string
\.{3,}\n Match 3 or more dots and a newline
(\S+)\n Capture 1+ non whitespace chars in group 1 for the filename and match a newline
\.{3,} Match 3 or more dots
(?: Non capture group to repeat as a whole part
\n Match a newline
(?!\.{3,}\n\S+\n\.{3,}) Negative lookahead, assert that from the current position we are not looking at a pattern that matches the dots with a filename in between
.* Match the whole line
)* Close the non capture group and optionally repeat it
Then you can use re.finditer to loop the matches, and use the group 1 value as part of the filename.
See a regex demo and a Python demo with the separate parts.
Example code
import re
pattern = r"^\.{3,}\n(\S+)\n\.{3,}(?:\n(?!\.{3,}\n\S+\n\.{3,}).*)*"
s = ("....your data here")
matches = re.finditer(pattern, s, re.MULTILINE)
your_path = "/your/path/"
for matchNum, match in enumerate(matches, start=1):
f = open(your_path + "group1_{}".format(match.group(1)), 'w')
f.write(match.group())
f.close()
i have a text file as the output of another program. I want to insert a white space above the line that starts with 'Sentence #'
this is what i currently have:
Sentence #26024 (5 tokens):
Today is a good day
[Text=Today CharacterOffsetBegin=1607176 CharacterOffsetEnd=1607178 PartOfSpeech=IN Lemma=if]
[Text=is CharacterOffsetBegin=1607179 CharacterOffsetEnd=1607181
PartOfSpeech=NN Lemma=yo]
[Text=a CharacterOffsetBegin=1607182 CharacterOffsetEnd=1607186 PartOfSpeech=NN Lemma=girl]
[Text=good CharacterOffsetBegin=1607187 CharacterOffsetEnd=1607193 PartOfSpeech=JJ Lemma=doesnt]
[Text=day CharacterOffsetBegin=1607202 CharacterOffsetEnd=1607205
root(ROOT-0, today-1)
root(today-1, is-2)
dobj(a-2, good-3)
amod(day-3, good-4)
Sentence #26025 (4 tokens):
if you can help
[Text=if CharacterOffsetBegin=1607223 CharacterOffsetEnd=1607225 PartOfSpeech=IN Lemma=if]
[Text=you CharacterOffsetBegin=1607226 CharacterOffsetEnd=1607229 PartOfSpeech=PRP Lemma=you]
[Text=can CharacterOffsetBegin=1607230 CharacterOffsetEnd=1607233 PartOfSpeech=MD Lemma=can
mark(help-4, if-1)
nsubj(help-4, you-2)
aux(help-4, can-3)
This is what i want it to look like:
Sentence #26024 (5 tokens):
Today is a good day
[Text=Today CharacterOffsetBegin=1607176 CharacterOffsetEnd=1607178 PartOfSpeech=IN Lemma=if]
[Text=is CharacterOffsetBegin=1607179 CharacterOffsetEnd=1607181
PartOfSpeech=NN Lemma=yo]
[Text=a CharacterOffsetBegin=1607182 CharacterOffsetEnd=1607186 PartOfSpeech=NN Lemma=girl]
[Text=good CharacterOffsetBegin=1607187 CharacterOffsetEnd=1607193 PartOfSpeech=JJ Lemma=doesnt]
[Text=day CharacterOffsetBegin=1607202 CharacterOffsetEnd=1607205
root(ROOT-0, today-1)
root(today-1, is-2)
dobj(a-2, good-3)
amod(day-3, good-4)
Sentence #26025 (4 tokens):
if you can help
[Text=if CharacterOffsetBegin=1607223 CharacterOffsetEnd=1607225 PartOfSpeech=IN Lemma=if]
[Text=you CharacterOffsetBegin=1607226 CharacterOffsetEnd=1607229 PartOfSpeech=PRP Lemma=you]
[Text=can CharacterOffsetBegin=1607230 CharacterOffsetEnd=1607233 PartOfSpeech=MD Lemma=can
mark(help-4, if-1)
nsubj(help-4, you-2)
aux(help-4, can-3)
Can anyone please provide pointers. Thanks
I can't do it manually because it's a large file that will need thoussand of spaces inserted.
This is what i did, in case anyone else has a similar problem.
file = open("nameofoldfile.txt", 'r')
filelines = file.readlines()
for lines in filelines:
lines = lines.strip()
if lines.startswith('Sentence #'):
print('\n')
print(lines)
else:
print(lines)
then i saved the file to a new text file by running on the command prompt
python nameoffile.py > nameoftextfile.txt
I would like to order this list.
From:
01104D-BB'42
01104D-BB42
01104D-BB43
01104D-CC'42
01104D-CC'72
01104D-CC32
01104D-CC42
01104D-CC62
01104D-CC72
01104D-DD'74
01104D-DD'75
01104D-DD'76
01104D-DD'77
01104D-DD'78
01104D-DD75
01104D-DD76
01104D-DD77
01104D-DD78
01104D-EE'102
01104D-EE'12
01104D-EE'2
01104D-EE'32
01104D-EE'42
01104D-EE'52
01104D-EE'53
01104D-EE'72
01104D-EE'82
01104D-EE'92
01104D-EE102
01104D-EE12
01104D-EE2
01104D-EE3
01104D-EE32
01104D-EE42
01104D-EE52
01104D-EE62
01104D-EE72
01104D-EE82
01104D-EE83
01104D-EE92
01104D-EE93
To:
01104D-BB42
01104D-BB43
01104D-BB'42
01104D-CC32
01104D-CC42
01104D-CC62
01104D-CC72
01104D-CC'42
01104D-CC'72
01104D-DD75
01104D-DD76
01104D-DD77
01104D-DD78
01104D-DD'74
01104D-DD'75
01104D-DD'76
01104D-DD'77
01104D-DD'78
01104D-EE102
01104D-EE12
01104D-EE2
01104D-EE3
01104D-EE32
01104D-EE42
01104D-EE52
01104D-EE62
01104D-EE72
01104D-EE82
01104D-EE83
01104D-EE92
01104D-EE93
01104D-EE'102
01104D-EE'12
01104D-EE'2
01104D-EE'32
01104D-EE'42
01104D-EE'52
01104D-EE'53
01104D-EE'72
01104D-EE'82
01104D-EE'92
Can you help me?
thanks
I'm guessing here, because you haven't explained how you want the sort to be done. But it looks like you want the character ' to sort after the digits 0-9, and the ascii sort order puts it before the digits. If that is correct, then you need to substitute a different character for '. A good choice might be ~ because it is the last printable ascii character.
If your data is in mylist, then
mylist.sort(key=lambda a: a.replace("'","~"))
will sort it in the order I'm guessing you want.
In my code I have the following line:
fprintf(logfile,'Parameters: Size: %d\tH: %.4f\tF: %.1f\tI: %.3f\tR: %d\tSigma: %d\tDisp: %.1f\r\n',parameter_sets(ps,:));
which is too long, so I want to break it to:
fprintf(logfile,'Parameters: Size: %d\tH: %.4f\tF: %.1f\tI: %.3f\tR: ...
%d\tSigma: %d\tDisp: %.1f\r\n',parameter_sets(ps,:));
However, since the brake is within a string, MATLAB see the formatting %d sign in the second line as a start of a comment, and ignore this line (and produce an error...).
So I tried to make it clearer with a [] that warp the string:
fprintf(logfile,['Parameters: Size: %d\tH: %.4f\tF: %.1f\tI: %.3f\tR: ...
%d\tSigma: %d\tDisp: %.1f\r\n'],parameter_sets(ps,:));
but no help, it still interpret the second line as a comment. I also tried with and without the ellipsis (...) in different places, with no success.
So how can I write a line in a formatted way (i.e. a reasonable length) if it has a % sign in it?
Divide it in two lines like this:
fprintf(logfile,['Parameters: Size: %d\tH: %.4f\tF: %.1f\tI: %.3f\tR:', ...
'%d\tSigma: %d\tDisp: %.1f\r\n'],parameter_sets(ps,:));
% notice the apostrophe and comma(',) before ellpsis(...) at the end of first line
% and apostrophe(') at the start of the second line
Just when I thought I had my head wrapped around converting unicode to strings Python 2.7 throws an exception.
The code below loops over a number of accented characters and converts them to their non-accented equivalents. I've put in an special case for the double s.
#!/usr/bin/python
# -*- coding: utf-8 -*-
import unicodedata
def unicodeToString(uni):
return unicodedata.normalize("NFD", uni).encode("ascii", "ignore")
accentList = [
#(grave accent)
u"à",
u"è",
u"ì",
u"ò",
u"ù",
u"À",
u"È",
u"Ì",
u"Ò",
u"Ù",
#(acute accent)
u"á",
u"é",
u"í",
u"ó",
u"ú",
u"ý",
u"Á",
u"É",
u"Í",
u"Ó",
u"Ú",
u"Ý",
#(arrete accent)
u"â",
u"ê",
u"î",
u"ô",
u"û",
u"Â",
u"Ê",
u"Î",
u"Ô",
u"Û",
#(tilde )
u"ã",
u"ñ",
u"õ",
u"Ã",
u"Ñ",
u"Õ",
#(diaresses)
u"ä",
u"ë",
u"ï",
u"ö",
u"ü",
u"ÿ",
u"Ä",
u"Ë",
u"Ï",
u"Ö",
u"Ü",
u"Ÿ",
#ring
u"å",
u"Å",
#ae ligature
u"æ",
u"Æ",
#oe ligature
u"œ",
u"Œ",
#c cidilla
u"ç",
u"Ç",
# D stroke?
u"ð",
u"Ð",
# o slash
u"ø",
u"Ø",
u"¿", # Spanish ?
u"¡", # Spanish !
u"ß" # Double s
]
for i in range(0, len(accentList)):
try:
u = accentList[i]
s = unicodeToString(u)
if u == u"ß":
s = "ss"
print("%s -> %s" % (u, s))
except:
pass
Without the try/except I get an error:
File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xc0' in position 0
: character maps to <undefined>
Is there anything I can do to make the code run without using the try/except? I'm using Sublime Text 2.
try/except does not make Unicode work. It just hides errors.
To fix the UnicodeEncodeError error, drop try/except and see Python, Unicode, and the Windows console.