I am a newbie in programming and have a question:
I try to edit some .vtt files, where I want to remove certain substrings from the text. The file should keep its structure. For this, I copied the .vtt files in the folder and changed it to a .txt ending. Now I run this simple code:
import os
file_index = 0
all_text = []
path = "/Users/username/Documents/programming/IMS/Translate/files/"
new_path = "/Users/username/Documents/programming/IMS/Translate/new_files/"
for filename in os.listdir(path):
if os.path.isfile(filename): #check if there is a file in the directory
with open(os.path.join(path, filename), 'r') as file: # open in read-only mode
for line in file.read().split("\n"): #read lines and split
line = " ".join(line.split())
start_index = line.find("[") #find the first character of string to remove, this returns the index number
last_index = start_index + 11 #define the last index to be removed
if start_index != -1:
line = line[:start_index] + line[last_index:] #The new line to slice the first charaters until the one to be removed, and add the others that need to stay
all_text.append(line)
else:
line = line[:]
all_text.append(line)'''
I get this error message:
> File "srt-files-strip.py", line 11, in <module>
> for line in file.read().split("\n"): #read lines and split File "/usr/local/Cellar/python#3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/codecs.py", line 322, in decode
> (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position
> 3131: invalid start byte
I have search through different forums, changed to encoding="utf16", but to no avail. Strange thing is that it did work earlier on. Then I wrote a program to rename my files automatically, after that, it threw this error. I have cleared all files in the folder, copied the original ones in again ... can't get it to work. Would really appreciate your help, as I have really no idea where to look. Thx
Related
I have a set of UTF-8 texts I have scraped from web pages. I am trying to extract keywords from these files like so:
import os
import json
from rake_nltk import Rake
rake_nltk_var = Rake()
directory = 'files'
results = {}
for filename in os.scandir(directory):
if filename.is_file():
with open("files/" + filename.name, encoding="utf-8", mode = 'r') as infile:
text = infile.read()
rake_nltk_var.extract_keywords_from_text(text)
keyword_extracted = rake_nltk_var.get_ranked_phrases()
results[filename.name] = keyword_extracted
with open("extracted-keywords.json", "w") as outfile:
json.dump(results, outfile)
One of the files I've managed to process so far is throwing the following error on read:
Traceback (most recent call last):
File "extract-keywords.py", line 11, in <module>
text = infile.read()
File "c:\python36\lib\codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 66: invalid start byte
0x92 is a right single quotation mark, but the 66th char of the file is a "u" so IDK where this error is coming from. Regardless, is there some way to make the codec tolerate such encoding errors? For example, Perl simply substitutes a question mark for any character it can't decode. Is there some way to get Python to do the same? I have a lot of files and can't afford to stop and debug every encoding error they might contain.
I have a set of UTF-8 texts I have scraped from web pages
If they can't be read with the script you've shown, then these are not actually UTF-8 encoded files.
We have to know about the code which wrote the files in the first place to tell the correct way to decode. However, the ’ character is 0x92 byte in code page 1252, so try using that encoding instead, i.e.:
with open("files/" + filename.name, encoding="cp1252") as infile:
text = infile.read()
Ignoring decoding errors corrupts the data, so it's best to use the correct decoder when possible, so try and do that first! However, about this part of the question:
Regardless, is there some way to make the codec tolerate such encoding errors? For example, Perl simply substitutes a question mark for any character it can't decode. Is there some way to get Python to do the same?
Yes, you can specify errors="replace"
>>> with open("/tmp/f.txt", "w", encoding="cp1252") as f:
... f.write('this is a right quote: \N{RIGHT SINGLE QUOTATION MARK}')
...
>>> with open("/tmp/f.txt", encoding="cp1252") as f:
... print(f.read()) # using correct encoding
...
this is a right quote: ’
>>> with open("/tmp/f.txt", encoding="utf-8", errors="replace") as f:
... print(f.read()) # using incorrect encoding and replacing errors
this is a right quote: �
I am training a word2vec model, using about 700 text files as my corpus. But, when I start reading the files after the preprocessing step, I get the mentioned error. The code is as follows
class MyCorpus(object):
def __iter__(self):
for i in ceo_path: /// ceo_path contains abs path of all text files
file = open(i, 'r', encoding='utf-8')
text = file.read()
###########
########### /// text preprocessing steps
###########
yield final_text /// returns preprocessed text
sentences = MyCorpus()
logging.basicConfig(format="%(levelname)s - %(asctime)s: %(message)s", datefmt= '%H:%M:%S', level=logging.INFO)
# training the model
cores = multiprocessing.cpu_count()
w2v_model = Word2Vec(min_count=5,
iter=30,
window=3,
size=200,
sample=6e-5,
alpha=0.025,
min_alpha=0.0001,
negative=20,
workers=cores-1,
sg=1)
w2v_model.build_vocab(sentences)
w2v_model.train(sentences, total_examples=w2v_model.corpus_count, epochs=30, report_delay=1)
w2v_model.save('ceo1.model')
The error that I am getting is:
Traceback (most recent call last):
File "C:/Users/name/PycharmProjects/prac2/hbs_word2vec.py", line 131, in <module>
w2v_model.build_vocab(sentences)
File "C:\Users\name\PycharmProjects\prac1\venv\lib\site-packages\gensim\models\base_any2vec.py", line 921, in build_vocab
total_words, corpus_count = self.vocabulary.scan_vocab(
File "C:\Users\name\PycharmProjects\prac1\venv\lib\site-packages\gensim\models\word2vec.py", line 1403, in scan_vocab
total_words, corpus_count = self._scan_vocab(sentences, progress_per, trim_rule)
File "C:\Users\name\PycharmProjects\prac1\venv\lib\site-packages\gensim\models\word2vec.py", line 1372, in _scan_vocab
for sentence_no, sentence in enumerate(sentences):
File "C:/Users/name/PycharmProjects/prac2/hbs_word2vec.py", line 65, in __iter__
text = file.read()
File "C:\Users\name\AppData\Local\Programs\Python\Python38-32\lib\codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
I am not able to understand the error as I am new to this. I was not getting the error in reading the text files when I wasn't using the iter function and sending the data in chunks as I am doing currently.
It looks like one of your files doesn't have proper utf-8-encoded text.
(Your Word2Vec-related code probably isn't necessary for hitting the error, at all. You could probably trigger the same error with just: sentences_list = list(MyCorpus()).)
To find which file, two different possibilities might be:
Change your MyCorpus class so that it prints the path of each file before it tries to read it.
Add a Python try: ... except UnicodeDecodeError: ... statement around the read, and when the exception is caught, print the offending filename.
Once you know the file involved, you may want to fix the file, or change the code to be able to handle the files you have.
Maybe they're not really in utf-8 encoding, in which case you'd specify a different encoding.
Maybe just one or a few have problems, and it's be OK to just print their names for later investigation, and skip them. (You could use the exception-handling approach above to do that.)
Maybe, those that aren't utf-8 are always in some other platform-specific encoding, so when utf-8 fails, you could try a 2nd encoding.
Separately, when you solve the encoding issue, your iterable MyCorpus is not yet returning whet the Word2Vec class expects.
It doesn't want full text plain strings. It needs those texts to already be broken up into individual word-tokens.
(Often, simply performing a .split() on a string is close-enough-to-real-tokenization to try as a starting point, but usually, projects use some more-sophisticated punctuation-aware tokenization.)
I have written the following script, in order to extract the text of a PDF file into plain text and save it into a TXT file:
import PyPDF2
def pdfToTxt(pdfFile):
pdfFileObject = open(pdfFile, 'rb')
pdfReader = PyPDF2.PdfFileReader(pdfFileObject)
numberOfPages = pdfReader.numPages
tempFile = open(r"temp.txt","a")
for p in range(numberOfPages):
pagesObject = pdfReader.getPage(p)
text = pagesObject.extractText()
tempFile.writelines(text)
tempFile.close()
pdfToTxt("PdfFile.pdf")
The code works fine for the first 15 pages, which are successfully written in temp.txt file, but after the 15th page I get the following error:
Traceback (most recent call last):
File "PdfToTextExtractor.py", line 35, in <module>
pdfToTxt("PdfFile.pdf")
File "PdfToTextExtractor.py", line 30, in pdfToTxt
tempFile.writelines(text)
File "C:\ProgramData\Anaconda3\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\ufb01' in position 0: characte
r maps to <undefined>
It seems that the character '\ufb01' is the problem.
In case you have any idea how to overcome this issue, please let me know.
In order to overcome this issue, you have to replace the character with another one (let's say a white space), before you write it into the file.
In that case you have to add the following line in the for loop:
text = text.replace('\ufb01', " ")
the method should look like this:
def pdfToTxt(pdfFile):
pdfFileObject = open(pdfFile, 'rb')
pdfReader = PyPDF2.PdfFileReader(pdfFileObject)
numberOfPages = pdfReader.numPages
tempFile = open(r"temp.txt","a")
for p in range(numberOfPages):
pagesObject = pdfReader.getPage(p)
text = pagesObject.extractText()
text = text.replace('\ufb01', " ")
tempFile.writelines(text)
tempFile.close()
When opening your tempFile, set the encoding like so:
tempFile = open(r"temp.txt","a", encoding='utf-8')
The issue is in the way you open file, so replace
tempFile = open(r"temp.txt","a")
With the same open + extra param:
tempFile = open(r"temp.txt","a", encoding="utf-8")
Additionally, I suggest you to use context manager in case of any file operations, which ensures that file will be closed correctly in case of unexpected exception:
with open(r"temp.txt","a") as tempFile:
...
Also, if you do so, you can remove file closing after for loop.
I'm doing a course of python on py4e, almost done, but chapter 11 seems like impossible because it gives me error every time.
Error:
line 4, in <module>
lines = ffail.read()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 8: invalid continuation byte
Code:
import re
ffail = open('regex_sum_340933.txt')
lines = ffail.read()
count = 0
match = re.findall('[0-9]+', lines)
for II in match:
number = int(II)
count = count + number
print(count)
Try this:
import re
lines = open('regex_sum_340933.txt', encoding='utf-8', errors='ignore').read()
count = sum(map(int, re.findall('[0-9]+', lines)))
You are not doing it right. first of all you need to close the file I would suggest
to just use with so you won't need to worry about closing the file.
replace how you read the file with this
ffail = ""
with open("regex_sum_340933.txt", mode = "r" ,encoding='UTF-8', errors='ignore', buffering=-1) as some_file:
ffail = some_file.read()
make sure that regex_sum_340933.txt is in the same directory as the file of the code.
if you are still having difficulties you could visit this question
thanks for helping, the code wasn't wrong, just my Mac didn't want to make it work. Tried with windows and the answer came immediately.
I am trying to scrape geo locations based on the urls and after about 500 searches and extraction of geo location I am getting encoding error. I have included the encoding utf-8 in the code and also followed the following command in the cmd.
chcp 65001
set PYTHONIOENCODING=utf-8
And yet I am getting the following error :
Traceback (most recent call last):
File "__main__.py", line 33, in <module>
outputfile.write(newline)
File "C:\Program Files\Anaconda3\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\ufe0f' in position 62: character maps to <undefined>
I am using Python 3.x updated version on anaconda of all the packages.
#!/usr/bin/python
import sys
from twitter_location import get_location
from google_location import get_coordinates
# Open output file
outputfile = open(sys.argv[2], 'w')
# Read input file
with open(sys.argv[1], 'r', encoding = "utf-8", errors='ignore') as csv:
# Skip headers line
next(csv)
# Loop in lines
for line in csv:
# Extract userid
print (line)
permalink = line.split(',')[-1].strip()
userid = permalink.split('/')[3]
# Get location as string if exists
location = get_location(userid)
if location is None:
print ('user {} can not be reached or do not exposes any location.'.format(userid))
continue
else:
# If location is ok, get coordinates
coordinates = get_coordinates(location)
print ('{}: {}'.format(userid, coordinates))
# Copy current input line and add coordinates at the end
newline = '{},{}\n'.format(line.strip(), coordinates)
# Write in output file
outputfile.write(newline)
I am looking for two things here
Help with the encoding error
I want to put back the input headers in the output + the new column header
My input files have following headers
username date retweets favorites text geo mentions hashtags id permalink
while writing the output I am able to get all the columns + the new geo coordinates column too. But I am not able to put back the headers in the output files.
Appreciate the help, Thanks in advance.