How to tokenize python code using the Tokenize module? - python-3.x

Question

Consider that I have a string that contains the python code.
input = "import nltk
from nltk.stem import PorterStemmer
porter_stemmer=PorterStemmer()
words=["connect","connected","connection","connections","connects"]
stemmed_words=[porter_stemmer.stem(word) for word in words]
stemmed_words"
How can I tokenize the code? I found the tokenize module (https://docs.python.org/3/library/tokenize.html). However, it is not clear to me how to use the module. It has tokenize.tokenize(readline) but the parameter takes a generator, not a string.

Answer 1

import tokenize
import io
inp = """import nltk
from nltk.stem import PorterStemmer
porter_stemmer=PorterStemmer()
words=["connect","connected","connection","connections","connects"]
stemmed_words=[porter_stemmer.stem(word) for word in words]
stemmed_words"""
for token in tokenize.generate_tokens(io.StringIO(inp).readline):
print(token)
tokenize.tokenize takes a method not a string. The method should be a readline method from an IO object.
In addition, tokenize.tokenize expects the readline method to return bytes, you can use tokenize.generate_tokens instead to use a readline method that returns strings.
Your input should also be in a docstring, as it is multiple lines long.
See io.TextIOBase, tokenize.generate_tokens for more info.

Answer 2

If you want to stick with tokenize.tokenize(), then this is what you can do:
from tokenize import tokenize
from io import BytesIO
code = """import nltk
from nltk.stem import PorterStemmer
porter_stemmer=PorterStemmer()
words=["connect","connected","connection","connections","connects"]
stemmed_words=[porter_stemmer.stem(word) for word in words]
stemmed_words"""
for tok in tokenize(BytesIO(code.encode('utf-8')).readline):
print(f"Type: {tok.type}\nString: {tok.string}\nStart: {tok.start}\nEnd: {tok.end}\nLine: {tok.line.strip()}\n======\n")
From the documentation you can see:
The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed (the last tuple item) is the physical line. The 5 tuple is returned as a named tuple with the field names: type string start end line.
The returned named tuple has an additional property named exact_type that contains the exact operator type for OP tokens. For all other token types exact_type equals the named tuple type field.