How can I add a feature using torchtext? - nlp

Question

torchtext is able to read a file with some columns, each one corresponding to a field. What if I want to create a new column (which I will use as a feature)? For example, imagine the file has two columns, text and target, and I want to extract some information from the text and generate a new feature (e.g. if it contains certain words), can I do this directly with torchtext or do I need to do it in the file before?
Thanks!

Answer 1

It can be done.
def postprocessing(arr,vocab,pad_token):
# required to pad the sequence
max_len = max([len(a) for a in arr])
l = []
for a in arr:
res = max_len - len(a)
if res > 0:
a.extend([[pad_token]*len(a[0])]*res)
l.append(a)
return l
def featurization(text_list):
# creates character level features
# text_list is a list of characters.
features = []
for ch in text_list:
l = []
l.append(1 if ch.isupper() else 0)
l.append(1 if ch in string.digits else 0)
l.append(1 if ch in string.punctuation else 0)
features.append(l)
return features
temp_data = pd.read_csv("../data/processed/data.csv")
The below step is necessary to take only those columns which we want to process and the column order matters
temp_data.loc[:,["text","label"]].to_csv("temp.csv",index=False)
Create the Text, Feature, and Target fields. Here I am tokenizing a sentence into characters.
TEXT = torchtext.data.Field(sequential=True, use_vocab=True,
tokenize=lambda x: list(x), include_lengths=True,
batch_first=True)
LABEL_PAD_TOKEN=-1
FEAT = torchtext.data.LabelField(use_vocab=False,batch_first=True,preprocessing=featurization,
pad_token=None,postprocessing=lambda x, _:postprocessing(x,_,LABEL_PAD_TOKEN))
LABELS = torchtext.data.Field(use_vocab=False,pad_token=LABEL_PAD_TOKEN,unk_token=None,
batch_first=True,dtype=torch.int64,tokenize=lambda x: list(x),
preprocessing=lambda x:[eval(i) for i in x])
In the TabularDataset, the correct field order should be given matching the temp.csv column order.
train_data = torchtext.data.TabularDataset(path="temp.csv",format="csv",skip_header=True,
fields=[(("text","feat"),(TEXT,FEAT)),
("labels",LABELS)])
TEXT.build_vocab(train_data)
train_data,valid_data = train_data.split() # create train val
Build the iterator
train_iter,valid_iter=torchtext.data.BucketIterator.splits((train_data,valid_data,),batch_size=2,device=device ,sort_within_batch=True,sort_key=lambda x:len(x.text))
a = next(iter(train_iter))
a.feat.shape, a.text[0].shape # printing the shape
(torch.Size([2, 36, 3]), torch.Size([2, 36]))
Next, you can pass the text to the embedding layer whose input is [batch_size, seq_len]
which will output [batch_size, seq_len, emb_dim]
The features have the shape of [batch_size, seq_len,3] because we have 3 features
Concatenate both of these on last dimension giving [batch_size, seq_len, emb_dim+3] and pass it either to LSTM or CNN