spaCy sentence separation with dictionary source from OpenAI Whisper / WhisperX? - nlp

Question

WhisperX is a whisper extension that does a really excellent job of text to speech with per-word timestamps.
I'd like to use spaCy to split up the text strings into sensible clauses but maintain a connection to the source dictionary so the result can inform subtitles and other video editing tools.
Is there a pathway to do this in spaCy? Most of the examples I see expect a text string of input.
My source dictionaries from WhisperX will be many multiples of this type of thing:
"word-level": [ {"text": "So", "start": 27.80031007751938, "end": 27.940671834625324}, { "text": "we've", "start": 27.98077519379845, "end": 28.201343669250647, }, {"text": "got", "start": 28.301602067183463, "end": 28.502118863049095}, { "text": "enough", "start": 28.58232558139535, "end": 28.842997416020673, }, { "text": "books.", "start": 28.983359173126615, "end": 29.223979328165374, }, {"text": "We", "start": 29.2640826873385, "end": 29.364341085271317}, { "text": "hopefully", "start": 29.384392764857882, "end": 29.765374677002583, }, { "text": "enough", "start": 29.885684754521964, "end": 30.18645994832041, }, {"text": "to", "start": 30.607545219638244, "end": 30.72785529715762}, {"text": "be", "start": 30.767958656330748, "end": 30.868217054263564}, {"text": "able", "start": 30.96847545219638, "end": 31.128888888888888}, {"text": "to", "start": 31.168992248062015, "end": 31.26925064599483}, { "text": "share", "start": 31.349457364341085, "end": 31.590077519379843, }, {"text": "some", "start": 31.670284237726097, "end": 31.81064599483204}, {"text": "of", "start": 31.850749354005167, "end": 31.890852713178294}, { "text": "these", "start": 31.93095607235142, "end": 32.071317829457364, }, {"text": "books", "start": 32.15152454780362, "end": 32.41219638242894}, {"text": "with", "start": 32.51245478036176, "end": 32.692919896640824}, {"text": "the", "start": 32.71297157622739, "end": 32.79317829457364}, { "text": "children", "start": 32.833281653746766, "end": 33.194211886304906, }, {"text": "our", "start": 33.254366925064595, "end": 33.37467700258398}, { "text": "community", "start": 33.43483204134367, "end": 33.73560723514212, }, {"text": "at", "start": 33.81581395348837, "end": 33.89602067183462}, {"text": "our", "start": 33.956175710594316, "end": 34.13664082687338}, { "text": "annual", "start": 34.216847545219636, "end": 34.4574677002584, }, { "text": "Christmas", "start": 34.49757105943152, "end": 34.75824289405685, }, {"text": "on", "start": 34.79834625322997, "end": 34.838449612403096}, {"text": "the", "start": 34.87855297157623, "end": 34.938708010335915}, { "text": "Boulevard", "start": 34.95875968992248, "end": 35.25953488372093, }, {"text": "event", "start": 35.27958656330749, "end": 35.37984496124031}, ]
I've played around with keying in and out of the dictionary, but this feels like a problem that already has a solution.

Answer 1

You can create docs from external tokenization like this:
import spacy
from spacy.tokens import Doc
nlp = spacy.blank("en")
doc = Doc(nlp.vocab, words=[entry["text"] for entry in entries])
If you need to, you can add additional metadata to the tokens for alignment:
from spacy.tokens import Token
Token.set_extension("my_metadata", default=None)
for i, entry in enumerate(entries):
doc[i]._.my_metadata = entry["some_index"]
If you're using this input for training, you might want to add this flag to the doc, so that the pipeline components know that the spaces between the tokens in the default doc.text aren't significant:
doc.has_unknown_spaces = True
We've and "books." are not tokenized the same way by default for the English pipelines, so provided pipelines like en_core_web_sm won't do a particularly great job of analyzing this input, especially around sentence boundaries and tokens like "books." where it would expect the tokens ["books", "."].