How to produce more than one clue from a record? - cluedin

Question

I am writing a CluedIn Crawler and when I get data, I get a record that contains an array of referenced records. The code looks like this:
public IEnumerable<object> GetData(CrawlJobData jobData)
{
if (!(jobData is MyCrawlJobData myCrawlJobData))
{
yield break;
}
var client = clientFactory.CreateNew(myCrawlJobData);
foreach (var myModel in client.GetMyModels())
{
yield return myModel;
}
}
}
Then in the clue producer, I want to create a main clue and a clue per each referenced record. But the problem is that the override for the MakeClueImpl returns only one Clue:
protected override Clue MakeClueImpl([NotNull] MyModel input, Guid accountId)
{
// ...
}
What can I do to avoid this limitation?

Answer 1

As the GetData is IEnumerable<object>, you can yield different model objects just from this method:
public IEnumerable<object> GetData(CrawlJobData jobData)
{
if (!(jobData is MyCrawlJobData myCrawlJobData))
{
yield break;
}
var client = clientFactory.CreateNew(myCrawlJobData);
foreach (var myModel in client.GetMyModels())
{
yield return myModel;
foreach (var relatedRecordModel in myModel.RelatedRecords)
{
yield return relatedRecordModel;
}
}
}
And then have two clue producers like:
protected override Clue MakeClueImpl([NotNull] MyModel input, Guid accountId)
{
// TODO:
return clue;
}
and
protected override Clue MakeClueImpl([NotNull] MyRelatedRecordModel input, Guid accountId)
{
// TODO:
return clue;
}
Yet another way would be just to produce and yield clues just from the GetData method and then create a dummy clue producer that will accept a Clue:
public IEnumerable<object> GetData(CrawlJobData jobData)
{
if (!(jobData is MyCrawlJobData myCrawlJobData))
{
yield break;
}
var client = clientFactory.CreateNew(myCrawlJobData);
foreach (var myModel in client.GetMyModels())
{
yield return myModel;
foreach (var relatedRecord in myModel.RelatedRecords)
{
// return a Clue per related record
yield return MakeRelatedRecordClue(relatedRecord);
}
}
}
protected override Clue MakeClueImpl([NotNull] Clue input, Guid accountId)
{
return clue;
}