How do I design a "conveyor" of operations with OmniThreadLibrary? - multithreading

Question

I have a Windows Delphi application with "Start" and "Stop" menu items accessible via a notification icon. After click on "Start", I need to do the following (as I see implementation):
ThreadMonitor: The first thread is waiting for the appearance of the specified file in the specified folder.
ThreadParse: Once the file appears, it should be transferred to another thread (for parsing content) and continue monitoring for the next file.
ThreadDB: Once all data are parsed, save them into MySQL DB. (Another background thread with active DB connection?)
ThreadLog: If there are any errors in the steps 1–3, write them to a log file (another background thread?) without interrupting the steps 1–3.
That is, it turns out that something like a continuous conveyor, whose work is stopped only by pressing Stop.
What should I use from a whole variety of methods of OmniThreadLibrary?

Answer 1

It would probably be best to use Parallel.BackgroundWorker for logging and Parallel.Pipeline for data processing. Here's a sketch of a solution (compiles, but is not fully implemented):
unit PipelineDemo1;
interface
uses
Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls,
OtlCommon, OtlCollections, OtlParallel;
type
TfrmPipelineDemo = class(TForm)
btnStart: TButton;
btnStop: TButton;
procedure btnStartClick(Sender: TObject);
procedure btnStopClick(Sender: TObject);
private
FLogger : IOmniBackgroundWorker;
FPipeline: IOmniPipeline;
strict protected //asynchronous workers
procedure Asy_LogMessage(const workItem: IOmniWorkItem);
procedure Asy_Monitor(const input, output: IOmniBlockingCollection);
procedure Asy_Parser(const input: TOmniValue; var output: TOmniValue);
procedure Asy_SQL(const input, output: IOmniBlockingCollection);
public
end;
var
frmPipelineDemo: TfrmPipelineDemo;
implementation
uses
OtlTask;
{$R *.dfm}
procedure TfrmPipelineDemo.Asy_LogMessage(const workItem: IOmniWorkItem);
begin
//log workItem.Data
end;
procedure TfrmPipelineDemo.Asy_Monitor(const input, output: IOmniBlockingCollection);
begin
while not input.IsCompleted do begin
if FileExists('0.0') then
output.TryAdd('0.0');
Sleep(1000);
end;
end;
procedure TfrmPipelineDemo.Asy_Parser(const input: TOmniValue; var output: TOmniValue);
begin
// output := ParseFile(input)
FLogger.Schedule(FLogger.CreateWorkItem('File processed: ' + input.AsString));
end;
procedure TfrmPipelineDemo.Asy_SQL(const input, output: IOmniBlockingCollection);
var
value: TOmniValue;
begin
//initialize DB connection
for value in input do begin
//store value into database
end;
//close DB connection
end;
procedure TfrmPipelineDemo.btnStartClick(Sender: TObject);
begin
FLogger := Parallel.BackgroundWorker.NumTasks(1).Execute(Asy_LogMessage);
FPipeline := Parallel.Pipeline
.Stage(Asy_Monitor)
.Stage(Asy_Parser)
.Stage(Asy_SQL)
.Run;
end;
procedure TfrmPipelineDemo.btnStopClick(Sender: TObject);
begin
FPipeline.Input.CompleteAdding;
FPipeline := nil;
FLogger.Terminate(INFINITE);
FLogger := nil;
end;
end.