Recommended way to achieve `rescue-ensure` kind of functionality with Kiba? - kiba-etl

Question

We have a Kiba pipeline where we need to do some task after the job has ended, no matter if there were errors or not (the whole pipeline doesn't fail, we just have couple of validation errors or similar).
This is what the documentation says:
:warning: Post-processors won't get called if an error occurred before them.
https://github.com/thbar/kiba/wiki/Implementing-pre-and-post-processors
Would this be recommended way to do this:
Kiba.run(
Kiba.parse do
source(...)
transform(...)
destination(....)
post_process do
# we cannot do it here, because it won't get called
end
end
)
# is this the location to do it?
Job.some_special_cleanup_task
Thanks!
PS what does it mean:
Post-processors won't get called if an error occurred before them.
Does this mean if the error occurred and wasn't rescued from?

Answer 1

Indeed post_process will not be called in case of errors, as documented and as you pointed out!
At this point, the best solution is to use a form of ensure statement:
A common way to structure that is:
module ETL
module Job
module_function
def setup(config)
Kiba.parse do
source(...)
transform(...)
destination(....)
end
end
def some_special_cleanup_task
# ...
end
def run(job)
Kiba.run(job)
ensure
Job.some_special_cleanup_task
end
end
end
Doing so allows to keep the code for the always-run task close to the ETL job, which is nice.
If your task is instead very independent from the job, and you want to encourage reuse between jobs, you can also create a generic block-form component:
module ETL
module Middlewares
module CommonTask
module_function
def with_common_task
yield
ensure
do_the_common_task
end
end
end
end
Which you would use like this:
ETL::Middlewares::CommonTask.with_common_task do
Kiba.run(...)
end
This second form is used by Kiba Pro FileLock, for instance.
In the future, Kiba ETL will introduce a form of middleware to make this even easier.
Hope this helps, and please mark the question as answered if it properly solves your problem!