Long-running / blocking operations in boost asio handlers - multithreading

Question

Current Situation
I implemented a TCP server using boost.asio which currently uses a single io_service object on which I call the run method from a single thread.
So far the server was able to answer the requests of the clients immediately, since it had all necessary information in the memory (no long-running operations in the receive handler were necessary).
Problem
Now requirements have changed and I need to get some information out of a database (with ODBC) - which is basically a long-running blocking operation - in order to create the response for the clients.
I see several approaches, but I don't know which one is best (and there are probably even more approaches):
First Approach
I could keep the long running operations in the handlers, and simply call io_service.run() from multiple threads. I guess I would use as many threads as I have CPU cores available?
While this approach would be easy to implement, I don't think I would get the best performance with this approach because of the limited number of threads (which are idling most of the time since database access is more an I/O-bound operation than a compute-bound operation).
Second Approach
In section 6 of this document it says:
Use threads for long running tasks
A variant of the single-threaded design, this design still uses a single io_service::run() thread for implementing protocol logic. Long running or blocking tasks are passed to a background thread and, once completed, the result is posted back to the io_service::run() thread.
This sounds promising, but I don't know how to implement that. Can anyone provide some code snippet / example for this approach?
Third Approach
Boris Schäling explains in section 7.5 of his boost introduction how to extend boost.asio with custom services.
This looks like a lot of work. Does this approach have any benefits compared to the other approaches?

Answer 1

The approaches are not explicitly mutually exclusive. I often see a combination of the first and second:
One or more thread are processing network I/O in one io_service.
Long running or blocking tasks are posted into a different io_service. This io_service functions as a thread pool that will not interfere with threads handling network I/O. Alternatively, one could spawn a detached thread every time a long running or blocking task is needed; however, the overhead of thread creation/destruction may a noticeable impact.
This answer that provides a thread pool implementation. Additionally, here is a basic example that tries to emphasize the interaction between two io_services.
#include <iostream>
#include <boost/asio.hpp>
#include <boost/bind.hpp>
#include <boost/chrono.hpp>
#include <boost/optional.hpp>
#include <boost/thread.hpp>
/// #brief Background service will function as a thread-pool where
/// long-standing blocking operations may occur without affecting
/// the network event loop.
boost::asio::io_service background_service;
/// #brief The main io_service will handle network operations.
boost::asio::io_service io_service;
boost::optional<boost::asio::io_service::work> work;
/// #brief ODBC blocking operation.
///
/// #brief data Data to use for query.
/// #brief handler Handler to invoke upon completion of operation.
template <typename Handler>
void query_odbc(unsigned int data,
Handler handler)
{
std::cout << "in background service, start querying odbc\n";
std::cout.flush();
// Mimic busy work.
boost::this_thread::sleep_for(boost::chrono::seconds(5));
std::cout << "in background service, posting odbc result to main service\n";
std::cout.flush();
io_service.post(boost::bind(handler, data * 2));
}
/// #brief Functions as a continuation for handle_read, that will be
/// invoked with results from ODBC.
void handle_read_odbc(unsigned int result)
{
std::stringstream stream;
stream << "in main service, got " << result << " from odbc.\n";
std::cout << stream.str();
std::cout.flush();
// Allow io_service to stop in this example.
work = boost::none;
}
/// #brief Mocked up read handler that will post work into a background
/// service.
void handle_read(const boost::system::error_code& error,
std::size_t bytes_transferred)
{
std::cout << "in main service, need to query odbc" << std::endl;
typedef void (*handler_type)(unsigned int);
background_service.post(boost::bind(&query_odbc<handler_type>,
21, // data
&handle_read_odbc) // handler
);
// Keep io_service event loop running in this example.
work = boost::in_place(boost::ref(io_service));
}
/// #brief Loop to show concurrency.
void print_loop(unsigned int iteration)
{
if (!iteration) return;
std::cout << " in main service, doing work.\n";
std::cout.flush();
boost::this_thread::sleep_for(boost::chrono::seconds(1));
io_service.post(boost::bind(&print_loop, --iteration));
}
int main()
{
boost::optional<boost::asio::io_service::work> background_work(
boost::in_place(boost::ref(background_service)));
// Dedicate 3 threads to performing long-standing blocking operations.
boost::thread_group background_threads;
for (std::size_t i = 0; i < 3; ++i)
background_threads.create_thread(
boost::bind(&boost::asio::io_service::run, &background_service));
// Post a mocked up 'handle read' handler into the main io_service.
io_service.post(boost::bind(&handle_read,
make_error_code(boost::system::errc::success), 0));
// Post a mockup loop into the io_service to show concurrency.
io_service.post(boost::bind(&print_loop, 5));
// Run the main io_service.
io_service.run();
// Cleanup background.
background_work = boost::none;
background_threads.join_all();
}
And the output:
in main service, need to query odbc
in main service, doing work.
in background service, start querying odbc
in main service, doing work.
in main service, doing work.
in main service, doing work.
in main service, doing work.
in background service, posting odbc result to main service
in main service, got 42 from odbc.
Note that the single thread processing the main io_service posts work into the background_service, and then continues to process its event loop while the background_service blocks. Once the background_service gets a result, it posts a handler into the main io_service.

Answer 2

We have same long-running tasks in our server (a legacy protocol with storages). So our server is running 200 threads to avoid blocking service (yes, 200 threads is running io_service::run). Its not too great thing, but works well for now.
The only problem we had is asio::strand which uses so-called "implementations" which gets locked when hadler is currently called. Solved this via increase this strands butckets and "deattaching" task via io_service::post without strand wrap.
Some tasks may run seconds or even minutes and this does work without issues at the moment.