Failed to start QThread when using QNetworkAccessManager - multithreading

Question

I'm currently trying to make a software that downloads a lot of files from Google Drive. Downloading is currently not a problem.
Nevertheless, I encounter an issue when launching 500+ simultaneous downloads. I use a slightly modified version of this tutorial : https://wiki.qt.io/Download_Data_from_URL.
Here is the .h file :
class FileDownloader : public QObject
{
Q_OBJECT
public:
explicit FileDownloader(QUrl url, QObject *parent = 0, int number = 0);
QByteArray downloadedData() const;
void launchNewDownload(QUrl url);
QByteArray m_DownloadedData;
QNetworkReply* reply;
static QNetworkAccessManager *m_WebCtrl;
signals:
void downloaded();
private slots:
void fileDownloaded(QNetworkReply* pReply);
};
And here is the .cpp file :
QNetworkAccessManager* FileDownloader::m_WebCtrl = nullptr;
FileDownloader::FileDownloader(QUrl url, QObject *parent) :
QObject(parent)
{
if (m_WebCtrl == nullptr) {
m_WebCtrl = new QNetworkAccessManager(this);
}
connect(m_WebCtrl, SIGNAL (finished(QNetworkReply*)),this, SLOT (fileDownloaded(QNetworkReply*)));
launchNewDownload(url);
}
void FileDownloader::launchNewDownload(QUrl url) {
QNetworkRequest request(url);
this->reply = m_WebCtrl->get(request);
}
void FileDownloader::fileDownloaded(QNetworkReply* pReply) {
m_DownloadedData = pReply->readAll();
//emit a signal
pReply->deleteLater();
emit downloaded();
}
QByteArray FileDownloader::downloadedData() const {
return m_DownloadedData;
}
The issue is "QThread::start: Failed to create thread ()" when reaching about the 500th download. I tried to limit the number of downloads which run at the same time - but I always get the same issue. Besides, I tried to delete every downloader when finishing its task - it did nothing else than crashing the program ;)
I think that it is coming from the number of threads allowed for an only process, but I'm not able to solve it !
Does anyone have an idea that could help me ?
Thank you !

Answer 1

QNetworkAccessManager::finished signal is documented to be emitted whenever a pending network reply is finished.
This means that if the QNetworkAccessManager is used to run multiple requests at a time (and this is perfectly normal usage). finished signal will be emitted once for every request. Since you have a shared instance of QNetworkAccessManager between your FileDownloader objects, the finished signal gets emitted for every get call you have made. So, all the FileDownloader objects get a finished signal as soon as the first FileDownloader finishes downloading.
Instead of using QNetworkAccessManager::finished, you can use QNetworkReply::finished to avoid mixing up signals. Here is an example implementation:
#include <QtNetwork>
#include <QtWidgets>
class FileDownloader : public QObject
{
Q_OBJECT
//using constructor injection instead of a static QNetworkAccessManager pointer
//This allows to share the same QNetworkAccessManager
//object with other classes utilizing network access
QNetworkAccessManager* m_nam;
QNetworkReply* m_reply;
QByteArray m_downloadedData;
public:
explicit FileDownloader(QUrl imageUrl, QNetworkAccessManager* nam,
QObject* parent= nullptr)
:QObject(parent), m_nam(nam)
{
QNetworkRequest request(imageUrl);
m_reply = m_nam->get(request);
connect(m_reply, &QNetworkReply::finished, this, &FileDownloader::fileDownloaded);
}
~FileDownloader() = default;
QByteArray downloadedData()const{return m_downloadedData;}
signals:
void downloaded();
private slots:
void fileDownloaded(){
m_downloadedData= m_reply->readAll();
m_reply->deleteLater();
emit downloaded();
}
};
//sample usage
int main(int argc, char* argv[]){
QApplication a(argc, argv);
QNetworkAccessManager nam;
FileDownloader fileDownloader(QUrl("http://i.imgur.com/Rt8fqpt.png"), &nam);
QLabel label;
label.setAlignment(Qt::AlignCenter);
label.setText("Downloading. . .");
label.setMinimumSize(640, 480);
label.show();
QObject::connect(&fileDownloader, &FileDownloader::downloaded, [&]{
QPixmap pixmap;
pixmap.loadFromData(fileDownloader.downloadedData());
label.setPixmap(pixmap);
});
return a.exec();
}
#include "main.moc"
If you are using this method to download large files, consider having a look at this question.

Answer 2

One solution could be to uses a QThreadPool. You simply feed it tasks (QRunnable) and it will handle the number of threads and the task queue for you.
However in your case this is not perfect because you will be limiting the number of simultaneous downloads to the number of threads created by QThreadPool (generally the number of CPU core you have).
To overcome this you will have to handle the QThread yourself and not use QThreadPool. You would create a small number of thread (see QThread::idealThreadCount()) and run multiple FileDownloader on each QThread.