Multiple requests on same IdHTTP inside thread - multithreading

Question

I have an application that makes thousands of HTTP requests along the day.
For best performance, I decided to create only once the IdHTTP object and use this same object for all requests.
This is where the problem begins. While creating one IdHTTP for each request, everything went fine.
The code is pretty basic:
constructor HTTPThread.Create;
begin
inherited Create(false);
httpObject:= TIdHTTP.Create(Nil);
sslObject:= TIdSSLIOHandlerSocketOpenSSL.Create(Nil);
sslObject.SSLOptions.Method:= sslvTLSv1_2;
httpObject.IOHandler:= sslObject;
httpObject.Request.Accept:= frmHTTPRequests.Edit1.Text;
httpObject.Request.UserAgent:= frmHTTPRequests.Edit3.Text;
httpObject.ReadTimeout:= 15000;
httpObject.HandleRedirects:= true;
FreeOnTerminate:= true;
OnTerminate:= TerminateProc;
end;
procedure HTTPThread.DoRequests;
var
htmlSource: string;
begin
try
htmlSource:= httpObject.Get(Link);
//a bunch of other stuff with HTML source
except
on E : Exception do
Synchronize(procedure
begin
errorList.Add(E.Message);
errorList.SaveToFile('Error.txt');
end);
end;
end;
I created this except saving the Error.txt file to watch what is happening...
The code runs fine sometimes for the first 1k requests, sometimes for the first 2k, it varies. Suddently, it starts writing on the TXT file the same error:
Connection reset by peer. domain.com - Socket Error # 10054
I tried to disconnect the httpObject, tried httpObject.Request.Clear, nothing seems to work.
Are there any chance to make this work?

Answer 1

For some reasons Indy doesn't close socket when server respond with Connection reset by peer, so you need to do it manually.
procedure HTTPThread.DoRequests;
const
MAX_TRIES_COUNT = 5;
var
htmlSource: string;
TriesCount: Integer;
begin
TriesCount := 0;
repeat
try
htmlSource:= httpObject.Get(Link);
//a bunch of other stuff with HTML source
except
on E: Exception do
begin
if E.Message.Contains('by peer') then
begin
httpObject.Disconnect;
// Try to solve network connection issues
Continue;
end
else
begin
// Some other error handlers
end;
end;
inc(TriesCount);
end;
until (httpObject.ResponseCode = 200) or (TriesCount > MAX_TRIES_COUNT);
end;
P.S. I don't recommend you to use Synchronize() for threads synchronization. Try to use TCriticalSection or TMonitor instead.