Webclient 401 Unauthorized Issue on above 5 minutes of processing time - sharepoint

Question

I have HttpHandler (abc.ashx), which I make a call to process (Bulk Upload) an excel file. This processing takes a certain amount of time, which is proportionate to the number of rows in the excel.
On top of this I have an Event Handler in Sharepoint which uses a WebClient to make a post service call to this handler using the code below;
NOTE: I generate an identity token using elevated privilege of Sharepoint, and use that token to impersonate to make the service call. net net I make the post service call using the identity of the System Account.
WindowsIdentity identity = null;
// Get an identity token using delegate activity of Elevated Privelleges of Sharepoint
SPHelper.ElevatedActivity(properties.Web.Site.Url, properties.Web.ServerRelativeUrl, web =>
{
identity = WindowsIdentity.GetCurrent();
});
//Impersonates the identity of System Account user, as received in token in the line above
using (identity.Impersonate())
{
// create a web client object which only increases the timeout of the web client call
var webClient = (new CustomWebClient());
webClient.Credentials = CredentialCache.DefaultNetworkCredentials;
webClient.UseDefaultCredentials = true;
//url of the httphandler as deployed in sharepoint, and the parameters that needs to be passed
string handlerUrl =
string.Format(
properties.Web.Url.Trim('/') + "/_layouts/Handlers/abc.ashx?Table=BulkUpload&Region={0}&UserId={1}&BatchId={2}&FileUrl={3}",
region, userValue.User.LoginName, batchId, fileUrl);
}
// execute the web service call using the webclient object
webClient.DownloadString(handlerUrl));
Essentially this code is written in the Event Handler for ItemAdded and ItemUpdated event.
The issue is in the webClient which seems to give an error "system.net.webexception: the remote server returned an error: (401) unauthorized", after precisely 5 mins of processing. Hence if the excel sheet to be processed has rows less than certain number (1700), the processing happens within 5 minutes, and everything runs fine without any error. However if it has more than that, then the processing takes more than 5 minutes, and fails with the error specificed above.
The strange behaviour is it seems like a timeout issue but the error message indicates authorization issue, which does not make sense, as if there was an authorization issue, it shouldnt have worked even when the processing time was less that 5 minutes.
We have tried to find any configuration that times out in 5 minutes, however we could not find any.
Any help or suggestion on this matter is appreciated.
UPDATE: I have now tried to make this work using HttpWebRequest, and have tried a bunch of setting which might cause this timeout/issue. However still getting the same issue, where after 5 minutes of processing I am getting "system.net.webexception: the remote server returned an error: (401) unauthorized". Below is the code I have tried
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(handlerUrl);
httpWebRequest.Credentials = CredentialCache.DefaultCredentials;
httpWebRequest.UseDefaultCredentials = true;
httpWebRequest.Timeout = 600000;
httpWebRequest.ReadWriteTimeout = 600000;
httpWebRequest.ServicePoint.ConnectionLeaseTimeout = 600000;
httpWebRequest.ServicePoint.MaxIdleTime = 600000;
httpWebRequest.ContentType = "application/json; charset=utf-8";
httpWebRequest.ContentLength = 0;
httpWebRequest.Method = WebRequestMethods.Http.Get;
BulkUploadProcessorResponse response = null;
var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (Stream stream = httpWebResponse.GetResponseStream())
{
using (StreamReader sr = new StreamReader(stream))
{
response =
JsonHelper.JsonDeserialize<BulkUploadProcessorResponse>(sr.ReadToEnd());
}
}