502 error upload csv file to web api - node.js

I have an Angular5 user interface with a file upload function. The user clicks a button and selects a file and the file is sent to the web api (asp.NET Core) method for processing.
This works fine with smaller files, but with larger files the request times out with a 502 error.
I can see the request always timesout at 120 seconds. (NOTE: I am hosting via node in development and via IIS in production).
In the case of large files I need to extend this timeout to a larger value. I've tried to achieve this in a number of ways:
Request Header - Timeout of request in angular code. I used the following code to try to set the timeout header value but it doesn't effect the 120 seconds:
export class AuthTokenInterceptor implements HttpInterceptor {
constructor(private authContext: AuthContext) {
}
intercept(req: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {
const authHeaderValue = this.authContext.getAuthenticationHeaderValue(req.url);
if (authHeaderValue) {
const authReq = req.clone({ headers:
req.headers.set('Authorization', authHeaderValue)
.set('Timeout', '100000000000000000') });
return next.handle(authReq);
}
return next.handle(req);
} }
web.config - I've tried setting the httpRuntime timeout value in the web.config file to the following (but still times out at 120 seconds):
Within IIS - I've tried setting the configuration of the "Limits" property in IIS and again, still times out at 120 seconds (and this has no relevance when I'm running through node server).
Has anyone been able to modify this 120 seconds in their Angular(2+) app requests?
Thanks for any pointers in advance!
NOTE: just for completeness, here's my asp.net core, controller method for uploading:
[HttpPost("Upload")]
public async Task<IActionResult> UploadAsync(IFormFile file)
{
// Ensure the file has contents before processing.
if (file == null || file.Length == 0)
throw new ApiException("Csv file should not be null", HttpStatusCode.BadRequest)
.AddApiExceptionResponseDetails(ErrorTypeCode.ValidationError, ErrorCode.BelowMinimumLength, SOURCE);
// Ensure the file is not over the allowed limit.
if (file.Length > (_settings.MaxCsvFileSize * 1024))
throw new ApiException("Max file size exceeded, limit of " + _settings.MaxCsvFileSize + "mb", HttpStatusCode.BadRequest)
.AddApiExceptionResponseDetails(ErrorTypeCode.ValidationError, ErrorCode.ExceedsMaximumLength, SOURCE);
// Ensure the file type is csv and content type is correct for the file.
if (Path.GetExtension(file.FileName) != ".csv" ||
!Constants.CsvAcceptedContentType.Contains(file.ContentType.ToLower(CultureInfo.InvariantCulture)))
throw new ApiException("Csv content only accepted").AddApiExceptionResponseDetails(ErrorTypeCode.ValidationError, ErrorCode.Invalid, SOURCE);
// Read csv content.
var content = await file.ReadCsvAsync<OrderCsvResponseDto>() as CsvProcessedResponseDto<OrderCsvResponseDto>;
await ProcessBulkUpload(content);
// Return information about the csv file.
return Ok(content);
}
Note - when I run the web api via IIS Express then it times out, I've run it using the command host and it doesn't time out - seem's like this may be related to an IIS setting of sorts. The web api doesn't have a web.config file due to the new version of ASP.net Core I'm using but this piece of code doesn't seem to have any bearing on IIS Express when I run through it:
var host = new WebHostBuilder()
.UseStartup<Startup>()
.UseKestrel(o => {
o.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(10);
o.ShutdownTimeout = TimeSpan.FromMinutes(10);
o.Limits.RequestHeadersTimeout = TimeSpan.FromMinutes(10);
})
.UseContentRoot(Directory.GetCurrentDirectory())
.UseIISIntegration()
.UseApplicationInsights()
.Build();

I will post this here in case anyone else comes across the same problem I was having. It turns out that running in IIS Express (or hosted IIS) that the config setting overrides whatever you have in code (maybe this is because the newer version of .net Core doesn't have a web.config file in the project - I'm not sure).
Anyway, I worked around this problem by carrying out the following steps:
Open IIS Express in your taskbar
Click on the app you are running (and wish to extend the request timeout for).
Clicking the app shows the config file of the application. Double click to open the config file.
Apply the following setting to the aspNetCore section:
requestTimeout="00:20:00"
Example:
<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
</handlers>
<aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" forwardWindowsAuthToken="false" requestTimeout="00:20:00" stdoutLogEnabled="false" />
<httpCompression>
<dynamicCompression>
<add mimeType="text/event-stream" enabled="false" />
</dynamicCompression>
</httpCompression>
</system.webServer>
And that's it!
NOTE: I am hosting the app in PaaS ASE - so cant configure IIS directly here. My solution for this was now to add a web.config file to the api project, and apply my setting within it. The build process honours the web.config you've got already instead of generating one on the fly and it will keep the changes needed. In my case, the web.config looked like this:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
</handlers>
<aspNetCore processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" forwardWindowsAuthToken="false" requestTimeout="00:20:00" stdoutLogEnabled="false" />
</system.webServer>
</configuration>
Hopefully this helps others!

Related

How to increase timeout of Azure App Service for my ASP.NET Core 2.0 API

I have an ASP.NET Core 2.0 API I am deploying to an Azure App Service. This has been working fine until recently when I had to process a request that took longer than 2 minutes to complete and I got a 502 Bad Gateway stating
"The specified CGI application encountered an error and the server terminated the process".
This consistently happens when I hit the 2 minute mark on this process.
My diagnostic logfile says
018-05-25 02:07:01.462 +00:00 [Error] Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: An unhandled exception has occurred while executing the request
System.Threading.Tasks.TaskCanceledException: A task was canceled.
I am working on the assumption that this is a timeout issue because it is always at the 2 minute mark and I know the request takes more than 2 minutes to complete. So I was looking into how to increase the timeout and found some posts on SO that talked about using an applicationHost.xdt file, placing it in the root of the Site folder for the site. I am using this XML;
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<system.applicationHost>
<webLimits xdt:Transform="SetAttributes(connectionTimeout)" connectionTimeout="00:05:00" />
</system.applicationHost>
</configuration>
When I upload it, using Kudu debug console, to the root of the Site folder for my deployment slot I am using to test my API and then look at the transform file, I see the following;
2018-05-24T19:34:19 Start 'site' site extension transform
2018-05-24T19:34:19 StartSection Executing SetAttributes (transform line 4, 16)
2018-05-24T19:34:19 on /configuration/system.applicationHost/webLimits
2018-05-24T19:34:19 Applying to 'webLimits' element (no source line info)
2018-05-24T19:34:19 Set 'connectionTimeout' attribute
2018-05-24T19:34:19 Set 1 attributes
2018-05-24T19:34:19 EndSection Done executing SetAttributes
2018-05-24T19:34:19 Successful 'D:\home\site\applicationHost.xdt' site extension transform
which to me looks like it successfully applied the XDT transform.
However even after restarting the base App Service and the related deployment slot. I am still getting the error.
So, am I using the wrong timeout setting?
When I look at the web.config file in my slots sites/wwwroot folder, it contains only this...
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
</handlers>
<aspNetCore processPath="dotnet" arguments=".\Mypp.dll"
stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" />
</system.webServer>
I expected to see the information about the connectiontimeout that was in the XDT file I applied. So, perhaps this is not the right web.config file?
I am not an Azure expert and at this point, I feel like I am wasting time so I wanted to check an see if anyone has any suggestions.
requestTimeout:
Specifies the duration for which the ASP.NET Core Module waits for a response from the process listening on %ASPNETCORE_PORT%.
In versions of the ASP.NET Core Module that shipped with the release of ASP.NET Core 2.0 or earlier, the requestTimeout must be specified in whole minutes only, otherwise it defaults to 2 minutes.
You could try to add the requestTimeout="00:20:00 in web.config in your slots sites/wwwroot folder.
<aspNetCore
requestTimeout="00:20:00"
processPath="%LAUNCHER_PATH%"
arguments="%LAUNCHER_ARGS%"
stdoutLogEnabled="false"
stdoutLogFile=".\logs\stdout">
<environmentVariables>
<environmentVariable name="ASPNETCORE_ENVIRONMENT" value="staging" /> <!-- value could be "development", "staging" or "production"-->
</environmentVariables>
</aspNetCore>
For more details, you could refer to this article to learn about ASP.NET Core Module configuration reference

ELMAH - Get SMTP credentials from Azure Application Settings

I've got an Azure Web App using ELMAH to log unhandled exceptions.
When I first deployed it, the web.config had the full SMTP setup defined in it, and ELMAH emailed exceptions:
<system.net>
<mailSettings>
<smtp from="me#mydomain.com">
<network host="smtp.mailprovider.com"
port="123"
userName="myUserName"
password="p#ssw0rd" />
</stmp>
</mailSettings>
</system.net>
The username and password have since been removed from the web.config, and they're now stored as application settings, configured through the Azure Portal.
Most of the emails I send still work fine, as the email code can access these application settings and use them when instantiating the SmtpClient, e.g.:
var userName = WebConfigurationManager.AppSettings["smtp.userName"];
var password = WebConfigurationManager.AppSettings["smtp.password"];
var credentials = new NetworkCredential(userName, password);
using (var smtpClient = new SmtpClient { Credentials = credentials })
{
await smtpClient.SendMailAsync(mailMessage);
}
What's the best way to get ELMAH to use the credentials stored in the application settings?
Options I can see:
There is a page on the wiki explaining how to use ELMAH's ErrorTweetModule to do an HTTP form post with the error details to any URL. The controller receiving the post could then use the stored credentials to email the details on.
The WebBase has a link to an article suggesting you can send emails directly to the recipient's SMTP server without authentication, but it says this may not work if you have DomainKeys set up, which I do.
This answer links to an article about intercepting the Mailing event, to customise the message.
I ended up creating a custom version of Elmah's ErrorMailModule, derived from the standard one, but overriding the SendMail method, based on some advice from Atif Aziz in a discussion on Google Groups.
The only changes required were to create the new module, and switch the Web.Config to use the custom module instead of the standard one.
Module
using System;
using System.Net.Mail;
namespace Test
{
public class ErrorMailModule : Elmah.ErrorMailModule
{
protected override void SendMail(MailMessage mail)
{
if (mail == null) throw new ArgumentNullException(nameof(mail));
// do what you want with the mail
// (in my case this fires up the email service, which
// gets the credentials from the Azure settings)
}
}
}
Web Config Changes
All that's required is to change the two occurrences of Elmah.ErrorLogModule, Elmah to your own module, in this case Test.ErrorMailModule.
So, instead of this...
<system.web>
<httpModules>
<add name="ErrorMail" type="Elmah.ErrorMailModule, Elmah" />
</httpModules>
</system.web>
<system.webServer>
<modules>
<add name="ErrorMail" type="Elmah.ErrorMailModule, Elmah" preCondition="managedHandler" />
</modules>
</system.webServer>
...you should now have this:
<system.web>
<httpModules>
<add name="ErrorMail" type="Test.ErrorMailModule" />
</httpModules>
</system.web>
<system.webServer>
<modules>
<add name="ErrorMail" type="Test.ErrorMailModule" preCondition="managedHandler" />
</modules>
</system.webServer>
You will still need the errorMail section, as Elmah is still responsible for creating the email. Mine looks like this:
<elmah>
<errorMail from="user#domain.com" to="user#domain.com" subject="Custom Email Module"/>
</elmah>
Creating a HTTP request could work, but that should be the solution if everything else doesn't work IMO. Intercepting the Mailing event doesn't work, since you do not have access to the SmtpClient with the credentials in that event.
I've looked at different ways to update the SMTP settings from code. At first I though that I could just get a reference to the smtp section and update the properties, since they all have setter. But the code throw a configuration exception on runtime.
From what I can find, the only way to update the username and password in smtp section, is to read the web.config, update it and write the new version. Here's an example of writing updates to web.config:
var configuration = WebConfigurationManager.OpenWebConfiguration("~");
var section = configuration.GetSection("system.net/mailSettings/smtp") as SmtpSection;
section.Network.UserName = ConfigurationManager.AppSettings["myusername"];
section.Network.Password = ConfigurationManager.AppSettings["mypassword"];
configuration.Save();
The code actually updates the web.config. The code can be run at startup, but that would modify your web.config file locally as well. Another approach would be to run the code as part of a post deployment task with Azure.

Azure Websites Dropping Custom Headers

An Azure Website I am working on inspects custom headers of incoming requests to decide what to do with the request internally. The request is being sent/received by the website server with the customer headers:
X-HEADER-1: ...
X-HEADER-2: ...
among other standard and non-standard headers.
I verified this by inspecting the FREB logs and looking at GENERAL_REQUEST_HEADERS, which correctly includes my custom headers.
When the application receives the request, those custom headers are not there. I explicitly check for one of them then throw and dump all available headers in the error message.
I have read around that Application Request Routing module can drop these headers. I tried adding this to the website's web.config but still doesn't work:
<system.webServer>
<rewrite>
<allowedServerVariables>
<add name="HTTP_X_HEADER_1" />
<add name="HTTP_X_HEADER_2" />
</allowedServerVariables>
</rewrite>
</system.webServer>
Any idea how I can whitelist my headers to let ARR/Azure let them through?
Update 1
Here is some more info.
This works locally on my dev box. I set up the site in IIS and point it to the project folder and headers are coming in and processed as expected.
It is an ASP.NET MVC website.
Here is the part of the code that reads the header. Again, this works locally.
public class BaseController : Controller
{
public AppControllerBase(...)
{
}
protected override void Initialize(RequestContext requestContext)
{
var header1Value = requestContext.HttpContext.Request.Headers["X-HEADER-1"];
if (string.IsNullOrEmpty(header1Value))
{
var stringBuilder = new StringBuilder();
// append all headers to stringBuilder
var errorMessage = string.Format("SiteId header is not set. Headers: {0}", stringBuilder);
throw new HttpRequestException(errorMessage);
}
base.Initialize(requestContext);
}
...
}
Update 2
I just deployed the same app as an azure cloud service and it worked well. The headers were received and the app read them successfully. Something with web apps is not letting those headers through.
The answer that worked for me was in the comments. Credit goes to #Tarek Ayna.
The custom headers are transmitted when you set X-LiveUpgrade to 0.
For example:
<httpProtocol>
<customHeaders>
<add name="X-LiveUpgrade" value="0" />**
<!-- Prevent iframes -->
<add name="X-Frame-Options" value="SAMEORIGIN" />
<add name="X-XSS-Protection" value="1" />
</customHeaders>
</httpProtocol>
One possibility is to disable ARR if your services are stateless... to do that:
(Inside web.config)
<system.webServer>
<httpProtocol>
<customHeaders>
<add name="Arr-Disable-Session-Affinity" value="True" />
</customHeaders>
</httpProtocol>
</system.webServer>

OData $batch endpoint not found when hosted in IIS

I'm struggling to figure this out - I have an OData V4 service which I have developed using .Net ODataConventionModelBuilder. Everything works fine when running "within visual studio" (i.e. localhost:port/service) but when I publish the service to IIS I can no longer make batch requests.
Example1 : Running under IISExpress while visual studio is running
GET http://localhost:1234/service/SomeResource
- works fine to access a resource and return JSON response
POST http://localhost:1234/service/$batch
- works fine to send a batch request (obviously with body data) for multiple GETs
Example2 : Hosting under IIS
GET http://localhost/service/SomeResource
- works fine to access a resource and return JSON response
POST http://localhost/service/$batch
- fails with a HTTP 404 Not Found
"Message": "No HTTP resource was found that matches the request URI 'http://localhost/service/$batch'.",
"MessageDetail": "No route providing a controller name was found to match request URI 'http://localhost/service/$batch'"
so I'm stuck I dont understand why routing is working ok when running under studio/IISExpress and not under proper IIS.
Ok seems this was down to a problem interpreting the url because of the $batch part.
Adding this to web.config solved it ok.
<system.webServer>
<handlers>
<remove name="ExtensionlessUrlHandler-Integrated-4.0" />
<add name="ExtensionlessUrlHandler-Integrated-4.0" path="*." verb="*" type="System.Web.Handlers.TransferRequestHandler" resourceType="Unspecified" requireAccess="Script" preCondition="integratedMode,runtimeVersionv4.0" />
</handlers>
</system.webServer>

Deny file access but server properly in IIS

I have a js file that includes in the master page.
I want to deny the file access when user type the direct link in the browser address bar.
I've tried the URL filtering IIS,like:
<security>
<requestFiltering>
<denyUrlSequences>
<add sequence="Scripts/Foo/bar.min.js" />
</denyUrlSequences>
</requestFiltering>
</security>
This does work, when i type 'localhost://blah/Scripts/Foo/bar.min.js' I get blocked.
but the page whitch need this js file can not render.
Does anyone have a workaround? Thanks in advance!
I finnaly found this.
and added these sections to web.config:
<security>
<requestFiltering>
<filteringRules>
<filteringRule name="protectjs" scanUrl="true" scanQueryString="true">
<scanHeaders>
<clear />
<add requestHeader="Accept" />
</scanHeaders>
<appliesTo>
<clear />
<add fileExtension=".js" />
</appliesTo>
<denyStrings>
<clear />
<add string="text/html" />
</denyStrings>
</filteringRule>
</filteringRules>
</requestFiltering>
</security>
It works at present, though I know this way is not that reliable.
Not 100% sure but I don’t think there is a way to do this. When browser renders your page it actually sends the same request to the server as the one you do when you manually type in JS file in the browser.
IIS doesn’t have means to distinguish requests you make and the one browser makes in the background while processing your page.
One thing that might work is
adding some kind of ID to your JS file like this Scripts/Foo/bar.min.js?ID=E3CF305B-4444-E011-8FD2-1CC1DEE89A7F
ID is time limited and works only 20 seconds or so after it’s created (enough time for browser to load the page)
creating custom handler that will parse ID and decide if server should return the request or not
So when someone types Scripts/Foo/bar.min.js handler will reject the request but it will also reject request when expired ID is used.
However this is most probably going to be difficult to maintain and also performance intensive for your server.
If you want to hide your JS file why not obfuscate it.
Use HttpModule and Check for HttpContext.Current.Request.UrlReferrer. it will always be null for direct access to the .js file.
public class JSAccessModule : IHttpModule
{
public void Init(System.Web.HttpApplication Appl)
{
Appl.BeginRequest += new System.EventHandler(Rewrite_BeginRequest);
}
public void Rewrite_BeginRequest(object sender, System.EventArgs args)
{
//Block if direct Url is accessed
if (HttpContext.Current.Request.UrlReferrer == null)
{
HttpApplication App = (HttpApplication)sender;
string path = App.Request.Path;
string strExt = System.IO.Path.GetExtension(path);
if (strExt == ".js")
{
HttpContext.Current.Response.Redirect("~/AccessDenied.html");
}
}
}
public void Dispose() { }
}

Resources