I've been running a process that consumes messages off of a Rackspace Cloud Queue for some time now. A while ago, I started seeing this exception in my logs at the line where I was creating a claim:
undefined method `split' for nil:NilClass
/home/ash/.rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/fog-1.19.0/lib/fog/rackspace/models/queues/claim.rb:112:in `create'
/home/ash/.rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/fog-1.19.0/lib/fog/rackspace/models/queues/claim.rb:46:in `save'
/home/ash/.rbenv/versions/2.1.0/lib/ruby/gems/2.1.0/gems/fog-1.19.0/lib/fog/rackspace/models/queues/claims.rb:36:in `create'
What's changed? Is this a fog problem, or is there something wrong with Cloud Queues?
It turns out that, after a server upgrade, Cloud Queues started to use lowercase HTTP headers in some of its responses. Specifically, Location is now location and Content-Type changed to content-type. HTTP specifies that HTTP headers are case-insensitive, but as of 1.19.0, Fog's header access is not.
This is now fixed in master, so you can depend on the latest code by adding this to your Gemfile:
gem 'fog', github: 'fog/fog'
Related
There is a strange problem that we have when we deploy our application on the Azure environment. When I start the application on my laptop, no Azure, no Docker or anything, on sending requests (which is a little bit big), I don't face any issues.
Our test and production environments are all on Azure right now. So when the application is deployed on it, I get this strange error:
log4javascript error: AjaxAppender.append: XMLHttpRequest request to URL ./common/logToServer.jsp?controllerName=6c3eaf3e-897d-4b30-a15e-62f9d3d3ce78 returned status code 413
Now I know what HTTP 413 error code is, but not sure, why my local is not showing the same error. Which leads me to believe that it might be some Azure configuration that I need to change. But don't know what.
It is simple web application on Java, Servlets and running on Tomcat.
Log4j is used as a logging framework for JavaScript with no runtime dependencies. As per the error statement, the issue was caused by the length of the payload, which is too large.
The HTTP status code 413 ("Payload Too Large") indicates that the request entity is larger than the limits defined by the server; the server might close the connection.
Fix:
Under java code -> application.properties add these two lines
server.tomcat.max-swallow-size=***MB //maximum size of the request body/payload
server.tomcat.max-http-post-size=*** MB //maximum size of entire POST request
NOTE:
*** is your desired integer representing megabyte.
reference article for more information and solution.
My IdentityServer being hosted in IIS seems to be recycled due to app_offline.htm according to the Windows application log. It seems to happen sometimes when the Discovery endpoint is being requested, but I can't deduce any obvious pattern to when it does or doesn't happen.
The event description is:
Application 'C:\Users\IP21Admin\Desktop\sts\' was recycled after detecting app_offline.htm.
The last few log entries in the IdentityServer log are:
[2021-01-14 17:01:32 DBG] IdentityServer4.Hosting.EndpointRouter
Request path /.well-known/openid-configuration/jwks matched to endpoint type Discovery
[2021-01-14 17:01:32 DBG] IdentityServer4.Hosting.EndpointRouter
Endpoint enabled: Discovery, successfully created handler: IdentityServer4.Endpoints.DiscoveryKeyEndpoint
[2021-01-14 17:01:32 INF] IdentityServer4.Hosting.IdentityServerMiddleware
Invoking IdentityServer endpoint: IdentityServer4.Endpoints.DiscoveryKeyEndpoint for /.well-known/openid-configuration/jwks
[2021-01-14 17:01:32 DBG] IdentityServer4.Endpoints.DiscoveryKeyEndpoint
Start key discovery request
I found a similar problem someone posted as an issue int he ASPNET core repository here:
https://github.com/dotnet/aspnetcore/issues/21528
I've tried using process monitor to find where app_offline.htm comes from, but every single entry is w2wp.exe operation 'CreateFile' with result 'NAME NOT FOUND'.
Process Monitor Output
I've tried manually putting in my own app_offline.htm and the process monitor will give completely different operations, which leads me to suspect that there wasn't actually an app_offline.htm being created previously. Perhaps the Windows Application event is misleading and there's some sort of crash happening?
Anyone have any suggestions as to what could be wrong?
Edit:
So having looked into this further, the IdentityServer4 instance in IIS seems to get recycled after successfully issuing an Authorisation Code Token from token endpoint. I hooked up remote debugging and saw that when the app is getting recycled, there is 100% CPU usage by the IIS worker, and the process memory of app stays constant; don't know if that's useful information or not.
The symptom of the problem is that when OidcClient of my client GUI application proceeds to get UserInfo from the UserInfo endpoint after getting the Authorisation Code Token; before UserInfo, its attempt to load the discovery document fails due to "Server has been shutdown".
Perhaps the TokenEndpoint causes the middleware to get stuck or something causing a recycle? I assume the server can't respond to the DiscoveryEndpoint request because something in(or after) the TokenEndpoint request caused the server app to get recycled before it could respond to the DiscoveryEndpoint request.
With today's attempts, the log now ends at:
[2021-01-15 13:01:23 DBG] IdentityServer4.Endpoints.TokenEndpoint
Token request success.
[2021-01-15 13:01:23 VRB] IdentityServer4.Hosting.IdentityServerMiddleware
Invoking result: IdentityServer4.Endpoints.Results.TokenResult
Turns out the issue was that IIS automatically recycles the app if any file changes in the directory. I had some log files and SQLite files in the folder that was being written to during requests.
Like this closed issue https://github.com/Azure/azure-functions-host/issues/5540 I have issues figuring out what setting I should be changing to allow 100MB files to be uploaded
The weird thing is that the system is deployed in Azure where big files are allowed, but no one have made any changes to settings that should affect this.
So is there some local.settings.json setting that I am missing that is default different when hosting in Azure when compared to localhost
Error:
Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception
while executing function: MessageReceiver --->
System.InvalidOperationException: Exception binding parameter
'request' --->
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException:
Request body too large.
There is https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserverlimits.maxrequestbodysize?view=aspnetcore-3.1
But I cant figure out how to set that when running Azure functions, in the startup I cant set it and setting [DisableRequestSizeLimit] or [RequestSizeLimit(100000000)] on top of my Azure function have no effect
A bug has been reported with problems on Windows https://github.com/Azure/azure-functions-core-tools/issues/2262
The HTTP request length is limited to 100 MB (104,857,600 bytes), and the URL length is limited to 4 KB (4,096 bytes). These limits are specified by the httpRuntime element of the runtime's Web.config file.
If a function that uses the HTTP trigger doesn't complete within 230 seconds, the Azure Load Balancer will time out and return an HTTP 502 error. The function will continue running but will be unable to return an HTTP response. For long-running functions, we recommend that you follow async patterns and return a location where you can ping the status of the request. For information about how long a function can run, see Scale and hosting - Consumption plan.
For more details, you could refer to this article.
I have several message handlers in a particular endpoint that do their work against a SQL Azure database (at the moment still using a local SQL 2012 instance). I have a command handler that publishes 2 events, call them X and Y. In the same endpoint I have a subscriber to X and a subscriber to Y. Both of these subscribers are internally using the same data access component, call that Z. Dependency injection is configured on a per-call basis, not shared.
Component Z is using Entity Framework 6 under the curtains. The issue I am having is that just opening the database is throwing a SqlException and complaining about MSDTC escalations.
I have temporarily wrapped the handlers in a TransactionScope.Suppress and that has stopped the error but I believe I'm missing something more fundamental.
Is it a simple matter of configuring the endpoint to be non-transactional? I would have thought this would just work seeing as I've configured to use Azure Service Bus as the transport mechanism. If I do this will NServiceBus still retry if an exception is thrown within the message handler? (Up to the SLR limits -- not part of the question, I also understand the idempotency issues).
#Phil,
First, you shouldn't be using MSDTC with SQL Azure - it's not supported. The feature is suggested, but only under review. DTC is not supported on Azure. Alternatively, you could look into the following suggestion to use SqlTransaction approach.
Second, transport you're using has nothing to do with your data access. Since you're using Azure Service Bus, it will not be part of your handler code. Making handler a transactional is to force an atomic change or roll-back. Regardless of your handler, will retry. Challenge is that when handler/endpoint is not transactional, and within handler first write to DB succeeded and second failed, first write won't be reverted. As for Azure Service Bus as a transport, it's not transactional in its nature (ie no DTC).
Which version of NServiceBus.Azure are you on? Do you have a stack trace of the exception? Where does it come from?
We push the sends and publishes out of the scope of the receive transaction scope explicitly to prevent promotion to the DTC, so that the transaction is local to the sql, so I doubt that is what is happening here.
From you description it looks like you are using a different data access instance for each handler (per call container config) and you have multiple handlers on the same message. If both of these open a new connection to the SQL you would see promotion as well (even if it is the same server)
Could that be it? That it throws on the second open?
Background / Issue
Having a strange issue running a Ghost blog on Azure. The site seems to run fine for a while, but every once in a while, I'll receive a 500 error with no further information. The next request always appears to succeed (in tests so far).
The error seems to happen after a period of inactivity. Since I'm currently just getting set up, I'm utilizing an Azure "Free" instance, so I'm wondering if some sort of resource conservation is causing it behind the scenes (which will be allevaited when I upgrade).
Any idea what could be causing this issue? I'm sort of at a loss for where to start since the logs don't necessarily help me in this case. I'm new to NodeJS (and nodeJS on Azure) and since this is my first foray, any tips/tricks on where to look would be helpful as well.
Some specific questions:
When receiving an error like this, is there anywhere I can go to see any output, or is it pretty much guaranteed that Node actually didn't output something?
On Azure free instances, does some sort of resource conservation take place which might cause the app to be shut down (and thus for me to see these errors only after a period of inactivity)?
The Full Error
The full text of the error is below (I've turned debugging on for this reason):
iisnode encountered an error when processing the request.
HRESULT: 0x2
HTTP status: 500
HTTP reason: Internal Server Error
You are receiving this HTTP 200 response because system.webServer/iisnode/#devErrorsEnabled configuration setting is 'true'.
In addition to the log of stdout and stderr of the node.exe process, consider using debugging and ETW traces to further diagnose the problem.
The node.exe process has not written any information to stderr or iisnode was unable to capture this information. Frequent reason is that the iisnode module is unable to create a log file to capture stdout and stderr output from node.exe. Please check that the identity of the IIS application pool running the node.js application has read and write access permissions to the directory on the server where the node.js application is located. Alternatively you can disable logging by setting system.webServer/iisnode/#loggingEnabled element of web.config to 'false'.
I think it might be something in the Azure web config rather than Ghost itself. So look for logs based on that because Ghost is not throwing that error. I found this question that might help you out:
How to debug Azure 500 internal server error
Good luck!