We have a cluster with 3 servers with Load Balancer in front (CloudFlare). Things worked well when we had 2 servers (A & B) in the cluster but after we added a 3-rd server (C) we noticed few odd things.
One of them is quite important and I do not understand how it happens at all.
Our web application makes AJAX requests to itself in order to get some JSON data back and if requests hit new server (C) response looks like that:
{
code: 404,
text: "Not Found",
message: "Database context not allowed."
}
Our application does not throw such error and so I searched in google a bit and noticed that it's mentioned on: OpenNTF XPagesExtensionLibrary
However, we do not use XPages at all so I wonder how could it be that our AJAX requests somehow involve that logic.
Any suggestion & tip would be appreciated.
UPDATE
The backend code of my agent is not important (it could be also an empty agent, I checked), because the request does not come to my agent.
The AJAX call is triggered by jQuery
let url = "domain.tld/api/key";
let params = {"a": 1};
$.post(url, params, function (data) {
// some code
},
"json"
).always(function() {
// some code
});
The URL, which I suspect is an issue starts with /api/key and I believe it's an issue (because all other ajax calls where endpoint do not start from /api/ work well).
Thanks.
Figured that our with help from comments (which you can see under my original post).
Apparently there is DAS servlet that handles all requests starting from /api/* and it runs if XPages engine is loaded.
In my case the 2 servers out of 3 have XPages shut down so the issue happened only on 1 server.
The solution would be:
Shut down XPages (or find a way to shut down DAS).
Alternatively change a URL from /api/path to something else (this is what we will do).
Related
I'm always reluctant to claim that a bug that I'm seeing is actually a .Net Core bug, but after spending 8+ hours investigating the following bug, it looks like a .Net Core SignalR bug to me. I need techniques for tracking this down further and for fixing it.
The first rule of honing in on a bug is to try to create a minimal amount of code that can consistently repro the bug. While I can't reproduce it in a small stand along project, I have worked hard try to zero in on what's happening.
I have a controller with the following action method
[HttpPost]
[Route("/hack/ajax/start")]
public JsonResult AjaxStart([FromBody] JObject data) {
//A call to some method that does some work
return Json(new {
started = true
});
}
Calling this code via a jquery ajax call or Postman works flawlessly every time if I do not have any SignalR Core 1.0 hubs registered in the startup.cs method. However, when I register the following in the startup.cs file I have intermittent issues.
namespace App.Site.Home {
public class HackHub : Hub {
public async Task SendMessage(string status, string progress) {
await Clients.All.SendAsync("serverMsg", status, progress);
}
}
}
Startup.cs ConfigureServices contains
services.AddSignalR();
Startup.cs Configure contains
app.UseSignalR(routes => {
routes.MapHub<App.Site.Home.HackHub>("/hub/hack");
});
If I were to comment out the one line above routes.MapHub<App.Site.Home.HackHub>("/hub/hack"); everything works fine every time. However with this line present, (I.e. some SignalR hub registered) then that's when the fun starts for me, even if I have no code executing on the client or server that makes use of the hub!
The issue is that sometimes when a HTTP POST request is made for the action method above, something in .Net Core (SignalR??) is converting the POST method to Post, and then because Post is not a valid HTTP Method it converts it to a blank method. And since My action method requires an HTTP POST a 404 status code is returned. Many of the HTTP POSTS for that endpoint work fine, but often the issue I just described occurres.
To ensure that my client code was not part of the problem, I was able to reproduce my issue using Postman to make the requests. Further to ensure that POST was actually being sent and not Post, I used Fiddler to watch what was going over the wire. All this is documented below.
Here is the first request (which always works) done via Postman:
Here is the second (identical!) request done via Postman, this one resulted in a 404:
Here is what the first request (the one that worked properly) looked like in fiddler:
Here is what the second request looked like in fiddler:
As you can see, the requests are identical. But the response certainly is not.
So to get a better idea what the server was seeing, I added the following code to the beginning of the startup.cs Configure method. Due to it's placement, for the request this code gets to run before any other application code or middleware.
public void Configure(IApplicationBuilder app, IHostingEnvironment env) {
//for debugging
app.Use(async (context, next) => {
if(context.Request.Method == "") {
string method = context.Request.Method;
string path = context.Request.Path;
IHttpRequestFeature requestFeature = context.Features.Get<IHttpRequestFeature>();
string kestralHttpMethod = requestFeature.Method;
string stop = path;
}
await next();
});
//more code here...
}
For the first request, the request.Method was POST as one would expect:
But for the second request request.Method was blank!!
To investigate this further, I accessed the requestFeature and checked the Http Method Method there. This is where things get really interesting. If I just hover over the property in the debuggger, it's blank too.
But, If I expand the requestFeature object and look at the Method property there, is it Post!!!
That alone seems like craziness. How can two views of the SAME property in the debugger have different values???! It would seem that some code converted the POST to Post, and at some level the system knows that Post is not a valid http method so in some views of that variable it's converted to a blank string. But that is so weird!
Also, we clearly saw via Postman and Fiddler that POST was sent, so how did it get changed to Post? What code did that? I'd like to claim that it can't be my code since I'm checking the value of the RequestFeature before any of my other code related to the request gets a chance to run. Also, if I comment out the one line of code that registers that SignalR hub, then POST is never converted to Post and I never get a 404. But with that SignalR hub registered I periodically get this behavior.
Are there any SignalR or other .net Core switches I can turn on to get better trace or logging info to see when the POST is getting changed to Post? Is there a way to fix this?
This question was looked into via this GitHub issue https://github.com/aspnet/KestrelHttpServer/issues/2591 which was originally opened up when someone else also observed random 404 errors
I want to especially thank #ben-adams for his help in understanding what was going on.
Let me start by saying that this did not turn out to be a bug in the framework. It was a bug in my code. How can that be given what I was observing?
Well, it's like this...
In some parts of the HttpRequest the method is a string, but it in other parts it's an enum. The enum value for POST is Post. So that's why the case conversion was happening.
The reason that one part of the request was saying Post while the other part showed a Method value of a blank string was because the request object was gummed up because I had accessed it at a time when it was in between requests.
How did I do THAT? you may wonder. Well let me tell you, because the plot thickens...
I turns out that I have some logging code that that gathers context information when it's called and one of the pieces of context info it gathers is the current request.Method. When this logging code is called from a main thread, there is no issue.
However, my system does have some code that runs on background threads that are either started via a Timer or via a ThreadPool.QueueUserWorkItem. If this code hits an exception it will call the same logger code.
When my logger code, running on a background thread, checks for the current httpContext via IHttpContextAccessor I fully expected it to receive null. And certainly this same code in the same situation when accessing the current HttpContext via HttpContext.Current in a non .Net Core website does receive null. But as it turns out, under .Net core, it wasn't receiving null, it was receiving an object. But that object was for a request that had already finished and who's request object had already been reset!!!
Starting in .Net Core 2.0 the HttpContext, and it's child objects like request, gets reset after the connection for a request closes. So the HttpContext object (and it's request object) the logger code was getting when running on a background thread was an object that had been reset. It's request.Path for example was null.
It turns out that a request in this state does not expects it's request.Method property to be accessed. And doing so gums up the works for the next request that comes in. Ultimately this is the source of why the next request that came in ended up returning a 404 error.
So how do we fix this? Why does IHttpContextAccessor return an object rather than null in this out of context situation especially given that the object may very possibly be between requests? The answer is that when I was used Timer or ThreadPool.QueueUserWorkItem to create a background task, the Execution Context was being flowed to the new thread. This is just what happens by default when you use these API methods. But, internally the IHttpContextAccessor uses an AsyncLocal to keep track of the current HttpContext and since my new thread received Execution Context from the main thread it had access to the same AsyncLocal. And so IHttpContextAccessor provided an object rather than the null I was expecting when called from a background thread.
The fix? (Thank you #Ben-Adams!) Instead of calling ThreadPool.QueueUserWorkItem I needed to call ThreadPool.UnsafeQueueUserWorkItem instead. This method DOES NOT flow the current Execution Context to the new thread, and therefore the new thread won't have access to those AsyncLocals from the main thread. Once I did this, IHttpContextAccessor then returned null when called from the background thread instead of returning a object that was in between requests and untouchable. Yea!
When creating a `Timer' I also needed to change my code to do it in a way that would not flow Execution Context. Here is the code I use (which was inspired by some #Ben-Adams suggested):
public static Timer GetNewTimer(TimerCallback callback, object state, int dueTime, int interval) {
bool didSuppress = false;
try {
if (!ExecutionContext.IsFlowSuppressed()) {
//We need to suppress the flow of the execution context so that it does not flow to our
//new asynchronous thread. This is important so that AsyncLocals (like the one used by
//IHttpaccessor) do not flow to the new thread we are pushing our work to. By not flowing the
//execution context, IHttpAccessor wil return null rather than bogusly returning a context for
//a request that is in between requests.
//Related info: https://github.com/aspnet/KestrelHttpServer/issues/2591#issuecomment-399978206
//Info on Execution Context: https://blogs.msdn.microsoft.com/pfxteam/2012/06/15/executioncontext-vs-synchronizationcontext/
ExecutionContext.SuppressFlow();
didSuppress = true;
}
return new Timer(callback, state, dueTime, interval);
} finally {
// Restore the current ExecutionContext
if (didSuppress) {
ExecutionContext.RestoreFlow();
}
}
}
This only leaves one remaining question unanswered. My original question noted that registering a SignalR hub was causing the system to exhibit this random 404 behavior but the system did not exhibit this behavior when no SignalR hub was registered (or so I thought). Why was this? I truly don't know. Perhaps it was putting more resource pressure on some part of the system and thus causing the issue to show up more easily. Not sure. All I know is that the root issue was that I was flowing Execution Context to my background threads without realizing it and that was causing the IHttpContextAccessor's AsyncLocal to be in scope. Not flowing the Execution Context to the background threads fixes that issue.
I am using this method from the azure mobile services tutorial:
await todoTable.LookupAsync(id). I have 2 rows in a table of id 1,2.
If i do await todoTable.LookupAsync(1), it works and return the record. If i do
await todoTable.LookupAsync(8) to see how it's going to handle null, it just blows up with Not Found exception.
Thanks for help on this.
NULL would mean there is a record for id = 8, but its value is `NULL'. But in your case you do not have a record. Which is different.
What you observe is what you should observe if you do not have a record.
And this is a standard for REST based HTTP services. If record is not there, you get an HTTP 404 from the service.
Azure mobile services is nothing more than a combination of Web API and a wrapping (plumbing) code for your application. And every Web API call to a non-existent record would result into an HTTP 404 error.
And as already said in the comments, you should wrap your code around try - catch blocks and inspect the exception.
In .NET 4.5/4.6 there is new HttpClient type along with HttpResponseMessage and HttpRequestMessatge. The former has EnsureSuccessStatusCode() method. Which, if called will trigger exception.
In the older versions of the Framework there WebClient class, which would throw an exception if the HTTP status code is not 200.
So, again, at the end - you observe absolutely normal behavoir. Just have to read a little more about HTTP REST services, HTTP VERBS and HTTP Status Codes. Then also understand how the particular framework you use (.NET) handles the HTTP Status Codes.
I have created my own REST service based on the examples from "Domino Sample REST Service Feature" from 901v00_11.20141217-1000 version of XPages Extension Library.
As far as I understand the design of the library each REST request will be run in its own thread on the server. This approach does not allow to handle parallel POST requests to the same document.
I have not found any examples in XPages Extension Library which would handle post requests as transactions on the server, i.e. which would block the server resource for the whole request processing time and would put put next requests in the queue?
Can anybody point to the source code of the service which would allow to handle parallel requests?
The skeleton for my post request processing function is this
#POST
#Path(PATH_SEPARATOR + MyURL)
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Response myPost(
String requestEntity,
#Context final UriInfo uriInfo)
{
LogMgr.traceEntry(this, "myPost");
RestContext.verifyUserContext();
String myJson = ... // Process post
Response response = buildResponse(myJson);
LogMgr.traceExit(this, "myPost", "OK");
return response;
}
And I would like to implement something like this
// Start transaction
String myJson = ... // Process post
// Stop transaction
Is there a way to do it in Java?
I suppose you could use document locking in traditional Notes/Domino context - and synchronized in Java :-)
Have you tried any of these? I cannot see why they should not work.
/John
I agree with John. You can use document locking to prevent simultaneous updates to the same document. You might also want to consider some changes to the definition of your REST API.
First, you imply you are using POST to update an existing document. Usually, POST is used to create a new resource. Consider using PUT instead of POST.
Second, even with document locking, you still might want to check for version conflicts. For example, let's say a client reads version 2 of a document and then attempts to update it. Meanwhile, another client has already updated the document to version 3. Many REST APIs use HTTP ETags to handle such version conflicts.
The app I'm working on has a controller that issues templates to the front end (single page app). It's very basic, and simply consists of
#path = params[:path]
render template: "templates/#{#path}", layout: nil
Here my concern however is the direct use of the users input. Everything about this to me feels like it can be attacked with something as simple as path traversal. The route for this is
get "/templates/:path.html" => "templates#file", constraints: { path: /.+/ }, defaults: { format: 'html' }
I've tried multiple things to attempt a path traversal attack, such as
request /templates/path/to/../somewhere/else.html
request /templates?path=/path/to/../../something.rb
request /templates/index.html?path=/path/to/../../config/something.html
request /templates/path/../../../file.html
Fortunately, I haven't had any success with this. The requests that just start with /templates and don't specify anything after it, don't match the route thanks to the constraint so that is good.
It seems as though when that route is matched, rails doesn't allow you to override the path parameter through a url parameter, so I don't seem to be able to inject it there.
The ones that interest are the first and last examples above, where rails seems to internally be changing the requested URL before invoking the routes file. When I request /templates/path/to/../somewhere/else.html, my console output shows a request for /templates/path/somewhere/else.html. When I make a request for /templates/path/../../../file.html, the log shows a request for /file.html.
Am I missing something somewhere that will leave the app open to security issues, or is this just rails being sensible and protecting itself for me?
UPDATE
I've done some more digging, and if I try doing some URL encoding then I can cause the server to simply not respond at all. If I request /templates/%2e%2e%2f%2e%2e%2f%2e%2e%2ffresult.html then I just get an empty response with a connection: close header.
I assume that the parameter parser higher up in the rack is checking all urls for this type of attack? Regardless, my original question still stands. Am I missing something here?
I'm writing a very small app to create and test caml querys for sharepoint. While executing the GetListItems method I'm receiving the following exception;
System.Net.WebException: "The request failed with an empty response."
The service is located on a https address (ssl). I setup the service as follows;
result = new ListService.Lists();
result.Url = siteUrl;
result.Credentials = new NetworkCredential(txtUserName.Text, txtPassword.Text, txtDomain.Text);
I invoke the GetListItems() method as follows;
xmlResult = spList.GetListItems(listName, string.Empty, camlQuery, null, string.Empty, null, string.Empty);
I'm trying to find out why I'm getting the empty result message. I've also tried other methods (i.e. GetListCollection) but to no avail.
At first I thought that the problem might be the URL (http instead of https), but that is not the case. I even checked it with wireshark to make sure the right URL is used.
Did someone come accross this problem and how did you solve it?
OMG...! I've solved it after all. After posting this question, I tried to get hold of the wsdl the check the service itself. When I checked it via internet explorer all was ok. When I tried to add it as a reference in VS it went wrong. So something was not ok. Then it occured to me that there was a new login screen for our company network.
After some quick phonecalls I learned what the problem was; IT Services installed a ISA 2006 server and 'forgot' to tell me. The ISA server was blocking all trafic on the HTTPS port (443) for unkown programs and or clients. That's why internet explorer presented me with a new and shiny login dialog.