I'm always reluctant to claim that a bug that I'm seeing is actually a .Net Core bug, but after spending 8+ hours investigating the following bug, it looks like a .Net Core SignalR bug to me. I need techniques for tracking this down further and for fixing it.
The first rule of honing in on a bug is to try to create a minimal amount of code that can consistently repro the bug. While I can't reproduce it in a small stand along project, I have worked hard try to zero in on what's happening.
I have a controller with the following action method
[HttpPost]
[Route("/hack/ajax/start")]
public JsonResult AjaxStart([FromBody] JObject data) {
//A call to some method that does some work
return Json(new {
started = true
});
}
Calling this code via a jquery ajax call or Postman works flawlessly every time if I do not have any SignalR Core 1.0 hubs registered in the startup.cs method. However, when I register the following in the startup.cs file I have intermittent issues.
namespace App.Site.Home {
public class HackHub : Hub {
public async Task SendMessage(string status, string progress) {
await Clients.All.SendAsync("serverMsg", status, progress);
}
}
}
Startup.cs ConfigureServices contains
services.AddSignalR();
Startup.cs Configure contains
app.UseSignalR(routes => {
routes.MapHub<App.Site.Home.HackHub>("/hub/hack");
});
If I were to comment out the one line above routes.MapHub<App.Site.Home.HackHub>("/hub/hack"); everything works fine every time. However with this line present, (I.e. some SignalR hub registered) then that's when the fun starts for me, even if I have no code executing on the client or server that makes use of the hub!
The issue is that sometimes when a HTTP POST request is made for the action method above, something in .Net Core (SignalR??) is converting the POST method to Post, and then because Post is not a valid HTTP Method it converts it to a blank method. And since My action method requires an HTTP POST a 404 status code is returned. Many of the HTTP POSTS for that endpoint work fine, but often the issue I just described occurres.
To ensure that my client code was not part of the problem, I was able to reproduce my issue using Postman to make the requests. Further to ensure that POST was actually being sent and not Post, I used Fiddler to watch what was going over the wire. All this is documented below.
Here is the first request (which always works) done via Postman:
Here is the second (identical!) request done via Postman, this one resulted in a 404:
Here is what the first request (the one that worked properly) looked like in fiddler:
Here is what the second request looked like in fiddler:
As you can see, the requests are identical. But the response certainly is not.
So to get a better idea what the server was seeing, I added the following code to the beginning of the startup.cs Configure method. Due to it's placement, for the request this code gets to run before any other application code or middleware.
public void Configure(IApplicationBuilder app, IHostingEnvironment env) {
//for debugging
app.Use(async (context, next) => {
if(context.Request.Method == "") {
string method = context.Request.Method;
string path = context.Request.Path;
IHttpRequestFeature requestFeature = context.Features.Get<IHttpRequestFeature>();
string kestralHttpMethod = requestFeature.Method;
string stop = path;
}
await next();
});
//more code here...
}
For the first request, the request.Method was POST as one would expect:
But for the second request request.Method was blank!!
To investigate this further, I accessed the requestFeature and checked the Http Method Method there. This is where things get really interesting. If I just hover over the property in the debuggger, it's blank too.
But, If I expand the requestFeature object and look at the Method property there, is it Post!!!
That alone seems like craziness. How can two views of the SAME property in the debugger have different values???! It would seem that some code converted the POST to Post, and at some level the system knows that Post is not a valid http method so in some views of that variable it's converted to a blank string. But that is so weird!
Also, we clearly saw via Postman and Fiddler that POST was sent, so how did it get changed to Post? What code did that? I'd like to claim that it can't be my code since I'm checking the value of the RequestFeature before any of my other code related to the request gets a chance to run. Also, if I comment out the one line of code that registers that SignalR hub, then POST is never converted to Post and I never get a 404. But with that SignalR hub registered I periodically get this behavior.
Are there any SignalR or other .net Core switches I can turn on to get better trace or logging info to see when the POST is getting changed to Post? Is there a way to fix this?
This question was looked into via this GitHub issue https://github.com/aspnet/KestrelHttpServer/issues/2591 which was originally opened up when someone else also observed random 404 errors
I want to especially thank #ben-adams for his help in understanding what was going on.
Let me start by saying that this did not turn out to be a bug in the framework. It was a bug in my code. How can that be given what I was observing?
Well, it's like this...
In some parts of the HttpRequest the method is a string, but it in other parts it's an enum. The enum value for POST is Post. So that's why the case conversion was happening.
The reason that one part of the request was saying Post while the other part showed a Method value of a blank string was because the request object was gummed up because I had accessed it at a time when it was in between requests.
How did I do THAT? you may wonder. Well let me tell you, because the plot thickens...
I turns out that I have some logging code that that gathers context information when it's called and one of the pieces of context info it gathers is the current request.Method. When this logging code is called from a main thread, there is no issue.
However, my system does have some code that runs on background threads that are either started via a Timer or via a ThreadPool.QueueUserWorkItem. If this code hits an exception it will call the same logger code.
When my logger code, running on a background thread, checks for the current httpContext via IHttpContextAccessor I fully expected it to receive null. And certainly this same code in the same situation when accessing the current HttpContext via HttpContext.Current in a non .Net Core website does receive null. But as it turns out, under .Net core, it wasn't receiving null, it was receiving an object. But that object was for a request that had already finished and who's request object had already been reset!!!
Starting in .Net Core 2.0 the HttpContext, and it's child objects like request, gets reset after the connection for a request closes. So the HttpContext object (and it's request object) the logger code was getting when running on a background thread was an object that had been reset. It's request.Path for example was null.
It turns out that a request in this state does not expects it's request.Method property to be accessed. And doing so gums up the works for the next request that comes in. Ultimately this is the source of why the next request that came in ended up returning a 404 error.
So how do we fix this? Why does IHttpContextAccessor return an object rather than null in this out of context situation especially given that the object may very possibly be between requests? The answer is that when I was used Timer or ThreadPool.QueueUserWorkItem to create a background task, the Execution Context was being flowed to the new thread. This is just what happens by default when you use these API methods. But, internally the IHttpContextAccessor uses an AsyncLocal to keep track of the current HttpContext and since my new thread received Execution Context from the main thread it had access to the same AsyncLocal. And so IHttpContextAccessor provided an object rather than the null I was expecting when called from a background thread.
The fix? (Thank you #Ben-Adams!) Instead of calling ThreadPool.QueueUserWorkItem I needed to call ThreadPool.UnsafeQueueUserWorkItem instead. This method DOES NOT flow the current Execution Context to the new thread, and therefore the new thread won't have access to those AsyncLocals from the main thread. Once I did this, IHttpContextAccessor then returned null when called from the background thread instead of returning a object that was in between requests and untouchable. Yea!
When creating a `Timer' I also needed to change my code to do it in a way that would not flow Execution Context. Here is the code I use (which was inspired by some #Ben-Adams suggested):
public static Timer GetNewTimer(TimerCallback callback, object state, int dueTime, int interval) {
bool didSuppress = false;
try {
if (!ExecutionContext.IsFlowSuppressed()) {
//We need to suppress the flow of the execution context so that it does not flow to our
//new asynchronous thread. This is important so that AsyncLocals (like the one used by
//IHttpaccessor) do not flow to the new thread we are pushing our work to. By not flowing the
//execution context, IHttpAccessor wil return null rather than bogusly returning a context for
//a request that is in between requests.
//Related info: https://github.com/aspnet/KestrelHttpServer/issues/2591#issuecomment-399978206
//Info on Execution Context: https://blogs.msdn.microsoft.com/pfxteam/2012/06/15/executioncontext-vs-synchronizationcontext/
ExecutionContext.SuppressFlow();
didSuppress = true;
}
return new Timer(callback, state, dueTime, interval);
} finally {
// Restore the current ExecutionContext
if (didSuppress) {
ExecutionContext.RestoreFlow();
}
}
}
This only leaves one remaining question unanswered. My original question noted that registering a SignalR hub was causing the system to exhibit this random 404 behavior but the system did not exhibit this behavior when no SignalR hub was registered (or so I thought). Why was this? I truly don't know. Perhaps it was putting more resource pressure on some part of the system and thus causing the issue to show up more easily. Not sure. All I know is that the root issue was that I was flowing Execution Context to my background threads without realizing it and that was causing the IHttpContextAccessor's AsyncLocal to be in scope. Not flowing the Execution Context to the background threads fixes that issue.
Related
I have a method for authentication and it will be called for all the requests made by UI. When a multiple requests sent from client same time I am getting an exception
HResult=-2146233079
Message=There is already an open DataReader associated with this Command which must be closed first.
We are using NHibernate. And here I'm trying to get a single row data for all the requests.
And one more thing while mapping tables and in Ioc we are using lifeStyle as Transient.
Can some one please tell how to handle this in NHibernate ?
It seems that you're using the same session across several client requests that are processed in different threads (sorry, it's not clear from your question). Check your design - you should use different session instances in different threads
Thanks for the reply.
While opening a session will check 3 conditions.
ISession GetSession()
{
if (_session == null)
_session = _sessionFactory.OpenSession();
if (_session.IsOpen == false)
_session = _sessionFactory.OpenSession();
if (_session.Connection.State != System.Data.ConnectionState.Open)
{
_session.Connection.Close();
_session.Connection.Open();
}
return _session;
} code here
Based on this I'll get the session.
For single api request it works fine. But when multiple web api requests were sent then I am facing a problem in getting data from DB. getting error as explained above.
I am using this method from the azure mobile services tutorial:
await todoTable.LookupAsync(id). I have 2 rows in a table of id 1,2.
If i do await todoTable.LookupAsync(1), it works and return the record. If i do
await todoTable.LookupAsync(8) to see how it's going to handle null, it just blows up with Not Found exception.
Thanks for help on this.
NULL would mean there is a record for id = 8, but its value is `NULL'. But in your case you do not have a record. Which is different.
What you observe is what you should observe if you do not have a record.
And this is a standard for REST based HTTP services. If record is not there, you get an HTTP 404 from the service.
Azure mobile services is nothing more than a combination of Web API and a wrapping (plumbing) code for your application. And every Web API call to a non-existent record would result into an HTTP 404 error.
And as already said in the comments, you should wrap your code around try - catch blocks and inspect the exception.
In .NET 4.5/4.6 there is new HttpClient type along with HttpResponseMessage and HttpRequestMessatge. The former has EnsureSuccessStatusCode() method. Which, if called will trigger exception.
In the older versions of the Framework there WebClient class, which would throw an exception if the HTTP status code is not 200.
So, again, at the end - you observe absolutely normal behavoir. Just have to read a little more about HTTP REST services, HTTP VERBS and HTTP Status Codes. Then also understand how the particular framework you use (.NET) handles the HTTP Status Codes.
I have created my own REST service based on the examples from "Domino Sample REST Service Feature" from 901v00_11.20141217-1000 version of XPages Extension Library.
As far as I understand the design of the library each REST request will be run in its own thread on the server. This approach does not allow to handle parallel POST requests to the same document.
I have not found any examples in XPages Extension Library which would handle post requests as transactions on the server, i.e. which would block the server resource for the whole request processing time and would put put next requests in the queue?
Can anybody point to the source code of the service which would allow to handle parallel requests?
The skeleton for my post request processing function is this
#POST
#Path(PATH_SEPARATOR + MyURL)
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Response myPost(
String requestEntity,
#Context final UriInfo uriInfo)
{
LogMgr.traceEntry(this, "myPost");
RestContext.verifyUserContext();
String myJson = ... // Process post
Response response = buildResponse(myJson);
LogMgr.traceExit(this, "myPost", "OK");
return response;
}
And I would like to implement something like this
// Start transaction
String myJson = ... // Process post
// Stop transaction
Is there a way to do it in Java?
I suppose you could use document locking in traditional Notes/Domino context - and synchronized in Java :-)
Have you tried any of these? I cannot see why they should not work.
/John
I agree with John. You can use document locking to prevent simultaneous updates to the same document. You might also want to consider some changes to the definition of your REST API.
First, you imply you are using POST to update an existing document. Usually, POST is used to create a new resource. Consider using PUT instead of POST.
Second, even with document locking, you still might want to check for version conflicts. For example, let's say a client reads version 2 of a document and then attempts to update it. Meanwhile, another client has already updated the document to version 3. Many REST APIs use HTTP ETags to handle such version conflicts.
I am using signal R 1.0 with c#
the issue is most of the time with IE(v 10) and sometimes with chrome(v 28).
My client methods are not being executed.
I have chat functionality and on page load in
$(document).ready(function(){
//here i call server method to create group between two users
});
chat.client.groupcreated = function(){} //this is not invoked
on server side i write Client.groupcreated()
It is working perfact in FF. More interesting thing is if i put a break point in my cs code at server side it works perfact in IE also
Oppss...... I was doing a silly mistake
In the server method to create group i was doing something like below
Groups.Add(connectionid1,groupname);
Groups.Add(connectionid2,groupname);
Clients.Group(strGroupName).hellworld(); //no users added yet to the group while this line executes
The problem with above code is adding members to the signalr group is actually asynchronous, so sending messages to that group immediately won't work as the adding members to group may have not yet finished.
Solution
Groups.Add(connectionid1, groupname).ContinueWith(t =>
Groups.Add(connectionid2, groupname).ContinueWith(t2 =>
Clients.Group(groupname).helloworld());
wait for the task to complete before calling client method
By the way, thanks for posting your comments
In my previous project, I use a framework (Agatha RRSL) similar to ServiceStack, in that everything is made of Request, Response and Handler. It also has Interceptors that can attach to handler and you can inject other interfaces to the handler as well. I can use this to open a transaction BeforeHandling, access to both request and response in AfterHandling, create audit, save to database and close transaction if needed.
I try to experiment similar with SerivceStack. But seems with Filters, I can't grab request and response together?
With custom ServiceRunner. When I try to debug OnAfterExecute(...), I can see the name of my request dto in IRequestContext {ServiceStack.ServiceHost.HttpRequestContext}. But just the name, I couldn't figure out how to retrieve the actual request object to work with the response object.
Another thing I haven't figure out is if it's possible to inject the auto wired service interface into it, like a db context or audit service. Maybe this one is too far ahead in the pipeline?
The final thing is, it seems I can only register one custom service runner? With Interceptor, I can drop a bunch of them, and they will wrap around each other.
Any thoughts? Thanks
The RequestContext also contains the HttpRequest and HttpResponse which you can get access with:
var httpReq = RequestContext.Get<IHttpRequest>();
var httpRes = RequestContext.Get<IHttpResponse>();
See the docs on Accessing HTTP Specific features for more info.