I am using retrofit to manage my remote api calls
api "com.squareup.retrofit2:retrofit:2.9.0"
api "com.squareup.retrofit2:converter-scalars:2.9.0"
api 'com.squareup.okhttp3:logging-interceptor:5.0.0-alpha.5'
api 'com.squareup.okhttp3:okhttp:5.0.0-alpha.5'
one of these calls downloads large pdf files as shown here
#Streaming
#GET
suspend fun download(
#HeaderMap headers: Map<String, String> = emptyMap(),
#Url url: String
): ResponseBody
this all works fine until i experience any HTTP error such as 404, i have to wrap my code in a try catch block and receive very little information regarding the exception
how can I detect errors when employing ResponseBody as a return type on my retrofit calls?
I employed ResponseBody to support streaming the files and so improving my overall download performance.
Related
---UPDATED 10/20/2022---
After some further digging, it turns out the problem described below has nothing to do with aiohttp. The issue is that the response (which is a FHIR resource bundle) contains paginated data. And the URL that the server was returning for the pagination links was invalid (it has the hshweb domain referenced in my question below; my guess is it referred to some internal address that's not exposed outside their firewall). My sample code below shows fetching the initial response, what's not shown is that there is additional processing to get subsequent pages. And because the pagination links were invalid, it was those subsequent page fetches that were failing.
---ORIGINAL POST---
Background
I've been using aiohttp to fetch JSON data from APIs of different companies; my code has been running fine for over a year now.
In the code snippet below, I query the endpoint, and if the status is OK I then get the JSON content using response.json
async with await session.get(url, headers=self.headers) as response:
logger.info(f'Received response to {url}',
extra={**self.log_dict, 'httpstatus': response.status}
)
if response.status in [200, 206]:
try:
response_json = await response.json(content_type=None)
break
except json.decoder.JSONDecodeError:
response_json = None
logger.warning((f'Invalid JSON querying {url}; '
f'status code {response.status}. Skipping.'),
extra=self.log_dict
)
break
This has been working as expected so far.
Problem
With the latest company whose API I'm attempting to integrate with, I'm hitting API https://api.mvphealthcare.com.... and am getting a 200 response status, however when attempting to then get the JSON payload with response.json() I get the following error:
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host hshweb.mvphealthcare.com:443 ssl:default [getaddrinfo failed]
The identical GET request from Postman works exactly as expected.
Note that the URL in this error message (hshweb.mvphealthcare.com) is different from the original that I queried (api.mvphealthcare.com).
Attempting to ping hshweb.mvphealthcare.com from the command line results in Ping request could not find host hshweb.mvphealthcare.com. Please check the name and try again. so the getaddrinfo failed error itself sort of makes sense. What's unclear is why aiohttp is attempting to hit that other address.
While not exactly the same issue, this SO question suggested that there are instances where using response.json() after response.ok could be problematic. So I temporarily modified my code to call await response.json() before the call to response.status and was able to get the response! As the SO answer suggested the issue is fixed in aiohttp 3.8, I upgraded my library (was previously using 3.7.4). However, that did not resolve the problem if calling response.json() was after response.status.
Questions
Why does calling reponse.json() work if it is before response.status, but not after?
What's going on behind the scenes that's causing the URL to change (and if hshweb.mvphealthcare.com is not resolvable anyway, why does it work if response.json() is called before response.status)
What do I need to do to make this work?
I'm having some trouble getting a GET Action with body to work in BluePrism (using web api service).
It seems that when I try to include an Action that sends a GET with body when I reach that stage this error gets thrown:
Internal : Unexpected error Cannot send a content-body with this verb-type.
What I've tried:
Using a different verb-type / passing parameters in the query instead of the body, unfortunately i don't have control over the endpoint i'm trying to reach so this didnt work as it only accepts a GET containing data in the body
Using BluePrism HTTP Utility to send the call, this has the same problem as the Web API Service
Compiling the body via code instead of using a template
I haven't been able to find anyone that made it work in BluePrism and there doesn't seem to be much documentation on this issue in BluePrism so any help would be appreciated.
Thanks!
.NET's WebRequest class that Blue Prism uses under the hood to conduct
these requests will not allow you to send a request body with any GET
request. There is no (documented) way to overcome this limitation.
While other related answers on Stack
Overflow correctly state
that there exists no such prohibition on including request bodies with
GET requests per RFC
9110§9.3.1, it is
very unusual for a production-grade service to require that the request
itself include anything in the request body. It's also possible that
intermediaries like HTTP proxies may strip or otherwise mangle the request
body in transit anyway.
There is no out-of-the-box way to force the .NET Framework (which Blue
Prism uses) to send GET requests with a request body. If you're able,
you can install
WinHttpHandler
and implement it as a drop-in replacement for HTTPRequest (this SO
thread will help).
Because this type of solution requires the install of a new library, it's
important to consider the caveats of doing so:
Blue Prism's support for external DLLs is unstable at best, and there's
no guarantee it will even import correctly to begin with. Vendor support
for this type of setup is, anecdotally, limited to nonexistent (and
rightfully so, IMO).
If you're able to successfully implement the functionality described
with WinHttpHandler, you'll need to install it on every Blue Prism
developer's machine and runtime resource in all your environments
(development/SIT/UAT/production). For some organizations, strict IT
security posture makes this rather impractical or outright infeasible.
I managed to get it working using a code block containing C# code and using Reflection, here is a my GET method:
public string GetWithBodyAndAuth(string uri, string data, string token, out int statusCode)
{
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("Authorization", "Basic " + token);
request.ContentType = "application/json; charset=utf-8";
request.Method = "GET";
request.Accept = "application/json";
var type = request.GetType();
var currentMethod = type.GetProperty("CurrentMethod", BindingFlags.NonPublic | BindingFlags.Instance).GetValue(request);
var methodType = currentMethod.GetType();
methodType.GetField("ContentBodyNotAllowed", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(currentMethod, false);
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
{
streamWriter.Write(data);
}
var response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
statusCode = ((int)response.StatusCode);
return reader.ReadToEnd();
}
It's a bit of a hack and can't say if it'll be supported in the future, but for BluePrism 6.9 this allows you to send GET requests containing a body.
It has the advantage of not requiring any external DLLs, this is the import list:
I have tried to use api connectors in sign-up flow, But it's not working.
I have tried following
(1) Api Connectors registration :
(2) Linked api connectors in sign-up flow (Before creating the user)
(3) End point (API)
** I am not using parameter values in api, returning static status result. Just for testing purpose.
[HttpPost]
public async Task<IActionResult> Post()
{
using (StreamReader stream = new StreamReader(HttpContext.Request.Body))
{
Task<string> body = stream.ReadToEndAsync();
return new BadRequestObjectResult(
new ResponseContent("ShowBlockPage", "You are blocked by authority"));
}
}
It is working as expected, you’re returning a bad request HTTP code, without a properly formed response to show the friendly error.
Here is an example error response:
https://github.com/azure-ad-b2c/samples/blob/master/policies/relying-party-rbac/source-code/AADB2C.RBAC.Sample/AADB2C.RBAC.Sample/Controllers/IdentityController.cs#L34
Carefully examine the b2crsponsemodel model and mimic its behaviour.
More info here
https://learn.microsoft.com/en-us/azure/active-directory-b2c/restful-technical-profile#returning-validation-error-message
I'm always reluctant to claim that a bug that I'm seeing is actually a .Net Core bug, but after spending 8+ hours investigating the following bug, it looks like a .Net Core SignalR bug to me. I need techniques for tracking this down further and for fixing it.
The first rule of honing in on a bug is to try to create a minimal amount of code that can consistently repro the bug. While I can't reproduce it in a small stand along project, I have worked hard try to zero in on what's happening.
I have a controller with the following action method
[HttpPost]
[Route("/hack/ajax/start")]
public JsonResult AjaxStart([FromBody] JObject data) {
//A call to some method that does some work
return Json(new {
started = true
});
}
Calling this code via a jquery ajax call or Postman works flawlessly every time if I do not have any SignalR Core 1.0 hubs registered in the startup.cs method. However, when I register the following in the startup.cs file I have intermittent issues.
namespace App.Site.Home {
public class HackHub : Hub {
public async Task SendMessage(string status, string progress) {
await Clients.All.SendAsync("serverMsg", status, progress);
}
}
}
Startup.cs ConfigureServices contains
services.AddSignalR();
Startup.cs Configure contains
app.UseSignalR(routes => {
routes.MapHub<App.Site.Home.HackHub>("/hub/hack");
});
If I were to comment out the one line above routes.MapHub<App.Site.Home.HackHub>("/hub/hack"); everything works fine every time. However with this line present, (I.e. some SignalR hub registered) then that's when the fun starts for me, even if I have no code executing on the client or server that makes use of the hub!
The issue is that sometimes when a HTTP POST request is made for the action method above, something in .Net Core (SignalR??) is converting the POST method to Post, and then because Post is not a valid HTTP Method it converts it to a blank method. And since My action method requires an HTTP POST a 404 status code is returned. Many of the HTTP POSTS for that endpoint work fine, but often the issue I just described occurres.
To ensure that my client code was not part of the problem, I was able to reproduce my issue using Postman to make the requests. Further to ensure that POST was actually being sent and not Post, I used Fiddler to watch what was going over the wire. All this is documented below.
Here is the first request (which always works) done via Postman:
Here is the second (identical!) request done via Postman, this one resulted in a 404:
Here is what the first request (the one that worked properly) looked like in fiddler:
Here is what the second request looked like in fiddler:
As you can see, the requests are identical. But the response certainly is not.
So to get a better idea what the server was seeing, I added the following code to the beginning of the startup.cs Configure method. Due to it's placement, for the request this code gets to run before any other application code or middleware.
public void Configure(IApplicationBuilder app, IHostingEnvironment env) {
//for debugging
app.Use(async (context, next) => {
if(context.Request.Method == "") {
string method = context.Request.Method;
string path = context.Request.Path;
IHttpRequestFeature requestFeature = context.Features.Get<IHttpRequestFeature>();
string kestralHttpMethod = requestFeature.Method;
string stop = path;
}
await next();
});
//more code here...
}
For the first request, the request.Method was POST as one would expect:
But for the second request request.Method was blank!!
To investigate this further, I accessed the requestFeature and checked the Http Method Method there. This is where things get really interesting. If I just hover over the property in the debuggger, it's blank too.
But, If I expand the requestFeature object and look at the Method property there, is it Post!!!
That alone seems like craziness. How can two views of the SAME property in the debugger have different values???! It would seem that some code converted the POST to Post, and at some level the system knows that Post is not a valid http method so in some views of that variable it's converted to a blank string. But that is so weird!
Also, we clearly saw via Postman and Fiddler that POST was sent, so how did it get changed to Post? What code did that? I'd like to claim that it can't be my code since I'm checking the value of the RequestFeature before any of my other code related to the request gets a chance to run. Also, if I comment out the one line of code that registers that SignalR hub, then POST is never converted to Post and I never get a 404. But with that SignalR hub registered I periodically get this behavior.
Are there any SignalR or other .net Core switches I can turn on to get better trace or logging info to see when the POST is getting changed to Post? Is there a way to fix this?
This question was looked into via this GitHub issue https://github.com/aspnet/KestrelHttpServer/issues/2591 which was originally opened up when someone else also observed random 404 errors
I want to especially thank #ben-adams for his help in understanding what was going on.
Let me start by saying that this did not turn out to be a bug in the framework. It was a bug in my code. How can that be given what I was observing?
Well, it's like this...
In some parts of the HttpRequest the method is a string, but it in other parts it's an enum. The enum value for POST is Post. So that's why the case conversion was happening.
The reason that one part of the request was saying Post while the other part showed a Method value of a blank string was because the request object was gummed up because I had accessed it at a time when it was in between requests.
How did I do THAT? you may wonder. Well let me tell you, because the plot thickens...
I turns out that I have some logging code that that gathers context information when it's called and one of the pieces of context info it gathers is the current request.Method. When this logging code is called from a main thread, there is no issue.
However, my system does have some code that runs on background threads that are either started via a Timer or via a ThreadPool.QueueUserWorkItem. If this code hits an exception it will call the same logger code.
When my logger code, running on a background thread, checks for the current httpContext via IHttpContextAccessor I fully expected it to receive null. And certainly this same code in the same situation when accessing the current HttpContext via HttpContext.Current in a non .Net Core website does receive null. But as it turns out, under .Net core, it wasn't receiving null, it was receiving an object. But that object was for a request that had already finished and who's request object had already been reset!!!
Starting in .Net Core 2.0 the HttpContext, and it's child objects like request, gets reset after the connection for a request closes. So the HttpContext object (and it's request object) the logger code was getting when running on a background thread was an object that had been reset. It's request.Path for example was null.
It turns out that a request in this state does not expects it's request.Method property to be accessed. And doing so gums up the works for the next request that comes in. Ultimately this is the source of why the next request that came in ended up returning a 404 error.
So how do we fix this? Why does IHttpContextAccessor return an object rather than null in this out of context situation especially given that the object may very possibly be between requests? The answer is that when I was used Timer or ThreadPool.QueueUserWorkItem to create a background task, the Execution Context was being flowed to the new thread. This is just what happens by default when you use these API methods. But, internally the IHttpContextAccessor uses an AsyncLocal to keep track of the current HttpContext and since my new thread received Execution Context from the main thread it had access to the same AsyncLocal. And so IHttpContextAccessor provided an object rather than the null I was expecting when called from a background thread.
The fix? (Thank you #Ben-Adams!) Instead of calling ThreadPool.QueueUserWorkItem I needed to call ThreadPool.UnsafeQueueUserWorkItem instead. This method DOES NOT flow the current Execution Context to the new thread, and therefore the new thread won't have access to those AsyncLocals from the main thread. Once I did this, IHttpContextAccessor then returned null when called from the background thread instead of returning a object that was in between requests and untouchable. Yea!
When creating a `Timer' I also needed to change my code to do it in a way that would not flow Execution Context. Here is the code I use (which was inspired by some #Ben-Adams suggested):
public static Timer GetNewTimer(TimerCallback callback, object state, int dueTime, int interval) {
bool didSuppress = false;
try {
if (!ExecutionContext.IsFlowSuppressed()) {
//We need to suppress the flow of the execution context so that it does not flow to our
//new asynchronous thread. This is important so that AsyncLocals (like the one used by
//IHttpaccessor) do not flow to the new thread we are pushing our work to. By not flowing the
//execution context, IHttpAccessor wil return null rather than bogusly returning a context for
//a request that is in between requests.
//Related info: https://github.com/aspnet/KestrelHttpServer/issues/2591#issuecomment-399978206
//Info on Execution Context: https://blogs.msdn.microsoft.com/pfxteam/2012/06/15/executioncontext-vs-synchronizationcontext/
ExecutionContext.SuppressFlow();
didSuppress = true;
}
return new Timer(callback, state, dueTime, interval);
} finally {
// Restore the current ExecutionContext
if (didSuppress) {
ExecutionContext.RestoreFlow();
}
}
}
This only leaves one remaining question unanswered. My original question noted that registering a SignalR hub was causing the system to exhibit this random 404 behavior but the system did not exhibit this behavior when no SignalR hub was registered (or so I thought). Why was this? I truly don't know. Perhaps it was putting more resource pressure on some part of the system and thus causing the issue to show up more easily. Not sure. All I know is that the root issue was that I was flowing Execution Context to my background threads without realizing it and that was causing the IHttpContextAccessor's AsyncLocal to be in scope. Not flowing the Execution Context to the background threads fixes that issue.
So we have a requirement to upload file to sharepoint server using iOS client . We are able to upload file size till 40kb but the problem is the file data we are passing is in Base64 encoding,and it fails for file size more than 40kb because we cannot pass large data in soap message,so we tried different approaches possible and mentioned on web . These are the ways we tried out
and we are left with only sending data using MTOM .
After lots of experiments and search I think MTOM is a way to upload data using soap . We are using CopyIntoItems sharepoint service for upload .
Soap request looks like
<"xmlns:soap12=\"http://schemas.xmlsoap.org/soap/envelope/\">\n"
"<soap12:Body>\n"
"<CopyIntoItems xmlns=\"http://schemas.microsoft.com/sharepoint/soap/\">\n"
"<SourceUrl>%#</SourceUrl>\n"
"<DestinationUrls>\n"
"<string>%#</string>\n"
"</DestinationUrls>\n"
"<Fields>\n"
"%#"
"</Fields>\n"
"<Stream>MY_FILE_DATA</Stream>\n"
"</CopyIntoItems>\n"
"</soap12:Body>\n"
"</soap12:Envelope>\n"
Now the problem is how do we format this soap request in the way MTOM accepts,because CopyIntoItems may not work in some other format.
Few samples of MTOM which i looked into
http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.wsfep.multiplatform.doc/info/ae/ae/cwbs_soapmtom.html
http://cxf.apache.org/docs/mtom.html
Any help would be much appreciated..Thanks..!!
What about using HTTP post and developing an Http handler to store posted data?
It is quite easy to setup an http handler in Sharepoint (MSDN) and in the Process request method you can use query string to get the information you need.
Say you create an Upload.ashx handler.
public void ProcessRequest(HttpContext context)
{
var targetList = context.Request["targetList"];
var targetFileName = context.Request["targetFn"];
// Get the stream to content
var stream = context.Request.GetBufferedInputStream();
// Save the file somewhere
}
From the client you do an HTTP post to http://server/_layouts/15/Upload.ashx?targetList=myFiles&...