Handling Acumatica timeout on API Invoke action - acumatica

I have code in a standalone application that invokes an Acumatica action to generate reports; I am running into timeouts on large documents while the action completes.
What is the best method to handle these timeouts? I need to wait for the action to complete in order to retrieve the files I've generated.
Standalone application code:
public SalesOrder GenerateAcumaticaLabels(string orderNbr, string reportType)
{
SalesOrder salesOrder = null;
using (ISoapClientProvider clientProvider = soapClientFactory.Create())
{
try
{
SalesOrder salesOrderToFind = new SalesOrder
{
OrderType = new StringSearch { Value = orderNbr.Split(OrderSeparator.SalesOrder).First() },
OrderNbr = new StringSearch { Value = orderNbr.Split(OrderSeparator.SalesOrder).Last() },
ReturnBehavior = ReturnBehavior.OnlySpecified,
};
salesOrder = clientProvider.Client.Get(salesOrderToFind) as SalesOrder;
InvokeResult invokeResult = new InvokeResult();
invokeResult = clientProvider.Client.Invoke(salesOrder, new exportSFPReport());
ProcessResult processResult = clientProvider.Client.GetProcessStatus(invokeResult);
//Wait for the update to complete before we attempt to retrieve the files
while (processResult.Status == ProcessStatus.InProcess)
{
Thread.Sleep(1000); //pause for 1 second
processResult = clientProvider.Client.GetProcessStatus(invokeResult);
}
}
And the action in Acumatica:
public PXAction<SOOrder> ExportSFPReport;
[PXButton]
[PXUIField(DisplayName = "Generate Robot SFP PDF")]
protected IEnumerable exportSFPReport(PXAdapter adapter)
{
//Report Paramenters
Dictionary<String, String> parameters = new Dictionary<String, String>();
parameters["SOOrder.OrderType"] = Base.Document.Current.OrderType;
parameters["SOOrder.OrderNbr"] = Base.Document.Current.OrderNbr;
IEnumerable reportFileInfo = ExportReport(adapter, "IN619217", parameters);
exportTrayLabelReport(adapter, "SFP");
return reportFileInfo;
}

The problem here is that your action is synchronous, so it is trying to complete within the Invoke call (which is not a good thing for long processes). You have to explicitly make your operation long-running by using PXLongOperation.StartOperation inside your handler, and then your client code should work properly, as it already handles the waiting and checking.

I believe the reason why you encounter time-out is because there is no TCP communication between the time you sent the request and receive the response. With TCP KeepAlive flag set to true, the client will periodically ping the server to reset the time-out period.
That would be the best way. However Acumatica connections are rather high level so I don't think you'll be able to easily access that flag. What I would try first in a scenario that doesn't involve external application is to wrap your action event-handler code in a PXLongOperation block which has to do something similar to keep connection alive under the hood:
PXLongOperation.StartOperation(this or Base, delegate
{
your code here
});
When I do encounter time-outs in Acumatica that can't be solved with PXLongOperation I go for the simplest method which is increasing IIS timeout in Web.Config file. I'm not sure if your use case with external application will go well with async PXLongOperation. The handler would return prematurely and the client could not be able to retrieve the async payload.
So you might have to increase time-out instead. As far as I know there's no real practical drawback to doing this unless your website is under threat of DOS attacks.
You can locate and edit the Web.Config file of your Acumatica instance using inetmgr program if you are self-hosting Acumatica. Otherwise talk to your SAAS contact to see if that's an option.
I'm pretty sure you are hitting IIS time-out. A tell-tale sign would be lost connection after exactly 5 minutes which is the default 300 seconds value. You can edit Web.Config file to increase executionTimeout value. It's not a bad idea to increase maxRequestLength too if you are requesting large amount of data from Acumatica API as this is also a common cause of failure that you miss in testing and occurs in real-life scenarios:
<httpRuntime executionTimeout="300" requestValidationMode="2.0" maxRequestLength="1048576" />

Related

Multiple REST calls timing out in Spring Boot web application

I created a Spring Boot (1.4.2) REST application. One of the #RestController methods needs to invoke a 3rd party API REST operation (RestOp1) which returns, say between 100-250 records. For each of those records returned by RestOp1, within the same method, another REST operation of the same 3rd party API (RestOp2) must be invoked. My first attempt involved using a Controller class level ExecutorService based on a Fixed Thread Pool of size 100, and a Callable returning a record corresponding to the response of RestOp2:
// Executor thread pool - declared and initialized at class level
ExecutorService executor = Executors.newFixedThreadPool(100);
// Get records from RestOp1
ResponseEntity<RestOp1ResObj[]> restOp1ResObjList
= this.restTemplate.exchange(url1, HttpMethod.GET, httpEntity, RestOp1ResObj[].class);
RestOp1ResObj[] records = restOp1ResObjList.getBody();
// Instantiate a list of futures (to call RestOp2 for each record)
List<Future<RestOp2ResObj>> futureList = new ArrayList<>();
// Iterate through the array of records and call RestOp2 in a concurrent manner, using Callables.
for (int count=0; count<records.length; count++) {
Future<RestOp2ResObj> future = this.executorService.submit(new Callable<RestOp2ResObj>() {
#Override
public RestOp2ResObj call() throws Exception {
return this.restTemplate.exchange(url2, HttpMethod.GET, httpEntity, RestOp2Obj.class);
}
};
futureList.add(future);
});
// Iterate list of futures and fetch response from RestOp2 for each
// record. Build a final response and send back to the client.
for (int count=0; count<futureList.size(); count++) {
RestOp2ResObj response = futureList.get(count).get();
// use above response to build a final response for all the records.
}
The performance of the above code is abysmal to say the least. The response time for a RestOp1 call (invoked only once) is around 2.5 seconds and that for a RestOp2 call (invoked for each record) is about 1.5 seconds. But the code execution time is between 20-30 seconds, as opposed to an expected range of 5-6 seconds! Am I missing something fundamental here?
Is the service you are calling fast enough to handle that many requests per second?
There is an async version of RestService is available called AsyncRestService. Why are you not using that?
I would probably go like this:
AsyncRestTemplate asyncRestTemplate = new AsyncRestTemplate(new ConcurrentTaskExecutor(Executors.newFixedThreadPool(100)));
asyncRestTemplate.exchange("http://www.example.com/myurl", HttpMethod.GET, new HttpEntity<>("message"), String.class)
.addCallback(new ListenableFutureCallback<ResponseEntity<String>>() {
#Override
public void onSuccess(ResponseEntity<String> result) {
//TODO: Add real response handling
System.out.println(result);
}
#Override
public void onFailure(Throwable ex) {
//TODO: Add real logging solution
ex.printStackTrace();
}
});
Your question involves two parts :
multiple API callbacks asynchronously
handle timeouts (fallback)
both parts are related as you've to handle the timeout of each call.
you may consider use Spring Cloud (based on spring boot) and use some out of the box solution based on OSS Netflix stacks.
The first (timeouts) on should be a circuit breaker hystrix based on feign client
The second (multiple requests) this is an architecture issue, using native Executors isn't a good idea as it will not scale and has a huge maintenance costs. You may relay on Spring Asynchrounous Methods you'll have better results and fully spring compliant.
Hope this will help.

Custom maintenance mode module does not work on Azure Web Role

I've created and registered custom http module to show maintenance message to user after administrator turns on maintenance mode via configuration change.
When I pass request for html it should return custom html loaded from file, but it returns message: "The service is unavailable." I can't find that string in my entire solution. Custom log message from custom maintenance module is written to log4net logs.
... INFO DdiPlusWeb.Common.MaintenanceResponder - Maintenance mode is on. Request rejected. RequestUrl=...
Seems something is miss configured in IIS on Azure. Something intercepts my 503 response. How to fix it?
Module code
void context_BeginRequest(object sender, EventArgs e)
{
HttpApplication application = (HttpApplication)sender;
HttpContext context = application.Context;
if (AppConfig.Azure.IsMaintenance)
{
MaintenanceResponder responder = new MaintenanceResponder(context, MaintenaceHtmlFileName);
responder.Respond();
}
}
Interesting part of responder code.
private void SetMaintenanceResponse(string message = null)
{
_context.Response.Clear();
_context.Response.StatusCode = 503;
_context.Response.StatusDescription = "Maintenance";
if (string.IsNullOrEmpty(message))
{
_context.Response.Write("503, Site is under maintenance. Please try again a bit later.");
}
else
{
_context.Response.Write(message);
}
_context.Response.Flush();
_context.Response.End();
}
EDIT: I lied. Sorry. Maintenance module returns the same message for requests that expect json or html.
This answer led me to the solution.
I've added another line to SetMaintenanceResponse method.
_context.Response.TrySkipIisCustomErrors = true;
It works now. Here is more about what it exactly means.

Azure Autoscale Restarts Running Instances

I've been using Autoscale to shift between 2 and 1 instances of a cloud service in a bid to reduce costs. This mostly works except that from time to time (not sure what the pattern seems to be here), the act of scaling up (1->2) causes both instances to recycle, generating a service outage for users.
Assuming nothing fancy is going on in RoleEntry in response to topology changes, why would scaling from 1->2 restart the already running instance?
Additional notes:
It's clear both instances are recycling by looking at the Instances
tab in Management Portal. Outage can also be confirmed by hitting the
public site.
It doesn't happen consistently but I'm not sure what the pattern is. It feels like when the 1-instance configuration has been running for multiple days, attempts to scale up recycle both. But if the 1-instance configuration has only been running for a few hours, you can scale up and down without outages.
The first instance always comes back much faster than the 2nd instance being introduced.
This has always been this way. When you have 1 server running and you go to 2+, the initial server is restarted. In order to have a full SLA, you need to have 2+ servers at all time.
Nariman, see my comment on Brent's post for some information about what is happening. You should be able to resolve this with the following code:
public class WebRole : RoleEntryPoint
{
public override bool OnStart()
{
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
IPHostEntry ipEntry = Dns.GetHostEntry(Dns.GetHostName());
string ip = null;
foreach (IPAddress ipaddress in ipEntry.AddressList)
{
if (ipaddress.AddressFamily.ToString() == "InterNetwork")
{
ip = ipaddress.ToString();
}
}
string urlToPing = "http://" + ip;
HttpWebRequest req = HttpWebRequest.Create(urlToPing) as HttpWebRequest;
WebResponse resp = req.GetResponse();
return base.OnStart();
}
}
You should be able to control this behavior. In the roleEntrypoint, there's an event you can trap for, RoleEnvironmentChanging.
A shell of some code to put into your solution will look like...
RoleEnvironment.Changing += RoleEnvironmentChanging;
private void RoleEnvironmentChanging(object sender, RoleEnvironmentChangingEventArgs e)
{
}
RoleEnvironment.Changed += RoleEnvironmentChanged;
private void RoleEnvironmentChanged(object sender, RoleEnvironmentChangedEventArgs e)
{
}
Then, inside the RoleEnvironmentChanged method, we can detect what the change is and tell Azure if we want to restart or not.
if ((e.Changes.Any(change => change is RoleEnvironmentConfigurationSettingChange)))
{
e.Cancel = true; // don't recycle the role
}

ServiceStack/Funq not disposing RavenDB document session after request is complete

In trying to integrate RavenDB usage with Service Stack, I ran across the following solution proposed for session management:
A: using RavenDB with ServiceStack
The proposal to use the line below to dispose of the DocumentSession object once the request is complete was an attractive one.
container.Register(c => c.Resolve<IDocumentStore>().OpenSession()).ReusedWithin(ReuseScope.Request);
From what I understand of the Funq logic, I'm registering a new DocumentSession object with the IoC container that will be resolved for IDocumentSession and will only exist for the duration of the request. That seemed like a very clean approach.
However, I have since run into the following max session requests exception from RavenDB:
The maximum number of requests (30) allowed for this session has been
reached. Raven limits the number of remote calls that a session is
allowed to make as an early warning system. Sessions are expected to
be short lived, and Raven provides facilities like Load(string[] keys)
to load multiple documents at once and batch saves.
Now, unless I'm missing something, I shouldn't be hitting a request cap on a single session if each session only exists for the duration of a single request. To get around this problem, I tried the following, quite ill-advised solution to no avail:
var session = container.Resolve<IDocumentStore>().OpenSession();
session.Advanced.MaxNumberOfRequestsPerSession = 50000;
container.Register(p => session).ReusedWithin(ReuseScope.Request);
Here is a sample of how I'm using the resolved DocumentSession instance:
private readonly IDocumentSession _session;
public UsersService(IDocumentSession session)
{
_session = session;
}
public ServiceResponse<UserProfile> Get(GetUser request)
{
var response = new ServiceResponse<UserProfile> {Successful = true};
try
{
var user = _session.Load<UserProfile>(request.UserId);
if (user == null || user.Deleted || !user.IsActive || !user.IsActive)
{
throw HttpError.NotFound("User {0} was not found.".Fmt(request.UserId));
}
response.Data = user;
}
catch (Exception ex)
{
_logger.Error(ex.Message, ex);
response.StackTrace = ex.StackTrace;
response.Errors.Add(ex.Message);
response.Successful = false;
}
return response;
}
As far as I can see, I'm implementing SS + RavenDB "by the book" as far as the integration point goes, but I'm still getting this max session request exception and I don't understand how. I also cannot reliably replicate the exception or the conditions under which it is being thrown, which is very unsettling.

RMI calls by multiple concurrent users to JBoss AS 4.2.3

I want to write a web frontend that wants to "propagate" the HTTP authentication received from the browser to a JBoss AS 4.2.3 that exposes numerous #Remote interfaces.
Consider the following trivial simulation of RMI call concurrency:
Properties user1 = new Properties();
user1.setProperty(Context.INITIAL_CONTEXT_FACTORY,
"org.jboss.security.jndi.JndiLoginInitialContextFactory");
user1.setProperty(Context.URL_PKG_PREFIXES, "org.jboss.naming");
user1.setProperty(Context.PROVIDER_URL, "127.0.0.1:1099");
user1.setProperty(Context.SECURITY_PRINCIPAL, "user1");
user1.setProperty(Context.SECURITY_CREDENTIALS, "pass1");
Properties user2 = new Properties();
user2.setProperty(Context.INITIAL_CONTEXT_FACTORY,
"org.jboss.security.jndi.JndiLoginInitialContextFactory");
user2.setProperty(Context.URL_PKG_PREFIXES, "org.jboss.naming");
user2.setProperty(Context.PROVIDER_URL, "127.0.0.1:1099");
user2.setProperty(Context.SECURITY_PRINCIPAL, "user2");
user2.setProperty(Context.SECURITY_CREDENTIALS, "pass2");
InitialContext ctx1 = new InitialContext(user1);
Mine bean1 = (Mine) ctx1.lookup("myear/MyBean/remote");
InitialContext ctx2 = new InitialContext(user2);
Mine bean2 = (Mine) ctx2.lookup("myear/MyBean/remote");
System.out.println(bean1.whoami());
System.out.println(bean2.whoami());
Call uses jbossall-client 4.2.3 and goes to a JBoss AS 4.2.3.
The .whoami() method simply echoes the logged-in username. As it turns our, this results in both calls saying they are made by "user2". Presumably, the underlying connection is shared and only authenticated using the last seen properties bundle.
In short, this sucks. Some preliminary testing indicates that the same problem remains in JBoss AS 7 so no luck.
Is there any other RMI client implementation I can use or any parameter I can pass in the prop bundle to make the InitialContexts not share their login info? Alternatively, can someone point me to the code that needs to be hacked to make this possible?
UPDATE:
As per request:
public class Worker extends Thread {
private final String pass, user;
private int correct = 0;
public Worker(String user, String pass) { this.user = user; this.pass = pass; }
public void run() {
Properties props = new Properties();
props.setProperty(Context.INITIAL_CONTEXT_FACTORY,
"org.jboss.security.jndi.JndiLoginInitialContextFactory");
props.setProperty(Context.URL_PKG_PREFIXES, "org.jboss.naming");
props.setProperty(Context.PROVIDER_URL, "127.0.0.1:1099");
props.setProperty(Context.SECURITY_PRINCIPAL, this.user);
props.setProperty(Context.SECURITY_CREDENTIALS, this.pass);
try {
InitialContext ctx = new InitialContext(props);
for(int i = 0; i < 100; i++) {
Mine bean = (Mine) ctx.lookup("myear/MyBean/remote");
if(bean.whoami().equals(this.user)) this.correct++;
Thread.sleep(2); }
ctx.close();
} catch (Exception e) { throw new RuntimeException(e); }
System.out.println("Done [id="+this.getId()+", good="+this.correct+"]");
}
}
Running with two workers yields:
public static void main(String[] args) throws Exception {
new Worker("user1", "pass1").start();
new Worker("user2", "pass2").start();
}
Done [t=9, good=0]
Done [t=10, good=100]
Running with 5 threads yields:
public static void main(String[] args) throws Exception {
new Worker("user1", "pass1").start();
new Worker("user2", "pass2").start();
new Worker("user3", "pass3").start();
new Worker("user4", "pass4").start();
new Worker("user5", "pass5").start();
}
Caused by: javax.ejb.EJBAccessException: Authentication failure
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.handleGeneralSecurityException(Ejb3AuthenticationInterceptor.java:68)
at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:70)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:304)
at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
at org.jboss.remoting.Client.invoke(Client.java:1634)
at org.jboss.remoting.Client.invoke(Client.java:548)
at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:107)
at $Proxy0.whoami(Unknown Source)
at net.windwards.Worker.run(TestRMIClient.java:31)
at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:74)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:107)
at $Proxy0.whoami(Unknown Source)
at net.windwards.Worker.run(TestRMIClient.java:31)
Making the initial connection takes about 100 ms, so I tried the following (sleeping 10 ms between calls to get good overlap):
public static void main(String[] args) throws Exception {
new Worker("user1", "pass1").start();
Thread.sleep(200);
new Worker("user2", "pass2").start();
Thread.sleep(200);
new Worker("user3", "pass3").start();
Thread.sleep(200);
new Worker("user4", "pass4").start();
Thread.sleep(200);
new Worker("user5", "pass5").start();
}
Done [t=9, good=1]
Done [t=14, good=12]
Done [t=15, good=14]
Done [t=16, good=15]
Done [t=17, good=100]
From the docs for org.jboss.security.jndi.JndiLoginInitialContextFactory :
During the getInitialContext callback from the JNDI naming, layer security context identity is populated with the username ... and the credentials ... There is no actual authentication of this information. It is merely made available to the jboss transport layer for incorporation into subsequent invocations
in this case, by the time you get to invoke your beans, user2 is the last principal set and so is the one available to be used by the jboss transport layer.
However, from the jboss4 source, it looks like you can make the security context scoped to the thread context, in which case your threaded test should work, simply add this property:
userN.setProperty("jnp.multi-threaded", "true");
Another solution would be using org.jboss.security.jndi.LoginInitialContextFactory instead of org.jboss.security.jndi.JndiLoginInitialContextFactory, unlike JndiLoginInitialContextFactory, LoginInitialContextFactory will try to authenticate when the look up is made, not when the EJB is invoked, you could give it a try, even though in the docs, they recommend JndiLoginInitialContextFactory when it comes to EJB authorization on remote clients
The basic problem here is that you haven't close the first context before you use the second one in the same thread. I doubt that this is a fair test. It would be more interesting to actually make the two concurrent, by running them both in separate threads.
When the getInitialContext() is being called from the JNDI, the Security Layer invokes a wrapper with the credential tiles; which is factually never verified with a source, it is just type of a virtual representation of the tiles to JBOSS for subsequent calls to the same entity model.
In your case, user2 is the last one to be available to JBOSS.
Alternatively, you can also use multiple instances of JBOSS on the
same machine by using ServiceBindingManager. This could help you
keep a track of all RMI calls you make, also the properties for the
Connector Object do work because itself is a JMX Bean Object.
You can also use a threaded model which can give you additional security by
adding a property
userN.setProperty("jnp.multi-threaded", "true");
And just as a suggestion, I found online Use JndiLoginInitialContextFactory for EJB Authentication on remote clients.
Hope this helps!

Resources