Importing Thousands of Records Into Acumatica via SOAP Contract-Based API - acumatica

Question

I’m using contract-based SOAP APIs to try to import about 25,000 Journal Entry lines from a banking system into a single Acumatica GL batch.
If I try to add all the records at once to the same GL batch, my
request times out after a few hours. Since it uses the same GL
batch, this solution does not leverage multi-threading.
I've also tried adding the 25000 lines one line at a time to a
single GL batch and the requests does not time out, but
performance-speed starts decreasing significantly after
approximately 3000 records or so are added to the GL batch. This
process takes several hours to run and since it uses the same GL
batch, this solution does not leverage multi-threading.
I looked into multi-threading as well to import the data into
several smaller GL-batches of 5000 lines each and that works without
any timeout issues. but it still takes about an hour and a half to
run. Also, the customer does not accept this multi-batch approach;
they want all their daily data in a single GL batch.
25,000 records does not seem like a lot to me, so I wonder if Acumatica’s APIs were not built for this volume of lines in a single transaction. All I’m doing in my code is building the entity info by reading a text file and then calling the put method to create the GL batch using that entity with 25,000 line records.
I've read a couple of articles about optimizing the APIs, but they primarily deal with different instances of an entity, as in several different GL batches or several different Stock Items for example. In those cases, multi-threading is a great asset because you can have multiple threads creating multiple "different" GL batches, but multi-threading is not helpful when updating the same GL batch.
Here's what I've read so far:
https://asiablog.acumatica.com/2016/12/optimizing-large-import.html
https://adn.acumatica.com/blog/contractapioptimization/
I'm at a loss here, so any pointers would be greatly appreciated.
I look forward to your response.
Here's my code:
public static void CreateMultipleLinesPerJournalEntryBatchContractTEST(MyStoreContract.DefaultSoapClient soapClient, List<JournalEntry> journalEntries)
{
string myModuleForBatchLookup = "GL";
//list holding the values of all the records belonging to the batch in process
List<JournalEntry> allBatchItems = journalEntries;
//List used to store objects in format required by Acumatica
List<MyStoreContract.JournalTransactionDetail> myJournalTransactionsFormatted = new List<MyStoreContract.JournalTransactionDetail>();
try
{
//Creating a header and returning a batch value to be used for all line iterations.
JournalEntry myHeaderJournalEntryContract = allBatchItems.First();
string myBatchNumberToProcess = AddGLBatchHeaderContractTEST(soapClient, myHeaderJournalEntryContract);
// Do something with then n number of items defined in processing subBatch size or remaining items if smaller
foreach (JournalEntry je in allBatchItems)
{
//Moving the items in each batch from the original unformatted list to the formatted list one at a time
myJournalTransactionsFormatted.Add(new MyStoreContract.JournalTransactionDetail
{
BranchID = new MyStoreContract.StringValue { Value = je.Branch },
Account = new MyStoreContract.StringValue { Value = je.Account },
Subaccount = new MyStoreContract.StringValue { Value = je.Subaccount },
ReferenceNbr = new MyStoreContract.StringValue { Value = je.RefNumber },
DebitAmount = new MyStoreContract.DecimalValue { Value = je.DebitAmount },
CreditAmount = new MyStoreContract.DecimalValue { Value = je.CreditAmount },
TransactionDescription = new MyStoreContract.StringValue { Value = je.TransactionDescription },
UsrTransactionTime = new MyStoreContract.StringValue { Value = je.UsrTransactionTime },
UsrTransactionType = new MyStoreContract.StringValue { Value = je.UsrTransactionType },
UsrTranSequence = new MyStoreContract.StringValue { Value = je.UsrTranSequence },
UsrTellerID = new MyStoreContract.StringValue { Value = je.UsrTellerID }
});
}
//Specify the values of a new Jornal Entry using all the collected elements from the batch(list) created
MyStoreContract.JournalTransaction journalToBeCreated = new MyStoreContract.JournalTransaction
{
//Header data and details added by list generated by loop
BatchNbr = new MyStoreContract.StringSearch { Value = myBatchNumberToProcess }, //This is one of two lines used to lookup/search the batch needing to be updated
Module = new MyStoreContract.StringSearch { Value = myModuleForBatchLookup }, //This is one of two lines used to lookup/search the batch needing to be updated
Details = myJournalTransactionsFormatted.ToArray() // this is the line adding the array containing all the line details
};
soapClient.Put(journalToBeCreated);
Console.WriteLine("Added " + allBatchItems.Count.ToString() + " line transactions");
Console.WriteLine();
Console.WriteLine("Press any key to continue");
Console.ReadLine();
}
catch (Exception e)
{
Console.WriteLine("The following error was encountered and all entries for this batch need to be logged in error table");
Console.WriteLine();
Console.WriteLine(e.Message);
Console.WriteLine();
Console.WriteLine("Press any key to continue");
Console.ReadLine();
}
}
public static string AddGLBatchHeaderContractTEST(MyStoreContract.DefaultSoapClient soapClient, JournalEntry je)
{
try
{
//Specify the values of a new Jornal Entry Batch header
MyStoreContract.JournalTransaction journalToBeCreated = new MyStoreContract.JournalTransaction
{
//Header data
BranchID = new MyStoreContract.StringValue { Value = "PRODWHOLE" }, //This is the default branch
TransactionDate = new MyStoreContract.DateTimeValue { Value = je.TransactionDate.AddDays(-1) }, //Reduced 1 day from the batch
CurrencyID = new MyStoreContract.StringValue { Value = je.CurrencyCode }, //Currency to be used for the batch
Description = new MyStoreContract.StringValue { Value = je.TransactionDescription },
Hold = new MyStoreContract.BooleanValue { Value = true }
};
//Create a Journal Entry with the specified values
MyStoreContract.JournalTransaction newJournalTransaction = (MyStoreContract.JournalTransaction)soapClient.Put(journalToBeCreated);
string myBatchToProcess = newJournalTransaction.BatchNbr.Value;
return myBatchToProcess;
}
catch (Exception e)
{
Console.WriteLine("Error was caught while trying to create the header for the batch...");
Console.WriteLine();
Console.WriteLine(e);
Console.WriteLine();
return null;
}
}
My custom class for legacy system line items which I then need format into Acumatica's format:
class JournalEntry
{
public DateTime TransactionDate { get; set; }
public string CurrencyCode { get; set; }
public string Description { get; set; }
public string Branch { get; set; }
public string Account { get; set; }
public string Subaccount { get; set; }
public string RefNumber { get; set; }
public decimal DebitAmount { get; set; }
public decimal CreditAmount { get; set; }
public string TransactionDescription { get; set; }
//Added custom fields for customer
public string UsrTellerID { get; set; }
public string UsrTransactionType { get; set; }
public string UsrTransactionTime { get; set; }
public string UsrTranSequence { get; set; }
//Adding original file data for the line
public string FileLineData { get; set; }
}
I tried Yuriy's approach described below, but my custom fields are not updating. Only standard fields are being updated. Which command should I use to update the extension (custom) fields. See code below:
//Here I create instance of GLTran
GLTran row = graph.GLTranModuleBatNbr.Cache.CreateInstance() as GLTran;
//here I get a handle to graph extension GLTranExt to be able to use the added fields.
var rowExt = row.GetExtension<GLTranExt>();
row = graph.GLTranModuleBatNbr.Insert(row);
graph.GLTranModuleBatNbr.Cache.SetValueExt(row, "AccountID", JE.Account);
graph.GLTranModuleBatNbr.Cache.SetValueExt(row, "SubID", JE.Subaccount);
row.TranDesc = "my line description";
row.Qty = 1.0m;
row.CuryDebitAmt = (JE.DebitAmount);
row.CuryCreditAmt = (JE.CreditAmount);
rowExt.UsrTellerID = "Test teller";
rowExt.UsrTransactionTime = "Test Transaction Time";
rowExt.UsrTransactionType = "Test Transaction Type";
rowExt.UsrTranSequence = "Test Transaction Sequence";
row = graph.GLTranModuleBatNbr.Update(row);
graph.Actions.PressSave();

Answer 1

In multi threaded import of Sales Orders I've got 18000 lines per hour ( 4 cores, 32Gb RAM ). So your 25000 is very similar to what I've get ( one Sales order had 1 - 6 lines ). For second link that you provided, what were parameters of your API call, what was number of your Acumatica instances ( CPU, RAM, parameters of SQL Server )?
I propose you to consider scaling Acumatica horizontally and also scale your database via SQL sharding.
Edit
In case if you need to have one GL Batch with 25000 lines on it, then I propose you following workaround:
Create one more Acumatica page that has text box and button Import.
In code of button Import button
2.1 read text box information as xml ( or JSON )
2.2 Create instance of GL Graph
2.3 Insert via Graph needed amount ( in your case 25000 ) lines
2.4 Call to graph.PressSave()
Send Web API your request not to GL Batch but to created by you page.

Answer 2

I know this is an old question, but I am answering here for the benefit of anyone who stumbles across this page. There's a performance issue in the Journal Transaction screen where the time to create a transaction increases non-linearly with the number of rows to insert.
A customization-based workaround was provided to us by Acumatica support which significantly improved performance. I don't have the exact version where this fix was included, but builds newer than today (September 2021) should include this already.
Customization fix:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using PX.Api;
using PX.Data;
using PX.Common;
using PX.Objects.Common;
using PX.Objects.Common.Extensions;
using PX.Objects.CS;
using PX.Objects.CM;
using PX.Objects.CA;
using PX.Objects.Common.Bql;
using PX.Objects.Common.GraphExtensions.Abstract;
using PX.Objects.Common.GraphExtensions.Abstract.DAC;
using PX.Objects.Common.GraphExtensions.Abstract.Mapping;
using PX.Objects.GL.DAC;
using PX.Objects.GL.FinPeriods;
using PX.Objects.GL.JournalEntryState;
using PX.Objects.GL.JournalEntryState.PartiallyEditable;
using PX.Objects.GL.Overrides.PostGraph;
using PX.Objects.GL.Reclassification.UI;
using PX.Objects.PM;
using PX.Objects.TX;
using PX.Objects.Common.Tools;
using PX.Objects.GL.DAC.Abstract;
using PX.Objects.Common.EntityInUse;
using PX.Objects.GL.FinPeriods.TableDefinition;
using PX.Data.SQLTree;
using PX.Objects.CR;
using PX.Data.BQL.Fluent;
using PX.Data.BQL;
using PX.Objects;
using PX.Objects.GL;
namespace PX.Objects.GL
{
public class JournalEntry_Extension : PXGraphExtension<JournalEntry>
{
public delegate void PopulateSubDescrDelegate(PXCache sender, GLTran Row, Boolean ExternalCall);
[PXOverride]
public void PopulateSubDescr(PXCache sender, GLTran Row, Boolean ExternalCall, PopulateSubDescrDelegate baseMethod)
{
if (Base.IsImport || Base.IsExport || Base.IsContractBasedAPI)
{
return;
}
baseMethod(sender,Row,ExternalCall);
}
}
}