ArangoDb .Net driver performance - arangodb

I am trying on the new ArangoDb-Net reimplement driver https://github.com/yojimbo87/ArangoDB-NET/tree/reimplement. Today is the first time I tried on the performance. When I used the araongosh to perform the insert. It can insert about 5000 records per second. However, when I used the .Net driver to perform the same update. I took about 2 minutes to perform the same insertion. May I know what I have done wrong? Thanks.
[EDIT] completing the question with the github discussion
I have tested the code below with my arangosh
count=1;
startTime=+new Date();
console.log(startTime);
while(count <= 10000)
{
db.someCollection.save({"Id":"1234567890123456789012345678901234",
"Key":1234567,
"Revision":1234567,
"Name":"Mohamad Abu Bakar",
"IC Number":"1234567-12-3444",
"Department":"IT Department",
"Height":1234,
"DateOfBirth":"2015-01-27 03:33",
"Salary":3333});
count++;
}
endTime=+new Date();
console.log(endTime);
console.log("Total time taken:" + (endTime - startTime)/1000);
It took 3.375 seconds to complete the operation.
I do the similar thing with the .Net driver and it took almost 9.5797819. Almost triple of the arangosh. Here's the code in .Net:
public static void TestArangoDb()
{
//ASettings.AddConnection("_system", "127.0.0.1", 8529, false, "_system");
//var db = new ADatabase("_system");
//db.Create("trial_sample");
ASettings.AddConnection("trial_sample",
"127.0.0.1", 8529, false, "trial_sample");
var db2 = new ADatabase("trial_sample");
db2.Collection.Create("someCollection");
DateTime startTime = DateTime.Now;
Console.WriteLine("Start Time: " + startTime.ToLongTimeString());
for(int count=1; count <= 10000; count++)
{
var employee = new Employee();
employee.Id = "1234567890123456789012345678901234";
employee.Key = "1234567";
employee.Revision = "1234567";
employee.Name = "Mohamad Abu Bakar";
employee.IcNumber = "1234567-12-3444";
employee.Department = "IT Department";
employee.Height = 1234;
employee.DateOfBirth = new DateTime(2015, 1, 27, 3, 33, 3);
employee.Salary = 3333;
var result = db2.Document.Create<Employee>("someCollection", employee);
//var updateDocument = new Dictionary<string, object>()
// .String("DocumentId", "SomeId");
//db2.Document.Update(result.Value.String("_id"), updateDocument);
}
DateTime endTime = DateTime.Now;
TimeSpan duration = endTime - startTime;
Console.WriteLine("End Time: " + endTime.ToLongTimeString());
Console.WriteLine("Total time taken: " + duration.TotalSeconds);
}
public class Employee
{
public string Id { get; set; }
public string Key { get; set; }
public string Revision { get; set; }
public string Name { get; set; }
public string IcNumber { get; set; }
public string Email { get; set; }
public string Department { get; set; }
public double Height { get; set; }
public DateTime DateOfBirth { get; set; }
public decimal Salary { get; set; }
}
If I remove the comment for:
var updateDocument = new Dictionary<string, object>()
.String("DocumentId", "SomeId");
db2.Document.Update(result.Value.String("_id"), updateDocument);
The performance is almost 30 times. It took 99.8789133 seconds to complete. In fact, I just perform additional update to add additional column.
Could you suggest on the problem on the code above? Thanks.

yojimbo87 researched the issue deeper. Testing The different layers uncovered the problem.
Merge request #32 improves performance when creating, updating and replacing documents/edges from generic objects by ~57%.
On local machine single document creation with given Employee example object now takes on average ~0.4ms in 10k iteration loop using the driver. Raw .NET HTTP request (without any ArangoDB driver abstraction) takes ~0.35ms in 10k loop. Difference is made by conversion from generic object into Dictionary which needs to be done because of attributes processing (such as IgnoreField, IgnoreNullValue and AliasField).
The NuGet package was updated to reflect this improvement.

Related

Importing Thousands of Records Into Acumatica via SOAP Contract-Based API

I’m using contract-based SOAP APIs to try to import about 25,000 Journal Entry lines from a banking system into a single Acumatica GL batch.
If I try to add all the records at once to the same GL batch, my
request times out after a few hours. Since it uses the same GL
batch, this solution does not leverage multi-threading.
I've also tried adding the 25000 lines one line at a time to a
single GL batch and the requests does not time out, but
performance-speed starts decreasing significantly after
approximately 3000 records or so are added to the GL batch. This
process takes several hours to run and since it uses the same GL
batch, this solution does not leverage multi-threading.
I looked into multi-threading as well to import the data into
several smaller GL-batches of 5000 lines each and that works without
any timeout issues. but it still takes about an hour and a half to
run. Also, the customer does not accept this multi-batch approach;
they want all their daily data in a single GL batch.
25,000 records does not seem like a lot to me, so I wonder if Acumatica’s APIs were not built for this volume of lines in a single transaction. All I’m doing in my code is building the entity info by reading a text file and then calling the put method to create the GL batch using that entity with 25,000 line records.
I've read a couple of articles about optimizing the APIs, but they primarily deal with different instances of an entity, as in several different GL batches or several different Stock Items for example. In those cases, multi-threading is a great asset because you can have multiple threads creating multiple "different" GL batches, but multi-threading is not helpful when updating the same GL batch.
Here's what I've read so far:
https://asiablog.acumatica.com/2016/12/optimizing-large-import.html
https://adn.acumatica.com/blog/contractapioptimization/
I'm at a loss here, so any pointers would be greatly appreciated.
I look forward to your response.
Here's my code:
public static void CreateMultipleLinesPerJournalEntryBatchContractTEST(MyStoreContract.DefaultSoapClient soapClient, List<JournalEntry> journalEntries)
{
string myModuleForBatchLookup = "GL";
//list holding the values of all the records belonging to the batch in process
List<JournalEntry> allBatchItems = journalEntries;
//List used to store objects in format required by Acumatica
List<MyStoreContract.JournalTransactionDetail> myJournalTransactionsFormatted = new List<MyStoreContract.JournalTransactionDetail>();
try
{
//Creating a header and returning a batch value to be used for all line iterations.
JournalEntry myHeaderJournalEntryContract = allBatchItems.First();
string myBatchNumberToProcess = AddGLBatchHeaderContractTEST(soapClient, myHeaderJournalEntryContract);
// Do something with then n number of items defined in processing subBatch size or remaining items if smaller
foreach (JournalEntry je in allBatchItems)
{
//Moving the items in each batch from the original unformatted list to the formatted list one at a time
myJournalTransactionsFormatted.Add(new MyStoreContract.JournalTransactionDetail
{
BranchID = new MyStoreContract.StringValue { Value = je.Branch },
Account = new MyStoreContract.StringValue { Value = je.Account },
Subaccount = new MyStoreContract.StringValue { Value = je.Subaccount },
ReferenceNbr = new MyStoreContract.StringValue { Value = je.RefNumber },
DebitAmount = new MyStoreContract.DecimalValue { Value = je.DebitAmount },
CreditAmount = new MyStoreContract.DecimalValue { Value = je.CreditAmount },
TransactionDescription = new MyStoreContract.StringValue { Value = je.TransactionDescription },
UsrTransactionTime = new MyStoreContract.StringValue { Value = je.UsrTransactionTime },
UsrTransactionType = new MyStoreContract.StringValue { Value = je.UsrTransactionType },
UsrTranSequence = new MyStoreContract.StringValue { Value = je.UsrTranSequence },
UsrTellerID = new MyStoreContract.StringValue { Value = je.UsrTellerID }
});
}
//Specify the values of a new Jornal Entry using all the collected elements from the batch(list) created
MyStoreContract.JournalTransaction journalToBeCreated = new MyStoreContract.JournalTransaction
{
//Header data and details added by list generated by loop
BatchNbr = new MyStoreContract.StringSearch { Value = myBatchNumberToProcess }, //This is one of two lines used to lookup/search the batch needing to be updated
Module = new MyStoreContract.StringSearch { Value = myModuleForBatchLookup }, //This is one of two lines used to lookup/search the batch needing to be updated
Details = myJournalTransactionsFormatted.ToArray() // this is the line adding the array containing all the line details
};
soapClient.Put(journalToBeCreated);
Console.WriteLine("Added " + allBatchItems.Count.ToString() + " line transactions");
Console.WriteLine();
Console.WriteLine("Press any key to continue");
Console.ReadLine();
}
catch (Exception e)
{
Console.WriteLine("The following error was encountered and all entries for this batch need to be logged in error table");
Console.WriteLine();
Console.WriteLine(e.Message);
Console.WriteLine();
Console.WriteLine("Press any key to continue");
Console.ReadLine();
}
}
public static string AddGLBatchHeaderContractTEST(MyStoreContract.DefaultSoapClient soapClient, JournalEntry je)
{
try
{
//Specify the values of a new Jornal Entry Batch header
MyStoreContract.JournalTransaction journalToBeCreated = new MyStoreContract.JournalTransaction
{
//Header data
BranchID = new MyStoreContract.StringValue { Value = "PRODWHOLE" }, //This is the default branch
TransactionDate = new MyStoreContract.DateTimeValue { Value = je.TransactionDate.AddDays(-1) }, //Reduced 1 day from the batch
CurrencyID = new MyStoreContract.StringValue { Value = je.CurrencyCode }, //Currency to be used for the batch
Description = new MyStoreContract.StringValue { Value = je.TransactionDescription },
Hold = new MyStoreContract.BooleanValue { Value = true }
};
//Create a Journal Entry with the specified values
MyStoreContract.JournalTransaction newJournalTransaction = (MyStoreContract.JournalTransaction)soapClient.Put(journalToBeCreated);
string myBatchToProcess = newJournalTransaction.BatchNbr.Value;
return myBatchToProcess;
}
catch (Exception e)
{
Console.WriteLine("Error was caught while trying to create the header for the batch...");
Console.WriteLine();
Console.WriteLine(e);
Console.WriteLine();
return null;
}
}
My custom class for legacy system line items which I then need format into Acumatica's format:
class JournalEntry
{
public DateTime TransactionDate { get; set; }
public string CurrencyCode { get; set; }
public string Description { get; set; }
public string Branch { get; set; }
public string Account { get; set; }
public string Subaccount { get; set; }
public string RefNumber { get; set; }
public decimal DebitAmount { get; set; }
public decimal CreditAmount { get; set; }
public string TransactionDescription { get; set; }
//Added custom fields for customer
public string UsrTellerID { get; set; }
public string UsrTransactionType { get; set; }
public string UsrTransactionTime { get; set; }
public string UsrTranSequence { get; set; }
//Adding original file data for the line
public string FileLineData { get; set; }
}
I tried Yuriy's approach described below, but my custom fields are not updating. Only standard fields are being updated. Which command should I use to update the extension (custom) fields. See code below:
//Here I create instance of GLTran
GLTran row = graph.GLTranModuleBatNbr.Cache.CreateInstance() as GLTran;
//here I get a handle to graph extension GLTranExt to be able to use the added fields.
var rowExt = row.GetExtension<GLTranExt>();
row = graph.GLTranModuleBatNbr.Insert(row);
graph.GLTranModuleBatNbr.Cache.SetValueExt(row, "AccountID", JE.Account);
graph.GLTranModuleBatNbr.Cache.SetValueExt(row, "SubID", JE.Subaccount);
row.TranDesc = "my line description";
row.Qty = 1.0m;
row.CuryDebitAmt = (JE.DebitAmount);
row.CuryCreditAmt = (JE.CreditAmount);
rowExt.UsrTellerID = "Test teller";
rowExt.UsrTransactionTime = "Test Transaction Time";
rowExt.UsrTransactionType = "Test Transaction Type";
rowExt.UsrTranSequence = "Test Transaction Sequence";
row = graph.GLTranModuleBatNbr.Update(row);
graph.Actions.PressSave();
In multi threaded import of Sales Orders I've got 18000 lines per hour ( 4 cores, 32Gb RAM ). So your 25000 is very similar to what I've get ( one Sales order had 1 - 6 lines ). For second link that you provided, what were parameters of your API call, what was number of your Acumatica instances ( CPU, RAM, parameters of SQL Server )?
I propose you to consider scaling Acumatica horizontally and also scale your database via SQL sharding.
Edit
In case if you need to have one GL Batch with 25000 lines on it, then I propose you following workaround:
Create one more Acumatica page that has text box and button Import.
In code of button Import button
2.1 read text box information as xml ( or JSON )
2.2 Create instance of GL Graph
2.3 Insert via Graph needed amount ( in your case 25000 ) lines
2.4 Call to graph.PressSave()
Send Web API your request not to GL Batch but to created by you page.
I know this is an old question, but I am answering here for the benefit of anyone who stumbles across this page. There's a performance issue in the Journal Transaction screen where the time to create a transaction increases non-linearly with the number of rows to insert.
A customization-based workaround was provided to us by Acumatica support which significantly improved performance. I don't have the exact version where this fix was included, but builds newer than today (September 2021) should include this already.
Customization fix:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using PX.Api;
using PX.Data;
using PX.Common;
using PX.Objects.Common;
using PX.Objects.Common.Extensions;
using PX.Objects.CS;
using PX.Objects.CM;
using PX.Objects.CA;
using PX.Objects.Common.Bql;
using PX.Objects.Common.GraphExtensions.Abstract;
using PX.Objects.Common.GraphExtensions.Abstract.DAC;
using PX.Objects.Common.GraphExtensions.Abstract.Mapping;
using PX.Objects.GL.DAC;
using PX.Objects.GL.FinPeriods;
using PX.Objects.GL.JournalEntryState;
using PX.Objects.GL.JournalEntryState.PartiallyEditable;
using PX.Objects.GL.Overrides.PostGraph;
using PX.Objects.GL.Reclassification.UI;
using PX.Objects.PM;
using PX.Objects.TX;
using PX.Objects.Common.Tools;
using PX.Objects.GL.DAC.Abstract;
using PX.Objects.Common.EntityInUse;
using PX.Objects.GL.FinPeriods.TableDefinition;
using PX.Data.SQLTree;
using PX.Objects.CR;
using PX.Data.BQL.Fluent;
using PX.Data.BQL;
using PX.Objects;
using PX.Objects.GL;
namespace PX.Objects.GL
{
public class JournalEntry_Extension : PXGraphExtension<JournalEntry>
{
public delegate void PopulateSubDescrDelegate(PXCache sender, GLTran Row, Boolean ExternalCall);
[PXOverride]
public void PopulateSubDescr(PXCache sender, GLTran Row, Boolean ExternalCall, PopulateSubDescrDelegate baseMethod)
{
if (Base.IsImport || Base.IsExport || Base.IsContractBasedAPI)
{
return;
}
baseMethod(sender,Row,ExternalCall);
}
}
}

How to retrieve only a predefined number of results in Azure Tables

I am trying to perform an azure table query.
My table (that saves logs) has thousands of rows of data, and it gets populated with more each second.
Right now i have only 1 partition key, but it doesn't affect the next question.
How can i get back lets say only the 100 latest results.
this is my Entity:
public MainServerLogEntity(string Message)
{
this.PartitionKey = "mainserverlogs";
this.RowKey = (DateTime.MaxValue.Ticks - DateTime.UtcNow.Ticks).ToString();
this.Message = Message;
this.Date = DateTime.UtcNow;
}
public MainServerLogEntity() { }
public string Message { get; set; }
public DateTime Date { get; set; }
Right now this is the query i am performing inside a web api i have:
[Route("MainServerLogs")]
[HttpGet]
public IEnumerable<MainServerLogEntity> GetMainServerLogs()
{
CloudTable table = AzureStorageHelpers.GetWebApiTable(connectionString, "mainserverlogs");
TableQuery<MainServerLogEntity> query = new TableQuery<MainServerLogEntity>().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "mainserverlogs"));
return table.ExecuteQuery(query);
}
But the problem is that i am getting alot of data, and i am requesting this api every few seconds in order to update the ui.
What should i do? is it possible to define in the query that i only want the 100 first rows?
If it is not possible then what other technique should i use?
Try implementing a .Take(100) on the query like so:
[Route("MainServerLogs")]
[HttpGet]
public IEnumerable<MainServerLogEntity> GetMainServerLogs()
{
CloudTable table = AzureStorageHelpers.GetWebApiTable(connectionString, "mainserverlogs");
TableQuery<MainServerLogEntity> query = new TableQuery<MainServerLogEntity>().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "mainserverlogs")).Take(100);
return table.ExecuteQuery(query);
}

Neo4jClient: decimal point dropped

I'm having trouble using properties of type decimal in my program, which uses Neo4jClient to interact with my Neo4J server.
using System;
using Neo4jClient;
namespace TestProject
{
class Program
{
static void Main()
{
var graphClient = new GraphClient(new Uri("http://localhost:7474/db/data"));
graphClient.Connect();
const decimal socialFactor = 0.5m;
var userIn = new User {Name = "John Doe", SocialFactor = socialFactor};
var nodeRef = graphClient.Create(userIn);
var userOut = graphClient.Get(nodeRef).Data;
Console.WriteLine(userOut.Name + ", " + userOut.SocialFactor);
Console.WriteLine(userOut.SocialFactor == socialFactor ? "win" : "fail!");
}
}
class User
{
public string Name { get; set; }
public decimal SocialFactor { get; set; }
}
}
What the program does is insert a user node with a social factor of 0.5 into Neo4j (userIn), and immediately retrieves the user node again (userOut). The problem is that userOut.SocialFactor is 5 instead of 0.5!
Strangely enough, when I change the type to double, there is no problem, but that's obviously something I don't want to do.
Is this a bug? Is there some kind of workaround?
Thanks,
Jan
Turns out that yes, it was a bug but it was recently fixed in version 1.0.0.595 of Neo4jClient.

How to read last message in MessageQueue

I am trying to read the messages in MessageQueue start from newest one. I know I can use GetAllMessages(). Then loop through. But it is quite slow when there are large amount of messages in the queue. Is there a better way to do this?
Thanks
In MSMQ 3 there is a feature that may be what you're looking for. The ReceiveByLookupId method can get the last message on the queue.
http://msdn.microsoft.com/en-us/library/3w50th9h.aspx
You would use the following action:
MessageLookupAction.Last: Receives the last message in the queue and
removes it from the queue. The lookupId parameter must be set to 0.
If you're using an earlier version of MSMQ you will probably have to use Peek similar to how this blog post explains how to get a count of the message queue. When you got to the last one, you could then ReceiveById.
http://jopinblog.wordpress.com/2008/03/12/counting-messages-in-an-msmq-messagequeue-from-c/
Assuming that the performance hit is in the looping through the messages:
There is no need to loop through all the messages if all you want is the last one. Since GetAllMessages() returns an array of type Message just go to the last element in the array by index.
public class PersonDetails
{
public string FirstName { get; set; }
public string LastName { get; set; }
public DateTime DateTime { get; set; }
}
const string queueName = #".\private$\PersonQueue";
public void GetMessageFromQueue(string queueName)
{
MessageQueue perosnMessageQueue = new MessageQueue(queueName);
try
{
XmlMessageFormatter xmlMessageFormatter = new XmlMessageFormatter(new Type[] {
(typeof(PersonDetails)) });
perosnMessageQueue.Formatter = xmlMessageFormatter;
perosnMessageQueue.Refresh();
for (int i = perosnMessageQueue.GetAllMessages().Count(); i !=0; i--)
{
var personDetailsFromQueue =
(PersonDetails)perosnMessageQueue.Receive
(MessageQueueTransactionType.Automatic).Body;
Console.WriteLine("FistName : {0} \n LastName : {1} \n Date Time :
{2}",personDetailsFromQueue.FirstName,
personDetailsFromQueue.LastName,
personDetailsFromQueue.DateTime);`
}

Retrieving values of ReadOnly fields from DynamicData DetailsView in Edit Mode on Updating using LinqDataSource

I have several tables in my database that have read-only fields that get set on Inserting and Updating, namely: AddDate (DateTime), AddUserName (string), LastModDate (DateTime), LastModUserName (string).
All of the tables that have these values have been set to inherit from the following interface:
public interface IUserTrackTable
{
string AddUserName { get; set; }
DateTime AddDate { get; set; }
string LastModUserName { get; set; }
DateTime LastModDate { get; set; }
}
As such, I have the following method on the Edit.aspx page:
protected void DetailsDataSource_Updating(object sender, LinqDataSourceUpdateEventArgs e)
{
IUserTrackTable newObject = e.NewObject as IUserTrackTable;
if (newObject != null)
{
newObject.LastModUserName = User.Identity.Name;
newObject.LastModDate = DateTime.Now;
}
}
However, by the time it hits this method, the e.OriginalObject has already lost the values for all four fields, so a ChangeConflictException gets thrown during the actual Update. I have tried adding the four column names to the DetailsView1.DataKeyNames array in the Init event handler:
protected void Page_Init(object sender, EventArgs e)
{
// other things happen before this
var readOnlyColumns = table.Columns.Where(c => c.Attributes.SingleOrDefaultOfType<ReadOnlyAttribute>(ReadOnlyAttribute.Default).IsReadOnly).Select(c => c.Name);
DetailsView1.DataKeyNames = DetailsView1.DataKeyNames.Union<string>(readOnlyColumns).ToArray<string>();
DetailsView1.RowsGenerator = new CustomFieldGenerator(table, PageTemplates.Edit, false);
// other things happen after this
}
I've tried making that code only happen on PostBack, and still nothing. I'm at a lose for how to get the values for all of the columns to make the round-trip.
The only thing the CustomFieldGenerator is handling the ReadOnlyAttribute, following the details on C# Bits.
UPDATE: After further investigation, the values make the round trip to the DetailsView_ItemUpdating event. All of the values are present in the e.OldValues dictionary. However, they are lost by the time it gets to the LinqDataSource_Updating event.
Obviously, there are the "solutions" of making those columns not participate in Concurrency Checks or other ways that involve hard-coding, but the ideal solution would dynamically add the appropriate information where needed so that this stays as a Dynamic solution.
i Drovani, I assume you want data auditing (see Steve Sheldon's A Method to Handle Audit Fields in LINQ to SQL), I would do this in the model in EF4 you can do it like this:
partial void OnContextCreated()
{
// Register the handler for the SavingChanges event.
this.SavingChanges += new EventHandler(context_SavingChanges);
}
private static void context_SavingChanges(object sender, EventArgs e)
{
// handle auditing
AuditingHelperUtility.ProcessAuditFields(objects.GetObjectStateEntries(EntityState.Added));
AuditingHelperUtility.ProcessAuditFields(objects.GetObjectStateEntries(EntityState.Modified), InsertMode: false);
}
internal static class AuditingHelperUtility
{
internal static void ProcessAuditFields(IEnumerable<Object> list, bool InsertMode = true)
{
foreach (var item in list)
{
IAuditable entity = item as IAuditable;
if (entity != null)
{
if (InsertMode)
{
entity.InsertedBy = GetUserId();
entity.InsertedOn = DateTime.Now;
}
entity.UpdatedBy = GetUserId();
entity.UpdatedOn = DateTime.Now;
}
}
}
}
Sadly this is not possible with EF v1

Resources