I have a very simple requirement to get All values stored in the IMap. I have a server and client on the same box (no cluster just one instance of server and one instance of the client).
Server start:
Config config = new Config();
NetworkConfig network = config.getNetworkConfig();
JoinConfig join = network.getJoin();
join.getMulticastConfig().setEnabled( false );
join.getTcpIpConfig().setEnabled( false );
join.getAwsConfig().setEnabled(false);
HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);
Client start:
ClientConfig clientConfig = new ClientConfig();
clientConfig.getNetworkConfig().addAddress("127.0.0.1");
HazelcastClient hazelIns = HazelcastClient.newHazelcastClient(clientConfig);
IMap<String,DeviceStatus> map = hazelIns.getMap("map");
Query:
class DeviceStatus{
String name;
String status;
DeviceStatus(String name,String status){
this.name=name;
this.status= status;
}
// PUT ALL OBJECTS
for(int i =0;i<1000;i++){
DeviceStatus status = new DeviceStatus("Device"+i,"UP");
map.put(status.getName(),status);
}
for(int i=0; i< 10000; i++){
List<DeviceStatus> list = (List<DeviceStatus>)hazelIns.getMap("map").values();
if(list.size() != 1000){
System.out.println("ALL SIZE "+ list.size());
}
String sql = "name ilike %D%";
List<DeviceStatus> list1 = hazelIns.getMap("map").values( new SqlPredicate( sql ));
if(list1.size() != 1000){
System.out.println("SQL SIZE "+ list1.size());
}
Thread.sleep(1000);
}
Hazelcast version:
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
<version>3.7.3</version>
</dependency>
<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-client</artifactId>
<version>3.7.3</version>
</dependency>
I am getting size miss-match size() sometimes from 1 to 1000 but I get 1000 size for majority of time. There is NO near-realtime cache. Why there is an inconsistent result set for both SQL and total size? Just FYI, cache key is updated in the background but never removed.
Any pointer is would be a great help!
Updated:
Background update in a separate thread:
run(){
for(int i=0; i < 10; i++){
DeviceStatus status = new DeviceStatus("Device"+i,"DOWN");
map.put(status.getName(),status);
}
Thanks,
Bhavesh
Related
I have an ASP.NET Core app, with a model, the aim is to allow user to upload an excel file and then save the file to the model/table. I have the below method
[HttpPost]
[ValidateAntiForgeryToken]
public async Task<IActionResult> Upload(IFormFile file)
{
string webRootPath = _hostEnvironment.WebRootPath;
var uploads = Path.Combine(webRootPath, "Upload");
var files = HttpContext.Request.Form.Files;
var extension = Path.GetExtension(files[0].FileName);
using (var filesStream = new FileStream(Path.Combine(uploads, file.FileName), FileMode.Create))
{
files[0].CopyTo(filesStream);
}
var list = new List<User>();
using (var stream = new MemoryStream())
{
await file.CopyToAsync(stream);
using (var package = new ExcelPackage(stream))
{
ExcelWorksheet worksheet = package.Workbook.Worksheets[0];
var rowcount = worksheet.Dimension.Rows;
for (int row = 2; row <= rowcount; row++)
{
list.Add(new User
{
Name = worksheet.Cells[row, 1]?.Value?.ToString().Trim(),
Address1 = worksheet.Cells[row, 2]?.Value?.ToString().Trim(),
PostCode = worksheet.Cells[row, 3]?.Value?.ToString().Trim(),
Mobile = worksheet.Cells[row, 4]?.Value?.ToString().Trim(),
});
}
}
}
foreach (var user in list)
{
_db.User.AddAsyncy(user);
}
_db.SaveChangesAsyncy();
return View();
}
This code works fine by processing an excel file uploaded by a user but the problem I'm having is that when the file is large say above 3 mb, it takes well over 8 minutes to upload.
Any idea how to speed this up please? Thanks.
There are two things you can do to increase speed.
1)Instead of reading excel file with ExcelWorksheet class go with a library called ExcelDataReader which can read around 600k records under a minute.
sample code
Model
class Person
{
public int id,
public string name
}
//and excel file has both columns in model ,the we can read with below code
using ExcelDataReader;
System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance);
var fileName = "./Person.xlsx";
var timer = new Stopwatch();
timer.Start();
int counter=0;
List<Person> persons = new List<Person>();
using (var stream = System.IO.File.Open(fileName, FileMode.Open, FileAccess.Read))
{
using (var reader = ExcelReaderFactory.CreateReader(stream))
{
while (reader.Read()) //Each row of the file
{
var person = new Person
{
id = reader.GetValue(0).ToString(),
name = reader.GetValue(1).ToString()
}
persons.Add(person)
counter++;
}
timer.Stop();
duration = timer.ElapsedMilliseconds / 1000;
//to check performace print duration and persons list
}
}
https://github.com/ExcelDataReader/ExcelDataReader
2)Once you read and store data in a list, you can store that data in DataTable class and insert into database using Oracle.ManagedDataAccess.Client Nuget package instead of EFcore. This method is fast. Please go through below link for doing this with Oracle database.
https://www.c-sharpcorner.com/article/two-ways-to-insert-bulk-data-into-oracle-database-using-c-sharp/
var db_timer = new Stopwatch();
db_timer.Start();
DataTable dt = new DataTable();
dt.Columns.Add("id");
dt.Columns.Add("name");
for (int i = 0; i < counter; i++)
{
DataRow dr = dt.NewRow();
dr["id"] = persons[i].id;
dr["name"] = persons[i].name;
dt.Rows.Add(dr);
}
using (var connection = new OracleConnection(oracleConString))
{
connection.Open();
int[] ids = new int[dt.Rows.Count];
string[] names = new string[dt.Rows.Count];
for (int j = 0; j < dt.Rows.Count; j++)
{
ids[j] = Convert.ToString(dt.Rows[j]["id"]);
names[j] = Convert.ToString(dt.Rows[j]["name"]);
}
OracleParameter id = new OracleParameter();
id.OracleDbType = OracleDbType.Int32;
id.Value = ids;
OracleParameter name = new OracleParameter();
name.OracleDbType = OracleDbType.Varchar2;
name.Value = names;
OracleCommand cmd = connection.CreateCommand();
cmd.CommandText = "INSERT INTO TEST(id,name) VALUES (:1,:2)";
cmd.ArrayBindCount = ids.Length;
cmd.Parameters.Add(id);
cmd.Parameters.Add(name);
cmd.ExecuteNonQuery();
}
just sample code you can user timer to check how much time it is taking to execute.
I set up a Cassandra cluster on AWS. What I want to get is increased I/O throughput (number of reads/writes per second) as more nodes are added (as advertised). However, I got exactly the opposite. The performance is reduced as new nodes are added.
Do you know any typical issues that prevents it from scaling?
Here is some details:
I am adding a text file (15MB) to the column family. Each line is a record. There are 150000 records. When there is 1 node, it takes about 90 seconds to write. But when there are 2 nodes, it takes 120 seconds. I can see the data is spread to 2 nodes. However, there is no increase in throughput.
The source code is below:
public class WordGenCAS {
static final String KEYSPACE = "text_ks";
static final String COLUMN_FAMILY = "text_table";
static final String COLUMN_NAME = "text_col";
public static void main(String[] args) throws Exception {
if (args.length < 2) {
System.out.println("Usage: WordGenCAS <input file> <host1,host2,...>");
System.exit(-1);
}
String[] contactPts = args[1].split(",");
Cluster cluster = Cluster.builder()
.addContactPoints(contactPts)
.build();
Session session = cluster.connect(KEYSPACE);
InputStream fis = new FileInputStream(args[0]);
InputStreamReader in = new InputStreamReader(fis, "UTF-8");
BufferedReader br = new BufferedReader(in);
String line;
int lineCount = 0;
while ( (line = br.readLine()) != null) {
line = line.replaceAll("'", " ");
line = line.trim();
if (line.isEmpty())
continue;
System.out.println("[" + line + "]");
String cqlStatement2 = String.format("insert into %s (id, %s) values (%d, '%s');",
COLUMN_FAMILY,
COLUMN_NAME,
lineCount,
line);
session.execute(cqlStatement2);
lineCount++;
}
System.out.println("Total lines written: " + lineCount);
}
}
The DB schema is the following:
CREATE KEYSPACE text_ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 };
USE text_ks;
CREATE TABLE text_table (
id int,
text_col text,
primary key (id)
) WITH COMPACT STORAGE;
Thanks!
Even if this an old post, I think it's worth posting a solution for these (common) kind of problems.
As you've already discovered, loading data with a serial procedure is slow. What you've been suggested is the right thing to do.
However, issuing a lot of queries without applying some sort of back pressure is likely looking for troubles, and you'll gonna lose data due to excessive overload on the server (and on the driver to some extent).
This solution will load data with async calls, and will try to apply some back pressure on the client to avoid data loss.
public class WordGenCAS {
static final String KEYSPACE = "text_ks";
static final String COLUMN_FAMILY = "text_table";
static final String COLUMN_NAME = "text_col";
public static void main(String[] args) throws Exception {
if (args.length < 2) {
System.out.println("Usage: WordGenCAS <input file> <host1,host2,...>");
System.exit(-1);
}
String[] contactPts = args[1].split(",");
Cluster cluster = Cluster.builder()
.addContactPoints(contactPts)
.build();
Session session = cluster.connect(KEYSPACE);
InputStream fis = new FileInputStream(args[0]);
InputStreamReader in = new InputStreamReader(fis, "UTF-8");
BufferedReader br = new BufferedReader(in);
String line;
int lineCount = 0;
// This is the futures list of our queries
List<Future<ResultSet>> futures = new ArrayList<>();
// Loop
while ( (line = br.readLine()) != null) {
line = line.replaceAll("'", " ");
line = line.trim();
if (line.isEmpty())
continue;
System.out.println("[" + line + "]");
String cqlStatement2 = String.format("insert into %s (id, %s) values (%d, '%s');",
COLUMN_FAMILY,
COLUMN_NAME,
lineCount,
line);
lineCount++;
// Add the "future" returned by async method the to the list
futures.add(session.executeAsync(cqlStatement2));
// Apply some backpressure if we issued more than X query.
// Change X to another value suitable for your cluster
while (futures.size() > 1000) {
Future<ResultSet> future = futures.remove(0);
try {
future.get();
} catch (Exception e) {
e.printStackTrace();
}
}
}
System.out.println("Total lines written: " + lineCount);
System.out.println("Waiting for writes to complete...");
// Wait until all writes are done.
while (futures.size() > 0) {
Future<ResultSet> future = futures.remove(0);
try {
future.get();
} catch (Exception e) {
e.printStackTrace();
}
}
System.out.println("Done!");
}
}
I am trying to change all the Business Unit references I got after importing a solution to the ones in the Acceptance environment.
QueryExpression ViewQuery = new QueryExpression("savedquery");
String[] viewArrayFields = { "name", "fetchxml" };
ViewQuery.ColumnSet = new ColumnSet(viewArrayFields);
ViewQuery.PageInfo = new PagingInfo();
ViewQuery.PageInfo.Count = 5000;
ViewQuery.PageInfo.PageNumber = 1;
ViewQuery.PageInfo.ReturnTotalRecordCount = true;
EntityCollection retrievedViews = service.RetrieveMultiple(ViewQuery);
//iterate though the values and print the right one for the current user
int oldValues = 0;
int accValuesUpdated = 0;
int prodValuesUpdated = 0;
int total = 0;
foreach (var entity in retrievedViews.Entities)
{
total++;
if (!entity.Contains("fetchxml"))
{ }
else
{
string fetchXML = entity.Attributes["fetchxml"].ToString();
for (int i = 0; i < guidDictionnary.Count; i++)
{
var entry = guidDictionnary.ElementAt(i);
if (fetchXML.Contains(entry.Key.ToString().ToUpperInvariant()))
{
Console.WriteLine(entity.Attributes["name"].ToString());
oldValues++;
if (destinationEnv.Equals("acc"))
{
accValuesUpdated++;
Console.WriteLine();
Console.WriteLine("BEFORE:");
Console.WriteLine();
Console.WriteLine(entity.Attributes["fetchxml"].ToString());
string query = entity.Attributes["fetchxml"].ToString();
query = query.Replace(entry.Key.ToString().ToUpperInvariant(), entry.Value.AccGuid.ToString().ToUpperInvariant());
entity.Attributes["fetchxml"] = query;
Console.WriteLine();
Console.WriteLine("AFTER:");
Console.WriteLine();
Console.WriteLine(entity.Attributes["fetchxml"].ToString());
}
else
{
prodValuesUpdated++;
string query = entity.Attributes["fetchxml"].ToString();
query = query.Replace(entry.Key.ToString().ToUpperInvariant(), entry.Value.ProdGuid.ToString().ToUpperInvariant());
entity.Attributes["fetchxml"] = query;
}
service.Update(entity);
}
}
}
}
Console.WriteLine("{0} values to be updated. {1} shall be mapped to acceptance, {2} to prod. Total = {3} : {4}", oldValues, accValuesUpdated, prodValuesUpdated, total, retrievedViews.Entities.Count);
I see that the new value is corrected, but it does not get saved. I get no error while updating the record and publishing the changes in CRM does not help.
Any hint?
According to your comments, it sounds like the value you're saving the entity as, is the value that you want it to be. I'm guessing your issue is with not publishing your change. If you don't publish it, it'll still give you the old value of the FetchXml I believe.
Try calling this method:
PublishEntity(service, "savedquery");
private void PublishEntity(IOrganizationService service, string logicalName)
{
service.Execute(new PublishXmlRequest()
{
ParameterXml = "<importexportxml>"
+ " <entities>"
+ " <entity>" + logicalName + "</entity>"
+ " </entities>"
+ "</importexportxml>"
});
}
I am trying understand concurrency in wcf and downloaded a sample code from here and done some changes for testing two way calls. To my surprise I could not see the effect of concurrency for two way calls (Although I can see concurrency for one way calls).
Is it the way WCF concurrency model works? (or)
Am I doing something terribly wrong?
This is the service code.
[ServiceContract]
public interface IHelloWorldService
{
[OperationContract(IsOneWay=true)]
void Call(string ClientName);
[OperationContract]
string GetData(int value);
[OperationContract]
CompositeType GetDataUsingDataContract(CompositeType composite);
}
[DataContract]
public class CompositeType
{
bool boolValue = true;
string stringValue = "Hello ";
[DataMember]
public bool BoolValue
{
get { return boolValue; }
set { boolValue = value; }
}
[DataMember]
public string StringValue
{
get { return stringValue; }
set { stringValue = value; }
}
}
[ServiceBehavior(InstanceContextMode=InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
public class HelloWorldService : IHelloWorldService
{
public static int counter;
public HelloWorldService()
{
counter++;
}
public void Call(string ClientName)
{
Console.WriteLine("Instance:" + counter.ToString() + " Thread:" + Thread.CurrentThread.ManagedThreadId.ToString() + " Time:" + DateTime.Now.ToString() + "\n\n");
Thread.Sleep(5000);
}
public string GetData(int value)
{
Console.WriteLine("Instance:" + counter.ToString() + " Thread:" + Thread.CurrentThread.ManagedThreadId.ToString() + " Time:" + DateTime.Now.ToString() + "\n\n");
Thread.Sleep(5000);
return value.ToString();
}
public CompositeType GetDataUsingDataContract(CompositeType composite)
{
Console.WriteLine("Instance:" + counter.ToString() + " Thread:" + Thread.CurrentThread.ManagedThreadId.ToString() + " Time:" + DateTime.Now.ToString() + "\n\n");
Thread.Sleep(5000);
return composite;
}
}
This is the service hosting code.
class Program
{
static void Main(string[] args)
{
//Create a URI to serve as the base address
//Uri httpUrl = new Uri("net.tcp://localhost:8001/HelloWorld");
Uri httpUrl = new Uri("http://localhost:8010/MyService/HelloWorld");
//Create ServiceHost
ServiceHost host
= new ServiceHost(typeof(ClassLibrary1.HelloWorldService), httpUrl);
//Add a service endpoint
host.AddServiceEndpoint(typeof(ClassLibrary1.IHelloWorldService)
, new WSHttpBinding(), "");
//Enable metadata exchange
ServiceMetadataBehavior smb = new ServiceMetadataBehavior();
smb.HttpGetEnabled = true;
host.Description.Behaviors.Add(smb);
ServiceThrottlingBehavior stb = new ServiceThrottlingBehavior();
stb.MaxConcurrentCalls = 100;
stb.MaxConcurrentInstances = 100;
stb.MaxConcurrentSessions = 100;
host.Description.Behaviors.Add(stb);
//Start the Service
host.Open();
Console.WriteLine("Service is host at " + DateTime.Now.ToString());
Console.WriteLine("Host is running... Press <Enter> key to stop");
Console.ReadLine();
}
}
This is the client code.
class Program
{
static int m_NumberOfWorkers = 10;
static readonly object m_Locker = new object();
static bool flag_GO = false;
static Stopwatch m_OverAllStopwatch = new Stopwatch();
static ConcurrentBag<Stopwatch> m_bagIndividualStopwatch = new ConcurrentBag<Stopwatch>();
static int m_CompletedWorkers = 0;
static ServiceReference1.HelloWorldServiceClient m_objProxy;
static int m_KindOfMethod;
static void Main(string[] args)
{
while(true)
{
try
{
flag_GO = false;
Console.WriteLine("Enter number of concurrent clients:");
m_NumberOfWorkers = Int32.Parse(Console.ReadLine());
Console.WriteLine("Kind of method (1: One way, 2: Two way 3: Two way using data contract):");
m_KindOfMethod = Int32.Parse(Console.ReadLine());
// Create Workers
List<Thread> lstThreads = new List<Thread>();
for (int i = 0; i < m_NumberOfWorkers; ++i)
{
lstThreads.Add(new Thread(WaitOnPulse));
}
// Start Workers
for (int i = 0; i < lstThreads.Count; ++i)
{
lstThreads[i].Start();
}
m_objProxy = new ServiceReference1.HelloWorldServiceClient();
m_OverAllStopwatch.Restart();
// Signal all workers
lock (m_Locker)
{
flag_GO = true;
Monitor.PulseAll(m_Locker);
}
// Wait all workers to finish
for (int i = 0; i < lstThreads.Count; ++i)
{
lstThreads[i].Join();
}
m_objProxy.Close();
m_objProxy = null;
}
catch
{
return;
}
}
}
private static void WaitOnPulse()
{
lock (m_Locker)
{
while (!flag_GO) Monitor.Wait(m_Locker);
}
TestWhatEverYouWant();
IamDone();
}
private static void TestWhatEverYouWant()
{
Stopwatch stopWatch = Stopwatch.StartNew();
//Thread.Sleep(1000);
switch (m_KindOfMethod)
{
case 1:
m_objProxy.Call(m_NumberOfWorkers.ToString() + "Client Calls");
break;
case 2:
m_objProxy.GetData(m_NumberOfWorkers);
break;
case 3:
ServiceReference1.CompositeType objData = new ServiceReference1.CompositeType();
m_objProxy.GetDataUsingDataContract(objData);
break;
}
stopWatch.Stop();
m_bagIndividualStopwatch.Add(stopWatch);
}
private static void IamDone()
{
Interlocked.Increment(ref m_CompletedWorkers);
// Summarize results if all workers are done
if (Interlocked.CompareExchange(ref m_CompletedWorkers, 0, m_NumberOfWorkers) == m_NumberOfWorkers)
{
m_OverAllStopwatch.Stop();
Console.WriteLine("OverAll Elapsed Time: {0}", m_OverAllStopwatch.ElapsedMilliseconds);
Stopwatch stopWatch;
while (m_bagIndividualStopwatch.TryTake(out stopWatch))
//foreach (Stopwatch stopWatch in m_bagIndividualStopwatch)
{
Console.WriteLine("Individual Elapsed Time: {0}", stopWatch.ElapsedMilliseconds);
}
}
}
}
This is the Cleint trace:
Enter number of concurrent clients:
8
Kind of method (1: One way, 2: Two way 3: Two way using data contract):
2
OverAll Elapsed Time: 42022
Individual Elapsed Time: 42021
Individual Elapsed Time: 37013
Individual Elapsed Time: 32008
Individual Elapsed Time: 26987
Individual Elapsed Time: 21981
Individual Elapsed Time: 16980
Individual Elapsed Time: 11968
Individual Elapsed Time: 6985
This is the server trace:
Instance:1 Thread:6 Time:12/17/2012 8:09:29 PM
Instance:1 Thread:5 Time:12/17/2012 8:09:34 PM
Instance:1 Thread:7 Time:12/17/2012 8:09:39 PM
Instance:1 Thread:7 Time:12/17/2012 8:09:44 PM
Instance:1 Thread:5 Time:12/17/2012 8:09:49 PM
Instance:1 Thread:7 Time:12/17/2012 8:09:54 PM
Instance:1 Thread:5 Time:12/17/2012 8:09:59 PM
Instance:1 Thread:7 Time:12/17/2012 8:10:04 PM
For these results you can clearly see that the requests were processed sequentially. Ideally I was expecting all the 8 concurrent requrest would finish in 5 sec. But it took around 42 sec to finish.
The problem in my code is the way the proxy is used. I have created just one proxy for all concurrent clients and all calls to the service were made through this proxy only. So, all these calls were getting queued in the channel. Creating one proxy for each client, the way one should simulate wcf load test, solved the problem.
I still think the problem is with settings; proxy and number of threads originating from each proxy, the link explains it well.
Also have a look at following link; may be test client can have a problem.
Seeking WCF Duplex "TwoWay" Subscribe+Callback Example
I need to test a Data Context and see what behavior it has under multiple simultaneous requests, for that I made a simple console application that [in theory] would send these requests:
private static DateTime startTime = DateTime.Now.AddSeconds(5);
public static Random rand = new Random();
static void Main(string[] args)
{
const byte testThreads = 10;
ThreadStart[] threadStarts = new ThreadStart[testThreads];
Thread[] threads = new Thread[testThreads];
for (byte i = 0; i < testThreads; i++)
{
threadStarts[i] = new ThreadStart(ExecutePOST);
threads[i] = new Thread(threadStarts[i]);
}
for (byte i = 0; i < testThreads; i++){ threads[i].Start(); }
for (byte i = 0; i < testThreads; i++){ threads[i].Join(); }
}
The called function is
private static void ExecutePOST()
{
while (DateTime.Now < startTime) { }
Console.WriteLine("{0} STARTING TEST", DateTime.Now.Millisecond);
WebRequest webRequest = WebRequest.Create(/*URL*/);
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Method = "POST";
string name = string.Format("Test {0}", Program.rand.Next(1000));
byte[] bytes = Encoding.ASCII.GetBytes(/*PARAMETERS*/);
Stream output = null;
try
{
webRequest.ContentLength = bytes.Length;
output = webRequest.GetRequestStream();
output.Write(bytes, 0, bytes.Length);
Console.WriteLine("{0}:{1}", DateTime.Now.Millisecond, name);
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (output != null)
{
output.Close();
}
}
}
The output I get is:
Can anyone please explain this behavior? Why is it stopping after two requests?
Thank you
Yes, this is because the number of connections per URL is limited to 2 by default - the connections are pooled.
You're hogging the connection by writing data to the request stream, but then never getting the response. A simple:
using (webRequest.GetResponse()) {}
at the end of the method is likely to sort it out. That will finish the request and release the connection to be used by another request.
Also note that a using statement for the output stream would make your code simpler too.