Have started to Make the conversion faster from dataTable to List as I have 20K record in datatable and converting it in normal way takes 5 to 7 Minutes. SO I thought to Make it faster by using Parallel.ForEach or Task but still no benefit - Any suggestion please. Mu code is as below :
public static List<T> FillFromStoredProc<T>(string storedproc, SqlParameter[] prms) where T : IReadOnlyDatabaseObject, new()
{
DataTable dt = DatabaseHelper.runStoredProc(Configuration.Instance.ConnectionString, storedproc, prms);
ConcurrentBag<T> bag = new ConcurrentBag<T>();
IList<PropertyInfo> properties = typeof(T).GetProperties().ToList();
Parallel.ForEach(dt.AsEnumerable(), new ParallelOptions { MaxDegreeOfParallelism = 10 }, Drow => {
bag.Add(GetFromDataRow<T>(Drow, properties));
});
return bag.ToList();
}
public static T GetFromDataRow<T>(DataRow dr, IList<PropertyInfo> properties) where T : IReadOnlyDatabaseObject, new()
{
T ret = new T();
ret.LoadFromDataRowAsync(dr, properties);
return ret;
}
public virtual void LoadFromDataRowAsync(DataRow dr, IList<PropertyInfo> properties)
{
Parallel.ForEach(properties, new ParallelOptions { MaxDegreeOfParallelism = 10 }, prop =>
{
try
{
if (dr.Table.Columns.Contains(prop.Name))
{
if (prop.PropertyType.BaseType.Name == "Enum")
{
prop.SetValue(this, Enum.Parse(prop.PropertyType, dr[prop.Name].ToString()));
}
else
{
var val = DatabaseHelper.ConvertFromDBVal(prop.GetType(), dr[prop.Name]);
if (prop.PropertyType == typeof(DateTime))
{
// Convert back from DB value
if ((DateTime)(val) == SqlDateTime.MinValue)
{
val = DateTime.MinValue;
}
}
prop.SetValue(this, val);
}
}
}
catch
{
}
});
}
Please help me to make this faster. Thanks
Nesting parallelism is useless and must be avoided, because usually it only makes the overall performance worse.
Your bottleneck is the reflection - it is slow. You should come up with an alternative. For example, you can create a base class for your generic types, and use a virtual method that maps the name of a property to an actual property. It sounds like quite some grinding and dirty work, but it will be more efficient.
Related
Let say we have an object:
#:checkDirty
class Test {
var a:Int;
var b(default, default):String;
var c(get, set):Array<Int>;
public function new() {
...
}
public function get_c() {
...
}
public function set_c(n) {
...
}
}
Could we write a macro checkDirty so that any change to field/properties would set property dirty to true. Macro would generate dirty field as Bool and clearDirty function to set it to false.
var test = new Test();
trace(test.dirty); // false
test.a = 12;
trace(test.dirty); // true
test.clearDirty();
trace(test.dirty); //false
test.b = "test"
trace(test.dirty); //true
test.clearDirty();
test.c = [1,2,3];
trace(test.dirty); //true
Just to note - whenever you consider proxying access to an object, in my experience, there are always hidden costs / added complexity. :)
That said, you have a few approaches:
First, if you want it to be pure Haxe, then either a macro or an abstract can get the job done. Either way, you're effectively transforming every property access into a function call that sets the value and also sets dirty.
For example, an abstract using the #:resolve getter and setter can be found in the NME source code, replicated here for convenience:
#:forward(decode,toString)
abstract URLVariables(URLVariablesBase)
{
public function new(?inEncoded:String)
{
this = new URLVariablesBase(inEncoded);
}
#:resolve
public function set(name:String, value:String) : String
{
return this.set(name,value);
}
#:resolve
public function get(name:String):String
{
return this.get(name);
}
}
This may be an older syntax, I'm not sure... also look at the operator overloading examples on the Haxe manual:
#:op(a.b) public function fieldRead(name:String)
return this.indexOf(name);
#:op(a.b) public function fieldWrite(name:String, value:String)
return this.split(name).join(value);
Second, I'd just point out that if the underlying language / runtime supports some kind of Proxy object (e.g. JavaScript Proxy), and macro / abstract isn't working as expected, then you could build your functionality on top of that.
I wrote a post (archive) about doing this kind of thing (except for emitting events) before - you can use a #:build macro to modify class members, be it appending an extra assignment into setter or replacing the field with a property.
So a modified version might look like so:
class Macro {
public static macro function build():Array<Field> {
var fields = Context.getBuildFields();
for (field in fields.copy()) { // (copy fields so that we don't go over freshly added ones)
switch (field.kind) {
case FVar(fieldType, fieldExpr), FProp("default", "default", fieldType, fieldExpr):
var fieldName = field.name;
if (fieldName == "dirty") continue;
var setterName = "set_" + fieldName;
var tmp_class = macro class {
public var $fieldName(default, set):$fieldType = $fieldExpr;
public function $setterName(v:$fieldType):$fieldType {
$i{fieldName} = v;
this.dirty = true;
return v;
}
};
for (mcf in tmp_class.fields) fields.push(mcf);
fields.remove(field);
case FProp(_, "set", t, e):
var setter = Lambda.find(fields, (f) -> f.name == "set_" + field.name);
if (setter == null) continue;
switch (setter.kind) {
case FFun(f):
f.expr = macro { dirty = true; ${f.expr}; };
default:
}
default:
}
}
if (Lambda.find(fields, (f) -> f.name == "dirty") == null) fields.push((macro class {
public var dirty:Bool = false;
}).fields[0]);
return fields;
}
}
which, if used as
#:build(Macro.build())
#:keep class Some {
public function new() {}
public var one:Int;
public var two(default, set):String;
function set_two(v:String):String {
two = v;
return v;
}
}
Would emit the following JS:
var Some = function() {
this.dirty = false;
};
Some.prototype = {
set_two: function(v) {
this.dirty = true;
this.two = v;
return v;
}
,set_one: function(v) {
this.one = v;
this.dirty = true;
return v;
}
};
I have a clustered program where each thread wants to lock a set of keys.
As I understood the simplest solution using hazelcast:
private void lock(SortedSet<String> objects) {
try {
IMap<String, String> lockMap = getLockMap();
for (; ; ) {
SortedSet<String> lockedObjects = new TreeSet<>();
for (String object : objects) {
try {
boolean locked = lockMap.tryLock(object, 0, null, maxBlockTime, TimeUnit.MILLISECONDS);
if (locked) {
lockedObjects.add(object);
} else {
for (String lockedObject : lockedObjects) {
lockMap.unlock(lockedObject);
}
lockedObjects.clear();
break;
}
} catch (Exception e) {
for (String lockedObject : lockedObjects) {
try {
lockMap.unlock(lockedObject);
} catch(Exception ignored) {
}
}
throw e;
}
}
if (!lockedObjects.isEmpty()) {
return lockedObjects;
}
Thread.sleep(waitTime);
}
}
}
The main problem of this code that this code generates a lot of network traffics and requests to hazelcast. Could somebody recommend how this code can be optimized?
I couldn't find bulk tryLock functionality in the Hazelcast IMap.
Hazelcast does not offer a bulkLock method.
You can optimize this code in various ways, but before we get to that it would be great to know why you want to lock these entries and what you are trying to achieve.
This code is based on Coroutines guide example: Fan-out
val inputProducer = produce<String>(CommonPool) {
(0..inputArray.size).forEach {
send(inputArray[it])
}
}
val resultChannel = Channel<Result>(10)
repeat(threadCount) {
launch(CommonPool) {
inputProducer.consumeEach {
resultChannel.send(getResultFromData(it))
}
}
}
What is the right way to create a Sequence<Result> that will provide results?
You can get the channel .iterator() from the ReceiveChannel and then wrap that channel iterator into a Sequence<T>, implementing its normal Iterator<T> that blocks waiting for the result on each request:
fun <T> ReceiveChannel<T>.asSequence(context: CoroutineContext) =
Sequence {
val iterator = iterator()
object : AbstractIterator<T>() {
override fun computeNext() = runBlocking(context) {
if (!iterator.hasNext())
done() else
setNext(iterator.next())
}
}
}
val resultSequence = resultChannel.asSequence(CommonPool)
I had the same problem, and in the end I came up with this rather unusual/convoluted solution:
fun Channel<T>.asSequence() : Sequence<T> {
val itr = this.iterator()
return sequence<Int> {
while ( runBlocking {itr.hasNext()} ) yield( runBlocking<T> { itr.next() } )
}
}
I do not think it is particular efficient (go with the one provided by #hotkey), but it has a certain appeal to me at least.
I need to know, what would be proper way to implement Maps with 64 bit keys. There will not be so many items in them, I just need to use various bits of the key for various things with large enough address space and I need it to be very fast, so String keys would probably be too slow. So far I tried:
import haxe.Int64;
import haxe.Unserializer;
import haxe.Serializer;
class Test {
static function main () {
var key:Int64 = 1 << 63 | 0x00000001;
var omap:Map<Int64, String> = new Map<Int64, String>();
omap.set(key, "test");
var smap:Map<Int64, String> = Unserializer.run(Serializer.run(omap));
var key2:Int64 = 1 << 63 | 0x00000001;
trace(key+" "+smap.get(key2));
}
}
http://try.haxe.org/#7CDb2
which obviously doesn't work, because haxe.Int64 creates an object instance. Using cpp.Int64 works, because it for some reason falls back to 32 bit integer in my cpp code and I don't know what am I doing wrong. How can I force it to "stay" 64 bit, or should I do it some other way?
EDIT: This is currently not working on native targets due to bug / current implementation in hxcpp: https://github.com/HaxeFoundation/hxcpp/issues/523
I figured out this workaround / wrapper, which may not be the most efficient solution possible, but it seems to work.
import haxe.Int64;
import haxe.Unserializer;
import haxe.Serializer;
class Test {
static function main () {
var key:Int64 = Int64.make(1000,1);
var omap:Int64Map<String> = new Int64Map();
omap.set(key, "test");
var smap:Int64Map<String> = Unserializer.run(Serializer.run(omap));
var key2:Int64 = Int64.make(1000,1);
trace(key+" "+smap.get(key2));
}
}
class Int64Map<V> {
private var map:Map<Int64,V>;
public function new() : Void {
this.map = new Map<Int64,V>();
}
public function set(key:Int64, value:V):Void {
this.map.set(key, value);
}
public inline function get(key:Int64):Null<V> {
var skey:Null<Int64> = getMapKey(key);
if (skey != null) return this.map.get(skey);
return null;
}
public inline function exists(key:Int64):Bool {
return (getMapKey(key) != null);
}
public function remove( key : Int64 ) : Bool {
var skey:Null<Int64> = getMapKey(key);
if (skey != null) return this.map.remove(skey);
return false;
}
public function keys() : Iterator<Int64> {
return this.map.keys();
}
public function toString() : String {
return this.map.toString();
}
public function iterator() : Iterator<V> {
return this.map.iterator();
}
private function getMapKey(key:Int64):Null<Int64> {
for (ikey in this.map.keys()){
if (Int64.eq(key, ikey)){
return ikey;
}
}
return null;
}
}
http://try.haxe.org/#57686
I read about Type dynamic in C# 2010. (the corresponding msdn entry)
I am confused about the practical difference between T and dynamic while developing generic functions. My current tests didn't show new ways to use dynamica way, that isn't possible with T. In addition it seems, that dynamic needs much more time in runtime to achieve the same tasks.
Some example code to make my point clear:
// Sample Object
public class SampleObj
{
public string Test { get; set; }
}
My Test-Method (measuring speed as well):
static void Main(string[] args)
{
var sampleObj1 = new SampleObj { Test = "Test1" };
var sampleObj2 = new SampleObj { Test = "Test2" };
Stopwatch c1 = Stopwatch.StartNew();
bool res1 = CompareObjectsT<SampleObj>(sampleObj1, sampleObj2);
c1.Stop();
Console.WriteLine("Comparison T: {0} time: {1} ms", res1, c1.ElapsedMilliseconds);
Stopwatch c2 = Stopwatch.StartNew();
bool res2 = CompareObjectsDyna(sampleObj1, sampleObj2);
Console.WriteLine("Comparison dynamic: {0} time: {1} ms", res2, c2.ElapsedMilliseconds);
c2.Stop();
var instance = new X<SampleObj>();
Console.ReadLine();
}
The result is:
Comparison T: False time: 6 ms
Comparison dynamic: False time: 3969 ms
The dynamic functions needs much more time, compared to the T comparison.
Decompiling my test application reveals heavy usage of reflection which may lead to this huge amount of time:
private static bool CompareObjectsDyna([Dynamic] object source1, [Dynamic] object source2)
{
if (<CompareObjectsDyna>o__SiteContainer2.<>p__Site3 == null)
{
<CompareObjectsDyna>o__SiteContainer2.<>p__Site3 = CallSite<Func<CallSite, object, bool>>.Create(Binder.Convert(CSharpBinderFlags.None, typeof(bool), typeof(Program)));
}
if (<CompareObjectsDyna>o__SiteContainer2.<>p__Site4 == null)
{
<CompareObjectsDyna>o__SiteContainer2.<>p__Site4 = CallSite<Func<CallSite, Type, object, object, object>>.Create(Binder.InvokeMember(CSharpBinderFlags.None, "Equals", null, typeof(Program), new CSharpArgumentInfo[] { CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.IsStaticType | CSharpArgumentInfoFlags.UseCompileTimeType, null), CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null), CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) }));
}
return <CompareObjectsDyna>o__SiteContainer2.<>p__Site3.Target(<CompareObjectsDyna>o__SiteContainer2.<>p__Site3, <CompareObjectsDyna>o__SiteContainer2.<>p__Site4.Target(<CompareObjectsDyna>o__SiteContainer2.<>p__Site4, typeof(object), source1, source2));
}
I considered this post but this hasn't ansered my question.
Can someone tell me, in what scenario dynamic is much more effective than T?
(hopefully with a small practical sample)
Short answer is that generic type T must be known at compile time, but dynamic is inferred at runtime.