In my XPages applications I frequently use SSJS objects (com.ibm.jscript.std.ObjectObject) to cache data. To check if the performance improves if I use java.util.HashMaps instead, I benchmarked the execution times of the following code snippets (SSJS):
All three code snippets do the same: they create and fill either a SSJS Object or a HashMap with different types of data / objects. For each of the snippets I measured the average execution times over 1000 runs, with n (=maximum loop index in the snippet) being 1000000 (1 million). The benchmarks were performed on a Domino 9.0.1 server using java.lang.System.nanoTime .
The ratios of the execution times are as follows:
154% for T[HashMap] / T[SSJS Object]
266% for T[HashMap with put method] / T[SSJS Object]
172% for T[HashMap with put method] / T[HashMap]
In other words:
filling a HashMap took ~54% longer than filling a SSJS object
filling a HashMap using the put method took ~166% longer than filling a SSJS object
filling a HashMap using the put method took ~72% longer than filling a HashMap using the SSJS "." notation
My questions are as follows:
I frequently see SSJS code where HashMaps are used to store data.
Why not use the standard SSJS objects if they provide better
performance?
Why is it so much more inefficient to use the put
method instead of the SSJS "." notation to set the value of a
HashMap?
Looking at your code (you should put code there, not an image), you are comparing apples and oranges.
oo = {};
initializes the JS object once, in the second loop it overwrites the existing object.
hw = new HashMap();
generates a new object each time and throws the old one to the garbage collector. That will be slower.
Your conclusion: "Filling a Hashmap took xx longer" is not supported by your data. The code you wrote supports "Creating and discarding HashMaps in quick sequence is slower".
If you want to run a reasonable test you need to:
Create ONE object (that's the usual use case) and put "a lot of things" into it and to get real ideas: put 100,000 things in. Then read 100,000, then update 100,000 things.
The main reason for HashMaps: you can use them in Java too and you can't accidentally add functionality to them as you can with a JavaScript object. The dot vs. put conversion is due to the different type conversion mechanism.
Related
I was following example shown in this youtube video (https://www.youtube.com/watch?v=WU_D2qNnuGg&index=7&list=PLc_1PNcpnV5742XyF8z7xyL9OF8XJNYnv) which illustrates superiority of filtering methods in Revit API over usual iteration. But my code is significantly slower than the the iteration method :
filter method-0.16 secs
iteration method-0.06 secs
My code using filter method is :
import Autodesk.Revit.DB as DB
doc=__revit__.ActiveUIDocument.Document
uidoc=__revit__.ActiveUIDocument
height_param_id=DB.ElementId(DB.BuiltInParameter.WALL_USER_HEIGHT_PARAM)
height_param_prov=DB.ParameterValueProvider(height_param_id)
param_equality=DB.FilterNumericEquals() # equality class
height_value_rule=DB.FilterDoubleRule(height_param_prov,param_equality,10,1e-02)
param_filter=DB.ElementParameterFilter(height_value_rule)
# This program significantly slows down for the next line
walls=DB.FilteredElementCollector(doc)\
.WherePasses(param_filter)\
.ToElementIds()
uidoc.Selection.SetElementIds(walls)
For iteration following code was used.
from System.Collections.Generic import List
import Autodesk.Revit.DB as DB
doc=__revit__.ActiveUIDocument.Document
uidoc=__revit__.ActiveUIDocument
sheet_collector=DB.FilteredElementCollector(doc)\
.OfCategory(DB.BuiltInCategory\
.OST_Sheets)\
.WhereElementIsNotElementType()\
.ToElements()
walls=DB.FilteredElementCollector(doc)\
.OfCategory(DB.BuiltInCategory.OST_Walls)\
.WhereElementIsNotElementType()\
.ToElements()
tallwallsids=[]
for wall in walls:
heightp=wall.LookupParameter('Unconnected Height')
if heightp and heightp.AsDouble()==10:
tallwallsids.append(wall.Id)
uidoc.Selection.SetElementIds(List[DB.ElementId](tallwallsids))
This makes sense if you consider the amount of elements that the two methods have to consider. First method:
walls=DB.FilteredElementCollector(doc)\
.WherePasses(param_filter)\
.ToElementIds()
In this method you are asking the filter to consider ALL elements in the model. That's potentially a lot of elements to pass through the filter. That's opposed to:
walls=DB.FilteredElementCollector(doc)\
.OfCategory(DB.BuiltInCategory.OST_Walls)\
.WhereElementIsNotElementType()\
.ToElements()
In this method you use the QUICK filter OfCategory() and another WhereElementIsNotElementType() to narrow down the selection to only Wall instances. Even though you follow that through with a simple for loop which is the slow component here, its still FASTER than passing ALL elements in the model through the first filter.
You can optimize it by creating a filter like so:
walls=DB.FilteredElementCollector(doc)\
.OfCategory(DB.BuiltInCategory.OST_Walls)\
.WhereElementIsNotElementType()\
.WherePasses(param_filter)
.ToElements()
This would actually combine the quick category filter, element type filter, and slow parameter filter to potentially be an overall faster and easier to read solution.
Give it a go, and let me know if this makes sense.
Cheers!
What iteration method?
Nowadays, filtered element collectors are normally the only way to retrieve and iterate over Revit database elements.
The filtered element collector in itself is probably fast.
If you have a huge number of walls and your memory is limited, the call to ToElementIds may consume significant resources.
SetElementIds may also cost time.
Check out the extensive Revit API forum discussion on filtered element collector by pipe system types for more on this.
I suggest you provide a complete minimal reproducible sample case equipped with benchmarking code for each of those method calls to prove the performance degradation.
What is to implement a custom string type in f# for interning strings. i have to read large csv files into memory. Given most of the columns are categorical, values are repeating and it makes sense to create new string first time it is encountered and only refer to it on subsequent occurrences to save memory.
In c# I do this by creating a global intern pool (concurrent dict) and before setting a value, lookup the dictionary if it already exists. if it exists, just point to the string already in the dictionary. if not, add it to the dictionary and set the value to the string just added to dictionary.
New to f# and wondering what is the best way to do this in f#. will be using the new string type in records named tuples etc and it will have to work with concurrent processes.
Edit:
String.Intern uses the Intern Pool. My understanding is, it is not very efficient for large pools and is not garbage collected i.e. any/all interned strings will remain in intern pool for lifetime of the app. Imagine a an application where you read a file, perform some operations and write data. Using Intern Pool solution will probably work. Now imagine you have to do the same 100 times and the strings in each file have little in common. If the memory is allocated on heap, after processing each file, we can force garbage collector to clear unnecessary strings.
I should have mentioned I could not really figure out how to do the C# approach in F# (other than implementing a C# type and using it in F#)
Memorisation pattern is slightly different from what I am looking for? We are not caching calculated results - we are ensuring each string object is created no more than once and all subsequent creations of same string are just references to the original. Using a dictionary to do this is a one way and using String.Intern is other.
sorry if is am missing something obvious here.
I have a few things to say, so I'll post them as an answer.
First, I guess String.Intern works just as well in F# as in C#.
let x = "abc"
let y = StringBuilder("a").Append("bc").ToString()
printfn "1 : %A" (LanguagePrimitives.PhysicalEquality x y) // false
let y2 = String.Intern y
printfn "2 : %A" (LanguagePrimitives.PhysicalEquality x y2) // true
Second, are you using a dictionary in combination with String.Intern in your C# solution? If so, why not just do s = String.Intern(s); after the string is ready following input from file?
To create a type for use in your business domain to handle string deduplication in general is a very bad idea. You don't want your business domain polluted by that kind of low level stuff.
As for rolling your own. I did that some years ago, probably to avoid that problem you mentioned with the strings not being garbage collected, but I never tested if that actually was a problem.
It might be a good idea to use a dictionary (or something) for each column (or type of column) where the same values are likely to repeat in great numbers. (This is pretty much what you said already.)
It makes sense to only keep these dictionaries live while you read the information from file, and stuff it into internal data structures. You might be thinking that you need the dictionaries for subsequent reads, but I am not so sure about that.
The important thing is to deduplicate the great majority of strings, and not necessarily every single duplicate. Because of this you can greatly simplify the solution as indicated. You most probably have nothing to gain by overcomplicating your solution to squeeze out the last fraction of memory savings.
Releasing the dictionaries after the file is read and structures filled, will have the advantage of not holding on to strings when they are no longer really needed. And of course you save memory by not holding onto the dictionaries.
I see no need to handle concurrency issues in the implementation here. String.Intern must necessarily be immune to concurrency issues. If you roll your own with the design suggested, you would not use it concurrently. Each file being read would have its own set of dictionaries for its columns.
Is there a faster way to search data in JavaScript (specifically on V8 via node.js, but without c/c++ modules) than using the JavaScript Object?
This may be outdated but it suggests a new class is dynamically generated for every single property. Which made me wonder if a binary tree implementation might be faster, however this does not appear to be the case.
The binary tree implementation isn't well balanced so it might get better with balancing (only the first 26 values are roughly balanced by hand.)
Does anyone have an idea on why or how it might be improved? On another note: does the dynamic class notion mean there are actually ~260,000 properties (in the jsperf benchmark test of the second link) and subsequently chains of dynamic class definitions held in memory?
V8 uses the concepts of 'maps', which describe the layout of the data in an object.
These maps can be "fast maps" which specify a fixed offset from the start of the object at which a particular property can be found, or they can be "dictionary map", which use a hashtable to provide a lookup mechanism.
Each object has a pointer to the map that describes it.
Generally, objects start off with a fast map. When a property is added to an object with a fast map, the map is transitioned to a new one which describes the location of the new property within the object. The object is re-allocated with enough space for the new data item if necessary, and the object's map pointer is set to the new map.
The old map keeps a record of the transitions from it, including a pointer to the new map and a description of the property whose addition caused the map transition.
If another object which has the old map gets the same property added (which is very common, since objects of the same type tend to get used in the same way), that object will just use the new map - V8 doesn't create a new map in this case.
However, once the number of properties goes over a certain theshold (in fact, the current metric is to do with the storage space used, not the actual number of properties), the object is changed to use a dictionary map. At this point the object is re-written using a hashtable. In general, it won't undergo any more map transitions - any more properties that are added will just go in the hashtable.
Fast maps allow V8 to generate optimized code (using Crankshaft) where the offset of a property within an object is hard-coded into the machine code. This makes it very fast for cases where it can do this - it avoids the need for doing any lookup.
Obviously, the generated machine code is then dependent on the map - if the object's data layout changes, the code has to be discarded and re-optimized when necessary. V8 has a type profiling mechanism which collects information about what the types of various objects are during execution of unoptimized code. It doesn't trigger optimization of the code until certain stability constraints are met - one of these is that the maps of objects used in the function aren't changing frequently.
Here's a more detailed description of how this stuff works.
Here's a video where one of the lead developers of V8 describes stuff like map transitions and lots more.
For your particular test case, I would think that it goes through a few hundred map transitions while properties are being added in the preparation loop, then it will eventually transition to a dictionary based object. It certainly won't go through 260,000 of them.
Regarding your question about binary trees: a properly sized hashtable (with a sensible hash function and a significant number of objects in it) will always outperform a binary tree for a use-case where you're just searching, as your test code seems to do (all of the insertion is done in the setup phase).
I'm trying out the random number generation from the new library in C++11 for a simple dice class. I'm not really grasping what actually happens but the reference shows an easy example:
std::default_random_engine generator;
std::uniform_int_distribution<int> distribution(1,6);
int dice_roll = distribution(generator);
I read somewhere that with the "old" way you should only seed once (e.g. in the main function) in your application ideally. However I'd like an easily reusable dice class. So would it be okay to use this code block in a dice::roll() method although multiple dice objects are instantiated and destroyed multiple times in an application?
Currently I made the generator as a class member and the last two lines are in the dice:roll() methods. It looks okay but before I compute statistics I thought I'd ask here...
Think of instantiating a pseudo-random number generator (PRNG) as digging a well - it's the overhead you have to go through to be able to get access to water. Generating instances of a pseudo-random number is like dipping into the well. Most people wouldn't dig a new well every time they want a drink of water, why invoke the unnecessary overhead of multiple instantiations to get additional pseudo-random numbers?
Beyond the unnecessary overhead, there's a statistical risk. The underlying implementations of PRNGs are deterministic functions that update some internally maintained state to generate the next value. The functions are very carefully crafted to give a sequence of uncorrelated (but not independent!) values. However, if the state of two or more PRNGs is initialized identically via seeding, they will produce the exact same sequences. If the seeding is based on the clock (a common default), PRNGs initialized within the same tick of the clock will produce identical results. If your statistical results have independence as a requirement then you're hosed.
Unless you really know what you're doing and are trying to use correlation induction strategies for variance reduction, best practice is to use a single instantiation of a PRNG and keep going back to it for additional values.
Pseudo-code:
for each x in someArray {
// possibly add an element to someArray
}
I forget the name of the exception this throws in some languages.
I'm curious to know why some languages prohibit this use case, whereas other languages allow it. Are the allowing languages unsafe -- open to some pitfall? Or are the prohibiting languages simply being overly cautious, or perhaps lazy (they could have implemented the language to gracefully handle this case, but simply didn't bother).
Thanks!
What would you want the behavior to be?
list = [1,2,3,4]
foreach x in list:
print x
if x == 2: list.remove(1)
possible behaviors:
list is some linked-list type iterator, where deletions don't affect your current iterator:
[1,2,3,4]
list is some array, where your iterator iterates via pointer increment:
[1,2,4]
same as before, only the system tries to cache the iteration count
[1,2,4,<segfault>]
The problem is that different collections implementing this enumerable/sequence interface that allows for foreach-looping have different behaviors.
Depending on the language (or platform, as .Net), iteration may be implemented differently.
Typically a foreach creates an Iterator or Enumerator object on the array, which internally keeps its state about the iteration details. If you modify the array (by adding or deleting an element), the iterator state would be inconsistent in regard to the new state of the array.
Platforms such as .Net allow you to define your own enumerators which may not be susceptible to adding/removing elements of the underlying array.
A generic solution to the problem of adding/removing elements while iterating is to collect the elements in a new list/collection/array, and add/remove the collected elements after the enumeration has completed.
Suppose your array has 10 elements. You get to the 7th element, and decide there that you need to add a new element earlier in the array. Uh-oh! That element doesn't get iterated on! for each has the semantics, to me at least, of operating on each and every element of the array, once and only once.
Your pseudo example code would lead to an infinite loop. For each element you look at, you add one to the collection, hence if you have at least 1 element to start with, you will have i (iterative counter) + 1 elements.
Arrays are typically fixed in the number of elements. You get flexible sized widths through wrapped objects (such as List) that allow the flexibility to occur. I suspect that the language may have issues if the mechanism they used created a whole new array to allow for the edit.
Many compiled languages implement "for" loops with the assumption that the number of iterations will be calculated once at loop startup (or better yet, compile time). This means that if you change the value of the "to" variable inside the "for i = 1 to x" loop, it won't change the number of iterations. Doing this allows a legion of loop optimizations, which are very important in speeding up number-crunching applications.
If you don't like that semantics, the idea is that you should use the language's "while" construct instead.
Note that in this view of the world, C and C++ don't have proper "for" loops, just fancy "while" loops.
To implement the lists and enumerators to handle this, would mean a lot of overhead. This overhead would always be there, and it would only be useful in a vast miniority of the cases.
Also, any implementation that were chosen would not always make sense. Take for example the simple case of inserting an item in the list while enumerating it, would the new item always be included in the enumeration, always excluded, or should that depend on where in the list the item was added? If I insert the item at the current position, would that change the value of the Current property of the enumerator, and should it skip the currently current item which is then the next item?
This only happens within foreach blocks. Use a for loop with an index value and you'll be allowed to. Just make sure to iterate backwards so that you can delete items without causing issues.
From the top of my head there could be two scenarios of implementing iteration on a collection.
the iterator iterates over the collection for which it was created
the iterator iterates over a copy of the collection for which it was created
when changes are made to the collection on the fly, the first option should either update its iteration sequence (which could be very hard or even impossible to do reliably) or just deny the possibility (throw an exception). The last of which obviously is the safe option.
In the second option changes can be made upon the original collection without bothering the iteration sequence. But any adjustments will not be seen in the iteration, this might be confusing for users (leaky abstraction).
I could imagine languages/libraries implementing any of these possibilities with equal merit.