i want to implement a new garbage collector (mark & sweep), but there're too many dup/free bits and pieces all over the code。I think i should turn off first then think about other things. This is my first time working in this field and unfortunately it's important.what should i do?
I don't have a clear understanding of this, because I didn't read all the source code about the dup/free part
Related
Our plugin maintains some instance parameter values across many elements, including those in groups.
Occasionally the end users will introduce data that activates an unused Category,
so we have to update the document parameter bindings, to include those categories. However, when we call
doc.ParameterBindings.ReInsert()
our existing parameter values inside groups are lost, because our VariesAcrossGroups flag is toggled back to false?
How did Revit intend this to work - are we supposed to use this in a different way, to not trigger this problem?
ReInsert() expects a base Definition argument, and would usualy get an ExternalDefinition supplied.
To learn, I instead tried to scan through the definition-keys of existing bindings and match those.
This way, I got the document's InternalDefinition, and tried calling Reinsert with that instead
(my hope was, that since its existing InternalDefinition DID include VariesAcrossGroups=true, this would help). Alas, Reinsert doesn't seem to care.
The problem, as you might guess, is that after VariesAcrossGroups=False, a lot of my instance parameters have collapsed into each other, so they all hold identical values. Given that they are IDs, this is less than ideal.
My current (intended) solution is to instead grab a backup of all existing parameter values BEFORE I update the bindings, then after the binding-update and variesAcrossGroups back to true, then inspect all values and re-assign all parameter-values that have been broken. But as you may surmise, this is less than ideal - it will be horribly slow for the users to use our plugin, and frankly it seems like something the revitAPI should take care of, not the plugin developer.
Are we using this the wrong way?
One approach I have considered, is to bind every possibly category I can think of, up front and once only. But I'm not sure that is possible. Categories in themselves are also difficult to work with, as you can only create them indirectly, by using your Project-Document as a factory (i.e. you cannot create a category yourself, you can only indirectly ask the Document to - maybe! - create a category for you, that you request). Because of this, I don't think you can bind for all categories up front - some categories only become available in the document, AFTER you have included a given family/type in your project.
To sum it up: First, I
doc.ParameterBindings.ReInsert()
my binding, with the updated categories. Then, I call
InternalDefinition.SetAllowVaryBetweenGroups()
(after having determined IDEF.VariesAcrossGroups has reverted back to false.)
I am interested to hear the best way to do this, without destroying the client's existing data.
Thank you very much in advance.
(I'm not sure I will accept my own answer).
My answer is just, that you can survive-circumvent this problem,
by scanning the entire revit database for your existing parmater values, before you update the document bindings.
Afterwards, you reset VariesAcrossGroups back to its lost value.
Then, you iterate through your collected parameters, and verify which ones have lost their original value, and reset them back to their intended value.
One trick that speeds this up a bit, is that you can check Element.GroupId <> -1. That is, those elements that are group members.
You only need to track elements which are group members, as it's precisely those that are affected by this Revit bug.
A further tip is, that you should not only watch out for parameter-values that have lost their original value. You must also watch out for parameter-values that have accidentally GOTTEN a value, but which should be left un-set.
I just use FilteredElementCollector with WhereElementIsNotElementType().
Performance-wise, it is of course horrible to do all this,
but given how Revit behaves, I see no other solution if you have to ship to your clients.
I am new to F# so maybe the solution can be clear to someone, but I can not find it.
Imagine a game world of world chunks (similar to Minecraft), but for more players.
In theory language like C++, java, or C# can modify multiple chunks of world at same time. Two or more players try to place or remove block in different chunks and all these actions can change the state of the world without affecting each other as long as no more than one action in each chunk is happening. Serializing will only happen when multiple players in one chunk perform the modification.
My understanding of F# is I need to serialize these actions on global level and no two actions can happen in same time in entire world, because the update function need actual world state update params(like add/remove blok) and return new world state.
For that example the world state contains chunk list.
Is there a way to do world update in parallel?
Can the world state be stored differently to allow update to multiple chunks at same time?
It sounds like you need to ensure that each chunk has one action run at a time. You can protect pieces of state by storing them inside mailbox processors (often referred to as just "agents"). You can send agents several messages from several threads. They will be queued and processed one at a time.
There is a detailed discussion of this here: https://fsharpforfunandprofit.com/posts/concurrency-actor-model/
Firstly, I don't this really adds any technical detail to a previous answer, so you if you like their solution you should go ahead and mark that as the answer. However, I hope this gives some extra context...
Underlying your problem is the question of how consistent do you require the state of your world to be in order to make decisions about modifying chunks.
Consider a world where I have two chunks, let's call them A and B. Consider the use case where I want to add or remove a block from chunk A. The all important question is:
Do I need to know about the blocks in chunk B in order to validate, and then perform the addition / removal of a block from chunk A.
For example, if I only have finite number of blocks in my world, I may well need this information to validate that I can actually add a block without going over my limit. The key here is that my "consistency boundary" is my entire world - in order to perform the addition of a new block to chunk A I need consistent information about everythign in my world. It is no good if halfway through my decision making another thread jumps in and adds a block to chunk B. If this is a requirement then you have no option - even in the C#/C++ case - you need to lock down access to your world so only one such action can be performed at anyone time.
From the way you phrase the question, I suspect this is not the case. In which case we need to examine exactly what your consistency requirements are. A weaker requirement is that if I'm adding blocks to chunk A, I at least have to have consistent information about the number (and position) of blocks in chunk A. In the C#/C++ case this would mean having to put locks around accessing to individual "chunk data", but not the whole world.
A simple way of modelling this in F# would be (using the suggestion in this answer):
open FSharp.Core
type ChunkMessage =
AddBlock
| RemoveBlock
type MyWorld =
{
Blocks : List<MailboxProcessor<ChunkMessage>>
}
Note that MyWorld is a mutable, but each MailboxProcessor encapsulates state which can only change through processing one message at a time.
The implementation of Blocks doesn't have to be a list of MailboxProcessor's, you could use a thread-safe collection of objects for which you had thread-safe methods on, but the use of them here as suggested by The Quick Brown Fox leads to a particularly nice programming model.
I'm now facing a problem and I'm not sure what the right solution is. I'll try to explain it, and I hope someone has some good solutions for me:
I have two big data arrays. One that I'm browsing, with something between 50^3 and 150^3 data samples (usually between 50 and 100, rare worst case scenario 150).
For every sample, I want to make a query on another structure that is around the same size (so huge number of total combinations, I can't explore them all).
The query can't be predicted exactly but usually, it is something like :
structure has fields A B C D E F G (EDIT : in total, it's something like 10 to 20 int fields).
query is something like :
10 < A < 20 and B > 100 and D > 200.
Yes, it's really close to SQL.
I thought to put this in a database, but actually it would be a standalone database, and I can work in RAM to make it even faster (speed is an essential criteria).
I thought to try something using GPGPU but it seems it's a terrible idea and despite search can be parallel, it does't seem to be a good idea, searching an unpredictable number of results isn't a good application (if someone can tell me if my understanding has been right it would help me confirm that I should forgive this solution).
EDIT : the nubmer of results is unpredictable because of the query nature, but the it is quite low, since the purpose is to find a low number of well suited combinations
Then since I could use a DB, why not make a RAM B-Tree? it seems close to the solution, but is it? If it is, how should I build my indexes? Can I really do multidimensional indexes, since multidimensional search will always exist? probably UB-Tree or R-tree could do the job (but in my second data sample, I could have some duplicates, so doesn't it make the R-TREE non applicable?).
The thing is, I'm not sure I understand properly all those right now, so if one of you knows trees (and gpgpu, and even solutions I didn't think to), perhaps you could let me know which solution I should explore, learn, and implement?
GPGPU is not a suitable choice due to the fact that you are limited by their capacity and since you are not telling us the data size of these samples I am assuming that a titan x tier card will not suffice. If you could go really wild, TESLA or FirePro, then it is actually worth it since you mentioned that speed really matters. But I am going to speculate that these things are out of your budget, and considering that you have to learn CUDA or OpenCL to make something that will generally be a pain to port here and there, my take is "No".
You mentioned that you have an unpredictable number of results and this is a bad thing. You should develop a formula that will "somewhat" calculate the amount of space which will be needed otherwise it will be disappointing to have your program work on something for quite some time only to get a capacity error/crash. On the other hand, if the RAM capacity is not sufficient, you could work "database style" fetching data from storage when needed(and this is quite bothersome to implement due to scheduling implementations etc).
If you do have the time to go bespoke, here is a helpfull link. Remember, you are going to stumble a lot, but when you make it you will have learnt a tone of stuff:
https://www.quora.com/What-are-some-fast-similarity-search-algorithms-and-data-structures-for-high-dimensional-vectors
In my opinion, an in memory database is the easiest and at the same time most reliable thing to do without compromising on speed. Which one to implement is on you. I think MemSQL is a good one.
So I'm essentially trying to implement an AIR Native Extension that does the physics simulation in C with interfaces through Actionscript.
I've gone through quite a few iterations which I'll list below for interest sake and I'm at what I think could be my final attempt at getting this working in a more performant way.
Ultimately I'm looking for help in how I should be setting up a threading environment for running the simulation of Box2D on a separate thread and then polling for state in AS3.
Methods:
Brute Force:
In this method I simply call into C from AS3 and tell it to create a world and pass it some boxes to add to this world. Every frame in AS3, I call into C to tell the world to Step, then loop through all the bodies in the World, get their position and rotation, convert them to actionscript objects and put them in an actionscript array and then send that back to AS3. Once there I loop through the returning array and assign those position and rotation values to my sprites so they visually update.
The results are actually quite decent with about 116 boxes being added before the framerate suffers. This is compared to 30 boxes in a pure AS3 implementation. Note that these stats are in Debug mode. In release mode, they both make it to about 120 boxes. There is little difference between the AS3 implementation and the Native Extension implementation.
ByteArray Sharing
In order to improve performance I decided it would be a good idea to try and limit the amount of data being marshalled across C and AS3. ANE's support sharing a byte array's memory space and so I would send the ByteArray created in AS3 to C and have C simply update the ByteArray. This saves us from having to construct AS3 objects in C and pass them back. Every frame, AS3 simply needs to iterate through it's ByteArray and see what C has written into it and then assign those values to the sprites to set the visual state.
The results here are sadly about the same. Improvements are only marginal.
Direct Object Setting From C
Another thing ANE's are capable of is setting the property of an object that lives in AS3. In this sense I aimed to eliminate the overhead of passing back data to AS3, the looping through the bodies to collect data in C and the looping through in AS3 to assign the values. I directly modified the Box2D code so that when it's values were changed it would write the new x, y, rotation values directly on the corresponding Sprite.
The results are amazing at very low amounts of objects since the call to set these properties is well under a millisecond. The problem is that this scales linearly and around 90 or so objects, the overhead is too severe and things start to slow down.
Threading
At this point I was a bit stumped. There's overhead in marshalling data, there's a cost in C for iterating and constructing the data to return and there's a cost in AS3 for iterating to assign values to the sprites.
Obviously there needs to be a trade-off so my current solution is the best I can come up with for now.
On the AS3 side you call into C to create your world, call in to add a box to that world, and call in to tell C you want a refresh of your data. When boxes are created in AS3 they get a unique id and they are stored in a dictionary with the key being the id.
On the C side, the world is created and a new pthread is spawned to do the Step. Essentially simulating the world on another thread. After it steps, it assembles all the data and writes it into a double array. Then it does so again and again and again. It just simulates forever basically on it's own thread.
When we call in to C to add a new box, I need to create a new box and add it to that world. Since the world is Stepping this could cause problems which means I need to use mutexes I'm pretty sure.
Same thing when we call to get the values refreshed in AIR, I'll want to do a memcpy from the array of doubles into my AS3 bytearray and then loop through the bytearray to set the values on the visual.
The mutexes were giving me trouble so I basically implemented my own which you can see below... and laugh at :)
However it does work, just not as fast as I would like it too. Around 90 we slow down again.
Anyone have any thoughts or pointers? It'd be greatly appreciated!
C Code
The parser was acting up so i've pasted it here:
http://pastebin.com/eBQGuGJX
AS3 Code
Same thing with the parser. I've only included the relevant method dealing with every frame in AS3.
http://pastebin.com/R1Qs2Tyt
I had forgotten I had this question. Fortunately I have figured it out.
The idea of using mutexes etc was over engineered in the first place and unnecessary.
Since we're running in Flash, everything runs in the main thread. Which means for each "frame" flash will natively handle any media, then our client code which we have written, then actually render to the screen and finally do any garbage collection if necessary.
I don't actually need to have the physics sim simulating forever, I simply need to have it be one step ahead of my client code.
So what happens now is when the Client calls into the ANE to setup the world, it creates a new thread that simulates the world and returns immediately back to Flash. Flash will continue to do its work of executing the rest of the client code and then rendering and then GC.
Then on each frame in Flash we can simply call into the ANE to retrieve the results. In the case that the Simulation thread wasn't finished we wait via a join, extract the values and return them to Flash. Making sure to spawn another thread for the next step before returning of course.
In this way we are maximizing our efficiency since the simulation is happening while Flash is busy doing other things we don't have control over (like rendering and GC).
The good news is that performance almost doubles with this approach. Going from approx 90 boxes in a synchronous pure AS3 implementation to approx 170 boxes in a threaded ANE approach.
The bottleneck eventually becomes the iteration through the data coming back from the ANE and assigning those values to the Display Objects.
I hope this helps someone else who was looking for something similar. I'll be giving a talk about it at FITC Toronto at the end of April so there may be more information and material I can post then.
http://www.fitc.ca/events/presentations/presentation.cfm?event=124&presentation_id=1973
I need to create feature which will iterate through all subsites of site collection and add some sample content to each of them (sample content=new pages,images,documents,possibly some lists). There is solution that uses recursion to loop through sites (it can be found here).
So i'm asking:
Did any of you implemented something similar, maybe in different way?
What was the performance of such iteration?
I can't really say how many subsites can be present in site collection, since this feature should be used in more projects. But i guess the number of subsites shouldn't excess 100 or such.
That is a very common method of performing looping of sites and web parts.
Speed with 100 sub-sites will not be a problem. It is really quick, if you do alot of processing that can slow it down.
The one thing to watch is the disposal of items if you leave the open or don't close you can suck all the memory from the WFE, especially if alot of users reload the page at the same time.
See http://www.sharepointnutsandbolts.com/2008/06/disposing-sharepoint-objects-what-they.html and http://blogs.msdn.com/rogerla/archive/2008/02/12/sharepoint-2007-and-wss-3-0-dispose-patterns-by-example.aspx for examples they also have the disposal tool at http://blogs.msdn.com/sharepoint/archive/2008/11/12/announcing-spdisposecheck-tool-for-sharepoint-developers.aspx
You can look at the answer I gave to this question. I think I've used the correct dispose patterns for that piece of code.
If you need to do a read-only loop through a big structure, here's a cool, yet a bit untraditional, way of doing it:
http://hristopavlov.wordpress.com/2008/10/20/a-very-fast-method-to-get-the-site-collections-web-structure/