Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
How do data processing engine like Spark, apache flink integrate structured, semi-structured and unstructured data together and affect computation?
General-purpose data processing engines like Flink or Spark let you define own data types and functions.
In case you have unstructured or semi-structured data, your data types can reflect these properties, e.g., by making some information optional or model it with flexible data structures (nested types, lists, maps, etc.). Your user-defined functions should be aware that some information might not always be present and know how to handle such cases.
So handling of semi-structured or unstructured data does not come for free. It must be explicitly specified. In fact, both systems put a focus on user-defined data and functions but have recently added APIs to ease the processing of structured data (Flink: Table API, Spark: DataFrames).
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
What's better to retrieve complex data from ArangoDB: A big query with all collection joins and graph traversal or multiple queries for each piece of data?
I think it depends on several aspects, e.g. the operation(s) you want to perform, scenario in which the querie(s) should be executed or if you favor performance over maintainability.
AQL provides the ability to write a single non-trivial query which might span through entire dataset and perform complex operation(s). Dissolving a big query into multiple smaller ones might improve maintainability and code readability, but on the other hand separate queries for each piece of data might have negative performance impact in the form of network latency associated with each request. One should also consider if the scenario allows to work with partial results returned from database while the other batch of queries is being processed.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Assume we have 100 gb of file. And my system is 60gb.How apache spark will handle this data?
We all know spark performs partitions on its own based on the cluster. But then when there is a reduced amount of memory I wanna know how spark handles it
In short: Spark does not require the full dataset to fit in memory at once. However, some operations may demand an entire partition of the dataset to fit in memory. Note that Spark allows you to control the number of partitions (and, consequently, the size of them).
See this topic for the details.
It is also worth to note that Java objects usually take more space than the raw data, so you may want to look at this.
Also i would recommend to look at Apache Spark : Memory management and Graceful degradation
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
In general, say you have a (<16mb) table in a database running on the same machine as your server. If you need to do lots of querying into this table (%100 reads), is it better to:
Get the entire table, and do all the searching/querying/ in the server code.
Make lots of queries into the local database.
If the database is local, can I take advantage of the dbms's highly-efficient internal data structures for querying, or is the delay such that it's faster to map the tables returned by the database into my own data structures.
Thanks.
This is going to depend heavily on what kind of searches you're doing.
If your data is all ID lookups, it's probably faster to have it in RAM.
If your data is all full scans (no indexes), it's probably faster to have it in RAM.
If your data uses indexes, it's probably faster to have it in the DB.
Of course, much of the appeal of a database is indexes and a common query interface, so you have to weigh how valuable those are versus raw speed.
There's no way to really answer this without knowing exactly the nature of the data and queries to be done on it. Over-the-wire time has its cost, as does BSON <-> native marshalling, but indexed searches can be O(log n) as opposed to a dumb O(n) (or worse) search over a simple in-memory data structure.
Have you tried benchmarking?
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
What would be a good or recommended way to model SVG DOM tree in Google's Realtime API? Specifically, stringify the SVG DOM tree and choose a collaborative string model or is there a better way? Thanks.
It depends on what you want to do with it. If all you want to do is to display something, without it being editable, then I would just store it is a blob. E.g., maybe just a static string.
If you want to be able to edit it, a collaborative string is problematic, as its hard to guarantee that the results of merging different collaborator's actions will result in well-formed XML.
Instead you could use custom objects to model the various nodes in the tree. You could do this either with a generic dom-like model where nodes have arbitrary attributes, or with specific classes different element types. I think the last would be the most powerful way to deal with it, and the nicest to work with, but also the most work to setup.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Can any one explain to me
what is Data Oriented programming?
Is Data oriented programming and functional programming the same?
How is Data Oriented programming different from Object Oriented programming?
Under what circumstances do we choose Data Oriented programming languages over Object Oriented programming languages?
First I want to say, that Data-oriented design and Data-driven programming is not the same!
In object-oriented programming you are focusing on a single object (class - its methods, members, etc.). In data-oriented design you are thinking how data is touched and processed.
You just have a box that processes your input data to your output data (the ideal input data is the same as output).
All this was created to write high-performance applications. You are working on homogeneous, linear data - all to take the full advantage of CPU cache (both instruction and data).
Whenever you can, try to avoid hierarchical structures (use arrays instead), try to write functions that works on multiple data and use hot and cold structure splitting.
int Foo(int* input_data, int count)
{
// do something with your data
}
As the name suggests, DOP intended for the development of data driven applications. It is not same as OOP. For further reference, go through the following links;
http://www.rti.com/whitepapers/Data-Oriented_Architecture.pdf
Alternate link here as the above one might not be working.
http://en.wikipedia.org/wiki/List_of_programming_languages_by_category#Data-oriented_languages
Data oriented programming is simply a programming language with database, you can create tables and queries, and program to manipulate the stored data on it, example of data oriented language are SQL, DBase and Visual Foxpro.