Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
what is "spark.executor.memoryOverhead" and "spark.memory.fraction"?
what is the default properties
spark.memory.fraction parameter can be used to separately understand memory available for storage and memory available for execution. If you are caching too many objects in memory then you will need more of storage (spark.memory.fraction can be 0.5/0.6). However, if you are using memory for largely execution purposes then you need memory to be available for execution (spark.memory.fraction can be 0.2/0.3).
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
General performance advice for Rust is to try to avoid placing things on the heap if possible.
An issue I am having is that I do not know where/when the size limit of a function stack will be reached, until my program panics unpredictably at runtime.
Two examples are:
Parsing deeply nested structs from JSON using Serde.
Creating many futures inside a function.
Questions:
Can I avoid this by detecting it at compile time?
How can I know what the limit of the stack is whilst I am writing code? Do others just know the exact size of their variables?
Why do people advise to try to avoid the heap?
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
Can we process 1tb of data using spark with 2 executors having 5 gb of memory each.if not how many executors are required, Assuming we don't have any time constraints.
This is very difficult question without looking at your data and code.
If you're ingesting raw files of 1TB without any caching then it MAY be possible with 5GB memory, but it will take very very long time as the parallelization is limited with only 2 executors unless you have multiple cores. Also, it depends wther you're asking for compressed 1GB or raw text files.
I hope this helps.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I would like to plot the CPU and memory usage of an application on linux vs time. What is the best way to do this?
Would greping these values out from top every 0.1s and writing them into some file work - or is there a better and easier way?
There is an easier way. All of the information displayed in top can be found in /proc/<pid>/, most of it in /proc/<pid>/stat. man proc describes the content of these files.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
How to create threads and assign tasks to them? Is there any way to do it, like usage of
thread.start_new_thread ( function, args[, kwargs] )
in Python?
thanks in advance
Haskell threads can be spawned using forkIO.
I recommend also reading the GHC concurrency guide, since it has all the relevant pointers.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
What is thread locality? I've been doing some research on a particular topic and thread locality has come a up a few times. I haven't been able to find a clear definition of what's meant though.
Thread locality refers to thread local storage. Data marked as threadlocal/threadstatic, is available as per thread copy to each thread accessing it. Any modification in data in one thread is isolated to any other thread.
Read this for details.