I'm new to this forum, and the reason I join is because out of curiosity I'm wondering how fast inno setup compression in other people's computer.
My speed while compressing using lzma2/ultra 64 is around 1.5 - 2.0 MB/s while I got the same speed when I'm just storing folder with different kind of file totaling in 150 GB (no compression / compresion = none).
My question: If I'm upgrading to (lets say) Intel Core i9 9900K, does it really increase the compression speed ?
My spec: 16 GB Ram, Intel i5-4690 with Windows 10 Pro
Thank you
Related
My task in spark uses images data for prediction I am working on a spark cluster standalone but I have an issue utilizing all the available memory capacity as here all available memory is 2.7 GB (coming from a memory executor that is configured 5 GB *0.6 *0.9= 2.7 it's okay ) but the usage memory is only 342 MB after that value my spark session being crashed and I did not know why this specific value!
I test my application on local and on a standalone cluster mode in addition whatever the memory executor configured value the limit of memory value for execution will be 342 MB. and here as shown my data size of 290691 KB led to the crash of my spark session and it works fine if I decrease the number of images
as follows screenshot issue:
This output error crashed with a data size of 290691 KB
Here my spark UI Storage Memory did not exceed 342 MB
so is there any advice or what is the correct spark configuration?
It's a warning, initially.
The general gist here is that you need to repartition to get more, but smaller size partitions, so as to get more parallelism and higher throughput. You can find many such issues out there on the Internet.
I am using pyspark / spark sql for performing very simple tasks. Data size is very less, highest being 215 MB. 90% of the data sources sizes are less than 15 MB. We do filtering, crunching and data aggregations and resultant data is also less than 5 MB for 90% of data. Only 2 data results are 120 MB and 260 MB.
Main hot-spot is coalesce(1) operation as we have requirement to produce only one file. I can understand 120 MB and 260 MB gziped file generation and writing taking time. But generation and writing less than 5MB file should be fast. When I monitor job I can see lot of time is taken by coalesce and save data file. I am clueless why it should take 60-70 secs for generating and writing 2-3 MB file.
Configuration:
I have achieved some performance gain with fat executors of 3 vcores per executor. I am using 1 master 3 worker cluster with 4 core node.
Regards
Manish Zope
Is there a relation between Cassandra's configuration parameters(given below with current values), Datastax's C++ driver configuration parameters(given below with current values) and the node's hardware specifications(no. of processors, RAM, no. of disks etc.)
Cassandra's Configuration Parameters(in YAML)
concurrent_reads set as 16
concurrent_writes set as 256
native_transport_max_threads set as 256
native_transport_max_frame_size_in_mb set as 512
Datastax's C++ Driver Configuration Parameters
cass_cluster_set_num_threads_io set as 10
cass_cluster_set_core_connections_per_host set as 1
cass_cluster_set_max_connections_per_host set as 20
cass_cluster_set_max_requests_per_flush set as 10000
Node's specs
No. of processors: 32
RAM: >150 GB
No. of hard disks: 1
Cassandra's Version: 3.11.2
Datastax C++ driver version: 2.7
RHEL version: 6.5
I have a cluster of 2 nodes and I've been getting dismal throughput(12000 ops/second). 1 operation = read + write(I can't use row cache). Is there any parameter which should've been set higher/lower(considering the nodes' specs)?
Please also note that my read+write application is multi-threaded(10
threads). Also, I'm doing asynchronous read+ asynchronous write(using future).
Replication factor is 2, both nodes are in the same DC, consistency
level for both read and write is also 2.
Some of the configuration properties in Cassandra are computed from available CPU cores and drives.
concurrent_reads = 16 * (number of drives)
concurrent_writes = 8 * (CPU cores)
It looks like you've done that, although I would question whether or not your 32 CPUs are all physical cores, or hyper-threaded.
I have a cluster of 2 nodes and I've been getting dismal throughput(12000 ops/second).
Just my opinion, but I think 12k ops/sec is pretty good. Actually REALLY good for a two node cluster. Cassandra scales horizontally, and linearly at that. So the solution here is an easy one...add more nodes.
What is your target operations per second? Right now, you're proving that you can get 6k ops/second per node. Which means, if you add another, the cluster should support 18K/sec. If you go to six nodes, you should be able to support 36k/sec. Basically, figure out your target, and do the math.
One thing you might consider, is to try ScyllaDB. Scylla is a drop-in replacement for Cassandra, which trumpets the ability to hit very high throughput requirements. The drawback, is that I think Scylla is only Cassandra 2.1 or 2.2 compatible ATM. But it might be worth a try based on what you're trying to do.
Does anybody have any tips when moving Spark execution from a few large nodes to many, smaller nodes?
I am running a system with 4 executors, each executor has 24Gb of ram and 12 cores. If I try to scale that out to 12 executors, 4 cores each and 8 Gb of ram (Same total RAM, same total cores, just distributed differently) I run into out of memory errors:
Container killed by YARN for exceeding memory limits. 8.8 GB of 8.8 GB physical memory used.
I have increased the number partitions by a factor of 3 to create more (yet smaller) partitions, but this didn't help.
Does anybody have any tips & tricks when trying to scale spark horizontally?
This is a pretty broad question, executor sizing in Spark is a very complicated kind of black magic, and the rules of thumb which were correct in 2015 for example are obsolete now, as will whatever I say be obsolete in 6 months with the next release of Spark. A lot comes down to exactly what you are doing and avoiding key skew in your data.
This is a good place to start to learn and develop your own understanding:
https://spark.apache.org/docs/latest/tuning.html
There are also a multitude of presentations on Slideshare about tuning Spark, try and read / watch the most recent ones. Anything older than 18 months be sceptical of, and anything older than 2 years just ignore.
I will make the assumption that you are using at least Spark 2.x.
The error you're encountering is indeed because of poor executor sizing. What is happening is that your executors are attempting to do too much at once, and running themselves into the ground as they run out of memory.
All other things being equal these are the current rules of thumb as I apply them:
The short version
3 - 4 virtual (hyperthreaded) cores and 29GB of RAM is a reasonable default executor size (I will explain why later). If you know nothing else, partition your data well and use that.
You should normally aim for a data partition size (in memory) on the order of ~100MB to ~3GB
The formulae I apply
Executor memory = number of executor cores * partition size * 1.3 (safety factor)
Partition size = size on disk of data / number of partitions * deser ratio
The deserialisation ratio is the ratio between the size of the data on disk and the size of data in memory. The Java memory representation of the same data tends to be a decent bit larger than on disk.
You also need to account for whether your data is compressed, many common formats like Parquet and ORC use compression like gzip or snappy.
For snappy compressed text data (very easily compressed), I use ~10X - 100X.
For snappy compressed data with a mix of text, floats, dates etc I see between 3X and 15X typically.
number of executor cores = 3 to 4
Executor cores totally depends on how compute vs memory intensive your calculation is. Experiment and see what is best for your use case. I have never seen anyone informed on Spark advocate more than 6 cores.
Spark is smart enough to take advantage of data locality, so the larger your executor, the better chance that your data is PROCESS_LOCAL
More data locality is good, up to a point.
When a JVM gets too large > 50GB, it begins to operate outside what it was originally designed to do, and depending on your garbage collection algorithm, you may begin to see degraded performance and high GC time.
https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html
There also happens to be a performance trick in Java that if your JVM is smaller than 32GB, you can use 32 bit compressed pointers rather than 64 bit pointers, which saves space and reduces cache pressure.
https://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/
It also so happens that YARN adds 7% or 384MB of RAM (whichever is larger) to your executor size for overhead / safety factor, which is where 29GB rule of thumb comes from: 29GB + 7% ~= 32GB
You mentioned that you are using 12 core, 24GB RAM executors. This sends up a red flags for me.
Why?
Because every "core" in an executor is assigned one "task" at time. A task is equivalent to the work required to compute the transformation of one partition from "stage" A to "stage" B.
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-taskscheduler-tasks.html
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-DAGScheduler-Stage.html
If your executor has 12 cores, then it is going to try and do 12 tasks simulatenously with a 24GB memory budget. 24GB / 12 cores = 2GB per core. If your partitions are greater than 2GB, you will get an out of memory error. If the particular transformation doubles the size of the input (even intermediately), then you need to account for that as well.
In Spark summit 2013 one of the yahoo presentation had this formula mentioned:
partitions needed = total data size/(memory size/number of cores)
Assuming a 64Gb memory host with 16 cores of CPU.
The presentation mentioned that to process 3Tb of data, the number of partitions needed is 46080. I am having hard time getting to the same result. Please explain the calculation, how the number 46080 came?
Looking at the presentation (available here), the information available is:
64Gb memory host
16 core cpu
Compression rato 30:1, 2 times overhead
Your formula should use the uncompressed data size when calculating, therefore, in this case you need to first uncompress it.
Data size = 3Tb * 30 * 2 = 180Tb = 184320Gb
Running it through the formula you get:
184320Gb/(64Gb/16) = 46080 partitions