By default, image-gc-high-threshold and image-gc-low-threshold values are 90 and 80% respectively.
We want to change them to 80 and 70, How we can change the Kubernetes image garbage collection threshold values.
Thanks in advance.
Changing the garbage collection thresholds can be done using switches on the kubelet.
From the docs
--image-gc-high-threshold int32 The percent of disk usage after which image garbage collection is always run. (default 85)
--image-gc-low-threshold int32 The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. (default 80)
Related
I am reading two datasets of sizes 9.5GB (df1) and 715.1MB(df2) on disk.
I merge them on a key and then run a global aggregation on a resultant column
This triggers a shuffle and then the shuffled results are sort merged together on the reduce side (This happens internally)
I was trying to get the sort stage to spill some data on the disk and for that I progressively reduced the executor sizes.
As I reduced the executor size, the stage started to consume more time and less memory and minimized the spill. Numbers to look at in the images below are a. "duration" that is right underneath the WholeStageCodegen and, b. peak memory total in the "sort" box
config("spark.executor.instances","6").
config("spark.executor.memory","6G").
config("spark.executormemoryOverhead","2G").
config("spark.memory.offHeap.size","2G").
config("spark.executor.pyspark.memory","2G")
config("spark.executor.instances","6").
config("spark.executor.memory","4G").
config("spark.executormemoryOverhead","2G").
config("spark.memory.offHeap.size","2G").
config("spark.executor.pyspark.memory","2G")
config("spark.executor.instances","6").
config("spark.executor.memory","2G").
config("spark.executormemoryOverhead","2G").
config("spark.memory.offHeap.size","2G").
config("spark.executor.pyspark.memory","2G")
I have spark.sql.adaptive.enabled set to False. I have not touched the shuffle partitions. It remains the default 200 throughout
Questions:
As you can see increasing the executor size did two things: Reduced the memory footprint (peak memory total) and increased the duration. What is happening under the hood? How is this brought about?
I am seeing a duration mentioned right underneath the WholeStageCodegen. I used to think that is the runtime statistics of the entire JAVA wholestagecode generated for the stage. But, here a) There is just one duration given for the second data. There are no statistics like min, med, max as are there for the first data. Why is that so? Also, What is the difference between the sort time total mentioned inside the Sort box and the duration mentioned underneath the WholeStageCodegen?
From the Spark configuration docs, we understand the following about the spark.memory.fraction configuration parameter:
Fraction of (heap space - 300MB) used for execution and storage. The lower this is, the more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records. Leaving this at the default value is recommended.
The default value for this configuration parameter is 0.6 at the time of writing this question. This means that for an executor with, for example, 32GB of heap space and the default configurations we have:
300MB of reserved space (a hardcoded value on this line)
(32GB - 300MB) * 0.6 = 19481MB of shared memory for execution + storage
(32GB - 300MB) * 0.4 = 12987MB of user memory
This "user memory" is (according to the docs) used for the following:
The rest of the space (40%) is reserved for user data structures, internal metadata in Spark, and safeguarding against OOM errors in the case of sparse and unusually large records.
On an executor with 32GB of heap space, we're allocating 12,7GB of memory for this, which feels rather large!
Do these user data structures/internal metadata/safeguarding against OOM errors really need that much space? Are there some striking examples of user memory usage which illustrate the need of this big of a user memory region?
I did some research and imo its 0.6 not to ensure enough memory for user memory but to ensure that execution + storage can fit into old gen region of jvm
Here i found something interesting: Spark tuning
The tenured generation size is controlled by the JVM’s NewRatio
parameter, which defaults to 2, meaning that the tenured generation is
2 times the size of the new generation (the rest of the heap). So, by
default, the tenured generation occupies 2/3 or about 0.66 of the
heap. A value of 0.6 for spark.memory.fraction keeps storage and
execution memory within the old generation with room to spare. If
spark.memory.fraction is increased to, say, 0.8, then NewRatio may
have to increase to 6 or more.
So by default in OpenJvm this ratio is set to 2 so you have 0,66% for old-gen, they choose to use 0,6 to have small margin
I found that in version 1.6 this was changed to 0,75 and it was causing some issues, here is Jira ticket
In the description you will find sample code which is adding records to cache just to use whole memory reserved for exeution + storage. With storage + execution set to higher amount than old gen overhead for gc was really high and code which was executed on older version (with this setting equal to 0.6) was 6 time faster (40-50 sec vs 6 min)
There was discussion and community decided to roll it back to 0.6 in Spark 2.0, here is PR with changes
I think that if you want to increase performance a little bit, you can try to change it up to 0.66 but if you want to have more memory for execution+storageyou need to also adjust your jvm and change old/new ratio as well otherwise you may face performance issues
I am creating this new post because I am looking for a solution to measure an mongo engine object size in MB within a python program, before to add anything to this object and save it within the DB. My idea is simply to make sure that the object size does not exceed the Mongo Engine limit (which is about 16 MB) before to add anything else to one of its list field.
Do you have any advise that could interest me?
Thank in advance
NOTE: Found the root cause in application code using hazelcast which started to execute after 15 min, the code retrieved almost entire data, so the issue NOT in hazelcast, leaving the question here if anyone will see same side effect or wrong code.
What can cause heavy traffic between Hazelcast (v3.12.12, also tried 4.1.1) 2 nodes ?
It holds maps with lot of data, no new entries are added/removed within that time, only map values are updated.
Java 11, Memory usage 1.5GB out of 12GB, no full GCs identified.
Following JFR the high IO is from:
com.hazelcast.internal.networking.nio.NioThread.processTaskQueue()
Below chart of Network IO, 15 min after start traffic jumps from 15 to 60 MB. From application perspective nothing changed after these 15 min.
This smells garbage collection, you are most likely to be running into long gc pauses. Check your gc logs, which you can enable using verbose gc settings on all members. If there are back-to-back GCs then you should do various things:
increase the heap size
tune your gc, I'd look into G1 (with -XX:MaxGCPauseMillis set to a reasonable number) and/or ZGC.
I am designing a Redis datastore with ~3000 sorted set keys, each with 60 - 300 items each around 250 bytes in size.
used_memory_overhead = 1055498028 bytes and used_memory_dataset= 9681332 bytes. This ratio seems way too high. used_memory_dataset_perc is less than 1%. Memory usage is exceeding the max of 1.16G and causing keys to be evicted.
Do sorted sets really have 99% memory overhead? Will I have to just find another solution? I just want a list of values that is sorted by a field in the value.
Here's the output of MEMORY INFO . used_memory_dataset_perc just keeps decreasing until it's <1% and eventually the max memory is exceeded
# Memory
used_memory:399243696
used_memory_human:380.75M
used_memory_rss:493936640
used_memory_rss_human:471.05M
used_memory_peak:1249248448
used_memory_peak_human:1.16G
used_memory_peak_perc:31.96%
used_memory_overhead:390394038
used_memory_startup:4263448
used_memory_dataset:8849658
used_memory_dataset_perc:2.24%
allocator_allocated:399390096
allocator_active:477728768
allocator_resident:499613696
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:1248854016
maxmemory_human:1.16G
maxmemory_policy:volatile-lru
allocator_frag_ratio:1.20
allocator_frag_bytes:78338672
allocator_rss_ratio:1.05
allocator_rss_bytes:21884928
rss_overhead_ratio:0.99
rss_overhead_bytes:-5677056
mem_fragmentation_ratio:1.24
mem_fragmentation_bytes:94804256
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:385555150
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
In case it is relevant, I am using AWS Elasticache