Wildfly 18 eviction tag is not parsing giving Failed to parse configuration error.
this is coming when i upgrade Wildfly 11 to 18. In wildfly 11 (infinispan 4) its working fine
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="security" default-cache="auth-cache">
<local-cache name="auth-cache">
<locking acquire-timeout="${infinispan.cache-container.security.auth-cache.locking.acquire-timeout}"/>
<eviction strategy="LRU" max-entries="${infinispan.cache-container.security.auth-cache.eviction.max-entries}"/>
<expiration max-idle="-1"/>
</local-cache>
</cache-container>
</subsystem>
In Wildfly 18 having below section (NOT WORKING)
<subsystem xmlns="urn:jboss:domain:infinispan:9.0">
<cache-container name="security" default-cache="auth-cache">
<local-cache name="auth-cache">
<locking acquire-timeout="${infinispan.cache-container.security.auth-cache.locking.acquire-timeout}"/>
<eviction strategy="LRU" max-entries="${infinispan.cache-container.security.auth-cache.eviction.max-entries}"/>
<expiration max-idle="-1"/>
</local-cache>
</cache-container>
</subsystem>
Its giving ^^^^ 'eviction' isn't an allowed element here.infinispan:9.4 its says Eviction is configured by adding the but even that gives unrecognized tag memory.
how to add eviction strategy=LRU or replacement to strategy:"LRU"=?
According to the docs in infinispan 9.0 eviction is configured by adding the <memory/> element to your <*-cache/> configuration sections. Eviction is handled by Caffeine utilizing the TinyLFU algorithm with an additional admission window. This was chosen as provides high hit rate while also requiring low memory overhead. This provides a better hit ratio than LRU while also requiring less memory than LIRS.
In general there are two types:
COUNT (This type of eviction will remove entries based on how many there are in the cache. Once the count of entries has grown larger than the size then an entry will be removed to make room.
MEMORY - This type of eviction will estimate how much each entry will take up in memory and will remove an entry when the total size of all entries is larger than the configured size. This type only works with primitive wrapper, String and byte[] types, thus if custom types are desired you must enable storeAsBinary. Also MEMORY based eviction only works with LRU policy.
So I would think you define it like that:
<cache-container name="security" default-cache="auth-cache">
<local-cache name="auth-cache">
<...your other options...>
<object-memory/>
</local-cache>
</cache-container>
OR
<binary-memory eviction-type="MEMORY/COUNT"/>
OR
off-heap-memory eviction-type="MEMORY/COUNT"/>
AND you can always specify the size:
size="${infinispan.cache-container.security.auth-cache.eviction.max-entries}"
Storeage types:
object-memory (Stores entries as objects in the Java heap. This is the default storage type. Storage type supports COUNT only so you do not need to explicitly set the eviction type. Policy=TinyLFU)
binary-memory (Stores entries as bytes[] in the Java heap. Eviction Type: COUNT OR MEMORY. Policy=TinyLFU)
off-heap-memory (Stores entries as bytes[] in native memory outside the Java. Eviction Type: COUNT OR MEMORY. Policy=LRU)
Lonzak's reponse is correct.
Additionally, you can just use your "urn:jboss:domain:infinispan:4.0" configuration from WildFly 9 in WildFly 19. WildFly will automatically update the configuration to its equivalent in the current schema version.
Related
From the Spark configuration docs, we understand the following about the spark.memory.fraction configuration parameter:
Fraction of (heap space - 300MB) used for execution and storage. The lower this is, the more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records. Leaving this at the default value is recommended.
The default value for this configuration parameter is 0.6 at the time of writing this question. This means that for an executor with, for example, 32GB of heap space and the default configurations we have:
300MB of reserved space (a hardcoded value on this line)
(32GB - 300MB) * 0.6 = 19481MB of shared memory for execution + storage
(32GB - 300MB) * 0.4 = 12987MB of user memory
This "user memory" is (according to the docs) used for the following:
The rest of the space (40%) is reserved for user data structures, internal metadata in Spark, and safeguarding against OOM errors in the case of sparse and unusually large records.
On an executor with 32GB of heap space, we're allocating 12,7GB of memory for this, which feels rather large!
Do these user data structures/internal metadata/safeguarding against OOM errors really need that much space? Are there some striking examples of user memory usage which illustrate the need of this big of a user memory region?
I did some research and imo its 0.6 not to ensure enough memory for user memory but to ensure that execution + storage can fit into old gen region of jvm
Here i found something interesting: Spark tuning
The tenured generation size is controlled by the JVM’s NewRatio
parameter, which defaults to 2, meaning that the tenured generation is
2 times the size of the new generation (the rest of the heap). So, by
default, the tenured generation occupies 2/3 or about 0.66 of the
heap. A value of 0.6 for spark.memory.fraction keeps storage and
execution memory within the old generation with room to spare. If
spark.memory.fraction is increased to, say, 0.8, then NewRatio may
have to increase to 6 or more.
So by default in OpenJvm this ratio is set to 2 so you have 0,66% for old-gen, they choose to use 0,6 to have small margin
I found that in version 1.6 this was changed to 0,75 and it was causing some issues, here is Jira ticket
In the description you will find sample code which is adding records to cache just to use whole memory reserved for exeution + storage. With storage + execution set to higher amount than old gen overhead for gc was really high and code which was executed on older version (with this setting equal to 0.6) was 6 time faster (40-50 sec vs 6 min)
There was discussion and community decided to roll it back to 0.6 in Spark 2.0, here is PR with changes
I think that if you want to increase performance a little bit, you can try to change it up to 0.66 but if you want to have more memory for execution+storageyou need to also adjust your jvm and change old/new ratio as well otherwise you may face performance issues
NOTE: Found the root cause in application code using hazelcast which started to execute after 15 min, the code retrieved almost entire data, so the issue NOT in hazelcast, leaving the question here if anyone will see same side effect or wrong code.
What can cause heavy traffic between Hazelcast (v3.12.12, also tried 4.1.1) 2 nodes ?
It holds maps with lot of data, no new entries are added/removed within that time, only map values are updated.
Java 11, Memory usage 1.5GB out of 12GB, no full GCs identified.
Following JFR the high IO is from:
com.hazelcast.internal.networking.nio.NioThread.processTaskQueue()
Below chart of Network IO, 15 min after start traffic jumps from 15 to 60 MB. From application perspective nothing changed after these 15 min.
This smells garbage collection, you are most likely to be running into long gc pauses. Check your gc logs, which you can enable using verbose gc settings on all members. If there are back-to-back GCs then you should do various things:
increase the heap size
tune your gc, I'd look into G1 (with -XX:MaxGCPauseMillis set to a reasonable number) and/or ZGC.
I see this error continuously in the debug.log in cassandra,
WARN [SharedPool-Worker-2] 2018-05-16 08:33:48,585 BatchStatement.java:287 - Batch of prepared statements for [test, test1] is of size 6419, exceeding specified threshold of 5120 by 1299.
In this
where,
6419 - Input payload size (Batch)
5120 - Threshold size
1299 - Byte size above threshold value
so as per this ticket in Cassandra, https://github.com/krasserm/akka-persistence-cassandra/issues/33 I see that it is due to the increase in input payload size so I Increased the commitlog_segment_size_in_mb in cassandra.yml to 60mb and we are not facing this warning anymore.
Is this Warning harmful? Increasing the commitlog_segment_size_in_mb will it affect anything in performance?
This is not related to the commit log size directly, and I wonder why its change lead to disappearing of the warning...
The batch size threshold is controlled by batch_size_warn_threshold_in_kb parameter that is default to 5kb (5120 bytes).
You can increase this parameter to higher value, but you really need to have good reason for using batches - it would be nice to understand the context of their usage...
commit_log_segment_size_in_mb represents your block size for commit log archiving or point-in-time backup. These are only active if you have configured archive_command or restore_command in your commitlog_archiving.properties file.
Default size is 32mb.
As per Expert Apache Cassandra Administration book:
you must ensure that value of commitlog_segment_size_in_mb must be twice the value of max_mutation_size_in_kb.
you can take reference of this:
Mutation of 17076203 bytes is too large for the maxiumum size of 16777216
I have a Liferay 6.2 server that has been running for years and is starting to take a lot of database space, despite limited actual content.
Table Size Number of rows
--------------------------------------
DLFileRank 5 GB 16 million
DLFileEntry 90 MB 60,000
JournalArticle 2 GB 100,000
The size of the DLFileRank table sounds to me as abnormally big (if it is totally normal please let me know).
While the file ranking feature of Liferay is nice to have, we would not really mind resetting it if it halves the size of the database.
Question: Would a DELETE * FROM DLFileRank be safe? (stop Liferay, run that SQL command, maybe set dl.file.rank.enabled=false in portal-ext.properties, start Liferay again)
Is there any better way to do it?
Bonus if there is a way to keep recent ranking data and throw away only the old data (not a strong requirement).
Wow. According to the documentation here (Ctrl-F rank), I'd not have expected the number of entries to be so high - did you configure those values differently?
Set the interval in minutes on how often CheckFileRankMessageListener
will run to check for and remove file ranks in excess of the maximum
number of file ranks to maintain per user per file. Defaults:
dl.file.rank.check.interval=15
Set this to true to enable file rank for document library files.
Defaults:
dl.file.rank.enabled=true
Set the maximum number of file ranks to maintain per user per file.
Defaults:
dl.file.rank.max.size=5
And according to the implementation of CheckFileRankMessageListener, it should be enough to just trigger DLFileRankLocalServiceUtil.checkFileRanks() yourself (e.g. through the scripting console). Why you accumulate that large number of files is beyond me...
As you might know, I can never be quoted by stating that direct database manipulation is the way to go - in fact I refuse thinking about the problem from that way.
Reading the docs on the new Eviction Algorithm, available from Hazelcast 3.7, it is not very clear to me how the parameters mentioned in that section, are linked to the actual Map eviction policy parameters.
Namely, the algorithm explanation uses:
GlobalCapacity: User defined maximum cache size (cluster-wide).
PartitionCount: Number of partitions in the cluster (defaults to 271).
BalancedPartitionSize: Number of elements in a balanced partition state, BalancedPartitionSize := GlobalCapacity / PartitionCount.
Deviation: An approximated standard deviation (tests proofed it to be pretty near), Deviation := sqrt(BalancedPartitionSize).
Whereas the eviction policy configuration mentions (amongst some others):
<hazelcast>
<map name="default">
...
<time-to-live-seconds>0</time-to-live-seconds>
<max-idle-seconds>0</max-idle-seconds>
<eviction-policy>LRU</eviction-policy>
<max-size policy="PER_NODE">5000</max-size>
...
</map>
</hazelcast>
One assumption to be made is that GlobalCapacity is somehow linked to the max-size property ??
Any help clarifying this is most welcome! : )
GlobalCapacity: User defined maximum cache size (cluster-wide).
PartitionCount: Number of partitions in the cluster (defaults to 271).
BalancedPartitionSize: Number of elements in a balanced partition
state, BalancedPartitionSize := GlobalCapacity / PartitionCount.
Deviation: An approximated standard deviation (tests proofed it to be
pretty near), Deviation := sqrt(BalancedPartitionSize).
The above are the variables used to explain algorithm in reference manual. They are not the API variables.
But specifically in your question: Yes, global capacity is equivalent and can defined by user with max-size config inside map-config.