I have varnish running with -s malloc,25G, but the process keeps growing until it consumes all the server memory and it crashes, restarting the cache.
Any idea what I can check to see what's wrong?
The server has 16 cores, and 32 GB of RAM.
I am using varnish 3.
28346 ? Ss 0:16 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :9290 -T localhost:6082 -f /etc/varnish/default.vcl -s malloc,25G -p thread_pools 16 -p thread_pool_min 160 -p thread_pool_max 16000 -p thread_pool_add_delay 2
The -s malloc,25G only limits the cache size, not the size of the varnishd process.
It is important to keep in mind that the size you specify with the -s argument is the size for the actual cache. Varnish has an overhead on top of this for keeping track of the cache, so the actual memory footprint of Varnish will exceed what the ‘-s’ argument specifies if the cache is full. The current estimate (subject to change on individual Varnish-versions) is that about 1kB of overhead needed for each object. For 1 million objects, that means 1GB extra memory usage.
If you have a large cache, the file backend may be preferable.
The documentation is correct, but problem is even worse due to memory fragmentation. To be on the safe side, you should allow for 1.5x to 2x the size configured for the malloc stevedore
Varnish might use the double of the given value because of fragmentation. You can read more here: https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage
By the way: Varnish 3 has a security leak and is pretty old. Please update
Related
I'm preparing for more traffic in the days to come, and I want to be sure server can handle it.
Running sar -q, the load of "3.5" doesn't seem much on 32 CPU architecture:
However, I'm not sure about the memory.
Running sar -r shows 98.5% for the %memused and only 13.60 for %commit:
running htop seems OK too: 14.9G/126G. Does this means only 14.9 Gigs are in use by the apps, out of the 126 available?
I'm more interested by the sar -r output.
%memused looks 98.5% and %commit is only 13.6%
I wonder what it means.
You see, linux will try to cache disk blocks read or written in memory when the memory is not in use. This is what you see reported by sar in columns kbcached and kbbuffers. When a new request comes in and requires memory, it is granted from this cache or from the free list.
kbmemused = memory consumed by running processes + cache + buffers
To find out the actual memory used by your application, you should subtract kbbuffers and kbcached from kbmemused.
Monitoring %commit makes more sense which is the actual memory used by current running processes. In your case this number approximately matches with the output of htop.
Another way to check actual free memory is by using command free -m.
free reports the same stats as sar.
To summarize
memused 98.5% shows you the memory utilised by your application + cache and buffers used to kernel to speed up disk access.
commit 13.6% is the actual memory committed by the kernel to you application processes.
WE have a 12 node cassandra cluster across 2 different datacenter. We are migrating the data from sql DB to cassandra through a net application and there is another .net app thats reads data from the cassandra. Off recently we are seeing one or the other node going down (nodetool status shows DN and the service is stopped on it). Below is the output of the nodetool status. WE have to start the service to again get it working but it stops again.
https://ibb.co/4P1T453
Path to the log: https://pastebin.com/FeN6uDGv
So in looking through your pastebin, I see a few things that can be adjusted.
First I'm reasonably sure that this is your primary issue:
Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out,
especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
From GNU Error Codes:
Macro: int ENOMEM
“Cannot allocate memory.” The system cannot allocate more virtual
memory because its capacity is full.
-Xms12G, -Xmx12G, -Xmn3000M,
How much RAM is on your instance? From what I'm seeing your node is dying from an OOM (Out of Memory error). My guess is that you're designating too much RAM to the heap, and there isn't enough for the OS/page-cache. In fact, I wouldn't designate much more than 50%-60% of RAM to the heap.
For example, I mostly build instances on 16GB of RAM, and I've found that a 10GB max heap is about as high as you'd want to go on that.
-XX:+UseParNewGC, -XX:+UseConcMarkSweepGC
In fact, as you're using CMS GC, I wouldn't go higher than 8GB for max heap size.
Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low,
recommended value: 1048575, you can change it with sysctl.
This means you haven't adjusted your limits.conf or sysctl.conf. Check through the guide (DSE 6.0 - Recommended Production Settings), but generally it's a good idea to add the following to these files:
/etc/limits.conf
* - memlock unlimited
* - nofile 100000
* - nproc 32768
* - as unlimited
/etc/sysctl.conf
vm.max_map_count = 1048575
Note: After adjusting sysctl.conf, you'll want to run a sudo sysctl -p or reboot.
Is swap disabled? : false,
You will want to disable swap. If Cassandra starts swapping contents of RAM to disk, things will get really slow. Run a swapoff -a and then edit /etc/fstab and remove any swap entries.
tl;dr; Summary
Set your initial and max heap sizes to 8GB (heap new size is fine).
Modify your limits.conf an sysctl.conf files appropriately.
Disable swap.
It's also a good idea to get on the latest version of 3.11 (3.11.4).
Hope this helps!
I'm running node.js on a Raspberry Pi 3 B with the following free memory:
free -m
total used free shared buffers cached
Mem: 973 230 742 6 14 135
-/+ buffers/cache: 80 892
Swap: 99 0 99
How can I configure node (v7) to not use all the free memory? To prolong the SD card life, I would like to prevent it from going to swap.
I am aware of --max_old_space_size:
node --v8-options | grep -A 5 max_old
--max_old_space_size (max size of the old space (in Mbytes))
type: int default: 0
I know some of the answer is application specific, however what are some general tips to limit node.js memory consumption to prevent swapping? Also any other tips to squeeze more free ram out of the pi would be appreciated.
I have already set the memory split so that the GPU has the minimum 16 megs of RAM allocated.
The only bulletproof way to prevent swapping is to turn off swapping in the operating system (delete or comment out any swap lines in /etc/fstab for permanent settings, or use swapoff -a to turn off all swap devices for the current session). Note that the kernel is forced to kill random processes when there is no free memory available (this is true both with and without swap).
In node.js, what you can limit is the size of V8's managed heap, and the --max-old-space-size flag you already mentioned is the primary way for doing that. A value around 400-500 (megabytes) probably makes sense for your Raspberry. There's also --max-semi-space-size which should be small and you can probably just stick with the default, and --max-executable-size for generated code (how much you need depends on the app you run; I'd just stick with the default).
That said, there's no way to limit the overall memory usage of the process, because there are other memory consumers outside the managed heap (e.g. node.js itself, V8's parser and compiler). There is no way to set limits on all kinds of memory usage. (Because what would such a limit do? Crash when memory is needed but not available? The kernel will take care of that anyway.)
I am running my app servers (one instance each of Karaf, Tomcat, Mongo and Zookeeper) in a RHEL environment and often see that (using free -m) of my total 12GB RAM almost 8GM is shown as cached. The app slows down as well. Why is this happening. I even tried to stop all of these services gracefully until i have only the Linux OS alone running on my box. Even then the cache is not freed. I have to manually free it to bring it down.
Why is the cache being accumulated like this and Does it have something to do with my application? Is it a good practise to run a chron job like this just to free the cache?
Try clearing the cache.
#sync; echo 3 > /proc/sys/vm/drop_caches
if you are talking about the "cache" from the last column here:
$ free -m
total used free shared buffers cached
Mem: 3954 3580 374 0 1 1448
then there is no reason to clear it. This cache is absolutely harmless, it retains (caches) for example previously opened files for faster access. When more memory is needed, this cache is automatically cleared. There is no reason why this cache would slow down any apps.
Update: Some apps store temporary files in memory, /dev/shm is usually a place for this, but you can check these on your system using:
$ mount|grep tmpfs
These files also show up in the cached column, but this data is not harmless in the sense that it cannot be dropped when more free memory is needed.
I want to force the Linux kernel to allocate more memory to applications after the cache starts taking up too much memory (as can be seen by the output of 'free').
I've run
sudo sync; sudo sysctl -w vm.drop_caches=3; free
(to free both disc dentry/inode cache and page cache) and I see that only about half of the used cache was freed - the rest remains. How can I tell what is taking up the rest of the cache and force it to be freed?
You may want to increase vfs_cache_pressure as well as set swappiness to 0.
Doing that will make the kernel reclaim cache faster, while giving processes equal or more favor when deciding what gets paged out.
You may only want to do this if processes you care about do very little disk I/O.
If a network I/O bound process has to swap in to serve requests, that's a problem and the real solution is to put it on a less competitive server.
With the default swappiness setting, the kernel is almost always going to favour keeping FS related cache in real memory.
As such, if you increase the cache pressure, be sure to equally adjust swappiness.
The contents of /proc/meminfo tell you what the kernel uses RAM for.
You can use /proc/sys/vm/vfs_cache_pressure to force the kernel to reclaim memory that is used for filesystem-related caches more lazily or eagerly.
Note that your application may only benefit from tuning this parameter if it does little or no disk I/O.
You might find John Nilsson's answer to my Question useful for purging the cache in order to test whether that is related to your problem:
sync && echo 1 > /proc/sys/vm/drop_caches
Though I'm guessing the only real difference is 1 vs 3