we have a kubernetes cluster environment which at any given time has around 10~20 pods/containers running on a single node. Each node has around 200k ish inodes available. However our micro service (nodejs/npm) app can each eats up around 20k+ inodes, and times that by say 10 pods/containers on a node that basically eats up all the available inodes on a server (node).
The question is if there is way to deal with this issue in the node_modules to minimize the number of files it contains or use some kind of bundler for the node_modules ??
There isn't really anything Kubernetes can do for you. You need to increase the number of inodes on your filesystem.
Without specifics on the size of your disks I can't give you exact advice. But, the details on the arch wiki about bytes per inode is the correct advice: https://wiki.archlinux.org/index.php/ext4#Bytes-per-inode_ratio
Related
I am currently running Gitlab CE. I have an issue where it is constantly gaining space,
There is 1 current user (myself). But sitting idle it gains 20gb of usage in under an hour for no apparent reason (not pushing or pulling or even using it, the service is simply live and idle) until eventually it fills my drive (411gb of free space before the installation of Gitlabs. takes less than 24hrs to fill it.).
I cannot locate the source of the issue, google seems to like referring me to size limitations, and that is fine if I needed to increase that which I don't, i have tried to disable some metrics and the safety features such as "Health checks" in an attempt to stop it from doing this but with no success
I have to keep reinstalling it to negate the idle data usage. There is a reason for me setting it up, but I cannot deploy this the way it is. Have any of you experienced this issue? Is there a way around this?
The system current running it: Fedora 36 running the installation on a 500GB SSD, 8 core Ryzen 7 Processor.
any advice to solve this problem would be great. Please note I am not an expert.
Answer to this question:
rsync was scheduled automatically and was in a loop.
Removed rsync, reinstalled it, rescheduled rsync to go on my schedule, removed the older 100 or so back ups and my space has been returned.
for those that are running rsync, just check that it is not running too closely and is detecting that its own backups are there. as the back ups i found were corrupted.
We are using Rendertron for Server-side rendering on one of our site and we're having problems with files created by puppeteer.
We have the Rendertron cache set to "memory" with a limit of 500, which seems to work properly.
But even with the cache set to memory in Rendertron, the underlying puppeteer processes will create tens of thousands of files under the /tmp/puppeteer_dev_chrome-[profileId]/Default/Cache directory.
This causes problems when there is a lot of requests hitting Rendertron. The disk I/O will quickly reach 100% and then the server will become unresponsive.
We have tried looking into flags we can pass to puppeteer (found here https://peter.sh/experiments/chromium-command-line-switches), but we can't seem to find one (or more) that would fix our problem.
Did anyone ever use rendertron for SSR and had this kind of problem? Any kind of help would be appreciated.
I ran into this recently on GCP where our iops were being eaten up on render boxes by /tmp usage, even though we had cache set to memory. Best thing to do is just use your RAM vs storage for the /tmp data by leveraging tmpfs. Here's an example from /etc/fstab for a "secure" 1GB /tmp leveraging tmpfs:
tmpfs /tmp tmpfs mode=1777,nosuid,nodev,size=1g 0 0
Once you have the fstab entry added, just 'mount /tmp', and restart services with any open file handles tied to previous /tmp.
I've been struggling to run multiple instances of Puppeteer on DigitalOcean for quite some time with little luck. I'm able to run ~5 concurrently using tools like puppeteer-cluster, but for some reason the whole thing just chokes with little helpful messaging. So, I switched to spawning ~5 child processes without any additional library -- just Puppeteer itself. Same issue. Chokes with no helpful errors.
I'm able to run all of these jobs just fine locally, but after I deploy, I hit these walls. So, my hunch is that it's a resource/performance issue, but I can't say for sure.
I'm running a droplet with 1GB and 3CPUs on Digital Ocean.
Basically, I'm just looking for ways to start troubleshooting something like this. is there a way I can know for sure that I'm hitting resource walls? I've tried pm2 and the DO dashboard graphs, but I feel like those are all leaving a lot of information out, or else I'm missing something else altogether.
Author of puppeteer-cluster here. You are right, 1 GB of memory is likely not enough for running 5 browser windows (or tabs) in addition to your operating system and maybe even other background tasks.
Here is a list of resources you should check:
Memory: Use a tool like htop to check your memory usage while your application is running.
CPU: Again, you can use htop for that, 3 vCPUs should be more than enough for 5 windows.
Disk space: Use a tool like df to check if there is enough space on the disk. I know of multiple cases in which there was not enough space on the disk (like some old kernels filling the disk), and Chrome needs at least some space to run.
Network throughput: Rarely the problem, but sometimes the network just does not have the bandwidth to support many open browser. Use a tool like nload to check the network throughput.
To use htop or nload, you start your script in the background (node script.js &) or use a terminal multiplexer (like tmux). Resource problems should then be easy to spot.
Most probably you're running out of memory, 5 puppeteer processes are a lot for a 1GB VM.
You can run
grep -i 'killed process' /var/log/messages
to confirm that the OOM killer terminated your processes.
Want to rent a small collection of VM's so I can play around with and learn Cassandra. Whats the minimum memory (and maybe) disk footprint you can get away with?
For the sake of the exercise, lets assume I'm starting with three nodes.
People have been running Cassandra on RaspberryPi's (watch a talk on YouTube, or look at some slides on SlideShare), so the minimum specs required are pretty low. The RaspberryPi has 500 megs of RAM and an ARM CPU running at 700 Mhz.
For disk space there's very little needed before you start writing data, then it scales roughly linerally with the data (but remember that removing things actually increases the disk usage temporarily, and compactions require some extra space too while they are running).
For playing you probably won't need much more than the equivalent of a RaspberryPi, but running it on say three EC2 m1.small works absolutely fine. I haven't run it on t1.micro, but it should work (t1.micro is actually spec'ed lower than a RaspberryPi, although it probably has faster disk IO).
Cassandra can scale down well for testing so EC2 small instances or equivalent will be fine. Later, if you plan on performance/load testing, you can scale up. Be sure to adjust/check the config - especially the JVM heap and size it according to the server spec.
If you're just wanting to learn the software, data modelling, CQL etc and replication isn't important just yet then a single server would work fine too - just create keyspaces with a replication factor of 1. I run cassandra on my laptop for development work and it's quite happy with 0.5 gig of jvm heap.
If you just want to play around with a cluster and not necessarily have something running all of the time you can try running a cluster on your own machine without running separate VMs for each node with ccm.
I'm using Ganglia and RRDTool to show charts in a web page.
Everything is fine, but for some machines the graphs about DISK are with some kind of bug.
Here is how they look in some machines (both machine are in the same cluster):
This one is correct, about the disk space:
But this one is showing 1.4Kb of disk space. Which is incorrect. How can I fix this ?
Any idea ? I already uninstall it and install it many times, but it doesn't seem to fix the problem.
I got in touch with Ganglia Developers e-Mail list, they told me that the value is correct, which means 1.4K GB, which are 1400GB or 1.4T. I was all a misunderstanding. I just think it should be a documentation more clear about what are the metrics.