libpcap performance and behavior differences between Ubuntu 14.40 and CentOS 6.5 - linux

I have been running a tcpdump based script on Ubuntu for some time, and recently I have been asked to run it on CentOS 6.5 and I'm noticing some very interesting differences
I'm running tcpdump 4.6.2, libpcap 1.6.2 on both setups, both are actually running on the same hardware (dual booted)
I'm running the same command on both OS'.
sudo /usr/sbin/tcpdump -s 0 -nei eth9 -w /mnt/tmpfs/eth9_rx.pcap -B 2000000
From "free -k", I see about 2G allocated on Ubuntu
Before:
free -k
total used free shared buffers cached
Mem: 65928188 1337008 64591180 1164 26556 68596
-/+ buffers/cache: 1241856 64686332
Swap: 67063804 0 67063804
After:
free -k
total used free shared buffers cached
Mem: 65928188 3341680 62586508 1160 26572 68592
-/+ buffers/cache: 3246516 62681672
Swap: 67063804 0 67063804
expr 3341680 - 1337184
2004496
One CentOS, I see twice the amount of memory (4G) being allocated from the same command
Before:
free -k
total used free shared buffers cached
Mem: 16225932 394000 15831932 0 15308 85384
-/+ buffers/cache: 293308 15932624
Swap: 8183804 0 8183804
After:
free -k
total used free shared buffers cached
Mem: 16225932 4401652 11824280 0 14896 84884
-/+ buffers/cache: 4301872 11924060
Swap: 8183804 0 8183804
expr 4401652 - 394000
4007652
From the command, I'm listening against an interface and dumping into a RAMdisk.
On Ubuntu, I can capture packets at line rate for large size packets (10G, 1024 byte frames)
But on CentOS, I can only capture packets at 60% of line rate (10G, 1024 byte frames)
Also, both OS's are running the same version of NIC drivers and driver configurations.
My goal is to achieve the same performance on CentOS as I have with Ubuntu.
I googled around and there seems to be the magic of libpcap behaving differently with different kernels. I'm curious if there's any kernel side options I have to tweek on the CentOS side to achieve the same performance on Ubuntu.

This has been answered. According to Guy Harris from tcpdump/libpcap, the difference is due to CentOS6.5 running 2.6.X kernel. Below is his response:
"
3.2 introduced the TPACKET_V3 version of the "T(urbo)PACKET" memory-mapped packet capture mechanism for PF_PACKET sockets; newer versions of libpcap (1.5 and later) support TPACKET_V3 and will use it if the kernel supports it. TPACKET_V3 makes much more efficient use of the capture buffer; in at least one test, it dropped fewer packets. It also might impose less overhead, so that asking for a 2GB buffer takes less kernel memory."

Related

Using qemu+kvm is slower than using qemu in rv6(xv6 rust porting)

We are currently working on the rv6 project which is porting MIT's educational operating system xv6 to Rust. Our code is located here.
We use qemu and qemu's virt platform to execute rv6, and it works well with using qemu.
Executing command on arm machine is this:
RUST_MODE=release TARGET=arm KVM=yes GIC_VERSION=3
qemu-system-aarch64 -machine virt -kernel kernel/kernel -m 128M -smp 80 -nographic -drive file=fs.img,if=none,format=raw,id=x0,copy-on-read=off -device virtio-blk-device,drive=x0,bus=virtio-mmio-bus.0 -cpu cortex-a53 -machine gic-version=3 -net none
To make some speed boost experiment with KVM, we made rv6 support the arm architecture on arm machine. The arm architecture's driver code locates in here.
The problem is, when we use qemu with kvm, the performance is significantly reduced.
Executing command on arm machine with KVM is this:
qemu-system-aarch64 -machine virt -kernel kernel/kernel -m 128M -smp 80 -nographic -drive file=fs.img,if=none,format=raw,id=x0,copy-on-read=off -device virtio-blk-device,drive=x0,bus=virtio-mmio-bus.0 -cpu host -enable-kvm -machine gic-version=3 -net none
We repeated
Write 500 bytes syscall 10,000 times and the result was: kvm disable: 4,500,000 us, kvm enable: 29,000,000 us. (> 5 times)
Open/Close syscall 10,000 times result: kvm disable: 12,000,000 us, kvm enable: 29,000,000 us. (> 5 times)
Getppid syscall 10,000 times result: kvm disable: 735,000 us, kvm enable: 825,000 us. (almost same)
Simple calculation(a = a * 1664525 + 1013904223) 100 million times result: kvm disable: 2,800,000 us, kvm enable: 65,000,000 us. (> 20 times)
And the elapsed time was estimated by uptime_as_micro syscall in rv6.
These results were so hard to understand. So first we tried to find the bottleneck on rv6's booting process, because finding bottleneck during processing user program was so difficult.
We found that the first noticeable bottleneck on rv6 booting process was here:
run.as_mut().init();
self.runs().push_front(run.as_ref());
As far as we know, this part is just kind of "list initialization and push element" part. So we thought that by some reason, the KVM is not actually working and it makes worse result. And also this part is even before turn on some interrupts, so we thought arm's GIC or interrupt related thing is not related with problem.
So, how can I get better performance when using kvm with qemu?
To solve this problem, we tried these already:
change qemu(4.2, 6.2), virt version, change some command for qemu-kvm like cpu, drive cache, copy-on-read something, kernel_irqchip.., cpu core.. etc
find some kvm hypercall to use - but not exists on arm64
Run lmbench by ubuntu on qemu with kvm to check KVM itself is okay. - We found KVM with ubuntu is super faster than only using qemu.
Check 16550a UART print code is really slow on enabling KVM which makes incorrect result on benchmark - Without bottleneck code, we found the progress time of rv6 booting were almost same with KVM enabled or not.
Check other people who suffer same situation like us - but this superuser page not works. Our clocksource is arch_sys_counter.
Our Environment
qemu-system-aarch64 version: 4.2.1 (Debian 1:4.2-3ubuntu6.19)
CPU model: Neoverse-N1
Architecture: arrch64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 80
Ubuntu 20.04.3 LTS(Focal Fossa)
/dev/kvm exists
The main reason is here: https://forum.osdev.org/viewtopic.php?f=1&t=56120
RV6's memory setting was incorrect.
Change ARM's device memory to cacheable memory for MAIR_EL1 ARM register solved the problem.

Attempting to boot new uImage

I am attempting to boot uImage via uboot and I am getting some seemingly conflicting log info:
update Kernel1 tftp uImage-2.5 6.35. -digi-armv7a.LONEPEAK-Ver-4_33
Using FEC0 device
TFTP from server 10.12.1.77; our IP address is 10.12.1.205
Filename 'uImage-2.6.35-digi-armv7a.LONEPEAK-Ver-4_33'.
Load address: 0x94000000
Loading: #################################################################
#################################################################
###########################################
done
Bytes transferred = 2533360 (26a7f0 hex)
Calculated checksum = 0x49669c61
Updating partition 'Kernel1'
Erasing 128 KiB # 0x08540000: 0%
Erasing 128 KiB # 0x085e0000: 20%
Erasing 128 KiB # 0x08680000: 41%
Erasing 128 KiB # 0x08720000: 62%
Erasing 128 KiB # 0x087c0000: 83%
Erasing: complete
Writing: 0%
Writing: 51%
Writing: complete
Verifying: 0%
Verifying: 51%
Verifying: complete
Writing Parameters to NVRAM
Update successful
Above it shows a successful update but then when issue a reboot command I get:
scanning bus for devices... 1 USB Device(s) found
scanning bus for storage devices... 0 Storage Device(s) found
** Invalid boot device **
Booting partition 'Kernel0'
## Booting kernel from Legacy Image at 90007fc0 ...
Image Name: Linux-2.6.35.14-tjerbmx51_0005+
Created: 2018-10-16 21:35:37 UTC
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 2533296 Bytes = 2.4 MB
Load Address: 90008000
Entry Point: 90008000
Loading Kernel Image ... OK
OK
Starting kernel ...
So my question is:
Is there a way for me to version my kernel when I build it s/t I can set the 'Image Name' so that I know its my kernel being loaded and not some type of Legacy Image??
Perhaps the CONFIG_LOCALVERSION-option of Linux kernel .config-file will help you.
From the Kernel.org:
Keep a backup kernel handy in case something goes wrong. This is
especially true for the development releases, since each new release
contains new code which has not been debugged. Make sure you keep a
backup of the modules corresponding to that kernel, as well. If you
are installing a new kernel with the same version number as your
working kernel, make a backup of your modules directory before you do
a make modules_install.
Alternatively, before compiling, use the kernel config option
“LOCALVERSION” to append a unique suffix to the regular kernel
version. LOCALVERSION can be set in the “General Setup” menu.
So during the kernel configuration you can add some clear suffix to your kernel e.g. CONFIG_LOCALVERSION="-test_some_stuff".
Some useful links: 1 and 2.

Node forever (npm package) memory leak on the server

I am using forever package to run my Node.js script. (not a web server). However, because of it, I have memory leak and even after stopping all processes, my memory is still taken:
root#aviok-cdc-elas-001:~# forever stopall
info: No forever processes running
root#aviok-cdc-elas-001:~# forever list
info: No forever processes running
root#aviok-cdc-elas-001:~# free -lm
total used free shared buffers cached
Mem: 11721 6900 4821 5 188 1242
Low: 11721 6900 4821
High: 0 0 0
-/+ buffers/cache: 5469 6252
Swap: 0 0 0
Also to mention, there is no memory leak from the script when ran locally without forever. I run it on Ubuntu server. And if I would reboot server now:
root#aviok-cdc-elas-001:~# reboot
Broadcast message from root#aviok-cdc-elas-001
(/dev/pts/0) at 3:19 ...
The system is going down for reboot NOW!
My RAM would be free again:
root#aviok-cdc-elas-001:~# free -lm
total used free shared buffers cached
Mem: 11721 1259 10462 5 64 288
Low: 11721 1259 10462
High: 0 0 0
-/+ buffers/cache: 905 10816
Swap: 0 0 0
I also want to mention that, when my script finishes what it is doing (and it does eventually) I have db.close and process.exit calls to make sure everything is killed from the side of my script. However, even after that RAM is taken away. Now I am aware that forever will run that script again after it is killed. So my questions are:
How do I tell forever to not execute script again if it is finished?
How do I stop forever properly so it does NOT take any RAM after I stopped it?
The reason I am using forever package for this is because my script needs a lot of time to do what it does and my SSH session would end, and so would Node script which I ran in a regular way.
From what I can see, the RAM isn't taken away, or leaking, it's being used by Linux as file system cache (because unused RAM is wasted RAM).
From the 6900 megs of "used" RAM, 5469 is used as buffer cache. Linux will reduce this amount automatically when processes request memory.
If you want a long-running process to keep running after you log out (or after your SSH session gets killed), you have various options that don't require forever:
Background the process, making sure that any "logout" signals are ignored:
$ nohup node script.js &
Use a terminal multiplexer like tmux or screen.

Java heap out of memory exception tomcat linux

Please help me,my live application sometimes throw exception out of memory java heap
however I set the max size to 512M half of virtual server size
I've searched on google and traced my Server like attached image
can anyone tell me where is the error please ?
the data in console is below
System load: 0.01 Processes: 74
Usage of /: 16.2% of 29.40GB Users logged in: 0
Memory usage: 60%
Swap usage: 0%
developer#pc:/$ free -m
total used free shared buffers cached
Mem: 994 754 239 0 24 138
-/+ buffers/cache: 592 401
Swap: 0 0 0

How to find the virtual memory address size in Linux Ubuntu?

I need to find the virtual address size supported in my PC running Linux Ubuntu. Following is my kernel version:
$ uname -a
Linux ubuntu 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC 2009 i686 GNU/Linux
I also ran 'free' to know this, as per some earlier posts in this forum:
***#ubuntu:~$ free
total used free shared buffers cached
Mem: 2963968 740752 2223216 0 217648 360296
-/+ buffers/cache: 162808 2801160
Swap: 262136 0 262136
However, the value that I obtained is not a power of 2, as I had expected.
Please let me know the correct way to obtain the total virtual memory address space. Thanks in advance for your help.
Use following command
cat /proc/meminfo
You will get a list of information
You can seeVmallocTotal, is the size of total virtual memory allocated.

Resources