Does linux make use of TLB when running as VM?
I wrote a C program to find 'Average access time per page' and found that with any number of frequent page access, the 'Average access time per page' is same.
For example,
With frequent access of 32 pages (10000000 times), its ~10 nano seconds.
With frequent access of 64 pages (10000000 times), its ~10 nano seconds.
With frequent access of 124 pages (10000000 times), its ~10 nano seconds.
With frequent access of 256 pages (10000000 times), its ~10 nano seconds.
With frequent access of 512 pages (10000000 times), its ~10 nano seconds.
and so on.
Does the above test reflect that linux does can not use TLB when running as a virtual machine and always goes to page table entry to get the translation for every 'virtual page number' to 'Physical page frame' for a process?
My machine details are:
[ ] uname -mpi
x86_64 x86_64 x86_64
[ ] getconf PAGE_SIZE
4096
[ ] uname -a
Linux VirtualBox 5.8.0-53-generic #60~20.04.1-Ubuntu SMP Thu May 6 09:52:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Thanks,
Related
My runtime environment is CentOS 7.9(kernel is version 5.16.11) in the VMBox virtual machine, it is allocated 1G memory and 8 CPU cores.
[root#dev236 ~]# uname -a
Linux dev236 5.16.11-1.el7.elrepo.x86_64 #1 SMP PREEMPT Tue Feb 22 10:22:37 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
I ran a computation-intensive program that used 8 threads to continuously use the CPU.
After some time, the operating system issues a bug alert, like this:
[root#dev236 src]# ./server --p:command-threads-count=8
[31274.179023] rcu: INFO: rcu_preempt self-detected stall on CPU
[31274.179039] watchdog: BUG: soft lockup - CPU#3 stuck for 210S! [server:1356]
[31274.179042] watchdog: BUG: soft lockup - CPU#1 stuck for 210S! [server:1350]
[31274.179070] watchdog: BUG: soft lockup - CPU#7 stuck for 210S! [server:1355]
[31274.179214] rcu: 0-...!: (1 GPs behind) idle=52f/1/0x4000000000000000 softirq=10073864/10073865
fqs=0
Message from syslogd#dev236 at Jan 25 18:59:49 ...
kernel:watchdog: BUG: soft lockup - CPU#3 stuck for 210S! [server:1356]
Message from syslogd#dev236 at Jan 25 18:59:49 ...
kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 210S! [server:1350]
Message from syslogd#dev236 at Jan 25 18:59:49 ...
kernel:watchdog: BUG: soft lockup - CPU#7 stuck for 210S! [server:1355]
^C
[root#dev236 src]#
Then, I looked at the program log, and the log file was constantly being appended, which indicated that my test program was still running.
I wonder if I can ignore this bug tip?
Or, do I have to do something?
for example:
Reduce the computational intensity of the program?
Give the CPU a break once in a while?
Reduce the number of threads started in the program?
Thank you all
We are currently working on the rv6 project which is porting MIT's educational operating system xv6 to Rust. Our code is located here.
We use qemu and qemu's virt platform to execute rv6, and it works well with using qemu.
Executing command on arm machine is this:
RUST_MODE=release TARGET=arm KVM=yes GIC_VERSION=3
qemu-system-aarch64 -machine virt -kernel kernel/kernel -m 128M -smp 80 -nographic -drive file=fs.img,if=none,format=raw,id=x0,copy-on-read=off -device virtio-blk-device,drive=x0,bus=virtio-mmio-bus.0 -cpu cortex-a53 -machine gic-version=3 -net none
To make some speed boost experiment with KVM, we made rv6 support the arm architecture on arm machine. The arm architecture's driver code locates in here.
The problem is, when we use qemu with kvm, the performance is significantly reduced.
Executing command on arm machine with KVM is this:
qemu-system-aarch64 -machine virt -kernel kernel/kernel -m 128M -smp 80 -nographic -drive file=fs.img,if=none,format=raw,id=x0,copy-on-read=off -device virtio-blk-device,drive=x0,bus=virtio-mmio-bus.0 -cpu host -enable-kvm -machine gic-version=3 -net none
We repeated
Write 500 bytes syscall 10,000 times and the result was: kvm disable: 4,500,000 us, kvm enable: 29,000,000 us. (> 5 times)
Open/Close syscall 10,000 times result: kvm disable: 12,000,000 us, kvm enable: 29,000,000 us. (> 5 times)
Getppid syscall 10,000 times result: kvm disable: 735,000 us, kvm enable: 825,000 us. (almost same)
Simple calculation(a = a * 1664525 + 1013904223) 100 million times result: kvm disable: 2,800,000 us, kvm enable: 65,000,000 us. (> 20 times)
And the elapsed time was estimated by uptime_as_micro syscall in rv6.
These results were so hard to understand. So first we tried to find the bottleneck on rv6's booting process, because finding bottleneck during processing user program was so difficult.
We found that the first noticeable bottleneck on rv6 booting process was here:
run.as_mut().init();
self.runs().push_front(run.as_ref());
As far as we know, this part is just kind of "list initialization and push element" part. So we thought that by some reason, the KVM is not actually working and it makes worse result. And also this part is even before turn on some interrupts, so we thought arm's GIC or interrupt related thing is not related with problem.
So, how can I get better performance when using kvm with qemu?
To solve this problem, we tried these already:
change qemu(4.2, 6.2), virt version, change some command for qemu-kvm like cpu, drive cache, copy-on-read something, kernel_irqchip.., cpu core.. etc
find some kvm hypercall to use - but not exists on arm64
Run lmbench by ubuntu on qemu with kvm to check KVM itself is okay. - We found KVM with ubuntu is super faster than only using qemu.
Check 16550a UART print code is really slow on enabling KVM which makes incorrect result on benchmark - Without bottleneck code, we found the progress time of rv6 booting were almost same with KVM enabled or not.
Check other people who suffer same situation like us - but this superuser page not works. Our clocksource is arch_sys_counter.
Our Environment
qemu-system-aarch64 version: 4.2.1 (Debian 1:4.2-3ubuntu6.19)
CPU model: Neoverse-N1
Architecture: arrch64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 80
Ubuntu 20.04.3 LTS(Focal Fossa)
/dev/kvm exists
The main reason is here: https://forum.osdev.org/viewtopic.php?f=1&t=56120
RV6's memory setting was incorrect.
Change ARM's device memory to cacheable memory for MAIR_EL1 ARM register solved the problem.
lsyncd is a fantastic alternative to NFS or NAS for replicating files among your Linux hosts. I have found the daemon works well with large Linux filesystems (many files, small to large sizes, xfs, ext4, luks) but requires some sysctl tuning as your filesystem grows.
This "question" is a note to myself so I can always find the answer via searching on stack overflow. Hope it helps you!
Github Project: https://github.com/axkibe/lsyncd
Exception in /var/log/lsyncd.log:
Thu Jun 18 17:48:52 2020 Normal: --- OVERFLOW in event queue ---
Thu Jun 18 17:48:52 2020 Normal: --- HUP signal, resetting ---
Thu Jun 18 17:48:52 2020 Normal: waiting for 1 more child processes.
Solution when you see "OVERFLOW in event queue" in lsyncd.log
From other knowledge bases, I had learned to tune max_user_watches, but by also tuning the max_queued_events, I corrected an OVERFLOW in event queue exception.
The temporary solution worked without needing to restart my lsyncd process.
I picked the number 1000000 as an arbitrarily large number. The default Ubuntu 18 value is 16384.
Temporary Solution
Check your current tuning values:
$ sysctl fs.inotify.max_queued_events
fs.inotify.max_queued_events = 16384
$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192
Update both max_user_watches and max_queued_events via shell
sudo sysctl fs.inotify.max_user_watches=1000000
sudo sysctl fs.inotify.max_queued_events=1000000
Permanent Solution, Persists after reboot
Update both max_user_watches and max_queued_events in /etc/sysctl.conf
fs.inotify.max_user_watches=1000000
fs.inotify.max_queued_events=1000000
Lsyncd.conf Basic Configuration
/etc/lsyncd/lsyncd.conf
settings {
logfile = "/var/log/lsyncd.log",
pidfile = "/var/run/lsyncd/lsyncd.pid",
insist = true
}
sync {
default.rsyncssh,
source="/var/application/data",
host="node2",
excludeFrom="/etc/lsyncd/exclude",
targetdir="/var/application/data",
rsync = {
archive = true,
compress = false,
whole_file = true
},
ssh = {
port = 22
}
}
System Details
Linux service1staging 5.0.0-36-generic #39~18.04.1-Ubuntu SMP Tue Nov 12 11:09:50 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 18.04.4 LTS
lsyncd --version
Version: 2.1.6
I have an application that I eventually want to run on a cloud computing service (e.g., such as AWS or Google Cloud) packaged inside a docker image. The reason the application will need to run in the cloud is because it's designed to process large data files, but before I actually deploy, I'd like to test it first on a local laptop, using a single large data file that I've stored (for test and development purposes) on an external USB drive.
My development machine is an OSX laptop, and I'm using a recent version of docker:
stachyra> uname -a
Darwin Andrews-MacBook-Pro-76.local 14.5.0 Darwin Kernel Version 14.5.0: Tue Sep 1 21:23:09 PDT 2015; root:xnu-2782.50.1~1/RELEASE_X86_64 x86_64
stachyra> docker --version
Docker version 1.10.2, build c3959b1
OSX has mounted my external USB drive, device /dev/disk2s2, as /Volumes/MGR DATA:
stachyra> df
Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted on
/dev/disk1 974770480 435721376 538537104 45% 54529170 67317138 45% /
devfs 375 375 0 100% 650 0 100% /dev
map -hosts 0 0 0 100% 0 0 100% /net
map auto_home 0 0 0 100% 0 0 100% /home
/dev/disk2s2 3906291632 3869523640 36767992 100% 483690453 4595999 99% /Volumes/MGR DATA
/dev/disk3s1 196608 193160 3448 99% 24143 431 98% /Volumes/VirtualBox
stachyra> diskutil list
/dev/disk0
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *500.3 GB disk0
1: EFI EFI 209.7 MB disk0s1
2: Apple_CoreStorage 499.4 GB disk0s2
3: Apple_Boot Recovery HD 650.0 MB disk0s3
/dev/disk1
#: TYPE NAME SIZE IDENTIFIER
0: Apple_HFS Macintosh HD *499.1 GB disk1
Logical Volume on disk0s2
DB70B91A-3B57-4C82-A758-C4BDEA4160FD
Unlocked Encrypted
/dev/disk2
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *2.0 TB disk2
1: EFI EFI 209.7 MB disk2s1
2: Apple_HFS MGR DATA 2.0 TB disk2s2
/dev/disk3
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *100.7 MB disk3
1: Apple_HFS VirtualBox 100.7 MB disk3s1
and it should also be noted, the drive has several directories and data which are visible inside it, at least when viewed directly through OSX:
stachyra> ls -l /Volumes/MGR\ DATA
total 0
drwxr-xr-x 6 stachyra staff 204 Apr 14 2015 1000genomes
drwxr-xr-x 5 stachyra staff 170 Oct 12 17:41 GIAB
drwxr-xr-x 4 stachyra staff 136 Apr 28 2015 genome_browser_tracks
drwxr-xr-x 24 stachyra staff 816 Oct 6 14:00 mitty
I have tried to follow the advice from this question, which describes how to mount a USB drive in docker when docker is running within a linux host. But my local laptop is OSX, not linux, so it doesn't seem to work.
Explicitly, when attempting to follow the advice of the accepted answer, I obtain the following result:
stachyra> docker run -i -t --privileged -v /dev/disk2s2:/dev/foo ubuntu bash
root#8da7b492a707:/# uname -a
Linux 8da7b492a707 4.1.18-boot2docker #1 SMP Sat Feb 20 08:24:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root#8da7b492a707:/# ls -l /dev/foo
total 0
root#8da7b492a707:/#
Based upon the response, one can see that docker does indeed launch a linux container correctly, and it also creates a volume /dev/foo inside of the container as requested, but the actual contents of the USB drive are not accessible via that location--the ls -l command claims there are no files or directories there.
I also tried the second method described in an alternate response to the same question, and that fails even worse:
stachyra> docker run -i -t --device=/dev/disk2s2 ubuntu bash
docker: Error response from daemon: error gathering device information while adding custom device "/dev/disk2s2": not a device node.
stachyra>
I have found another discussion thread on stackoverflow which suggests that raw USB access is handled quite differently in OSX than in linux, which I suspect is probably the reason why both of the above attempts at USB access are failing.
But, what should I actually do about it? That is to say, what is the correct sequence of actions or commands to allow docker to access a USB device mounted on an OSX host, rather than linux?
I was finally able to access my USB drive from /var/media inside my container by using the machine-diskutil.sh script mentioned in warmoverflow's comment like so
machine-diskutil.sh mount my-machine-name /Volumes/my-usb-drive
and then starting the container like so
docker run -v /Volumes/my-usb-drive:/var/media -it my/image:latest bash
Because I had tried to add /Volumes/my-usb-drive as a shared folder manually in VirtualBox, I first got this error.
Error: The shared folder /Volumes/Seagate already exists on the
docker machine, please unmount it first.
So I removed it manually and re-ran the machine-diskutil.sh mount command without any problems. Great stuff!
As per #pgayvallet comment on GitHub:
As the daemon runs inside a VM in Docker Desktop, it is not possible to actually share a mac host device with the container inside the VM, and this will most definitely never be possible.
I need to find the virtual address size supported in my PC running Linux Ubuntu. Following is my kernel version:
$ uname -a
Linux ubuntu 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC 2009 i686 GNU/Linux
I also ran 'free' to know this, as per some earlier posts in this forum:
***#ubuntu:~$ free
total used free shared buffers cached
Mem: 2963968 740752 2223216 0 217648 360296
-/+ buffers/cache: 162808 2801160
Swap: 262136 0 262136
However, the value that I obtained is not a power of 2, as I had expected.
Please let me know the correct way to obtain the total virtual memory address space. Thanks in advance for your help.
Use following command
cat /proc/meminfo
You will get a list of information
You can seeVmallocTotal, is the size of total virtual memory allocated.