I have an Arch Linux host which runs virtualised router.
When using a LXC guest as router, everything is fine. I get 100MBits Up/Down and almost no CPU usage at all.
However, when I use libvirt gest (pfSense FreeBSD) as a router, whenever there is heavy network traffic going through the guest, the CPU usage goes unreasonably high (up to 100%) but the worst thing is that the network throughput is halved! I get 45-49Mbits max.
Host doesn’t support PCI pass through, so this is my config for the libvirtd VM:
Nic1 (wan)
Network source: Direct ‘eth0’
Source mode: passthrough
Device model: virtio
Nic2 (lan)
Bridge name: br0
Device model: virtio
I tried e1000 instead but it changes absolutely nothing.
Host CPU: AMD A4-5000 Kabini
Guest CPU: default or Opteron_G3
This has been so since over a year now, since I started using KVM. If I do not solve this problem, I will have to dump libvirt because such performance is unacceptable.
It is pretty hard to diagnose these sort of problems with such limited information. Definitely don't use e1000 or any other NIC model - virtio-net will offer the best performance of any virtualized NIC. Make sure the host has /dev/vhost-net available as that accelerates guest NIC traffic in host kernel space.
If you want to use a guest as a high performance network routing appliance though, there's quite a few ways to tune it the VM in general. Pinning the guest vCPUs to specific host physical CPUs, and keeping other guests off these CPUs ensures the guest won't get its cache trashed by being pre-empted by other processes. Next up, use huge pages for the guest RAM to massively increase the TLB cache hit rate for guest memory access. If the host has multiple NUMA nodes, makes sure the guest CPU and guest RAM (hugepages) are fixed to come from the same host NUMA node. Similarly ensure IRQ handling for the host NIC used by the guest has affinity set to match the pCPUs used by the guest.
Related
I am using QEMU to virtualize KVM VPS. I have already turned off all the things like gso,tso,tx,rx at the host's network interface and my VPS use VirtIO as the NIC. When I do speedtest at the host, usually a result of approximately 800 Mbps downlink and 600 Mbps uplink. However, when I do the same test inside the VPS, only 300/200 Mbps can be obtained, as if something is limiting the speed to 300 Mbps. However after I check everything, I did not find the cause of the poor network performance.
Are there any way to further improve the network performance in the KVM VPS? My host is using double Xeon E5530 (8 Core 16 Threads) and has 64 GiB of physical memory and approximately 100 VPS (mostly 256 MiB Memory/1 Core) are running on it. Average load of the host is about 3.0. Both the host and the VPS is using the same NIC on the host and network bridge is correctly set up.
I was facing some problem in a Xen to KVM migration and studing the situation we got the next conclusions.
1.- Our best performance was obtained inserting some new NICs to the server and assigning a pci device to a VPS.
You will get same performance as it was not virtualized.
Problems:
You need a VPS linked to a new external NIC controller.Pci passthrough.
You need one network controller for each port you want to configure. Search iommu information.
Forget live migrations between hosts with assigned pci's.
2.- Using virtio drivers and performance tunning.
We got better performance but impossible to compare with a pci-passthrough.
There are some researches from KVM people who say that they reach great performance, I can't say it is not true but I couldn't replicate that performance.
http://www.linux-kvm.org/page/Using_VirtIO_NIC
Tunning:
Following the next guide you can find some tips to get the best performance.
We noticed an important improve with Multi-Queue virtio-net approach but I guess It won't be useful for you if your VPS is using just one core.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/chap-Virtualization_Tuning_Optimization_Guide-Networking.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-General_Tips
While creating a KVM Virtual Machine in proxmox from the GUI, for the hard disk and CPU tab there are a couple of options that are confusing.
For example, this is the hard disk tab,
In that tab, what does "No backup", "Discard" and "Iothread" signify?
And similarly, this is the CPU tab,
In this tab, what does "Sockets", "Cores" and "Enable numa" mean?
I did not have any luck with google and the results that I got were conflicting.
No backup instructs proxmox to not perform any backups for that VM.
Discard allows the guest to use fstrim or the discard option to free up the unused space from the underlying storage system. This is only usable on a virtio_scsi driver.
Iothread sets the AIO mode to threads (instead of native). For more information check this presentation.
Sockets is the number of CPUs that the guest will see as installed/available.
Cores is the number of cores that the guest will be able to use for each CPU.
If your server has 2 sockets each with a CPU that has 6 cores you could put 2 in the Sockets and 6 in the Cores fields and the guest will be able to use 100% of both CPU's. You can also put 1 in Sockets and 3 in Cores fields and the guest will be able to use only 50% from one CPU, so only 25% of the available CPU power on the server.
Enable Numa will allow the guest to make use of the NUMA architecture (servers with more than 2 sockets) on specific servers. Check www.admin-magazine.com/Archive/2014/20/Best-practices-for-KVM-on-NUMA-servers
I have successfully installed RedHat Linux and run them on harddrive using VMware simulation. Things work quite smooth if I put all the nodes VM on my physical machine.
For management purposes, I want to use USB devices to store ISO and plug one if more nodes are needed. I would like to run VMware on my physical machines.
Can I just build one virtual machine on one USB device? So I can plug one node if needed.
I mean, if I simulate machine A one USB 1 and another machine B on USB 2, can I build a network using my physical machine as server?
(1) If so, are there problems I should pay attention to?
(2) If not, are there any alternative solution for my management purpose?(I do not want to make VMs on partitions of my physical machine now) Can I use multiple mobile hard drives instead?
Actually I want to start up master-slaves Hadoop2.x deployments using virtual machines. Are there any good reference for this purpose?
I shall explain that am not too lazy to have a try on my idea, however, it is now rather expensive to do so if I do not even know something about the feasibility of this solution.
Thanks for your time.
I'm not an expert on VMWare, but I know that this is common on almost any virtualization system. You can install a system :
on a physical device (a hard disk, a hard disk partition)
or on a file
The physical device way allows normally better performances since you only use one driver between the OS and the device, while the file way offer greater simplicity to add one VM.
Now for your questions :
Can I just build one virtual machine on one USB device? Yes, you can always do it on a file, and depending on host OS directly on the physical device
... can I build a network using my physical machine as server? Yes, VMWare will allow the VM to communicate with each other and/or with the host and/or with external world depending on how you configure the network interfaces of your VMs.
If so, are there problems I should pay attention to?
USB devices are pluggable and unpluggable. If you unadvertantly unplug one while the OS is active bad things could happen. That's why I advised you to use files on the hard disk to host your VMs.
memory sticks (no concern for USB disks) support a limited number of writes and generally perform poorly on writes. Never put temp filesystem of swap there but use a memory filesystem for that usage, as is done for live filesystems on read-only CD or DVD
every VMs uses memory from the host system. That is often the first limitation for the number of simultaneous VMs on a personnal system
How does xen handle E820 memory map for domU kernels? In my specific problem I am trying to map Non-Volatile RAM to domU kernels. The dom0 memory map returns the following relevant parts.
100000000-17fffffff : System RAM (4GB to 6GB)
180000000-37fffffff : reserved (6GB to 14GB)
The second line corrosponds to the NVRAM which is a region from 6GB to 14GB in the dom0 kernel. How can I map this NVRAM region to the domU kernel which does not map this region at all.
Ultimately I want to the nvram region to be available in other domU VMs so any solutions or advice would be highly helpful.
P.S. :: If I attempt to write to this region from the domU kernel will Xen intercept this write operation. Actually this is just a memory region write which should not be a problem, but it might appear as a hardware access.
Guest domains in Xen have two different models for x86:
1. Hardware Virtual Machine (HVM) : It takes benefit of Intel VT or AMD SVM extensions to enable true virtualization on x86 platform
2. Paravirtualized (PV) : This mode adds modifications in the source code of the operating system to get rid of the x86 virtualization problems and also add performance boost to the system.
These two different models handle the E820 memory map differently. E820 memory map basically gives an OS the physical address space to operate on along with the location of I/O devices. In PV mode I/O devices are available through Xenstore. The domain builder only provides a console device during boot to the pv guest. All other I/O devices have to be mapped by the guest. The guest in this mode starts execution in protected mode instead of real mode for x86. The domain builder maps the start_info pages into the guest domain's physical address space. This start_info pages contain most of the information to initialize a kernel such as number of available pages, number of CPUs, console information, Xenstore etc. E820 memory map in this context would just consist of the number of available memory pages because BIOS is not emulated and I/O device information is provided separately through Xenstore.
On the otherhand, in HVM guest BIOS and other devices have to be emulated by Xen. This mode should support any unmodified OS, thus we cannot use the previous method. BIOS emulation is done via code borrowed from Bochs, while devices are emulated using QEMU code. Here an OS is provided with an E820 memory map, build by the domain builder. The HVM domain builder would typically pass the memory layout information to the Bochs emulator which then performs the required task.
To get hold of the NVRAM pages you will have to build a separate MMU for NVRAM. This MMU should handle all the NVM pages and allocate/free it on demand just like the RAM pages. It is a lot of work.
I've heared that hosted VMs make I/O virtualization easier. Because no device driver is needed for the VM and it can use the host OS device drivers for devices. My question is if the device driver is installed on the guest OS why do we need to use device drivers of the host OS?
Thanks in advance! :D
Depending of your virtualization solution some devices of the virtual machine are virtual (for example, hard disk) while other not (for example processor), for the virtualizated devices, they're generic enought so they may not need drivers (but VM tools contains drivers that make them perform better), but at the end Virtual Machines are running on top of the host operating system, so this operating system, need to have drivers for all the real hardware, to be able to provide that hardware to the virtual machine.