KVM Macvtap vs bridging
2014-03-03

I’ve been using KVM based virtual machine as a web server for a while. The VM needs a public IP of course, so I used a bridged setup. Recently (2014-03-03), however, one of the other machines on my subnet generated some 3Mbps of traffic (to an external server). This caused my host machine CPU usage to rise to 40%!

I’m guessing this is because the bridged setup put eth0 in promiscuous mode, and the host kernel has to inspect each packet and decide whether to pass it to the VM guest or not. I switched my networking setup to macvtap, and got a huge performance improvement. The 3Mbps of traffic barely puts a dent on my CPU usage.

macvtap Setup

Open virt-manager and set the following:

Add Hardware --> Network
Host device ethXX : macvtap
Device Model: virtio
Source Mode: VEPA

Here ethXX is your host machines outgoing interface.

Communication with the host machine.

The only caveat with VEPA is that your host machine CAN NOT communicate to the guest machine over this interface (unless you have some special hair pin switch.) I worked around this by creating a second NAT interface for the guest, and assigning static IP’s as follows:

  1. Use virt-manager to create a second (NAT) interface for the guest. Call it default. Get the interface up and running in the guest.

  2. Run virsh net-autostart default to make sure the interface comes up automatically after reboots.

  3. Run virsh net-edit default. It will pop up an XML file with basic network configuration. Find the <dhcp> section, and below the <range start... /> line add the line

    <host mac='guestmac' name='guestname' ip='192.168.122.2' />
    
  4. Edit /etc/hosts on the host, and add the line

    192.168.122.2   guest.host.name
    
  5. Edit /etc/hosts on the guest, and add the line

    192.168.122.1   host.host.name
    

Now you should have transparent host/guest access on all machines. (Accessing the guest via an IP address still won’t work though; you’ll have to use host-names.)


 Comments

  • Network unreachable after upgrade to qemu-1.1.2+dfsg-6a+deb7u2
    GI (2014-05-03 15:49:47)

    After upgrading on 2014-05-02 and rebooting my system my precious macvtap was unreachable. On my host I could see the interface, but was unable to send or receive anything on it. I realized that after an upgrade of QEmu (not libvirt!) on the host, my host interface eth1 wasn’t being put in promiscuous mode. A temporary fix is to run

    ifconfig eth1 promisc
    

    after which everything should work.

    For a permanent solution, put the following in /etc/network/interfaces:

    iface eth1 inet dhcp
        up ifconfig $IFACE promisc
        # Your custom network commands
        # ...
        down ifconfig $IFACE -promisc
    

    Hopefully this bug should go away soon.


  • Traffic to guest extremely slow after upgrade to 1.1.2+dfsg-6+deb7u3
    GI (2014-06-18 17:31:13)

    After upgrading I found that the network on my host machine was dropping up to 20% of incoming packets. This had the effect of reducing the incoming speed to less than 100kbps, while the outgoing speed was a good 20mbps+.

    I found that setting the device model to virtio fixed it. (The device model was Default earlier.)


  • One more reason to use this instead of a bridge
    Francois (2014-07-16 22:49:51)

    Thanks for the nice and simple post. Another reason to use macvtap is that if a bridge is added in a corporate network you are most likely to face a shutdown of the physical network port. This happens if the bridge announces itself by sending BPDUs.


  • Typo?
    Vincenzo (2014-10-15 10:16:09)

    I fear it is “VEPA” ( virtual ethernet port aggregator) and not “VPEA”.


  • Re: Typo?
    GI (2014-10-15 13:59:16)

    Oops. Indeed; I fixed it.


  • lost at the beginning
    Stefan (2015-07-05 20:35:50)

    I’d like to work through this manual, but at the very beinning I already get lost. Perhaps it is due to the translation of virt-manager’s interface (v1.0.1 running on Netrunner v16 distro) :

    I can not find the place / menu to

    Add Hardware –> Network

    In the following I had to translate back - so please excuse the different terms:

    1. So if i right - klick on localhost (QEMU) … I can go on with New VM - wrong place
    2. Menu File - Add connection - wrong place
    3. Menu File - New virtual machine - see 1. - wrong place
    4. Edit Connection details seems promising due to this contains the tabs: “overview / Virtual Netzworks / Memory(or storage)/ LAN-Adapters (containing lo) 4a) when I add a LAN connection the only choices are: Bridge / Bond / Ethernet / VLAN Here I added eth0 and neither can make it a macvtap nor can I delete / remove this choice. Even not by calling sudo virt-manager

    So where might I add the eth0 macvtap?


  • Thanks for Useful info
    Himanshu (2017-10-22 14:52:21 EDT)

    Thanks for quick tip for tuning my KVM machine, facing similar issue i will try the same. - Thanks Himanshu Blogger @


 Leave a comment (Spammers beware: All comments are moderated)