vhost_net: a kernel-level virtio server
in [Kernel]
|
Prev: bfa: Brocade BFA FC SCSI driver (bfad)
Next: Atheros Linux wireless drivers home page - and two new driver projects
From: Gregory Haskins on 11 Sep 2009 12:10 Ira W. Snyder wrote: > On Mon, Sep 07, 2009 at 01:15:37PM +0300, Michael S. Tsirkin wrote: >> On Thu, Sep 03, 2009 at 11:39:45AM -0700, Ira W. Snyder wrote: >>> On Thu, Aug 27, 2009 at 07:07:50PM +0300, Michael S. Tsirkin wrote: >>>> What it is: vhost net is a character device that can be used to reduce >>>> the number of system calls involved in virtio networking. >>>> Existing virtio net code is used in the guest without modification. >>>> >>>> There's similarity with vringfd, with some differences and reduced scope >>>> - uses eventfd for signalling >>>> - structures can be moved around in memory at any time (good for migration) >>>> - support memory table and not just an offset (needed for kvm) >>>> >>>> common virtio related code has been put in a separate file vhost.c and >>>> can be made into a separate module if/when more backends appear. I used >>>> Rusty's lguest.c as the source for developing this part : this supplied >>>> me with witty comments I wouldn't be able to write myself. >>>> >>>> What it is not: vhost net is not a bus, and not a generic new system >>>> call. No assumptions are made on how guest performs hypercalls. >>>> Userspace hypervisors are supported as well as kvm. >>>> >>>> How it works: Basically, we connect virtio frontend (configured by >>>> userspace) to a backend. The backend could be a network device, or a >>>> tun-like device. In this version I only support raw socket as a backend, >>>> which can be bound to e.g. SR IOV, or to macvlan device. Backend is >>>> also configured by userspace, including vlan/mac etc. >>>> >>>> Status: >>>> This works for me, and I haven't see any crashes. >>>> I have done some light benchmarking (with v4), compared to userspace, I >>>> see improved latency (as I save up to 4 system calls per packet) but not >>>> bandwidth/CPU (as TSO and interrupt mitigation are not supported). For >>>> ping benchmark (where there's no TSO) troughput is also improved. >>>> >>>> Features that I plan to look at in the future: >>>> - tap support >>>> - TSO >>>> - interrupt mitigation >>>> - zero copy >>>> >>> Hello Michael, >>> >>> I've started looking at vhost with the intention of using it over PCI to >>> connect physical machines together. >>> >>> The part that I am struggling with the most is figuring out which parts >>> of the rings are in the host's memory, and which parts are in the >>> guest's memory. >> All rings are in guest's memory, to match existing virtio code. > > Ok, this makes sense. > >> vhost >> assumes that the memory space of the hypervisor userspace process covers >> the whole of guest memory. > > Is this necessary? Why? The assumption seems very wrong when you're > doing data transport between two physical systems via PCI. FWIW: VBUS handles this situation via the "memctx" abstraction. IOW, the memory is not assumed to be a userspace address. Rather, it is a memctx-specific address, which can be userspace, or any other type (including hardware, dma-engine, etc). As long as the memctx knows how to translate it, it will work. Kind Regards, -Greg
From: Gregory Haskins on 11 Sep 2009 12:20 Gregory Haskins wrote: [snip] > > FWIW: VBUS handles this situation via the "memctx" abstraction. IOW, > the memory is not assumed to be a userspace address. Rather, it is a > memctx-specific address, which can be userspace, or any other type > (including hardware, dma-engine, etc). As long as the memctx knows how > to translate it, it will work. > citations: Here is a packet import (from the perspective of the host side "venet" device model, similar to Michaels "vhost") http://git.kernel.org/?p=linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git;a=blob;f=kernel/vbus/devices/venet-tap.c;h=ee091c47f06e9bb8487a45e72d493273fe08329f;hb=ded8ce2005a85c174ba93ee26f8d67049ef11025#l535 Here is the KVM specific memctx: http://git.kernel.org/?p=linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git;a=blob;f=kernel/vbus/kvm.c;h=56e2c5682a7ca8432c159377b0f7389cf34cbc1b;hb=ded8ce2005a85c174ba93ee26f8d67049ef11025#l188 and http://git.kernel.org/?p=linux/kernel/git/ghaskins/alacrityvm/linux-2.6.git;a=blob;f=virt/kvm/xinterface.c;h=0cccb6095ca2a51bad01f7ba2137fdd9111b63d3;hb=ded8ce2005a85c174ba93ee26f8d67049ef11025#l289 You could alternatively define a memctx for your environment which knows how to deal with your PPC boards PCI based memory, and the devices would all "just work". Kind Regards, -Greg
From: Michael S. Tsirkin on 13 Sep 2009 01:50 On Fri, Sep 11, 2009 at 11:17:33PM +0800, Xin, Xiaohui wrote: > Michael, > We are very interested in your patch and want to have a try with it. > I have collected your 3 patches in kernel side and 4 patches in queue side. > The patches are listed here: > > PATCHv5-1-3-mm-export-use_mm-unuse_mm-to-modules.patch > PATCHv5-2-3-mm-reduce-atomic-use-on-use_mm-fast-path.patch > PATCHv5-3-3-vhost_net-a-kernel-level-virtio-server.patch > > PATCHv3-1-4-qemu-kvm-move-virtio-pci[1].o-to-near-pci.o.patch > PATCHv3-2-4-virtio-move-features-to-an-inline-function.patch > PATCHv3-3-4-qemu-kvm-vhost-net-implementation.patch > PATCHv3-4-4-qemu-kvm-add-compat-eventfd.patch > > I applied the kernel patches on v2.6.31-rc4 and the qemu patches on latest kvm qemu. > But seems there are some patches are needed at least irqfd and ioeventfd patches on > current qemu. I cannot create a kvm guest with "-net nic,model=virtio,vhost=vethX". > > May you kindly advice us the patch lists all exactly to make it work? > Thanks a lot. :-) > > Thanks > Xiaohui The irqfd/ioeventfd patches are part of Avi's kvm.git tree: git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git I expect them to be merged by 2.6.32-rc1 - right, Avi? -- MST -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Michael S. Tsirkin on 13 Sep 2009 08:10 On Fri, Sep 11, 2009 at 12:00:21PM -0400, Gregory Haskins wrote: > FWIW: VBUS handles this situation via the "memctx" abstraction. IOW, > the memory is not assumed to be a userspace address. Rather, it is a > memctx-specific address, which can be userspace, or any other type > (including hardware, dma-engine, etc). As long as the memctx knows how > to translate it, it will work. How would permissions be handled? it's easy to allow an app to pass in virtual addresses in its own address space. But we can't let the guest specify physical addresses. -- MST -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Xin, Xiaohui on 14 Sep 2009 02:00
>The irqfd/ioeventfd patches are part of Avi's kvm.git tree: >git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git > >I expect them to be merged by 2.6.32-rc1 - right, Avi? Michael, I think I have the kernel patch for kvm_irqfd and kvm_ioeventfd, but missed the qemu side patch for irqfd and ioeventfd. I met the compile error when I compiled virtio-pci.c file in qemu-kvm like this: /root/work/vmdq/vhost/qemu-kvm/hw/virtio-pci.c:384: error: `KVM_IRQFD` undeclared (first use in this function) /root/work/vmdq/vhost/qemu-kvm/hw/virtio-pci.c:400: error: `KVM_IOEVENTFD` undeclared (first use in this function) Which qemu tree or patch do you use for kvm_irqfd and kvm_ioeventfd? Thanks Xiaohui -----Original Message----- From: Michael S. Tsirkin [mailto:mst(a)redhat.com] Sent: Sunday, September 13, 2009 1:46 PM To: Xin, Xiaohui Cc: Ira W. Snyder; netdev(a)vger.kernel.org; virtualization(a)lists.linux-foundation.org; kvm(a)vger.kernel.org; linux-kernel(a)vger.kernel.org; mingo(a)elte.hu; linux-mm(a)kvack.org; akpm(a)linux-foundation.org; hpa(a)zytor.com; gregory.haskins(a)gmail.com; Rusty Russell; s.hetze(a)linux-ag.com; avi(a)redhat.com Subject: Re: [PATCHv5 3/3] vhost_net: a kernel-level virtio server On Fri, Sep 11, 2009 at 11:17:33PM +0800, Xin, Xiaohui wrote: > Michael, > We are very interested in your patch and want to have a try with it. > I have collected your 3 patches in kernel side and 4 patches in queue side. > The patches are listed here: > > PATCHv5-1-3-mm-export-use_mm-unuse_mm-to-modules.patch > PATCHv5-2-3-mm-reduce-atomic-use-on-use_mm-fast-path.patch > PATCHv5-3-3-vhost_net-a-kernel-level-virtio-server.patch > > PATCHv3-1-4-qemu-kvm-move-virtio-pci[1].o-to-near-pci.o.patch > PATCHv3-2-4-virtio-move-features-to-an-inline-function.patch > PATCHv3-3-4-qemu-kvm-vhost-net-implementation.patch > PATCHv3-4-4-qemu-kvm-add-compat-eventfd.patch > > I applied the kernel patches on v2.6.31-rc4 and the qemu patches on latest kvm qemu. > But seems there are some patches are needed at least irqfd and ioeventfd patches on > current qemu. I cannot create a kvm guest with "-net nic,model=virtio,vhost=vethX". > > May you kindly advice us the patch lists all exactly to make it work? > Thanks a lot. :-) > > Thanks > Xiaohui The irqfd/ioeventfd patches are part of Avi's kvm.git tree: git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git I expect them to be merged by 2.6.32-rc1 - right, Avi? -- MST -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |