vhost_net: a kernel-level virtio server
in [Kernel]
|
Prev: bfa: Brocade BFA FC SCSI driver (bfad)
Next: Atheros Linux wireless drivers home page - and two new driver projects
From: Javier Guerra on 17 Sep 2009 10:20 On Wed, Sep 16, 2009 at 10:11 PM, Gregory Haskins <gregory.haskins(a)gmail.com> wrote: > It is certainly not a requirement to make said > chip somehow work with existing drivers/facilities on bare metal, per > se. Why should virtual systems be different? i'd guess it's an issue of support resources. a hardware developer creates a chip and immediately sells it, getting small but assured revenue, with it they write (or pays to write) drivers for a couple of releases, and stop to manufacture it as soon as it's not profitable. software has a much longer lifetime, especially at the platform-level (and KVM is a platform for a lot of us). also, being GPL, it's cheaper to produce but has (much!) more limited resources. creating a new support issue is a scary thought. -- Javier -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Ira W. Snyder on 21 Sep 2009 17:50 On Wed, Sep 16, 2009 at 11:11:57PM -0400, Gregory Haskins wrote: > Avi Kivity wrote: > > On 09/16/2009 10:22 PM, Gregory Haskins wrote: > >> Avi Kivity wrote: > >> > >>> On 09/16/2009 05:10 PM, Gregory Haskins wrote: > >>> > >>>>> If kvm can do it, others can. > >>>>> > >>>>> > >>>> The problem is that you seem to either hand-wave over details like > >>>> this, > >>>> or you give details that are pretty much exactly what vbus does > >>>> already. > >>>> My point is that I've already sat down and thought about these > >>>> issues > >>>> and solved them in a freely available GPL'ed software package. > >>>> > >>>> > >>> In the kernel. IMO that's the wrong place for it. > >>> > >> 3) "in-kernel": You can do something like virtio-net to vhost to > >> potentially meet some of the requirements, but not all. > >> > >> In order to fully meet (3), you would need to do some of that stuff you > >> mentioned in the last reply with muxing device-nr/reg-nr. In addition, > >> we need to have a facility for mapping eventfds and establishing a > >> signaling mechanism (like PIO+qid), etc. KVM does this with > >> IRQFD/IOEVENTFD, but we dont have KVM in this case so it needs to be > >> invented. > >> > > > > irqfd/eventfd is the abstraction layer, it doesn't need to be reabstracted. > > Not per se, but it needs to be interfaced. How do I register that > eventfd with the fastpath in Ira's rig? How do I signal the eventfd > (x86->ppc, and ppc->x86)? > Sorry to reply so late to this thread, I've been on vacation for the past week. If you'd like to continue in another thread, please start it and CC me. On the PPC, I've got a hardware "doorbell" register which generates 30 distiguishable interrupts over the PCI bus. I have outbound and inbound registers, which can be used to signal the "other side". I assume it isn't too much code to signal an eventfd in an interrupt handler. I haven't gotten to this point in the code yet. > To take it to the next level, how do I organize that mechanism so that > it works for more than one IO-stream (e.g. address the various queues > within ethernet or a different device like the console)? KVM has > IOEVENTFD and IRQFD managed with MSI and PIO. This new rig does not > have the luxury of an established IO paradigm. > > Is vbus the only way to implement a solution? No. But it is _a_ way, > and its one that was specifically designed to solve this very problem > (as well as others). > > (As an aside, note that you generally will want an abstraction on top of > irqfd/eventfd like shm-signal or virtqueues to do shared-memory based > event mitigation, but I digress. That is a separate topic). > > > > >> To meet performance, this stuff has to be in kernel and there has to be > >> a way to manage it. > > > > and management belongs in userspace. > > vbus does not dictate where the management must be. Its an extensible > framework, governed by what you plug into it (ala connectors and devices). > > For instance, the vbus-kvm connector in alacrityvm chooses to put DEVADD > and DEVDROP hotswap events into the interrupt stream, because they are > simple and we already needed the interrupt stream anyway for fast-path. > > As another example: venet chose to put ->call(MACQUERY) "config-space" > into its call namespace because its simple, and we already need > ->calls() for fastpath. It therefore exports an attribute to sysfs that > allows the management app to set it. > > I could likewise have designed the connector or device-model differently > as to keep the mac-address and hotswap-events somewhere else (QEMU/PCI > userspace) but this seems silly to me when they are so trivial, so I didn't. > > > > >> Since vbus was designed to do exactly that, this is > >> what I would advocate. You could also reinvent these concepts and put > >> your own mux and mapping code in place, in addition to all the other > >> stuff that vbus does. But I am not clear why anyone would want to. > >> > > > > Maybe they like their backward compatibility and Windows support. > > This is really not relevant to this thread, since we are talking about > Ira's hardware. But if you must bring this up, then I will reiterate > that you just design the connector to interface with QEMU+PCI and you > have that too if that was important to you. > > But on that topic: Since you could consider KVM a "motherboard > manufacturer" of sorts (it just happens to be virtual hardware), I don't > know why KVM seems to consider itself the only motherboard manufacturer > in the world that has to make everything look legacy. If a company like > ASUS wants to add some cutting edge IO controller/bus, they simply do > it. Pretty much every product release may contain a different array of > devices, many of which are not backwards compatible with any prior > silicon. The guy/gal installing Windows on that system may see a "?" in > device-manager until they load a driver that supports the new chip, and > subsequently it works. It is certainly not a requirement to make said > chip somehow work with existing drivers/facilities on bare metal, per > se. Why should virtual systems be different? > > So, yeah, the current design of the vbus-kvm connector means I have to > provide a driver. This is understood, and I have no problem with that. > > The only thing that I would agree has to be backwards compatible is the > BIOS/boot function. If you can't support running an image like the > Windows installer, you are hosed. If you can't use your ethernet until > you get a chance to install a driver after the install completes, its > just like most other systems in existence. IOW: It's not a big deal. > > For cases where the IO system is needed as part of the boot/install, you > provide BIOS and/or an install-disk support for it. > > > > >> So no, the kernel is not the wrong place for it. Its the _only_ place > >> for it. Otherwise, just use (1) and be done with it. > >> > >> > > > > I'm talking about the config stuff, not the data path. > > As stated above, where config stuff lives is a function of what you > interface to vbus. Data-path stuff must be in the kernel for > performance reasons, and this is what I was referring to. I think we > are generally both in agreement, here. > > What I was getting at is that you can't just hand-wave the datapath > stuff. We do fast path in KVM with IRQFD/IOEVENTFD+PIO, and we do > device discovery/addressing with PCI. Neither of those are available > here in Ira's case yet the general concepts are needed. Therefore, we > have to come up with something else. > > > > >>> Further, if we adopt > >>> vbus, if drop compatibility with existing guests or have to support both > >>> vbus and virtio-pci. > >>> > >> We already need to support both (at least to support Ira). virtio-pci > >> doesn't work here. Something else (vbus, or vbus-like) is needed. > >> > > > > virtio-ira. > > Sure, virtio-ira and he is on his own to make a bus-model under that, or > virtio-vbus + vbus-ira-connector to use the vbus framework. Either > model can work, I agree. > Yes, I'm having to create my own bus model, a-la lguest, virtio-pci, and virtio-s390. It isn't especially easy. I can steal lots of code from the lguest bus model, but sometimes it is good to generalize, especially after the fourth implemention or so. I think this is what GHaskins tried to do. Here is what I've implemented so far: * a generic virtio-phys-guest layer (my bus model, like lguest) - this runs on the crate server (x86) in my system * a generic virtio-phys-host layer (my /dev/lguest implementation) - this runs on the ppc boards in my system - this assumes that the kernel will allocate some memory and expose it over PCI in a device-specific way, so the guest can see it as a PCI BAR * a virtio-phys-mpc83xx driver - this runs on the crate server (x86) in my system - this interfaces virtio-phys-guest to my mpc83xx board - it is a Linux PCI driver, which detects mpc83xx boards, runs ioremap_pci_bar() on the correct PCI BAR, and then gives that to the virtio-phys-guest layer I think that the idea of device/driver (instead of host/guest) is a good one. It makes my problem easier to think about. I've given it some thought, and I think that running vhost-net (or similar) on the ppc boards, with virtio-net on the x86 crate server will work. The virtio-ring abstraction is almost good enough to work for this situation, but I had to re-invent it to work with my boards. I've exposed a 16K region of memory as PCI BAR1 from my ppc board. Remember that this is the "host" system. I used each 4K block as a "device descriptor" which contains: 1) the type of device, config space, etc. for virtio 2) the "desc" table (virtio memory descriptors, see virtio-ring) 3) the "avail" table (available entries in the desc table) Parts 2 and 3 are repeated three times, to allow for a maximum of three virtqueues per device. This is good enough for all current drivers. The guest side (x86 in my system) allocates some device-accessible memory, and writes the PCI address to the device descriptor. This memory contains: 1) the "used" table (consumed entries in the desc/avail tables) This exists three times as well, once for each virtqueue. The rest is basically a copy of virtio-ring, with a few changes to allow for cacheing, etc. It may not even be worth doing this from a performance standpoint, I haven't benchmarked it yet. For now, I'd be happy with a non-DMA memcpy only solution. I can add DMA once things are working. I've got the current code (subject to change at any time) available at the address listed below. If you think another format would be better for you, please ask, and I'll provide it. http://www.mmarray.org/~iws/virtio-phys/ I've gotten plenty of email about this from lots of interested developers. There are people who would like this kind of system to just work, while having to write just some glue for their device, just like a network driver. I hunch most people have created some proprietary mess that basically works, and left it at that. So, here is a desperate cry for help. I'd like to make this work, and I'd really like to see it in mainline. I'm trying to give back to the community from which I've taken plenty. Ira -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 22 Sep 2009 05:50 On 09/22/2009 12:43 AM, Ira W. Snyder wrote: > >> Sure, virtio-ira and he is on his own to make a bus-model under that, or >> virtio-vbus + vbus-ira-connector to use the vbus framework. Either >> model can work, I agree. >> >> > Yes, I'm having to create my own bus model, a-la lguest, virtio-pci, and > virtio-s390. It isn't especially easy. I can steal lots of code from the > lguest bus model, but sometimes it is good to generalize, especially > after the fourth implemention or so. I think this is what GHaskins tried > to do. > Yes. vbus is more finely layered so there is less code duplication. The virtio layering was more or less dictated by Xen which doesn't have shared memory (it uses grant references instead). As a matter of fact lguest, kvm/pci, and kvm/s390 all have shared memory, as you do, so that part is duplicated. It's probably possible to add a virtio-shmem.ko library that people who do have shared memory can reuse. > I've given it some thought, and I think that running vhost-net (or > similar) on the ppc boards, with virtio-net on the x86 crate server will > work. The virtio-ring abstraction is almost good enough to work for this > situation, but I had to re-invent it to work with my boards. > > I've exposed a 16K region of memory as PCI BAR1 from my ppc board. > Remember that this is the "host" system. I used each 4K block as a > "device descriptor" which contains: > > 1) the type of device, config space, etc. for virtio > 2) the "desc" table (virtio memory descriptors, see virtio-ring) > 3) the "avail" table (available entries in the desc table) > Won't access from x86 be slow to this memory (on the other hand, if you change it to main memory access from ppc will be slow... really depends on how your system is tuned. > Parts 2 and 3 are repeated three times, to allow for a maximum of three > virtqueues per device. This is good enough for all current drivers. > The plan is to switch to multiqueue soon. Will not affect you if your boards are uniprocessor or small smp. > I've gotten plenty of email about this from lots of interested > developers. There are people who would like this kind of system to just > work, while having to write just some glue for their device, just like a > network driver. I hunch most people have created some proprietary mess > that basically works, and left it at that. > So long as you keep the system-dependent features hookable or configurable, it should work. > So, here is a desperate cry for help. I'd like to make this work, and > I'd really like to see it in mainline. I'm trying to give back to the > community from which I've taken plenty. > Not sure who you're crying for help to. Once you get this working, post patches. If the patches are reasonably clean and don't impact performance for the main use case, and if you can show the need, I expect they'll be merged. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Ira W. Snyder on 22 Sep 2009 11:30 On Tue, Sep 22, 2009 at 12:43:36PM +0300, Avi Kivity wrote: > On 09/22/2009 12:43 AM, Ira W. Snyder wrote: > > > >> Sure, virtio-ira and he is on his own to make a bus-model under that, or > >> virtio-vbus + vbus-ira-connector to use the vbus framework. Either > >> model can work, I agree. > >> > >> > > Yes, I'm having to create my own bus model, a-la lguest, virtio-pci, and > > virtio-s390. It isn't especially easy. I can steal lots of code from the > > lguest bus model, but sometimes it is good to generalize, especially > > after the fourth implemention or so. I think this is what GHaskins tried > > to do. > > > > Yes. vbus is more finely layered so there is less code duplication. > > The virtio layering was more or less dictated by Xen which doesn't have > shared memory (it uses grant references instead). As a matter of fact > lguest, kvm/pci, and kvm/s390 all have shared memory, as you do, so that > part is duplicated. It's probably possible to add a virtio-shmem.ko > library that people who do have shared memory can reuse. > Seems like a nice benefit of vbus. > > I've given it some thought, and I think that running vhost-net (or > > similar) on the ppc boards, with virtio-net on the x86 crate server will > > work. The virtio-ring abstraction is almost good enough to work for this > > situation, but I had to re-invent it to work with my boards. > > > > I've exposed a 16K region of memory as PCI BAR1 from my ppc board. > > Remember that this is the "host" system. I used each 4K block as a > > "device descriptor" which contains: > > > > 1) the type of device, config space, etc. for virtio > > 2) the "desc" table (virtio memory descriptors, see virtio-ring) > > 3) the "avail" table (available entries in the desc table) > > > > Won't access from x86 be slow to this memory (on the other hand, if you > change it to main memory access from ppc will be slow... really depends > on how your system is tuned. > Writes across the bus are fast, reads across the bus are slow. These are just the descriptor tables for memory buffers, not the physical memory buffers themselves. These only need to be written by the guest (x86), and read by the host (ppc). The host never changes the tables, so we can cache a copy in the guest, for a fast detach_buf() implementation (see virtio-ring, which I'm copying the design from). The only accesses are writes across the PCI bus. There is never a need to do a read (except for slow-path configuration). > > Parts 2 and 3 are repeated three times, to allow for a maximum of three > > virtqueues per device. This is good enough for all current drivers. > > > > The plan is to switch to multiqueue soon. Will not affect you if your > boards are uniprocessor or small smp. > Everything I have is UP. I don't need extreme performance, either. 40MB/sec is the minimum I need to reach, though I'd like to have some headroom. For reference, using the CPU to handle data transfers, I get ~2MB/sec transfers. Using the DMA engine, I've hit about 60MB/sec with my "crossed-wires" virtio-net. > > I've gotten plenty of email about this from lots of interested > > developers. There are people who would like this kind of system to just > > work, while having to write just some glue for their device, just like a > > network driver. I hunch most people have created some proprietary mess > > that basically works, and left it at that. > > > > So long as you keep the system-dependent features hookable or > configurable, it should work. > > > So, here is a desperate cry for help. I'd like to make this work, and > > I'd really like to see it in mainline. I'm trying to give back to the > > community from which I've taken plenty. > > > > Not sure who you're crying for help to. Once you get this working, post > patches. If the patches are reasonably clean and don't impact > performance for the main use case, and if you can show the need, I > expect they'll be merged. > In the spirit of "post early and often", I'm making my code available, that's all. I'm asking anyone interested for some review, before I have to re-code this for about the fifth time now. I'm trying to avoid Haskins' situation, where he's invented and debugged a lot of new code, and then been told to do it completely differently. Yes, the code I posted is only compile-tested, because quite a lot of code (kernel and userspace) must be working before anything works at all. I hate to design the whole thing, then be told that something fundamental about it is wrong, and have to completely re-write it. Thanks for the comments, Ira > -- > error compiling committee.c: too many arguments to function > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 22 Sep 2009 12:00
On 09/22/2009 06:25 PM, Ira W. Snyder wrote: > >> Yes. vbus is more finely layered so there is less code duplication. >> >> The virtio layering was more or less dictated by Xen which doesn't have >> shared memory (it uses grant references instead). As a matter of fact >> lguest, kvm/pci, and kvm/s390 all have shared memory, as you do, so that >> part is duplicated. It's probably possible to add a virtio-shmem.ko >> library that people who do have shared memory can reuse. >> >> > Seems like a nice benefit of vbus. > Yes, it is. With some work virtio can gain that too (virtio-shmem.ko). >>> I've given it some thought, and I think that running vhost-net (or >>> similar) on the ppc boards, with virtio-net on the x86 crate server will >>> work. The virtio-ring abstraction is almost good enough to work for this >>> situation, but I had to re-invent it to work with my boards. >>> >>> I've exposed a 16K region of memory as PCI BAR1 from my ppc board. >>> Remember that this is the "host" system. I used each 4K block as a >>> "device descriptor" which contains: >>> >>> 1) the type of device, config space, etc. for virtio >>> 2) the "desc" table (virtio memory descriptors, see virtio-ring) >>> 3) the "avail" table (available entries in the desc table) >>> >>> >> Won't access from x86 be slow to this memory (on the other hand, if you >> change it to main memory access from ppc will be slow... really depends >> on how your system is tuned. >> >> > Writes across the bus are fast, reads across the bus are slow. These are > just the descriptor tables for memory buffers, not the physical memory > buffers themselves. > > These only need to be written by the guest (x86), and read by the host > (ppc). The host never changes the tables, so we can cache a copy in the > guest, for a fast detach_buf() implementation (see virtio-ring, which > I'm copying the design from). > > The only accesses are writes across the PCI bus. There is never a need > to do a read (except for slow-path configuration). > Okay, sounds like what you're doing it optimal then. > In the spirit of "post early and often", I'm making my code available, > that's all. I'm asking anyone interested for some review, before I have > to re-code this for about the fifth time now. I'm trying to avoid > Haskins' situation, where he's invented and debugged a lot of new code, > and then been told to do it completely differently. > > Yes, the code I posted is only compile-tested, because quite a lot of > code (kernel and userspace) must be working before anything works at > all. I hate to design the whole thing, then be told that something > fundamental about it is wrong, and have to completely re-write it. > Understood. Best to get a review from Rusty then. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |