USB transfer_buffer allocations on 64bit systems
in [Kernel]
|
Prev: mfd: Initialise WM831x IRQ masks on chip even if interrupts not in use
Next: INFO: task apache2:7601 blocked for more than 120 seconds.
From: Daniel Mack on 12 Apr 2010 09:00 On Mon, Apr 12, 2010 at 02:47:08PM +0200, Andi Kleen wrote: > On Mon, Apr 12, 2010 at 02:32:38PM +0200, Daniel Mack wrote: > > Another detail I can't explain is that on his machine, the kernel oopses > > when kmalloc() with GFP_DMA32 is used. The patch to try this also only > > touched the allocation in sound/usb/caiaq/audio.c. > > Where did it oops? Pedro sent me an image: http://caiaq.de/download/tmp/GFP_DMA32.JPG The patch I sent him for testing and that lead to this Oops was: > diff --git a/sound/usb/caiaq/audio.c b/sound/usb/caiaq/audio.c > index 4328cad..26013be 100644 > --- a/sound/usb/caiaq/audio.c > +++ b/sound/usb/caiaq/audio.c > @@ -560,7 +560,7 @@ static struct urb **alloc_urbs(struct snd_usb_caiaqdev *dev, int dir, int *ret) > � � � � � � � �} > > � � � � � � � �urbs[i]->transfer_buffer = > - � � � � � � � � � � � kmalloc(FRAMES_PER_URB * BYTES_PER_FRAME, GFP_KERNEL); > + � � � � � � � � � � � kmalloc(FRAMES_PER_URB * BYTES_PER_FRAME, GFP_KERNEL | GFP_DMA32); > � � � � � � � �if (!urbs[i]->transfer_buffer) { > � � � � � � � � � � � �log("unable to kmalloc() transfer buffer, OOM!?\n"); > � � � � � � � � � � � �*ret = -ENOMEM; > > > http://caiaq.de/download/tmp/pedro-dmesg > > The system seems to set up the soft iotlb correctly. > > [ 0.468472] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) > [ 0.468539] Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000 > [ 0.468610] software IO TLB at phys 0x20000000 - 0x24000000 > > Also if that was wrong a lot more things would go wrong. > > I would suspect the driver. Are you sure: > > - it sets up it's dma_masks correctly? > - it uses pci_map_single/sg correctly for all transferred data? Well, the sound driver itself doesn't care for any of those things, just like any other USB driver doesn't. The USB core itself of the host controller driver should do, and as far as I can see, it does that, yes. Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andi Kleen on 12 Apr 2010 11:50 > > � � � � � � � �urbs[i]->transfer_buffer = > > - � � � � � � � � � � � kmalloc(FRAMES_PER_URB * BYTES_PER_FRAME, GFP_KERNEL); > > + � � � � � � � � � � � kmalloc(FRAMES_PER_URB * BYTES_PER_FRAME, GFP_KERNEL | GFP_DMA32); Ah you can't use GFP_DMA32 with kmalloc, only GFP_DMA. Actually there should be a WARN_ON for this when slab debugging is enabled. Slab needs separate caches for dma, and it only has them for GFP_DMA, but not DMA32. Use __get_free_pages() for GFP_DMA32 > Well, the sound driver itself doesn't care for any of those things, just > like any other USB driver doesn't. The USB core itself of the host > controller driver should do, and as far as I can see, it does that, yes. Hmm, still things must go wrong somewhere. Perhaps need some instrumentation to see if all the transfer buffers really hit the PCI mapping functions. It might be interesting to test if the device works with enabled IOMMU. That would trigger any failures to properly map the buffers earlier. -Andi -- ak(a)linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Daniel Mack on 12 Apr 2010 13:20 On Mon, Apr 12, 2010 at 12:57:16PM -0400, Alan Stern wrote: > On Mon, 12 Apr 2010, Andi Kleen wrote: > > Hmm, thanks. But things must still go wrong somewhere, otherwise > > the GFP_DMA32 wouldn't be needed? > > Indeed, something must go wrong somewhere. Since Daniel's patch fixed > the problem by changing the buffer from a streaming mapping to a > coherent mapping, it's logical to assume that bad DMA addresses have > something to do with it. But we don't really know for certain. Given that - at least for non-64-aware host controllers - we want memory <4GB anyway for USB transfers to avoid DMA bouncing buffers, maybe we should just do that and fix the problem at this level? I already started to implement usb_[mz]alloc() and use it in some USB drivers. But even after all collected wisdom about memory management in this thread, I'm still uncertain of how to get suitable memory. Using dma_alloc_coherent() seems overdone as that type of memory is not necessarily needed and might be a costly good on some platforms. And as fas as I understand, kmalloc(GFP_DMA) does not avoid memory >4GB. Can anyone explain which is the right way to go? Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andi Kleen on 12 Apr 2010 13:30 On Mon, Apr 12, 2010 at 07:15:07PM +0200, Daniel Mack wrote: > On Mon, Apr 12, 2010 at 12:57:16PM -0400, Alan Stern wrote: > > On Mon, 12 Apr 2010, Andi Kleen wrote: > > > Hmm, thanks. But things must still go wrong somewhere, otherwise > > > the GFP_DMA32 wouldn't be needed? > > > > Indeed, something must go wrong somewhere. Since Daniel's patch fixed > > the problem by changing the buffer from a streaming mapping to a > > coherent mapping, it's logical to assume that bad DMA addresses have > > something to do with it. But we don't really know for certain. > > Given that - at least for non-64-aware host controllers - we want memory > <4GB anyway for USB transfers to avoid DMA bouncing buffers, maybe we > should just do that and fix the problem at this level? I already started > to implement usb_[mz]alloc() and use it in some USB drivers. If the area is not mapped correctly it will fail in other situations, e.g. with an IOMMU active or in virtualized setups. So the bug has to be eventually tracked down. > > But even after all collected wisdom about memory management in this > thread, I'm still uncertain of how to get suitable memory. Using > dma_alloc_coherent() seems overdone as that type of memory is not > necessarily needed and might be a costly good on some platforms. And as > fas as I understand, kmalloc(GFP_DMA) does not avoid memory >4GB. It does actually, but it only has 16MB to play with. Don't use it. If anything use __get_free_pages(GFP_DMA32), but it's a x86-64 specific code. > Can anyone explain which is the right way to go? The right thing would be to define a proper interface for it. I had an attempt for it a couple of years ago with the mask allocator, but it didn't make it into the tree. -Andi -- ak(a)linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Konrad Rzeszutek Wilk on 12 Apr 2010 14:10
On Mon, Apr 12, 2010 at 07:15:07PM +0200, Daniel Mack wrote: > On Mon, Apr 12, 2010 at 12:57:16PM -0400, Alan Stern wrote: > > On Mon, 12 Apr 2010, Andi Kleen wrote: > > > Hmm, thanks. But things must still go wrong somewhere, otherwise > > > the GFP_DMA32 wouldn't be needed? > > > > Indeed, something must go wrong somewhere. Since Daniel's patch fixed > > the problem by changing the buffer from a streaming mapping to a > > coherent mapping, it's logical to assume that bad DMA addresses have > > something to do with it. But we don't really know for certain. > > Given that - at least for non-64-aware host controllers - we want memory > <4GB anyway for USB transfers to avoid DMA bouncing buffers, maybe we > should just do that and fix the problem at this level? I already started > to implement usb_[mz]alloc() and use it in some USB drivers. You might want to run some benchmarks first to see if it is such a problem. Keep in mind that you would be addressing only the host-side of this: all DMA transfers from the USB controller to the memory. But for any transfer from the user space to the USB device you can't make the <4GB assumption as the stack/heap in the user-land is stiched from various memory areas - some of them above your 4GB mark. So when you write your response to this e-mail, and your /var/spool/clientmqueue is on your USB disk, the page with your response that is being written to the disk, can be allocated from a page above the 4GB mark and then has to be bounced-buffered for the USB controller. Note, I am only talking about 64-bit kernels, the 32-bit are a different beast altogether when it comes to Thought please keep in mind that this issue of bounce-buffer is less of a problem nowadays. Both AMD and Intel are outfitting their machines with hardware IOMMU's that replace the SWIOTLB (and IBM's high-end boxes with the Calgary ones). And on AMD the GART has been used for many years as a poor-man IOMMU. > > But even after all collected wisdom about memory management in this > thread, I'm still uncertain of how to get suitable memory. Using > dma_alloc_coherent() seems overdone as that type of memory is not > necessarily needed and might be a costly good on some platforms. And as > fas as I understand, kmalloc(GFP_DMA) does not avoid memory >4GB. > > Can anyone explain which is the right way to go? Fix whatever makes the DMA address have the wrong value. In the 0x08...00<bus address> address the 0x08 looks quite suspicious. Like it has been used as a flag or the generated casting code (by GCC) from 64-bit to 32-bit didn't get the right thing (I remember seeing this with InfiniBand with RHEL5.. which was GCC 4.1 I think?) It would be worth instrumenting the PCI-DMA API code and trigger a dump_stack when that flag (0x008) is detected in the return from the underlaying page mapping code. If you need help with this I can give you some debug patches. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |