ad7877: keep dma rx buffers in seperate cache lines
in [Kernel]
|
Prev: cpuset,mm: fix no node to alloc memory when changing cpuset's mems
Next: tracing: Fix function declarations if !CONFIG_STACKTRACE
From: Marc Gauthier on 11 May 2010 22:10 Mike Frysinger wrote: > lets look at the cacheline sizes for arches that dont set > ARCH_KMALLOC_MINALIGN to L1_CACHE_BYTES: > - alplha - 32 or 64 > - frv - 32 or 64 > - blackfin - 32 > - parisc - 32 or 64 > - mn10300 - 16 > - s390 - 256 > - score - 16 > - sparc - 32 > - xtensa - 16 or 32 > > assuming alpha and s390 handle cache coherency in hardware, it looks > to me like the proposed assumption (kmalloc returns cachealigned > pointers when cache management is in software) does not hold true. > > so should these other arches also be setting ARCH_KMALLOC_MINALIGN to > L1_CACHE_BYTES ? IMHO, yes. It just makes sense to avoid false-sharing issues, not to allocate unrelated blocks in the same cache line. Also as it turns out (hope he doesn't me telling), Christian Zankel recently found a bug that was fixed exactly that way, by setting ARCH_KMALLOC_MINALIGN to L1_CACHE_BYTES for the Xtensa architecture. (Too recent to have percolated to mainline.) A lot of the above might be cache line aligned (?). -Marc -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Nick Piggin on 11 May 2010 23:10 On Tue, May 11, 2010 at 07:07:35PM -0700, Marc Gauthier wrote: > Mike Frysinger wrote: > > lets look at the cacheline sizes for arches that dont set > > ARCH_KMALLOC_MINALIGN to L1_CACHE_BYTES: > > - alplha - 32 or 64 > > - frv - 32 or 64 > > - blackfin - 32 > > - parisc - 32 or 64 > > - mn10300 - 16 > > - s390 - 256 > > - score - 16 > > - sparc - 32 > > - xtensa - 16 or 32 > > > > assuming alpha and s390 handle cache coherency in hardware, it looks > > to me like the proposed assumption (kmalloc returns cachealigned > > pointers when cache management is in software) does not hold true. > > > > so should these other arches also be setting ARCH_KMALLOC_MINALIGN to > > L1_CACHE_BYTES ? > > IMHO, yes. It just makes sense to avoid false-sharing issues, not to > allocate unrelated blocks in the same cache line. > > Also as it turns out (hope he doesn't me telling), Christian Zankel > recently found a bug that was fixed exactly that way, by setting > ARCH_KMALLOC_MINALIGN to L1_CACHE_BYTES for the Xtensa architecture. > (Too recent to have percolated to mainline.) > > A lot of the above might be cache line aligned (?). I don't think it's necessarily a good idea. MINALIGN is an enforced minimum alignment and the allocator has no leeway in reducing this. In a UP system, or in a memory constrained system, it might be a better idea to pack objects more tightly, for example. If we allow drivers to assume kmalloc is cacheline aligned, it will be (practically) impossible to revert this because it would require driver audits. So whenever strengthening API guarantees like this, it is better to be very careful and conservative. Probably even introducing a new API with the stronger semantics (even if it is just a wrapper in the case where KMALLOC_MINALIGNED *is* cacheline sized). I think adding to the DMA API would be a better idea. If the arch knows that kmalloc is suitable for the job directly, it can be used. Drivers can use the new interface, and kmalloc doesn't get saddled with alignment requirements. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: FUJITA Tomonori on 11 May 2010 23:30 On Wed, 12 May 2010 13:03:50 +1000 Nick Piggin <npiggin(a)suse.de> wrote: > On Tue, May 11, 2010 at 07:07:35PM -0700, Marc Gauthier wrote: > > Mike Frysinger wrote: > > > lets look at the cacheline sizes for arches that dont set > > > ARCH_KMALLOC_MINALIGN to L1_CACHE_BYTES: > > > - alplha - 32 or 64 > > > - frv - 32 or 64 > > > - blackfin - 32 > > > - parisc - 32 or 64 > > > - mn10300 - 16 > > > - s390 - 256 > > > - score - 16 > > > - sparc - 32 > > > - xtensa - 16 or 32 > > > > > > assuming alpha and s390 handle cache coherency in hardware, it looks > > > to me like the proposed assumption (kmalloc returns cachealigned > > > pointers when cache management is in software) does not hold true. > > > > > > so should these other arches also be setting ARCH_KMALLOC_MINALIGN to > > > L1_CACHE_BYTES ? > > > > IMHO, yes. It just makes sense to avoid false-sharing issues, not to > > allocate unrelated blocks in the same cache line. > > > > Also as it turns out (hope he doesn't me telling), Christian Zankel > > recently found a bug that was fixed exactly that way, by setting > > ARCH_KMALLOC_MINALIGN to L1_CACHE_BYTES for the Xtensa architecture. > > (Too recent to have percolated to mainline.) > > > > A lot of the above might be cache line aligned (?). > > I don't think it's necessarily a good idea. MINALIGN is an enforced > minimum alignment and the allocator has no leeway in reducing this. > In a UP system, or in a memory constrained system, it might be a better > idea to pack objects more tightly, for example. I agree, however... > If we allow drivers to assume kmalloc is cacheline aligned, it will be > (practically) impossible to revert this because it would require driver > audits. SCSI and some block drivers already depend on it. If kmalloc'ed buffer is not DMA safe, they breaks. Seems that kmalloc is not cacheline aligned on some architectures but they works. Probably, we might be just lucky because in general they allocate larger buffers than 64 for DMA via kmalloc and the buffers are aligned on the size? > So whenever strengthening API guarantees like this, it is better to be > very careful and conservative. Probably even introducing a new API with > the stronger semantics (even if it is just a wrapper in the case where > KMALLOC_MINALIGNED *is* cacheline sized). > > I think adding to the DMA API would be a better idea. If the arch knows > that kmalloc is suitable for the job directly, it can be used. Drivers > can use the new interface, and kmalloc doesn't get saddled with > alignment requirements. Or a new GFP flag? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Johannes Weiner on 12 May 2010 06:40 On Wed, May 12, 2010 at 12:35:45AM -0400, Mike Frysinger wrote: > On Tue, May 11, 2010 at 23:23, FUJITA Tomonori wrote: > > Seems that kmalloc is not cacheline aligned on some architectures but > > they works. Probably, we might be just lucky because in general they > > allocate larger buffers than 64 for DMA via kmalloc and the buffers > > are aligned on the size? > > i think the magic combo is: > - DMA buffer is written to (receive) Check. > - some driver state is in the same cacheline as the DMA buffer > - that driver state is used after the flush but before the DMA finishes The kmalloc caches are system-wide. Any other kmalloc(samesize) user could interfer when touching its object between dma_map_single() on the neighbor object and the end of the transfer. > - only on arches that need software cache coherency > > so i could see this not being an obvious issue for many people > -mike -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Dmitry Torokhov on 12 May 2010 14:20
On Wed, May 12, 2010 at 10:37:05AM -0400, Mike Frysinger wrote: > On Wed, May 12, 2010 at 01:28, FUJITA Tomonori wrote: > > I guess that it would be better to start a new thread about this issue. > > will do In the mean time I will be applying the version of the patch with ___cacheline_aligned and buffers at the end, 'k? -- Dmitry -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |