ad7877: keep dma rx buffers in seperate cache lines
in [Kernel]
|
From: Mike Frysinger on 6 May 2010 14:50 On Thu, May 6, 2010 at 06:37, Oskar Schirmer wrote: >  struct ser_req { > +    u16           sample; > +    char           __padalign[L1_CACHE_BYTES - sizeof(u16)]; > + >     u16           reset; >     u16           ref_on; >     u16           command; > -    u16           sample; >     struct spi_message    msg; >     struct spi_transfer   xfer[6]; >  }; are you sure this is necessary ? ser_req is only ever used with spi_sync() and it's allocated/released on the fly, so how could anything be reading that memory between the start of the transmission and the return to adi7877 ? >  struct ad7877 { > +    u16           conversion_data[AD7877_NR_SENSE]; > +    char           __padalign[L1_CACHE_BYTES > +                    - AD7877_NR_SENSE * sizeof(u16)]; > + >     struct input_dev     *input; >     char           phys[32]; > > @@ -182,8 +188,6 @@ struct ad7877 { >     u8            averaging; >     u8            pen_down_acc_interval; > > -    u16           conversion_data[AD7877_NR_SENSE]; > - >     struct spi_transfer   xfer[AD7877_NR_SENSE + 2]; >     struct spi_message    msg; i can see the spi_message inside of this struct being a problem because the spi transfer is doing asynchronously with spi_async(). however, i would add a comment right above these two fields with a short explanation as to why they're at the start and why the pad exists so someone down the line doesnt move it. -mike
From: Oskar Schirmer on 7 May 2010 06:20 On Thu, May 06, 2010 at 14:46:04 -0400, Mike Frysinger wrote: > On Thu, May 6, 2010 at 06:37, Oskar Schirmer wrote: > > struct ser_req { > > + u16 sample; > > + char __padalign[L1_CACHE_BYTES - sizeof(u16)]; > > + > > u16 reset; > > u16 ref_on; > > u16 command; > > - u16 sample; > > struct spi_message msg; > > struct spi_transfer xfer[6]; > > }; > > are you sure this is necessary ? ser_req is only ever used with > spi_sync() and it's allocated/released on the fly, so how could > anything be reading that memory between the start of the transmission > and the return to adi7877 ? msg is handed over to spi_sync, it contains the addresses which will be used to programme the DMA: the spi master transfer function will read these fields to start DMA. E.g., drivers/spi/atmel_spi.c, assures cache coherency in function atmel_spi_dma_map_xfer, one xfer at a time. Multiple transfers are worked on in a loop in atmel_spi_transfer, so when coherency for transfer #1 is ensured, addresses for transfer #2 are read from the msg and xfer structs, caching lines which have been just before invalidated. And, citing Documentation/DMA-API.txt, Part Id: "mapped region must begin exactly on a cache line boundary and end exactly on one", which is our case. > > > struct ad7877 { > > + u16 conversion_data[AD7877_NR_SENSE]; > > + char __padalign[L1_CACHE_BYTES > > + - AD7877_NR_SENSE * sizeof(u16)]; > > + > > struct input_dev *input; > > char phys[32]; > > > > @@ -182,8 +188,6 @@ struct ad7877 { > > u8 averaging; > > u8 pen_down_acc_interval; > > > > - u16 conversion_data[AD7877_NR_SENSE]; > > - > > struct spi_transfer xfer[AD7877_NR_SENSE + 2]; > > struct spi_message msg; > > i can see the spi_message inside of this struct being a problem > because the spi transfer is doing asynchronously with spi_async(). > however, i would add a comment right above these two fields with a > short explanation as to why they're at the start and why the pad > exists so someone down the line doesnt move it. The code says "pad to align according to L1 cache, and keep away other stuff by exactly the amount so it is off the line". I'ld guess a comment would repeat just this, so it is superfluous. But if opinions differ on this topic, we can have a comment added, sure. Oskar -- oskar schirmer, emlix gmbh, http://www.emlix.com fon +49 551 30664-0, fax -11, bahnhofsallee 1b, 37081 göttingen, germany sitz der gesellschaft: göttingen, amtsgericht göttingen hr b 3160 geschäftsführer: dr. uwe kracke, ust-idnr.: de 205 198 055 emlix - your embedded linux partner -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Johannes Weiner on 7 May 2010 08:10 On Thu, May 06, 2010 at 02:46:04PM -0400, Mike Frysinger wrote: > On Thu, May 6, 2010 at 06:37, Oskar Schirmer wrote: > > �struct ser_req { > > + � � � u16 � � � � � � � � � � sample; > > + � � � char � � � � � � � � � �__padalign[L1_CACHE_BYTES - sizeof(u16)]; > > + > > � � � �u16 � � � � � � � � � � reset; > > � � � �u16 � � � � � � � � � � ref_on; > > � � � �u16 � � � � � � � � � � command; > > - � � � u16 � � � � � � � � � � sample; > > � � � �struct spi_message � � �msg; > > � � � �struct spi_transfer � � xfer[6]; > > �}; > > are you sure this is necessary ? ser_req is only ever used with > spi_sync() and it's allocated/released on the fly, so how could > anything be reading that memory between the start of the transmission > and the return to adi7877 ? The master driver can. atmel_spi flushes the cache of the buffers pretty early when queuing a message and touches the message members afterwards. > > �struct ad7877 { > > + � � � u16 � � � � � � � � � � conversion_data[AD7877_NR_SENSE]; > > + � � � char � � � � � � � � � �__padalign[L1_CACHE_BYTES > > + � � � � � � � � � � � � � � � � � � � - AD7877_NR_SENSE * sizeof(u16)]; > > + > > � � � �struct input_dev � � � �*input; > > � � � �char � � � � � � � � � �phys[32]; > > > > @@ -182,8 +188,6 @@ struct ad7877 { > > � � � �u8 � � � � � � � � � � �averaging; > > � � � �u8 � � � � � � � � � � �pen_down_acc_interval; > > > > - � � � u16 � � � � � � � � � � conversion_data[AD7877_NR_SENSE]; > > - > > � � � �struct spi_transfer � � xfer[AD7877_NR_SENSE + 2]; > > � � � �struct spi_message � � �msg; > > i can see the spi_message inside of this struct being a problem > because the spi transfer is doing asynchronously with spi_async(). > however, i would add a comment right above these two fields with a > short explanation as to why they're at the start and why the pad > exists so someone down the line doesnt move it. Good idea. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Mike Frysinger on 7 May 2010 14:30 On Fri, May 7, 2010 at 06:15, Oskar Schirmer wrote: > On Thu, May 06, 2010 at 14:46:04 -0400, Mike Frysinger wrote: >> On Thu, May 6, 2010 at 06:37, Oskar Schirmer wrote: >> > struct ser_req { >> > + u16 sample; >> > + char __padalign[L1_CACHE_BYTES - sizeof(u16)]; >> > + >> > u16 reset; >> > u16 ref_on; >> > u16 command; >> > - u16 sample; >> > struct spi_message msg; >> > struct spi_transfer xfer[6]; >> > }; >> >> are you sure this is necessary ? ser_req is only ever used with >> spi_sync() and it's allocated/released on the fly, so how could >> anything be reading that memory between the start of the transmission >> and the return to adi7877 ? > > msg is handed over to spi_sync, it contains the addresses > which will be used to programme the DMA: the spi master > transfer function will read these fields to start DMA. so the issue is coming from the SPI master drivers and not the AD7877 driver >> > struct ad7877 { >> > + u16 conversion_data[AD7877_NR_SENSE]; >> > + char __padalign[L1_CACHE_BYTES >> > + - AD7877_NR_SENSE * sizeof(u16)]; >> > + >> > struct input_dev *input; >> > char phys[32]; >> > >> > @@ -182,8 +188,6 @@ struct ad7877 { >> > u8 averaging; >> > u8 pen_down_acc_interval; >> > >> > - u16 conversion_data[AD7877_NR_SENSE]; >> > - >> > struct spi_transfer xfer[AD7877_NR_SENSE + 2]; >> > struct spi_message msg; >> >> i can see the spi_message inside of this struct being a problem >> because the spi transfer is doing asynchronously with spi_async(). >> however, i would add a comment right above these two fields with a >> short explanation as to why they're at the start and why the pad >> exists so someone down the line doesnt move it. > > The code says "pad to align according to L1 cache, and > keep away other stuff by exactly the amount so it is > off the line". I'ld guess a comment would repeat just > this, so it is superfluous. But if opinions differ on > this topic, we can have a comment added, sure. not everyone knows to read every single piece of documentation that may or may not be affected implicitly in the call stack. a simple comment here is not superfluous. since the other struct is also going to be changed, a comment should be placed there as well. -mike -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Johannes Weiner on 8 May 2010 18:40
On Fri, May 07, 2010 at 02:28:16PM -0400, Mike Frysinger wrote: > On Fri, May 7, 2010 at 06:15, Oskar Schirmer wrote: > > On Thu, May 06, 2010 at 14:46:04 -0400, Mike Frysinger wrote: > >> On Thu, May 6, 2010 at 06:37, Oskar Schirmer wrote: > >> > �struct ser_req { > >> > + � � � u16 � � � � � � � � � � sample; > >> > + � � � char � � � � � � � � � �__padalign[L1_CACHE_BYTES - sizeof(u16)]; > >> > + > >> > � � � �u16 � � � � � � � � � � reset; > >> > � � � �u16 � � � � � � � � � � ref_on; > >> > � � � �u16 � � � � � � � � � � command; > >> > - � � � u16 � � � � � � � � � � sample; > >> > � � � �struct spi_message � � �msg; > >> > � � � �struct spi_transfer � � xfer[6]; > >> > �}; > >> > >> are you sure this is necessary ? �ser_req is only ever used with > >> spi_sync() and it's allocated/released on the fly, so how could > >> anything be reading that memory between the start of the transmission > >> and the return to adi7877 ? > > > > msg is handed over to spi_sync, it contains the addresses > > which will be used to programme the DMA: the spi master > > transfer function will read these fields to start DMA. > > so the issue is coming from the SPI master drivers and not the AD7877 driver No, the issue is coming from ad7877 placing a transmission buffer into the same cache line with memory locations that are accessed outside the driver's scope. It must assume that the SPI driver does DMA, that cache coherency is maintained at cache line granularity, and must not make any assumptions about how the struct spi_message members are used. The following is about slave drivers from Documentation/spi/spi-summary: - Follow standard kernel rules, and provide DMA-safe buffers in your messages. That way controller drivers using DMA aren't forced to make extra copies unless the hardware requires it (e.g. working around hardware errata that force the use of bounce buffering). > >> > �struct ad7877 { > >> > + � � � u16 � � � � � � � � � � conversion_data[AD7877_NR_SENSE]; > >> > + � � � char � � � � � � � � � �__padalign[L1_CACHE_BYTES > >> > + � � � � � � � � � � � � � � � � � � � - AD7877_NR_SENSE * sizeof(u16)]; > >> > + > >> > � � � �struct input_dev � � � �*input; > >> > � � � �char � � � � � � � � � �phys[32]; > >> > > >> > @@ -182,8 +188,6 @@ struct ad7877 { > >> > � � � �u8 � � � � � � � � � � �averaging; > >> > � � � �u8 � � � � � � � � � � �pen_down_acc_interval; > >> > > >> > - � � � u16 � � � � � � � � � � conversion_data[AD7877_NR_SENSE]; > >> > - > >> > � � � �struct spi_transfer � � xfer[AD7877_NR_SENSE + 2]; > >> > � � � �struct spi_message � � �msg; > >> > >> i can see the spi_message inside of this struct being a problem > >> because the spi transfer is doing asynchronously with spi_async(). > >> however, i would add a comment right above these two fields with a > >> short explanation as to why they're at the start and why the pad > >> exists so someone down the line doesnt move it. > > > > The code says "pad to align according to L1 cache, and > > keep away other stuff by exactly the amount so it is > > off the line". I'ld guess a comment would repeat just > > this, so it is superfluous. But if opinions differ on > > this topic, we can have a comment added, sure. > > not everyone knows to read every single piece of documentation that > may or may not be affected implicitly in the call stack. a simple > comment here is not superfluous. > > since the other struct is also going to be changed, a comment should > be placed there as well. How about /* * DMA (thus cache coherency maintainance) requires the * transfer buffers to live in their own cache lines. */ char __padalign[...]; ? It might be obvious what the code does, but I agree with Mike that it might not be immediately apparent why it's needed. Hannes -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |