Re: [microblaze-uclinux] Device drivers for a Linux kernel with cachesbut no MMU support

[John Williams <jwilliams@xxxxxxxxxxxxxx>]

Hi Paul,

Paul Hartke wrote:

> I'm trying to wrap my mind around writing device drivers for a Linux kernel
> with caches but no MMU support.  Although Microblaze currently only offers
> write-through caches, coherency with a DMA engine must be enforced by
> invalidating any cache lines that the DMA engine could end up touching. 
> This coherency issue is generally bypassed by allocating DMA buffer
> descriptors in non-cachable memory:
> 
> drivers/net/xilinx_enet/adapter.c
>         /* calc size of descriptor space pool; alloc from non-cached memory
> */
>         dftsize = (XEM_DFT_RECV_DESC + XEM_DFT_SEND_DESC) *
>             sizeof (XBufDescriptor);
> 
>         lp->desc_space = consistent_alloc(GFP_ATOMIC, dftsize,
>                                           &lp->desc_space_handle);
>         if (lp->desc_space == 0) {
>                 return -1;
>         }
>         lp->desc_space_size = dftsize;
> 
> However, how does Microblaze uclinux do this exactly without an MMU?  One
> solution is to have Microblaze itself have a chunk of noncached memory and
> put the descriptors there.  But, how would the kernel know about this
> memory?  In addition, most microblaze designs I've seen cache the whole
> memory space.
> 
> The other approach is to invalidate the desciptors every time they are
> passed to the DMA engine but, at least for the case of the Xilinx Ethernet
> drivers, means mucking with them.
> 
> How are folks handling this?

Microblaze has (or can have) faked support for consistent_alloc - you 
can see it in arch/microblaze/mm/consistent.c:

http://cvs.uclinux.org/cgi-bin/cvsweb.cgi/~checkout~/uClinux-2.4.x/arch/microblaze/mm/consistent.c?rev=1.2;content-type=text%2Fplain

It's a simple trick - you define the bus mapping of the external memory 
controller to be twice as large as the actual physical memory it 
supports, yet make microblaze only cache the true physical range.  By 
twiddling a top-order address bit you can get uncached access to the 
memory region.  So, consistent_alloc() essentially does a normal 
kmalloc(), then twiddles the appropriate bit with the following code:

#if CONFIG_XILINX_UNCACHED_SHADOW
	ret = (void *)((unsigned) ret | UNCACHED_SHADOW_MASK);
#endif

UNCACHED_SHADOW_MASK is defined in linux/include/asm-microblaze somewhere.

consistent_free() checks to see if this shadow bit is set, if so it 
clears it and passes to pointer to kfree() to do the hard work.

There is experimental support in the uClinux MLD/TCL to define this 
uncached shadow memory, and if enabled, it is supported by the 
consistent_alloc() function.

It's a bit of a hack, but it works.  I've used it to avoid major hackery 
on the ethernet driver for SGDMA mode.  Otherwise, as you say, you must 
explicitly invalidate cachelines all over the place, it's messy and 
horribly slow.

Regards,

John


___________________________
microblaze-uclinux mailing list
microblaze-uclinux@itee.uq.edu.au
Project Home Page : http://www.itee.uq.edu.au/~jwilliams/mblaze-uclinux
Mailing List Archive : http://www.itee.uq.edu.au/~listarch/microblaze-uclinux/