Re: [microblaze-uclinux] Device drivers for a Linux kernel with caches but no MMU support

[Paul Hartke <phartke@xxxxxxxxxxxx>]

Quoting John Williams <jwilliams@itee.uq.edu.au>:
> Hi Paul,
>
> Paul Hartke wrote:
>
> > I'm trying to wrap my mind around writing device drivers for a Linux
> kernel
> > with caches but no MMU support.  Although Microblaze currently only
> offers
> > write-through caches, coherency with a DMA engine must be enforced by
> > invalidating any cache lines that the DMA engine could end up touching.
> > This coherency issue is generally bypassed by allocating DMA buffer
> > descriptors in non-cachable memory:
> >
> > drivers/net/xilinx_enet/adapter.c
> >         /* calc size of descriptor space pool; alloc from non-cached
> memory
> > */
> >         dftsize = (XEM_DFT_RECV_DESC + XEM_DFT_SEND_DESC) *
> >             sizeof (XBufDescriptor);
> >
> >         lp->desc_space = consistent_alloc(GFP_ATOMIC, dftsize,
> >                                           &lp->desc_space_handle);
> >         if (lp->desc_space == 0) {
> >                 return -1;
> >         }
> >         lp->desc_space_size = dftsize;
> >
> > However, how does Microblaze uclinux do this exactly without an MMU?
> One
> > solution is to have Microblaze itself have a chunk of noncached memory
> and
> > put the descriptors there.  But, how would the kernel know about this
> > memory?  In addition, most microblaze designs I've seen cache the whole
> > memory space.
> >
> > The other approach is to invalidate the desciptors every time they are
> > passed to the DMA engine but, at least for the case of the Xilinx
> Ethernet
> > drivers, means mucking with them.
> >
> > How are folks handling this?
>
> Microblaze has (or can have) faked support for consistent_alloc - you
> can see it in arch/microblaze/mm/consistent.c:
>
>
http://cvs.uclinux.org/cgi-bin/cvsweb.cgi/~checkout~/uClinux-2.4.x/arch/microblaze/mm/consistent.c?rev=1.2;content-type=text%2Fplain
>
> It's a simple trick - you define the bus mapping of the external memory
> controller to be twice as large as the actual physical memory it
> supports, yet make microblaze only cache the true physical range.  By
> twiddling a top-order address bit you can get uncached access to the
> memory region.  So, consistent_alloc() essentially does a normal
> kmalloc(), then twiddles the appropriate bit with the following code:
>
> #if CONFIG_XILINX_UNCACHED_SHADOW
> 	ret = (void *)((unsigned) ret | UNCACHED_SHADOW_MASK);
> #endif
>
> UNCACHED_SHADOW_MASK is defined in linux/include/asm-microblaze
> somewhere.
>
> consistent_free() checks to see if this shadow bit is set, if so it
> clears it and passes to pointer to kfree() to do the hard work.
>
> There is experimental support in the uClinux MLD/TCL to define this
> uncached shadow memory, and if enabled, it is supported by the
> consistent_alloc() function.
>
> It's a bit of a hack, but it works.  I've used it to avoid major hackery
> on the ethernet driver for SGDMA mode.  Otherwise, as you say, you must
> explicitly invalidate cachelines all over the place, it's messy and
> horribly slow.

Of course--I've done this myself!  The only gotcha is that this is not a
guaranteed feature of the EDK memory controllers but rather a fortuitous 
side effect.

Gameplan: Configure the HIGHADDR for the memory controller to be twice as
large as the available memory.  Configure the MB I/D cache mhs parameters
to cache only the first half of the memory region.  The result is that
accesses to the first half of the memory space will be cached and the
second half will not.  Now somehow this info gets into the kernel but I
don't see how.
I didn't notice any support of this in the designs on your website nor do I
see where in the uClinux-auto mld/tcl this happens.

Somehow this needs to get fed into BSB system-generation.  Maybe the xilinx
BSB folks would add a "Create Uncached Shadow memory space parameter" for
MB configuration and the above steps would be performed.  Probably this
would set a parameter for the standalone BSP in the BSB-generated MSS which
would clue the user to set the same parameter for the uClinux-auto BSP.

P.S. uClinux-auto is very, _very_ cool...

___________________________
microblaze-uclinux mailing list
microblaze-uclinux@itee.uq.edu.au
Project Home Page : http://www.itee.uq.edu.au/~jwilliams/mblaze-uclinux
Mailing List Archive : http://www.itee.uq.edu.au/~listarch/microblaze-uclinux/