From nobody@FreeBSD.org  Thu Nov  7 07:07:11 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTP id 64EB48C6
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  7 Nov 2013 07:07:11 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from oldred.freebsd.org (oldred.freebsd.org [8.8.178.121])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id 530812FBA
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  7 Nov 2013 07:07:11 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id rA777BDE084148
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 7 Nov 2013 07:07:11 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id rA777BvG084136;
	Thu, 7 Nov 2013 07:07:11 GMT
	(envelope-from nobody)
Message-Id: <201311070707.rA777BvG084136@oldred.freebsd.org>
Date: Thu, 7 Nov 2013 07:07:11 GMT
From: John Wehle <john@feith.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: mutex on some arm hardware requires dcache enabled
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         183740
>Category:       arm
>Synopsis:       mutex on some arm hardware requires dcache enabled
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-arm
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Nov 07 07:10:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Wed Nov 20 16:00:00 UTC 2013
>Originator:     John Wehle
>Release:        FreeBSD svn 257205
>Organization:
Personal
>Environment:
Not Yet
>Description:
In my spare time I'm working on bringing FreeBSD up on amlogic based
arm processors.  While working through various bootstrap issues I got
to the point where initarm calls pmap_bootstrap only to have it splat.

What happens is pmap_bootstrap calls pmap_extract which does PMAP_LOCK(pmap).
This trys to acquire a mutex which invokes atomics that use ldrex / strex.

The ARM documentation at:

  http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0464d/CHDIBJGE.html

notes that the L1 memory system has an internal exclusive monitor which
handles ldrex / strex accesses.

The README at:

  https://github.com/dwelch67/raspberrypi/tree/master/extest

has a nice write up which explains some of the wrinkles regarding hardware
and ldrex / strex.

The attached patch enables the dcache and suffices to get me through
initarm into mi_startup (I'm not to a console login yet ... however
I'm significantly further along).

-- John

>How-To-Repeat:

>Fix:
Enable the dcache prior to using a mutex.

Patch attached with submission follows:

--- sys/arm/arm/machdep.c.ORIGINAL	2013-10-27 01:15:39.000000000 -0400
+++ sys/arm/arm/machdep.c	2013-11-07 01:43:01.000000000 -0500
@@ -1481,6 +1483,12 @@ initarm(struct arm_boot_params *abp)
 	 */
 	cpu_idcache_wbinv_all();
 
+	/*
+	 * Atomics invoked by mutex use ldrex / strex which on some
+	 * hardware requires the L1 cache.
+	 */
+	cpu_control(CPU_CONTROL_DC_ENABLE, CPU_CONTROL_DC_ENABLE);
+
 	/* Set stack for exception handlers */
 	data_abort_handler_address = (u_int)data_abort_handler;
 	prefetch_abort_handler_address = (u_int)prefetch_abort_handler;


>Release-Note:
>Audit-Trail:

From: Ganbold Tsagaankhuu <ganbold@gmail.com>
To: John Wehle <john@feith.com>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: arm/183740: mutex on some arm hardware requires dcache enabled
Date: Thu, 7 Nov 2013 15:17:23 +0800

 --001a11c1feb69c14b604ea910ed6
 Content-Type: text/plain; charset=ISO-8859-1
 
 John,
 
 
 On Thu, Nov 7, 2013 at 3:07 PM, John Wehle <john@feith.com> wrote:
 
 >
 > >Number:         183740
 > >Category:       arm
 > >Synopsis:       mutex on some arm hardware requires dcache enabled
 > >Confidential:   no
 > >Severity:       non-critical
 > >Priority:       low
 > >Responsible:    freebsd-arm
 > >State:          open
 > >Quarter:
 > >Keywords:
 > >Date-Required:
 > >Class:          sw-bug
 > >Submitter-Id:   current-users
 > >Arrival-Date:   Thu Nov 07 07:10:00 UTC 2013
 > >Closed-Date:
 > >Last-Modified:
 > >Originator:     John Wehle
 > >Release:        FreeBSD svn 257205
 > >Organization:
 > Personal
 > >Environment:
 > Not Yet
 > >Description:
 > In my spare time I'm working on bringing FreeBSD up on amlogic based
 > arm processors.  While working through various bootstrap issues I got
 > to the point where initarm calls pmap_bootstrap only to have it splat.
 >
 > What happens is pmap_bootstrap calls pmap_extract which does
 > PMAP_LOCK(pmap).
 > This trys to acquire a mutex which invokes atomics that use ldrex / strex.
 >
 > The ARM documentation at:
 >
 >
 > http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0464d/CHDIBJGE.html
 >
 > notes that the L1 memory system has an internal exclusive monitor which
 > handles ldrex / strex accesses.
 >
 > The README at:
 >
 >   https://github.com/dwelch67/raspberrypi/tree/master/extest
 >
 > has a nice write up which explains some of the wrinkles regarding hardware
 > and ldrex / strex.
 >
 > The attached patch enables the dcache and suffices to get me through
 > initarm into mi_startup (I'm not to a console login yet ... however
 > I'm significantly further along).
 >
 > -- John
 >
 > >How-To-Repeat:
 >
 > >Fix:
 > Enable the dcache prior to using a mutex.
 >
 > Patch attached with submission follows:
 >
 > --- sys/arm/arm/machdep.c.ORIGINAL      2013-10-27 01:15:39.000000000 -0400
 > +++ sys/arm/arm/machdep.c       2013-11-07 01:43:01.000000000 -0500
 > @@ -1481,6 +1483,12 @@ initarm(struct arm_boot_params *abp)
 >          */
 >         cpu_idcache_wbinv_all();
 >
 > +       /*
 > +        * Atomics invoked by mutex use ldrex / strex which on some
 > +        * hardware requires the L1 cache.
 > +        */
 > +       cpu_control(CPU_CONTROL_DC_ENABLE, CPU_CONTROL_DC_ENABLE);
 > +
 >         /* Set stack for exception handlers */
 >         data_abort_handler_address = (u_int)data_abort_handler;
 >         prefetch_abort_handler_address = (u_int)prefetch_abort_handler;
 >
 >
 What kind of Amlogic board you are working on? That could be interesting.
 
 As for the patch can you try to put it in initarm_late_init() of your board
 machdep code and try?
 I meant like in /usr/src/sys/arm/rockchip/rk30xx_machdep.c
 Please let us know.
 
 thanks,
 
 Ganbold
 
 
 
 >
 > >Release-Note:
 > >Audit-Trail:
 > >Unformatted:
 > _______________________________________________
 > freebsd-arm@freebsd.org mailing list
 > http://lists.freebsd.org/mailman/listinfo/freebsd-arm
 > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org"
 >
 
 --001a11c1feb69c14b604ea910ed6
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 <div dir=3D"ltr">John,<div class=3D"gmail_extra"><br><br><div class=3D"gmai=
 l_quote">On Thu, Nov 7, 2013 at 3:07 PM, John Wehle <span dir=3D"ltr">&lt;<=
 a href=3D"mailto:john@feith.com" target=3D"_blank">john@feith.com</a>&gt;</=
 span> wrote:<br>
 <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
 x #ccc solid;padding-left:1ex"><br>
 &gt;Number: =A0 =A0 =A0 =A0 183740<br>
 &gt;Category: =A0 =A0 =A0 arm<br>
 &gt;Synopsis: =A0 =A0 =A0 mutex on some arm hardware requires dcache enable=
 d<br>
 &gt;Confidential: =A0 no<br>
 &gt;Severity: =A0 =A0 =A0 non-critical<br>
 &gt;Priority: =A0 =A0 =A0 low<br>
 &gt;Responsible: =A0 =A0freebsd-arm<br>
 &gt;State: =A0 =A0 =A0 =A0 =A0open<br>
 &gt;Quarter:<br>
 &gt;Keywords:<br>
 &gt;Date-Required:<br>
 &gt;Class: =A0 =A0 =A0 =A0 =A0sw-bug<br>
 &gt;Submitter-Id: =A0 current-users<br>
 &gt;Arrival-Date: =A0 Thu Nov 07 07:10:00 UTC 2013<br>
 &gt;Closed-Date:<br>
 &gt;Last-Modified:<br>
 &gt;Originator: =A0 =A0 John Wehle<br>
 &gt;Release: =A0 =A0 =A0 =A0FreeBSD svn 257205<br>
 &gt;Organization:<br>
 Personal<br>
 &gt;Environment:<br>
 Not Yet<br>
 &gt;Description:<br>
 In my spare time I&#39;m working on bringing FreeBSD up on amlogic based<br=
 >
 arm processors. =A0While working through various bootstrap issues I got<br>
 to the point where initarm calls pmap_bootstrap only to have it splat.<br>
 <br>
 What happens is pmap_bootstrap calls pmap_extract which does PMAP_LOCK(pmap=
 ).<br>
 This trys to acquire a mutex which invokes atomics that use ldrex / strex.<=
 br>
 <br>
 The ARM documentation at:<br>
 <br>
 =A0 <a href=3D"http://infocenter.arm.com/help/index.jsp?topic=3D/com.arm.do=
 c.ddi0464d/CHDIBJGE.html" target=3D"_blank">http://infocenter.arm.com/help/=
 index.jsp?topic=3D/com.arm.doc.ddi0464d/CHDIBJGE.html</a><br>
 <br>
 notes that the L1 memory system has an internal exclusive monitor which<br>
 handles ldrex / strex accesses.<br>
 <br>
 The README at:<br>
 <br>
 =A0 <a href=3D"https://github.com/dwelch67/raspberrypi/tree/master/extest" =
 target=3D"_blank">https://github.com/dwelch67/raspberrypi/tree/master/extes=
 t</a><br>
 <br>
 has a nice write up which explains some of the wrinkles regarding hardware<=
 br>
 and ldrex / strex.<br>
 <br>
 The attached patch enables the dcache and suffices to get me through<br>
 initarm into mi_startup (I&#39;m not to a console login yet ... however<br>
 I&#39;m significantly further along).<br>
 <br>
 -- John<br>
 <br>
 &gt;How-To-Repeat:<br>
 <br>
 &gt;Fix:<br>
 Enable the dcache prior to using a mutex.<br>
 <br>
 Patch attached with submission follows:<br>
 <br>
 --- sys/arm/arm/machdep.c.ORIGINAL =A0 =A0 =A02013-10-27 01:15:39.000000000=
  -0400<br>
 +++ sys/arm/arm/machdep.c =A0 =A0 =A0 2013-11-07 01:43:01.000000000 -0500<b=
 r>
 @@ -1481,6 +1483,12 @@ initarm(struct arm_boot_params *abp)<br>
 =A0 =A0 =A0 =A0 =A0*/<br>
 =A0 =A0 =A0 =A0 cpu_idcache_wbinv_all();<br>
 <br>
 + =A0 =A0 =A0 /*<br>
 + =A0 =A0 =A0 =A0* Atomics invoked by mutex use ldrex / strex which on some=
 <br>
 + =A0 =A0 =A0 =A0* hardware requires the L1 cache.<br>
 + =A0 =A0 =A0 =A0*/<br>
 + =A0 =A0 =A0 cpu_control(CPU_CONTROL_DC_ENABLE, CPU_CONTROL_DC_ENABLE);<br=
 >
 +<br>
 =A0 =A0 =A0 =A0 /* Set stack for exception handlers */<br>
 =A0 =A0 =A0 =A0 data_abort_handler_address =3D (u_int)data_abort_handler;<b=
 r>
 =A0 =A0 =A0 =A0 prefetch_abort_handler_address =3D (u_int)prefetch_abort_ha=
 ndler;<br>
 <br></blockquote><div><br></div><div>What kind of Amlogic board you are wor=
 king on? That could be interesting.</div><div><br></div><div>As for the pat=
 ch can you try to put it in initarm_late_init() of your board machdep code =
 and try?</div>
 <div>I meant like in /usr/src/sys/arm/rockchip/rk30xx_machdep.c</div><div>P=
 lease let us know.</div><div><br></div><div>thanks,</div><div><br></div><di=
 v>Ganbold</div><div><br></div><div>=A0</div><blockquote class=3D"gmail_quot=
 e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
 
 <br>
 &gt;Release-Note:<br>
 &gt;Audit-Trail:<br>
 &gt;Unformatted:<br>
 _______________________________________________<br>
 <a href=3D"mailto:freebsd-arm@freebsd.org">freebsd-arm@freebsd.org</a> mail=
 ing list<br>
 <a href=3D"http://lists.freebsd.org/mailman/listinfo/freebsd-arm" target=3D=
 "_blank">http://lists.freebsd.org/mailman/listinfo/freebsd-arm</a><br>
 To unsubscribe, send any mail to &quot;<a href=3D"mailto:freebsd-arm-unsubs=
 cribe@freebsd.org">freebsd-arm-unsubscribe@freebsd.org</a>&quot;<br>
 </blockquote></div><br></div></div>
 
 --001a11c1feb69c14b604ea910ed6--

From: John Wehle <john@feith.com>
To: ganbold@gmail.com
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: arm/183740: mutex on some arm hardware requires dcache enabled
Date: Mon, 11 Nov 2013 01:31:38 -0500 (EST)

 > What kind of Amlogic board you are working on?
 
 I'm currently using a Visson ATV-102 "Android TV Box" which has an
 Amlogic aml8726-m3 SoC.  I have it to the point of attempting to
 mount the root filesystem so it's probably time for me to write a
 driver for the Amlogic MMC hardware.
 
 I also have a Ainol Elf 2 tablet which has an Amlogic aml8726-mx SoC
 which I plan to work on next.
 
 Both chips are dual core Arm Cortex A9 processors.
 
 > As for the patch can you try to put it in initarm_late_init() of
 > your board machdep code and try?
 
 Yep ... enabling the dcache in initarm_late_init takes care of the
 mutex ldrex / strex problem.
 
 -- John
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: arm/183740: commit references a PR
Date: Wed, 20 Nov 2013 15:53:57 +0000 (UTC)

 Author: ian
 Date: Wed Nov 20 15:53:50 2013
 New Revision: 258392
 URL: http://svnweb.freebsd.org/changeset/base/258392
 
 Log:
   Call cpu_setup() immediately after the page tables are installed.  This
   enables data cache and other chip-specific features.  It was previously
   done via an early SYSINIT, but it was being done after pmap and vm setup,
   and those setups need to use mutexes.  On some modern ARM platforms,
   the ldrex/strex instructions that implement mutexes require the data cache
   to be enabled.
   
   A nice side effect of enabling caching earlier is that it eliminates the
   multi-second pause that used to happen early in boot while physical memory
   and pmap and vm were being set up.  On boards with 1 GB or more of ram
   this pause was very noticible, sometimes 5-6 seconds.
   
   PR:		arm/183740
 
 Modified:
   head/sys/arm/arm/machdep.c
 
 Modified: head/sys/arm/arm/machdep.c
 ==============================================================================
 --- head/sys/arm/arm/machdep.c	Wed Nov 20 13:22:22 2013	(r258391)
 +++ head/sys/arm/arm/machdep.c	Wed Nov 20 15:53:50 2013	(r258392)
 @@ -361,7 +361,6 @@ cpu_startup(void *dummy)
  #endif
  #endif
  
 -	cpu_setup("");
  	identify_arm_cpu();
  
  	printf("real memory  = %ju (%ju MB)\n", (uintmax_t)ptoa(physmem),
 @@ -1431,6 +1430,12 @@ initarm(struct arm_boot_params *abp)
  	cpu_domains(DOMAIN_CLIENT << (PMAP_DOMAIN_KERNEL * 2));
  
  	/*
 +	 * Now that proper page tables are installed, call cpu_setup() to enable
 +	 * instruction and data caches and other chip-specific features.
 +	 */
 +	cpu_setup("");
 +
 +	/*
  	 * Only after the SOC registers block is mapped we can perform device
  	 * tree fixups, as they may attempt to read parameters from hardware.
  	 */
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
>Unformatted:
