From cperciva@xps.daemonology.net  Tue Jan  4 11:05:24 2011
Return-Path: <cperciva@xps.daemonology.net>
Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35])
	by hub.freebsd.org (Postfix) with ESMTP id 4FCC7106566C
	for <FreeBSD-gnats-submit@freebsd.org>; Tue,  4 Jan 2011 11:05:24 +0000 (UTC)
	(envelope-from cperciva@xps.daemonology.net)
Received: from xps.daemonology.net (freefall.freebsd.org [IPv6:2001:4f8:fff6::28])
	by mx2.freebsd.org (Postfix) with SMTP id 1588B15470B
	for <FreeBSD-gnats-submit@freebsd.org>; Tue,  4 Jan 2011 11:05:24 +0000 (UTC)
Received: (qmail 42569 invoked by uid 1001); 4 Jan 2011 11:05:23 -0000
Message-Id: <20110104110523.42568.qmail@xps.daemonology.net>
Date: 4 Jan 2011 11:05:23 -0000
From: Colin Percival <cperciva@freebsd.org>
Reply-To: Colin Percival <cperciva@freebsd.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: i386/XEN panics under heavy fork load
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         153672
>Category:       kern
>Synopsis:       [xen] [panic] i386/XEN panics under heavy fork load
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-xen
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Jan 04 11:10:09 UTC 2011
>Closed-Date:    
>Last-Modified:  Tue Mar 01 10:23:49 EST 2011
>Originator:     Colin Percival
>Release:        FreeBSD 9.0-CURRENT i386/XEN
>Organization:
>Environment:
>Description:

Running 'make index', i386/XEN frequently panics.  There are three
common backtraces:
  vmspace_fork -> pmap_copy -> pmap_qenter
  vmspace_fork -> pmap_copy -> pmap_zero_page
  vmspace_fork -> pmap_pinit -> pmap_qenter

Notably, in every vmcore I've examined, a different thread was inside
pmap_release, suggesting that there is a race happening between
pmap_release and pmap_{copy, pinit}.

>How-To-Repeat:

Launch an EC2 instance running ami-f4db2a9d; portsnap fetch extract;
cd /usr/ports && make index.

On average there is one panic per 100 minutes.

>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-xen 
Responsible-Changed-By: cperciva 
Responsible-Changed-When: Tue Jan 4 11:11:24 UTC 2011 
Responsible-Changed-Why:  
Assign Xen bug to freebsd-xen list. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153672 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/153672: commit references a PR
Date: Tue,  4 Jan 2011 15:55:21 +0000 (UTC)

 Author: cperciva
 Date: Tue Jan  4 15:55:15 2011
 New Revision: 216960
 URL: http://svn.freebsd.org/changeset/base/216960
 
 Log:
   Add hamfisted locking to the Xen/PV pmap code: Only allow one thread to
   be in {pmap_pinit, pmap_copy, pmap_release} at a time.
   
   This reduces the rate of panics when running 'make index' from ~0.6/hour
   to ~0.02/hour (p < 10^-30).
   
   At a later date this locking will be removed, and for this reason, it is
   wrapped in #ifdef HAMFISTED_LOCKING; this temporary hack is being put in
   place with the intention of shipping somewhat-stable Xen bits in FreeBSD
   8.2-RELEASE.
   
   PR:		kern/153672
   MFC after:	3 days
 
 Modified:
   head/sys/i386/xen/pmap.c
 
 Modified: head/sys/i386/xen/pmap.c
 ==============================================================================
 --- head/sys/i386/xen/pmap.c	Tue Jan  4 15:53:38 2011	(r216959)
 +++ head/sys/i386/xen/pmap.c	Tue Jan  4 15:55:15 2011	(r216960)
 @@ -202,6 +202,11 @@ __FBSDID("$FreeBSD$");
  
  #define pmap_pte_set_prot(pte, v) ((*(int *)pte &= ~PG_PROT), (*(int *)pte |= (v)))
  
 +#define HAMFISTED_LOCKING
 +#ifdef HAMFISTED_LOCKING
 +static struct mtx createdelete_lock;
 +#endif
 +
  struct pmap kernel_pmap_store;
  LIST_HEAD(pmaplist, pmap);
  static struct pmaplist allpmaps;
 @@ -502,6 +507,10 @@ pmap_bootstrap(vm_paddr_t firstaddr)
  	/* Turn on PG_G on kernel page(s) */
  	pmap_set_pg();
  #endif
 +
 +#ifdef HAMFISTED_LOCKING
 +	mtx_init(&createdelete_lock, "pmap create/delete", NULL, MTX_DEF);
 +#endif
  }
  
  /*
 @@ -1462,6 +1471,10 @@ pmap_pinit(pmap_t pmap)
  	static int color;
  	int i;
  
 +#ifdef HAMFISTED_LOCKING
 +	mtx_lock(&createdelete_lock);
 +#endif
 +
  	PMAP_LOCK_INIT(pmap);
  
  	/*
 @@ -1473,6 +1486,9 @@ pmap_pinit(pmap_t pmap)
  		    NBPTD);
  		if (pmap->pm_pdir == NULL) {
  			PMAP_LOCK_DESTROY(pmap);
 +#ifdef HAMFISTED_LOCKING
 +			mtx_unlock(&createdelete_lock);
 +#endif
  			return (0);
  		}
  #ifdef PAE
 @@ -1545,6 +1561,9 @@ pmap_pinit(pmap_t pmap)
  	TAILQ_INIT(&pmap->pm_pvchunk);
  	bzero(&pmap->pm_stats, sizeof pmap->pm_stats);
  
 +#ifdef HAMFISTED_LOCKING
 +	mtx_unlock(&createdelete_lock);
 +#endif
  	return (1);
  }
  
 @@ -1776,6 +1795,10 @@ pmap_release(pmap_t pmap)
  	    pmap->pm_stats.resident_count));
  	PT_UPDATES_FLUSH();
  
 +#ifdef HAMFISTED_LOCKING
 +	mtx_lock(&createdelete_lock);
 +#endif
 +
  	pmap_lazyfix(pmap);
  	mtx_lock_spin(&allpmaps_lock);
  	LIST_REMOVE(pmap, pm_list);
 @@ -1811,6 +1834,10 @@ pmap_release(pmap_t pmap)
  	pmap_qremove((vm_offset_t)pmap->pm_pdpt, 1);
  #endif
  	PMAP_LOCK_DESTROY(pmap);
 +
 +#ifdef HAMFISTED_LOCKING
 +	mtx_unlock(&createdelete_lock);
 +#endif
  }
  
  static int
 @@ -3136,6 +3163,10 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pm
  	CTR5(KTR_PMAP, "pmap_copy:  dst_pmap=%p src_pmap=%p dst_addr=0x%x len=%d src_addr=0x%x",
  	    dst_pmap, src_pmap, dst_addr, len, src_addr);
  	
 +#ifdef HAMFISTED_LOCKING
 +	mtx_lock(&createdelete_lock);
 +#endif
 +
  	vm_page_lock_queues();
  	if (dst_pmap < src_pmap) {
  		PMAP_LOCK(dst_pmap);
 @@ -3225,6 +3256,10 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pm
  	vm_page_unlock_queues();
  	PMAP_UNLOCK(src_pmap);
  	PMAP_UNLOCK(dst_pmap);
 +
 +#ifdef HAMFISTED_LOCKING
 +	mtx_unlock(&createdelete_lock);
 +#endif
  }	
  
  static __inline void
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: eadler 
State-Changed-When: Tue Mar 1 10:14:00 EST 2011 
State-Changed-Why:  
committed in head 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153672 
State-Changed-From-To: patched->open 
State-Changed-By: eadler 
State-Changed-When: Tue Mar 1 10:23:07 EST 2011 
State-Changed-Why:  
Upon request from cperciva - the change committed was a partial workaround - not a fix 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153672 
>Unformatted:
