From nobody@FreeBSD.org  Thu Mar 22 06:09:58 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 47FB616A4EF
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 22 Mar 2007 06:09:58 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id 3845913C45D
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 22 Mar 2007 06:09:58 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l2M69wk8081101
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 22 Mar 2007 06:09:58 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l2M64ueJ075601;
	Thu, 22 Mar 2007 06:04:56 GMT
	(envelope-from nobody)
Message-Id: <200703220604.l2M64ueJ075601@www.freebsd.org>
Date: Thu, 22 Mar 2007 06:04:56 GMT
From: "Martin M. Mladenov"<fm@mtweb.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: 32 bit threaded applications crash on amd64 SMP kernel.
X-Send-Pr-Version: www-3.0

>Number:         110655
>Category:       amd64
>Synopsis:       [threads] 32 bit threaded applications crash on amd64 SMP kernel.
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-amd64
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Mar 22 06:10:06 GMT 2007
>Closed-Date:    Mon Apr 20 12:48:14 UTC 2009
>Last-Modified:  Mon Apr 20 12:48:14 UTC 2009
>Originator:     Martin M. Mladenov
>Release:        FreeBSD 6.2-RELEASE-p3 amd64
>Organization:
>Environment:
System: FreeBSD c8 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #2: Wed Mar 21 06:31:50 MST 2007 root@c8:/usr/obj/usr/src/sys/SM6015TTV amd64
>Description:
32 bit applications using the posix threads library crash when calling
pthread_create on an SMP amd64 kernel. This happens with existing binaries,
built natively on a x86 machine, as well as binaries built with cc -m32
on the amd64 system. 

  	 A backtrace for the example in the next section yields:
	
	Core was generated by `pthread-crash'.
	Program terminated with signal 11, Segmentation fault.
	Reading symbols from /usr/lib32/libpthread.so.2...(no debugging symbols found)...done.
	Loaded symbols for /usr/lib32/libpthread.so.2
	Reading symbols from /usr/lib32/libc.so.6...(no debugging symbols found)...done.
	Loaded symbols for /usr/lib32/libc.so.6
	Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
	Loaded symbols for /libexec/ld-elf.so.1
	#0  0x0804d100 in ?? ()
	[New Thread 0x805301408053200 (runnable)]
	[New Thread 0x2809f1c808053000 (runnable)]
	Cannot get thread info: generic error
	(gdb) bt
	#0  0x0804d100 in ?? ()
	
This has been reproduced on two machines with completely different hardware. 	


>How-To-Repeat:
The problem can be reproduced by the following code (pthread_crash.c):
        #include <stdio.h>
        #include <pthread.h>

        void *thread(void *data) {
          puts("Thread.");
          return NULL;
        }
 
        int main() {
          pthread_t pth;
          void *pv;
          pthread_create(&pth,NULL,thread,NULL);
          pthread_join(pth,&pv);
          return 0;
        }
        The code is compiled with:
        cc -m32 -B/usr/lib32 -pthread -o pthread-crash pthread-crash.c
	This code runs fine when compiled to 64 bit.

>Fix:
Non known.
>Release-Note:
>Audit-Trail:

From: Peter Jeremy <peterjeremy@optushome.com.au>
To: "Martin M. Mladenov" <fm@mtweb.org>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: amd64/110655: 32 bit threaded applications crash on amd64 SMP kernel.
Date: Thu, 22 Mar 2007 19:34:12 +1100

 This problem also affects a non-SMP RELENG_6 kernel from 30th January.
 
 -- 
 Peter Jeremy

From: Andriy Gapon <avg@icyb.net.ua>
To: bug-followup@FreeBSD.org
Cc: freebsd-threads@freebsd.org
Subject: Re: amd64/110655: 32 bit threaded applications crash on amd64 SMP
 kernel.
Date: Thu, 22 Mar 2007 12:40:25 +0200

 I can confirm this problem for 6.2-RELEASE-p2 amd64 UP system, for both
 libpthread and libthr.
 i386 simple test programs die with the following symptoms:
 
 [libpthread]
 program dies with sigsegv. What's interesting, when run under ktrace my
 very short program sometimes completes successfully and sometime crashes
 with the same symptoms. There is one strange line in ktrace:
 
  66443 thr_test CALL  kse_create(0x804b50c,0x1)
  66443 thr_test RET   kse_create 0
  66443 thr_test CALL
 __sysctl(0xffffd7b8,0x2,0xffffd7c0,0xffffd7b4,0x2809c63c,0x18)
  66443 thr_test RET   __sysctl 0
  66443 thr_test CALL  __sysctl(0xffffd7c0,0x3,0xffffd86c,0xffffd870,0,0)
  66443 thr_test RET   __sysctl 0
  66443 thr_test CALL  break(0x805c000)
  66443 thr_test RET   break 0
  66443 thr_test CALL  mmap(0xffbfc000,0x101000,0x3,0x400,0xffffffff,0,0,0)
  66443 thr_test RET   mmap -4210688/0xffbfc000
  66443 thr_test CALL  mprotect(0xffbfc000,0x1000,0)
  66443 thr_test RET   mprotect 0
  66443 thr_test CALL  clock_gettime(0,0x8051f5c)
  66443 thr_test RET   clock_gettime 0
  66443 thr_test CALL  fstat(0x1,0xffcfc7c0)
  66443 thr_test RET   fstat 0
 =================
  66443 thr_test RET   fork 0
 =================
  66443 thr_test PSIG  SIGSEGV SIG_DFL
  66443 thr_test NAMI  "thr_test.core"
 
 
 [libthr]
 the problem here seems to be more obvious, but unexpected (at least for me):
  69654 thr_test CALL  sysarch(0xa,0xffffd700)
  69654 thr_test RET   sysarch 0
  69654 thr_test CALL  sigprocmask(0x3,0xffffd730,0xffffd720)
  69654 thr_test RET   sigprocmask 0
  69654 thr_test CALL  sigaction(0x20,0xffffd6d0,0)
  69654 thr_test RET   sigaction 0
  69654 thr_test CALL  sigprocmask(0x3,0xffffd720,0)
  69654 thr_test RET   sigprocmask 0
  69654 thr_test CALL  sigprocmask(0x1,0x28070a20,0xffffd760)
  69654 thr_test RET   sigprocmask 0
  69654 thr_test CALL  sigprocmask(0x3,0x28070a30,0)
  69654 thr_test RET   sigprocmask 0
  69654 thr_test CALL  _umtx_op
  69654 thr_test RET   _umtx_op -1 errno 78 Function not implemented
  69654 thr_test PSIG  SIGSYS SIG_DFL
  69654 thr_test NAMI  "thr_test.core"
 
 
 -- 
 Andriy Gapon

From: Martin Mladenov <mladenov.martin@googlemail.com>
To: bug-followup@FreeBSD.org,
 Martin Mladenov <fm@mtweb.org>
Cc:  
Subject: Re: amd64/110655: 32 bit threaded applications crash on amd64 SMP kernel.
Date: Mon, 26 Mar 2007 15:02:56 +0200

 Andriy Gapon <avg@icyb.net.ua> wrote:
  > There is one strange line in ktrace:
  > [...]
  > 66443 thr_test RET fork 0
 
 http://lists.freebsd.org/pipermail/freebsd-threads/2007-February/ 
 003858.html
 
 According to this post upcalls from KSE are reported as returns from  
 fork.

From: Andriy Gapon <avg@icyb.net.ua>
To: bug-followup@FreeBSD.org,  fm@mtweb.org
Cc:  
Subject: Re: amd64/110655: 32 bit threaded applications crash on amd64 SMP
 kernel.
Date: Thu, 29 Mar 2007 15:14:36 +0300

 Thank you for the pointer, that explains it.
 In this view I think that the most probable cause of the crash is binary
 incompatibility between KSE structures in amd64 kernel and i386
 userland, which is not so unexpected, unfortunately.
 
 
 -- 
 Andriy Gapon

From: Jeff Wheelhouse <jdw_list@wheelhouse.org>
To: bug-followup@FreeBSD.org,  fm@mtweb.org
Cc:  
Subject: Re: amd64/110655: 32 bit threaded applications crash on amd64 SMP
 kernel.
Date: Wed, 04 Apr 2007 11:30:50 -0400

 Would it be appropriate to follow up on this with the KSE folks or the 
 i386-on-amd64 folks?  (Or is there some lucky individual around who 
 knows about both?)
 
 This bug is causing a lot of problems for us, and we're happy to do 
 whatever we can to help fix it, but both subsystems are pretty much 
 opaque to me, so I am at a loss as to how we can help.
 
 Thanks,
 Jeff

From: "Jeffrey D. Wheelhouse" <jdw@wheelhouse.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: amd64/110655: [threads] 32 bit threaded applications crash on
 amd64 SMP kernel.
Date: Tue, 05 Jun 2007 16:49:59 -0400

 We are still having this problem on 6.2-RELEASE-p4.
 
 I have confirmed that using the example source code above with the thr 
 library also does not work:
 
 $ cc -m32 -B/usr/lib32 -lthr -o crash32 crash32.c
 $ ./crash32
 Bad system call(coredump)
 $ cc -lthr -o crash32 crash32.c
 $ ./crash32
 Thread.
 $
 
 Thanks,
 Jeff
 
 -- 
 Jeff Wheelhouse
 jdw@wheelhouse.org

From: Jeff Wheelhouse <jdw_list@wheelhouse.org>
To: Kris Kennaway <kris@obsecurity.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: amd64/110655: [threads] 32 bit threaded applications crash on
 amd64 SMP kernel.
Date: Wed, 06 Jun 2007 17:11:25 -0400

 Kris Kennaway wrote:
 > -m32 doesn't DTRT on FreeBSD, can you confirm that this problem exists
 > with a binary compiled by an i386 compiler?
 
 I can.
 
 $ uname -m
 i386
 $ cc -o crash32-thr crash32.c -lthr
 $ cc -o crash32-pthread crash32.c -pthread
 $ md5 crash32-thr crash32-pthread
 MD5 (crash32-thr) = 91aea395dd2c5798d7cba9a216c4c227
 MD5 (crash32-pthread) = a590f1c97e5ec11ddc94fa3d7f4bf167
 $ ./crash32-thr
 Thread.
 $ ./crash32-pthread
 Thread.
 $
 
 $ uname -m
 amd64
 $ md5 crash32-thr crash32-pthread
 MD5 (crash32-thr) = 91aea395dd2c5798d7cba9a216c4c227
 MD5 (crash32-pthread) = a590f1c97e5ec11ddc94fa3d7f4bf167
 $ ./crash32-thr
 Bad system call(coredump)
 $ ./crash32-pthread
 Memory fault(coredump)
 $
 
 Thanks,
 Jeff
 

From: Kris Kennaway <kris@obsecurity.org>
To: Jeff Wheelhouse <jdw_list@wheelhouse.org>
Cc: Kris Kennaway <kris@obsecurity.org>, bug-followup@FreeBSD.org
Subject: Re: amd64/110655: [threads] 32 bit threaded applications crash on amd64 SMP kernel.
Date: Wed, 6 Jun 2007 17:42:38 -0400

 On Wed, Jun 06, 2007 at 05:11:25PM -0400, Jeff Wheelhouse wrote:
 > Kris Kennaway wrote:
 > >-m32 doesn't DTRT on FreeBSD, can you confirm that this problem exists
 > >with a binary compiled by an i386 compiler?
 > 
 > I can.
 > 
 > $ uname -m
 > i386
 > $ cc -o crash32-thr crash32.c -lthr
 > $ cc -o crash32-pthread crash32.c -pthread
 > $ md5 crash32-thr crash32-pthread
 > MD5 (crash32-thr) = 91aea395dd2c5798d7cba9a216c4c227
 > MD5 (crash32-pthread) = a590f1c97e5ec11ddc94fa3d7f4bf167
 > $ ./crash32-thr
 > Thread.
 > $ ./crash32-pthread
 > Thread.
 > $
 > 
 > $ uname -m
 > amd64
 > $ md5 crash32-thr crash32-pthread
 > MD5 (crash32-thr) = 91aea395dd2c5798d7cba9a216c4c227
 > MD5 (crash32-pthread) = a590f1c97e5ec11ddc94fa3d7f4bf167
 > $ ./crash32-thr
 > Bad system call(coredump)
 > $ ./crash32-pthread
 > Memory fault(coredump)
 > $
 
 OK, that's useful to know.  I ran into a problem myself that looked
 similar but wanted to check first (I didn't isolate it to libthr yet).
 
 Kris

From: John Baldwin <jhb@FreeBSD.org>
To: bug-followup@freebsd.org,
 fm@mtweb.org
Cc:  
Subject: Re: amd64/110655: [threads] 32 bit threaded applications crash on amd64 SMP kernel.
Date: Fri, 18 Jan 2008 09:09:39 -0500

 6.3 should have freebsd32 compat for libthr and 7.0 should have freebsd32
 compat for both libthr and libkse.  Can you retest on newer OSs?
 
 -- 
 John Baldwin

From: Rolf Grossmann <rg@progtech.net>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: amd64/110655: [threads] 32 bit threaded applications crash on
 amd64 SMP kernel.
Date: Fri, 17 Apr 2009 19:17:20 +0200

 It now works for me on FreeBSD 7.1 (GENERIC kernel, which includes 
 options SMP, but single cpu):
 
 $ uname -m
 i386
 $ cc -o crash32-thr crash32.c -lthr
 $ cc -o crash32-pthread crash32.c -pthread
 $ md5 crash32-thr crash32-pthread
 MD5 (crash32-thr) = 30b58238379b4c3496413f22863c2e86
 MD5 (crash32-pthread) = 734e11117fbfd63efb06376cf74430d5
 $ ./crash32-thr
 Thread.
 $ ./crash32-pthread
 Thread.
 
 $ uname -m
 amd64
 $ md5 crash32-thr crash32-pthread
 MD5 (crash32-thr) = 30b58238379b4c3496413f22863c2e86
 MD5 (crash32-pthread) = 734e11117fbfd63efb06376cf74430d5
 $ ./crash32-thr
 Thread.
 $ ./crash32-pthread
 Thread.
 
 Can we close this ticket?
 
State-Changed-From-To: open->closed 
State-Changed-By: jhb 
State-Changed-When: Mon Apr 20 12:47:45 UTC 2009 
State-Changed-Why:  
This is fixed in 6.3 (libthr only) and 7.0+ (libthr + libkse). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=110655 
>Unformatted:
