From nobody@FreeBSD.org  Wed Jan 14 02:01:04 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 8ABB3106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 14 Jan 2009 02:01:04 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 6BC1D8FC12
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 14 Jan 2009 02:01:04 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n0E2140U006768
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 14 Jan 2009 02:01:04 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n0E214RK006767;
	Wed, 14 Jan 2009 02:01:04 GMT
	(envelope-from nobody)
Message-Id: <200901140201.n0E214RK006767@www.freebsd.org>
Date: Wed, 14 Jan 2009 02:01:04 GMT
From: Gary Byers <gb@clozure.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: fsbase issues for i386 processes running on 7.1-RELEASE/amd64
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         130526
>Category:       amd64
>Synopsis:       [kernel] [patch] fsbase issues for i386 processes running on 7.1-RELEASE/amd64
>Confidential:   no
>Severity:       critical
>Priority:       low
>Responsible:    kib
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jan 14 02:10:01 UTC 2009
>Closed-Date:    Mon Feb 23 11:16:16 UTC 2009
>Last-Modified:  Thu Aug 11 14:30:10 UTC 2011
>Originator:     Gary Byers
>Release:        7.1-RELEASE/amd64
>Organization:
Clozure Associates
>Environment:
FreeBSD boddhi.abq.clozure.com 7.1-RELEASE FreeBSD 7.1-RELEASE #0: Thu Jan  1 08:58:24 UTC 2009     root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64

>Description:
After using i386_set_fsbase() to make the %fs segment register point to a specified linear address, 32-bit processes running on 7.1-RELEASE amd64 can't reliably use %fs to access memory.

Exactly when the process loses the ability to use %fs is unclear, but the enclosed example program seems to show that that ability is lost after a call to sleep().

I'm not familiar enough with x8664 system-level architecture to be fully comfortable in describing the symptom, but my limited understanding suggests that the symptom is consistent with the fsbase MSR not being restored correctly.
>How-To-Repeat:
The attached shell archive contains source to a small C program which seems to demonstrate the problem.  (It likely needs to be compiled on a 32-bit FreeBSD system.)

The program establishes a signal handler for SIGBUS and SIGSEGV; the handler simply prints some context information and exits.  The program then allocates a 100-byte pointer ("p") via malloc() and uses the pointer returned as an argument to i386_set_fsbase(); this should have the effect of making the %fs segment register address the pointer, so the byte or word at %fs:0 is equivalent to p[0].

The program then executes 100 iterations of a loop which stores a 32-bit value at the malloc'ed pointer's address, reads the 32-bit word at %fs:0, sleeps for 1 second, and reads the word at %fs:0 again.  After each read of %fs:0, the value read relative to %fs is compared to the value stored at the pointer address; if the values differ, the code calls i386_set_fsbase() to see the adderss it returns by reference matches the pointer address, prints these values, and exits.

The program should therefore do little of interest and take about 100 seconds to do so.  When run on a 7.1 amd64 release kernel, the program segfaults, usually after sleeping on the first iteration.  (So the first attempt to reference memory at %fs:0 succeeded as expected; attempting to read the same value after calling sleep() fails.)  This may indicate that the fsbase MSR is not restored correctly after context switch and/or syscall return.
>Fix:
Unknown.

A few days ago, I submitted an invalid bug report (130355) which claimed that this symptom was caused by failure to set %fs to the proper selector on return from i386_set_fsbase() on 7.1/amd64.  No amd64 kernel changes the %fs selector in the implementation of i386_set_fsbase; 64-bit kernels do arrange that the "fsbase" MSR is set appropriately.  It seems that this works correctly (the test program can use %fs before sleeping on the first iteration), but that the fsbase MSR isn't set correctly (or something like that ...) on syscall return or after context switch, in at least some cases.


Patch attached with submission follows:

# This is a shell archive.  Save it in a file, remove anything before
# this line, and then unpack it by entering "sh file".  Note, it may
# create directories; files and directories will be owned by you and
# have default permissions.
#
# This archive contains:
#
#	i386_set_fsbase_test.c
#
echo x - i386_set_fsbase_test.c
sed 's/^X//' >i386_set_fsbase_test.c << 'e02ffb4f5ccf0ea67f579a83c1cccce4'
X/* This program should be compiled in a 32-bit i386 FreeBSD environment. */
X
X#include <stdio.h>
X#include <machine/segments.h>
X#include <machine/sysarch.h>
X#include <stdlib.h>
X#include <unistd.h>
X#include <sys/signal.h>
X#include <stdbool.h>
X
X
Xvoid
Xinstall_signal_handler(int signo, void * handler)
X{
X  struct sigaction sa;
X  
X  sa.sa_sigaction = (void *)handler;
X  sigfillset(&sa.sa_mask);
X  sa.sa_flags = SA_SIGINFO;
X
X  sigaction(signo, &sa, NULL);
X}
X
Xvoid *p, *check = NULL;
Xint i;
Xbool after_sleep = false;
X
Xvoid
Xsignal_handler(int signo, siginfo_t *info, void *context)
X{
X  i386_get_fsbase(&check);
X
X  fprintf(stderr, "terminating with signal %d on iteration %d, %s sleep, address = 0x%x, p = 0x%x, fsbase = 0x%x\n", 
X          signo,
X          i,
X          after_sleep ? "after" : "before",
X          (uintptr_t)info->si_addr, 
X          (uintptr_t)p, (uintptr_t)check);
X  exit(5);
X
X}
X
X/* return the contents of the first 32-bit word addressed by %fs */
Xunsigned long
Xread_fs_contents()
X{
X  unsigned long res;
X
X  __asm__ volatile ("movl %%fs:0,%0" : "=r" (res));
X  return res;
X}
X
X
Xmain()
X{
X  int status;
X
X  install_signal_handler(SIGBUS, signal_handler);
X  install_signal_handler(SIGSEGV, signal_handler);
X  p = malloc(100);
X  status = i386_set_fsbase(p);
X  if (status != 0) {
X    perror("i386_set_fsbase");
X    exit(1);
X  }
X  /* it should now be true that %fs addresses the pointer 'p'; if we
X     store a 32-bit value at 'p', we should be able to read that 
X     value back via read_fs_contents(), without getting any sort
X     of bus fault.
X     This seems to be true of all 32-bit FreeBSD kernels that I've
X     tried it on and is true of the 64-bit 6.4-RELEASE and 7.0-RELEASE;
X     it does not seem to be true of the 7.1/amd64-RELEASE kernel.
X  */
X  for (i = 0; i < 100; i++) {
X    *((unsigned long *)p) = i;
X    after_sleep = false;
X    if (read_fs_contents() != i) {
X      i386_get_fsbase(&check);
X      fprintf (stderr, "%%fs base may have changed, now 0x%x, should be 0x%x\n",(uintptr_t)check,(uintptr_t)p);
X      exit(2);
X    }
X    sleep(1);
X    after_sleep = true;
X    if (read_fs_contents() != i) {
X      i386_get_fsbase(&check);
X      fprintf (stderr, "after sleep, %%fs base may have changed, now 0x%x, should be 0x%x\n",(uintptr_t)check,(uintptr_t)p);
X      exit(3);
X    }
X  }
X  exit(0);
X}
X
e02ffb4f5ccf0ea67f579a83c1cccce4
exit



>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->analyzed 
State-Changed-By: kib 
State-Changed-When: Thu Jan 15 18:03:16 UTC 2009 
State-Changed-Why:  
I have a patch for the issue. 


Responsible-Changed-From-To: freebsd-amd64->kib 
Responsible-Changed-By: kib 
Responsible-Changed-When: Thu Jan 15 18:03:16 UTC 2009 
Responsible-Changed-Why:  
I have a patch for the issue. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=130526 

From: Kostik Belousov <kostikbel@gmail.com>
To: Gary Byers <gb@clozure.com>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: amd64/130526: fsbase issues for i386 processes running on 7.1-RELEASE/amd64
Date: Thu, 15 Jan 2009 20:09:26 +0200

 --jL2BoiuKMElzg3CS
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 Yes, I confirm. The issue is that amd64/amd64/cpu_switch.S, code at
 done_load_seg assumes that %r9 contains the value of the fsbase.
 This is not true after the %fs load is done for 32bit process
 at the load_seg labeled block.
 
 Patch below fixed your test.
 
 diff --git a/sys/amd64/amd64/cpu_switch.S b/sys/amd64/amd64/cpu_switch.S
 index 99d6716..8f5096b 100644
 --- a/sys/amd64/amd64/cpu_switch.S
 +++ b/sys/amd64/amd64/cpu_switch.S
 @@ -199,6 +199,7 @@ done_load_seg:
  	cmpq	PCB_FSBASE(%r8),%r9
  	jz	1f
  	/* Restore userland %fs */
 +load_fs:
  	movl	$MSR_FSBASE,%ecx
  	movl	PCB_FSBASE(%r8),%eax
  	movl	PCB_FSBASE+4(%r8),%edx
 @@ -281,7 +282,7 @@ load_seg:
  	movl	PCB_DS(%r8),%ds
  	movl	PCB_ES(%r8),%es
  	movl	PCB_FS(%r8),%fs
 -	jmp	done_load_seg
 +	jmp	load_fs
  	/* Restore userland %gs while preserving kernel gsbase */
  2:	movq	PCPU(GS32P),%rax
  	movq	PCB_GS32SD(%r8),%rcx
 
 --jL2BoiuKMElzg3CS
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.9 (FreeBSD)
 
 iEUEARECAAYFAklve9YACgkQC3+MBN1Mb4gRswCeO5pyhBq193WQAzDYFaHUoHJi
 IjUAmOIUeZGNNZSj6eSz+he4DFt4ehE=
 =Hk0p
 -----END PGP SIGNATURE-----
 
 --jL2BoiuKMElzg3CS--

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: amd64/130526: commit references a PR
Date: Tue, 20 Jan 2009 12:08:12 +0000 (UTC)

 Author: kib
 Date: Tue Jan 20 12:07:49 2009
 New Revision: 187470
 URL: http://svn.freebsd.org/changeset/base/187470
 
 Log:
   The context switch to the 32bit binary does not properly restore
   the fsbase value. The switch loads the fs segment register, that
   invalidates the value in fsbase msr, thus value in %r9 can not be
   considered the current value for fsbase anymore.
   
   Unconditionally reload fsbase when switching to 32bit binary.
   
   PR:	130526
   MFC after:	3 weeks
 
 Modified:
   head/sys/amd64/amd64/cpu_switch.S
 
 Modified: head/sys/amd64/amd64/cpu_switch.S
 ==============================================================================
 --- head/sys/amd64/amd64/cpu_switch.S	Tue Jan 20 11:34:28 2009	(r187469)
 +++ head/sys/amd64/amd64/cpu_switch.S	Tue Jan 20 12:07:49 2009	(r187470)
 @@ -199,6 +199,7 @@ done_load_seg:
  	cmpq	PCB_FSBASE(%r8),%r9
  	jz	1f
  	/* Restore userland %fs */
 +restore_fsbase:
  	movl	$MSR_FSBASE,%ecx
  	movl	PCB_FSBASE(%r8),%eax
  	movl	PCB_FSBASE+4(%r8),%edx
 @@ -281,7 +282,7 @@ load_seg:
  	movl	PCB_DS(%r8),%ds
  	movl	PCB_ES(%r8),%es
  	movl	PCB_FS(%r8),%fs
 -	jmp	done_load_seg
 +	jmp	restore_fsbase
  	/* Restore userland %gs while preserving kernel gsbase */
  2:	movq	PCPU(GS32P),%rax
  	movq	PCB_GS32SD(%r8),%rcx
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: analyzed->closed 
State-Changed-By: kib 
State-Changed-When: Mon Feb 23 11:14:57 UTC 2009 
State-Changed-Why:  
The patch was MFCed to RELENG_7. Without any feedback, I am closing 
PR for now, since I believe that the problem shall be fixed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=130526 

From: Robert Millan <rmh@debian.org>
To: bug-followup@FreeBSD.org, gb@clozure.com
Cc:  
Subject: Re: amd64/130526: [kernel] [patch] fsbase issues for i386 processes
 running on 7.1-RELEASE/amd64
Date: Thu, 11 Aug 2011 16:27:15 +0200

 It appears that this bug has reborn.  I'm using GNU/kFreeBSD with 8.1
 amd64 kernel (32-bit chroot):
 
 $ gcc i386_set_fsbase_test.c -o test && ./test
 terminating with signal 11 on iteration 0, before sleep, address =
 0x0, p = 0x8049b20, fsbase = 0x0
 
 -- 
 Robert Millan
>Unformatted:
