From nobody@FreeBSD.org  Mon Aug  4 06:40:10 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 944A7106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  4 Aug 2008 06:40:10 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 818158FC20
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  4 Aug 2008 06:40:10 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m746eAn8080222
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 4 Aug 2008 06:40:10 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m746eAGN080221;
	Mon, 4 Aug 2008 06:40:10 GMT
	(envelope-from nobody)
Message-Id: <200808040640.m746eAGN080221@www.freebsd.org>
Date: Mon, 4 Aug 2008 06:40:10 GMT
From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: freebsd-gnats-submit@FreeBSD.org
Subject: weird process cpu usage stats and weird load average
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         126230
>Category:       kern
>Synopsis:       [libc] select(2) causing weird process cpu usage and load average stats
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 04 06:50:02 UTC 2008
>Closed-Date:    
>Last-Modified:  Wed Nov 24 01:12:01 UTC 2010
>Originator:     Jakub Jasinski
>Release:        7.0
>Organization:
>Environment:
FreeBSD mail2.parkin.ca 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 10:35:36 UTC 2008     root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
This is a Proliant 380 G5, running the 64-bit FreeBSD 7.0-RELEASE.

Top's output seems like total nonsense.  The server is not deployed yet.  So I ran rsync to see what strangeness might occur.  Only rsync is really active.

Here's a snippet of top's output:

last pid:  7591;  load averages:  7.70,  8.78,  9.87    up 0+09:14:46  02:34:03
61 processes:  1 running, 59 sleeping, 1 stopped
CPU states:  1.1% user,  0.0% nice, 34.4% system,  3.1% interrupt, 61.4% idle
Mem: 208M Active, 7031M Inact, 377M Wired, 257M Cache, 214M Buf, 40M Free
Swap: 8192M Total, 4K Used, 8192M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
 7315 root          1 102    0 17196K 11908K select 1   6:35  9.03% rsync
  664 root          2   4    0 15396K  2196K select 0   6:12  7.32% apcupsd
  745 root          1   4    0  9940K  1072K select 3   3:47  3.96% authdaemond
 6619 root          1   4    0 30108K  3912K select 0   2:54  3.81% sendmail
 7310 root          1   4    0 17196K 11620K select 2   2:22  3.76% rsync
 5212 root          1   4    0 10700K  2184K select 3   3:04  3.56% gam_server
 6534 mailnull      1   4    0  9252K  1732K select 1   2:42  3.52% mimedefang-
  473 root          1   4    0  5808K  1000K select 2   2:42  3.52% moused
  768 root          1   4    0  9940K  1204K select 2   2:23  3.47% authdaemond
 6551 mailnull      2  20    0  7196K  1648K sigwai 2   2:26  3.37% mimedefang
 7575 root          1  98    0  7656K  2016K CPU1   0   0:09  3.32% top
  772 root          1   4    0  9940K  1204K select 2   2:21  3.22% authdaemond
 5458 root          1   4    0 76168K 17452K select 3   3:53  3.17% kdeinit
  500 root          1   4    0  5808K  1000K select 1   2:41  3.17% moused
  770 root          1   4    0  9940K  1072K select 3   2:21  3.12% authdaemond
  811 root          1   4    0 20960K  3004K select 2   2:17  3.08% sshd
 7092 root          1   4    0 27300K  9668K select 2   2:25  3.03% perl
  771 root          1   4    0  9940K  1204K select 0   2:21  2.98% authdaemond
 5845 admin         1   4    0 32936K  3552K select 2   2:29  2.93% sshd
  769 root          1   4    0  9940K  1204K select 1   2:21  2.88% authdaemond
  788 root          1   4    0  3604K  1024K select 1   2:16  2.73% couriertcpd
  615 root          1   4    0  4684K  1172K select 0   2:28  2.69% syslogd
  763 root          1   4    0  3604K  1020K select 1   2:18  2.64% couriertcpd
  548 root          1   4    0  1616K   484K select 1   2:19  2.54% devd
  778 root          1   4    0  3604K  1056K select 3   2:16  2.54% couriertcpd
  754 root          1   4    0  3604K  1020K select 3   2:16  2.54% couriertcpd
  793 mailnull      1   4    0 88116K 75880K accept 1   0:08  0.00% clamd
 6536 mailnull      1  -8    0 44464K 22288K piperd 3   0:00  0.00% perl
 6556 mailnull      1  -8    0 43696K 21524K piperd 3   0:00  0.00% perl
 5857 root          1   8    0  8692K  2316K wait   3   0:00  0.00% bash
 5540 root          1   4   19 75800K 19312K STOP   0   0:00  0.00% kblankscrn.
  836 root          1   8    0  5736K  1208K nanslp 1   0:00  0.00% cron
 1003 root          1  20    0  9616K  2612K pause  1   0:00  0.00% csh
  954 root          1   5    0  9616K  2564K ttyin  1   0:00  0.00% csh
  798 mailnull      1  20    0 11976K  1796K pause  1   0:00  0.00% freshclam
 4013 root          1   5    0  8692K  2232K ttyin  1   0:00  0.00% bash
 5842 root          1   4    0 32936K  3552K sbwait 1   0:00  0.00% sshd
  945 root          1   8    0 19412K  1532K wait   2   0:00  0.00% login
 5852 root          1  20    0  9616K  2432K pause  0   0:00  0.00% csh
  946 root          1   8    0 19412K  1492K wait   1   0:00  0.00% login
 5847 admin         1   8    0  8692K  2304K wait   3   0:00  0.00% bash

>How-To-Repeat:
Run top on a Proliant 380 G5 (or similar) with FreeBSD 7.0-RELEASE (amd64).
>Fix:


>Release-Note:
>Audit-Trail:

From: "Remko Lodder" <remko@elvandar.org>
To: "Jakub Jasinski" <jakub.jasinski@utoronto.ca>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load 
     average
Date: Mon, 4 Aug 2008 09:16:18 +0200 (CEST)

 Hello Jakub,
 
 First of all thanks for taking the time to write down this report. In all
 honesty I am not yet sure whether this is indeed a problem or not. I think
 that the active daemons are all actually active. I have multiple systems
 running on 7.0 with amd64 and I personally do not see any processes active
 which 'arent' active. Perhaps you can trace the processes to see what they
 are doing, but I think'it's very likely they are doing things, eventhough
 you do not expect it.
 
 Cheers
 remko
 
 -- 
 /"\   Best regards,                      | remko@FreeBSD.org
 \ /   Remko Lodder                       | remko@EFnet
  X    http://www.evilcoder.org/          |
 / \   ASCII Ribbon Campaign              | Against HTML Mail and News
 
 

From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: remko@elvandar.org
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load 
 average
Date: Mon, 4 Aug 2008 05:17:39 -0400

 After following Remko's advice, i.e., ktrace-ing some of the processes
 (i.e., moused and couriertcpd), I can see that they're definitely not
 doing anything with ktrace.out being empty and top showing >1% cpu
 usage.  I ran ktrace on rsync _ONLY_ (after ktrace -C'ing and deleting
 ktrace.out) and it grew in size very quickly.  AND all ports are
 LISTENing (other than ssh and rsync).
 
From: Kris Kennaway <kris@FreeBSD.org>
To: Jakub Jasinski <jakub.jasinski@utoronto.ca>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load average
Date: Mon, 04 Aug 2008 13:34:44 +0200
 
 This shows that various processes have recently been running, explaining 
 the load average.  You can also see that the same processes that have 
 non-zero WCPU have non-zero TIME, indicating that they have accumulated 
 CPU runtime.  All of this is self-consistent and points to the fact that 
 you have processes running on your server.
 
 Kris

From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: Kris Kennaway <kris@FreeBSD.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load
 average
Date: Mon, 4 Aug 2008 11:52:08 -0400

 No.
 
 I ran ktrace on the processes and ktrace.out was empty.
 
 Read my follow up message: I already had a lengthy conversation with
 Remko about this.
 
 NONE of the processes were running.  The non-zero TIME is nonsense as
 well and only accumulated after rsync was left running.
 
From: Kris Kennaway <kris@FreeBSD.org>
To: Jakub Jasinski <jakub.jasinski@utoronto.ca>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load average
Date: Mon, 04 Aug 2008 18:59:15 +0200

 That would be a very strange situation indeed, requiring multiple things 
 to go wrong in bizarre ways.  The more likely explanation is that the 
 processes are in fact running and being reported correctly.
 
 Kris

From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: Kris Kennaway <kris@freebsd.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load
 average
Date: Tue, 5 Aug 2008 12:36:57 -0400

 I recompiled the kernel with the ULE scheduler and the only difference
 now is that every 'working' process has higher WCPU _after_ rsync is
 activated. Before rsync is activated all process have 0% WCPU.
 
 BTW, thanks for the help.
 
 [root@mail2 ~]# vmstat -i
 interrupt                          total       rate
 irq1: atkbd0                          78          0
 irq4: sio0                          2023          5
 irq14: ata0                           58          0
 irq16: uhci0 ehci0                     1          0
 irq18: ciss0 uhci2                  2618          6
 irq22: uhci4                          19          0
 cpu0: timer                       751994       1994
 irq257: bce1                        2271          6
 cpu1: timer                       744491       1974
 cpu2: timer                       744494       1974
 cpu3: timer                       744513       1974
 Total                            2992560       7937
 
 (snipped top output:)
 
 60 processes:  1 running, 59 sleeping
 CPU states:  1.0% user,  0.0% nice, 38.4% system,  2.4% interrupt, 58.1%
 Mem: 142M Active, 333M Inact, 284M Wired, 52K Cache, 214M Buf, 7154M Fre
 Swap: 8192M Total, 8192M Free
 
   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMM
   662 root          2   4    0 14112K  2504K select 1   0:10 13.43% apcu
  1020 root          1  55    0 17196K 12096K select 3   0:08 11.57% rsyn
   756 root          1   4    0  9940K  1260K select 2   0:05  7.08% auth
   498 root          1   4    0  5808K  1196K select 3   0:05  6.40% mous
   471 root          1   4    0  5808K  1196K select 0   0:04  5.86% mous
   890 mailnull      1   4    0  9252K  2024K select 0   0:04  5.08% mime
   981 root          1   4    0 30108K  5292K select 2   0:04  4.98% send
   969 admin         1   4    0 32936K  4608K select 1   0:04  4.88% sshd
  1001 admin         1   4    0 32936K  4612K select 1   0:04  4.88% sshd
   791 root          1   4    0  9940K  1260K select 3   0:04  4.88% auth
   613 root          1   4    0  4684K  1352K select 0   0:04  4.79% sysl
   784 root          1   4    0  3604K  1224K select 0   0:04  4.79% cour
   792 root          1   4    0  9940K  1260K select 2   0:04  4.79% auth
  1018 root          1   4    0 17196K 11808K select 0   0:04  4.69% rsyn
   793 root          1   4    0  9940K  1260K select 3   0:04  4.59% auth
   719 root          1   4    0 27284K 10392K select 2   0:04  4.59% perl
   799 root          1   4    0  3604K  1224K select 2   0:04  4.59% cour
   789 root          1   4    0  9940K  1260K select 1   0:04  4.49% auth
   790 root          1   4    0  9940K  1260K select 2   0:04  4.49% auth
   907 mailnull      2  20    0  4888K  1536K sigwai 1   0:04  4.49% mime
   833 root          1   4    0 20960K  3920K select 1   0:04  4.49% sshd
   774 root          1   4    0  3604K  1220K select 3   0:04  4.39% cour
 
 On Mon, 4 Aug 2008 17:00:11 GMT
 Kris Kennaway <kris@freebsd.org> wrote:
 
 > The following reply was made to PR misc/126230; it has been noted by
 > GNATS.
 > 
 > From: Kris Kennaway <kris@FreeBSD.org>
 > To: Jakub Jasinski <jakub.jasinski@utoronto.ca>
 > Cc: freebsd-gnats-submit@FreeBSD.org
 > Subject: Re: misc/126230: weird process cpu usage stats and weird
 > load average Date: Mon, 04 Aug 2008 18:59:15 +0200
 > 
 >  Jakub Jasinski wrote:
 >  > No.
 >  > 
 >  > I ran ktrace on the processes and ktrace.out was empty.
 >  > 
 >  > Read my follow up message: I already had a lengthy conversation
 >  > with Remko about this.
 >  > 
 >  > NONE of the processes were running.  The non-zero TIME is nonsense
 >  > as well and only accumulated after rsync was left running.
 >  
 >  That would be a very strange situation indeed, requiring multiple
 > things to go wrong in bizarre ways.  The more likely explanation is
 > that the processes are in fact running and being reported correctly.
 >  
 >  Kris
 > _______________________________________________
 > freebsd-bugs@freebsd.org mailing list
 > http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 > To unsubscribe, send any mail to
 > "freebsd-bugs-unsubscribe@freebsd.org"

From: Kris Kennaway <kris@FreeBSD.org>
To: Jakub Jasinski <jakub.jasinski@utoronto.ca>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load average
Date: Tue, 05 Aug 2008 18:48:57 +0200

 Jakub Jasinski wrote:
 > I recompiled the kernel with the ULE scheduler and the only difference
 > now is that every 'working' process has higher WCPU _after_ rsync is
 > activated. Before rsync is activated all process have 0% WCPU.
 > 
 > BTW, thanks for the help.
 > 
 > [root@mail2 ~]# vmstat -i
 > interrupt                          total       rate
 > irq1: atkbd0                          78          0
 > irq4: sio0                          2023          5
 > irq14: ata0                           58          0
 > irq16: uhci0 ehci0                     1          0
 > irq18: ciss0 uhci2                  2618          6
 > irq22: uhci4                          19          0
 > cpu0: timer                       751994       1994
 > irq257: bce1                        2271          6
 > cpu1: timer                       744491       1974
 > cpu2: timer                       744494       1974
 > cpu3: timer                       744513       1974
 > Total                            2992560       7937
 
 Your system is sharing an interrupt between the ciss and one of the USB 
 controllers.  This means that every time there is an interrupt from disk 
 I/O the USB drivers will also wake up to process it.  This could be 
 causing things like moused and cups to run.  What devices are attached 
 to uhci2?
 
 Kris
 
 
 > (snipped top output:)
 > 
 > 60 processes:  1 running, 59 sleeping
 > CPU states:  1.0% user,  0.0% nice, 38.4% system,  2.4% interrupt, 58.1%
 > Mem: 142M Active, 333M Inact, 284M Wired, 52K Cache, 214M Buf, 7154M Fre
 > Swap: 8192M Total, 8192M Free
 > 
 >   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMM
 >   662 root          2   4    0 14112K  2504K select 1   0:10 13.43% apcu
 >  1020 root          1  55    0 17196K 12096K select 3   0:08 11.57% rsyn
 >   756 root          1   4    0  9940K  1260K select 2   0:05  7.08% auth
 >   498 root          1   4    0  5808K  1196K select 3   0:05  6.40% mous
 >   471 root          1   4    0  5808K  1196K select 0   0:04  5.86% mous
 >   890 mailnull      1   4    0  9252K  2024K select 0   0:04  5.08% mime
 >   981 root          1   4    0 30108K  5292K select 2   0:04  4.98% send
 >   969 admin         1   4    0 32936K  4608K select 1   0:04  4.88% sshd
 >  1001 admin         1   4    0 32936K  4612K select 1   0:04  4.88% sshd
 >   791 root          1   4    0  9940K  1260K select 3   0:04  4.88% auth
 >   613 root          1   4    0  4684K  1352K select 0   0:04  4.79% sysl
 >   784 root          1   4    0  3604K  1224K select 0   0:04  4.79% cour
 >   792 root          1   4    0  9940K  1260K select 2   0:04  4.79% auth
 >  1018 root          1   4    0 17196K 11808K select 0   0:04  4.69% rsyn
 >   793 root          1   4    0  9940K  1260K select 3   0:04  4.59% auth
 >   719 root          1   4    0 27284K 10392K select 2   0:04  4.59% perl
 >   799 root          1   4    0  3604K  1224K select 2   0:04  4.59% cour
 >   789 root          1   4    0  9940K  1260K select 1   0:04  4.49% auth
 >   790 root          1   4    0  9940K  1260K select 2   0:04  4.49% auth
 >   907 mailnull      2  20    0  4888K  1536K sigwai 1   0:04  4.49% mime
 >   833 root          1   4    0 20960K  3920K select 1   0:04  4.49% sshd
 >   774 root          1   4    0  3604K  1220K select 3   0:04  4.39% cour
 > 
 > On Mon, 4 Aug 2008 17:00:11 GMT
 > Kris Kennaway <kris@freebsd.org> wrote:
 > 
 >> The following reply was made to PR misc/126230; it has been noted by
 >> GNATS.
 >>
 >> From: Kris Kennaway <kris@FreeBSD.org>
 >> To: Jakub Jasinski <jakub.jasinski@utoronto.ca>
 >> Cc: freebsd-gnats-submit@FreeBSD.org
 >> Subject: Re: misc/126230: weird process cpu usage stats and weird
 >> load average Date: Mon, 04 Aug 2008 18:59:15 +0200
 >>
 >>  Jakub Jasinski wrote:
 >>  > No.
 >>  > 
 >>  > I ran ktrace on the processes and ktrace.out was empty.
 >>  > 
 >>  > Read my follow up message: I already had a lengthy conversation
 >>  > with Remko about this.
 >>  > 
 >>  > NONE of the processes were running.  The non-zero TIME is nonsense
 >>  > as well and only accumulated after rsync was left running.
 >>  
 >>  That would be a very strange situation indeed, requiring multiple
 >> things to go wrong in bizarre ways.  The more likely explanation is
 >> that the processes are in fact running and being reported correctly.
 >>  
 >>  Kris
 >> _______________________________________________
 >> freebsd-bugs@freebsd.org mailing list
 >> http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 >> To unsubscribe, send any mail to
 >> "freebsd-bugs-unsubscribe@freebsd.org"
 > 
 > 
 

From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: Kris Kennaway <kris@FreeBSD.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load
 average
Date: Tue, 5 Aug 2008 13:35:45 -0400

 uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
 0x1040-0x105f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED]
 uhci2: [ITHREAD]
 usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
 usb2: USB revision 1.0
 uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
 uhub2: 2 ports with 2 removable, self powered
 ukbd0: <No brand KVM A1, class 0/0, rev 1.10/0.00, addr 2> on uhub2
 kbd2 at ukbd0
 ums0: <No brand KVM A1, class 0/0, rev 1.10/0.00, addr 2> on uhub2
 ums0: 5 buttons and Z dir.
 
 Ok, so the keyboard and mouse are there.  I detached them, killed
 moused, rerun rsync and I got the same weird top output:
 
 FreeBSD 7.0-RELEASE (CUSTOM) #2: Mon Aug  4 12:44:23 EDT 2008
 last pid:  1258;  load averages:  0.00,  0.00,  0.00   up 0+01:09:53
 13:20:17 last pid:  1266;  load averages:  1.98,  0.55,  0.20   up 0
 +01:11:33  13:21:57 58 processes:  1 running, 57 sleeping
 CPU states:  0.9% user,  0.0% nice, 36.0% system,  3.5% interrupt,
 59.6% idle Mem: 142M Active, 59M Inact, 278M Wired, 48K Cache, 214M
 Buf, 7433M Free Swap: 8192M Total, 8192M Free
 
   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
 COMMAND
 662 root 2   4    0 14112K  2504K select 3   0:15 12.65% apcupsd
 1266 root 1  54    0 17196K 12112K select 0 0:03 10.50% rsync
 756 root 1   4    0  9940K  1260K select 2 0:08  6.40% authdaemon
 1259 root 1  48    0  7656K  2328K CPU1   1   0:02  5.27% top
 981 root 1   4    0 30108K  5292K select 2   0:06  4.59% sendmail
 1265 root 1   4    0 17196K 11824K select 3   0:02  4.59% rsync
 790 root 1   4    0 9940K  1260K select 1   0:06  4.49% authdaemon
 774 root 1 4    0  3604K  1220K select 2   0:06  4.49% couriertcp
 791 root 1   4    0  9940K  1260K select 0   0:06  4.49% authdaemon
 
 etc.
 
 ...and why are apcupsd (connected via serial port),  syslogd,
 couriertcpd, sendmail, etc. working a the _exact_ same time for the
 same duration at around the same WPCU%?
 
 Just for comparison, this is a summary of top on FreeBSD 6.2 (32 bit) on
 another non-deployed box running pretty much the same software (courier
 imap, clamav, spamassassin, etc.) after rsync is activated:
 43.48% for one rsync process
 1.96% for another
 0.00% for everything else.
 
 On Tue, 5 Aug 2008 16:50:04 GMT
 Kris Kennaway <kris@FreeBSD.org> wrote:
 
 >  Jakub Jasinski wrote:
 >  > I recompiled the kernel with the ULE scheduler and the only
 >  > difference now is that every 'working' process has higher WCPU
 >  > _after_ rsync is activated. Before rsync is activated all process
 >  > have 0% WCPU.
 >  > 
 >  > BTW, thanks for the help.
 >  > 
 >  > [root@mail2 ~]# vmstat -i
 >  > interrupt                          total       rate
 >  > irq1: atkbd0                          78          0
 >  > irq4: sio0                          2023          5
 >  > irq14: ata0                           58          0
 >  > irq16: uhci0 ehci0                     1          0
 >  > irq18: ciss0 uhci2                  2618          6
 >  > irq22: uhci4                          19          0
 >  > cpu0: timer                       751994       1994
 >  > irq257: bce1                        2271          6
 >  > cpu1: timer                       744491       1974
 >  > cpu2: timer                       744494       1974
 >  > cpu3: timer                       744513       1974
 >  > Total                            2992560       7937
 >  
 >  Your system is sharing an interrupt between the ciss and one of the
 > USB controllers.  This means that every time there is an interrupt
 > from disk I/O the USB drivers will also wake up to process it.  This
 > could be causing things like moused and cups to run.  What devices
 > are attached to uhci2?
 >  

From: Kris Kennaway <kris@FreeBSD.org>
To: Jakub Jasinski <jakub.jasinski@utoronto.ca>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load average
Date: Tue, 05 Aug 2008 20:08:36 +0200

 Jakub Jasinski wrote:
 > uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
 > 0x1040-0x105f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED]
 > uhci2: [ITHREAD]
 > usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
 > usb2: USB revision 1.0
 > uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
 > uhub2: 2 ports with 2 removable, self powered
 > ukbd0: <No brand KVM A1, class 0/0, rev 1.10/0.00, addr 2> on uhub2
 > kbd2 at ukbd0
 > ums0: <No brand KVM A1, class 0/0, rev 1.10/0.00, addr 2> on uhub2
 > ums0: 5 buttons and Z dir.
 > 
 > Ok, so the keyboard and mouse are there.  I detached them, killed
 > moused, rerun rsync and I got the same weird top output:
 
 Are the devices still attached (i.e. did they detach)?  What if you 
 compile a kernel with no USB support at all (obviously you won't be able 
 to log in with a USB keyboard, but you can still log in remotely).
 
 Kris
 
 
 > 
 > FreeBSD 7.0-RELEASE (CUSTOM) #2: Mon Aug  4 12:44:23 EDT 2008
 > last pid:  1258;  load averages:  0.00,  0.00,  0.00   up 0+01:09:53
 > 13:20:17 last pid:  1266;  load averages:  1.98,  0.55,  0.20   up 0
 > +01:11:33  13:21:57 58 processes:  1 running, 57 sleeping
 > CPU states:  0.9% user,  0.0% nice, 36.0% system,  3.5% interrupt,
 > 59.6% idle Mem: 142M Active, 59M Inact, 278M Wired, 48K Cache, 214M
 > Buf, 7433M Free Swap: 8192M Total, 8192M Free
 > 
 >   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
 > COMMAND
 > 662 root 2   4    0 14112K  2504K select 3   0:15 12.65% apcupsd
 > 1266 root 1  54    0 17196K 12112K select 0 0:03 10.50% rsync
 > 756 root 1   4    0  9940K  1260K select 2 0:08  6.40% authdaemon
 > 1259 root 1  48    0  7656K  2328K CPU1   1   0:02  5.27% top
 > 981 root 1   4    0 30108K  5292K select 2   0:06  4.59% sendmail
 > 1265 root 1   4    0 17196K 11824K select 3   0:02  4.59% rsync
 > 790 root 1   4    0 9940K  1260K select 1   0:06  4.49% authdaemon
 > 774 root 1 4    0  3604K  1220K select 2   0:06  4.49% couriertcp
 > 791 root 1   4    0  9940K  1260K select 0   0:06  4.49% authdaemon
 > 
 > etc.
 > 
 > ...and why are apcupsd (connected via serial port),  syslogd,
 > couriertcpd, sendmail, etc. working a the _exact_ same time for the
 > same duration at around the same WPCU%?
 > 
 > Just for comparison, this is a summary of top on FreeBSD 6.2 (32 bit) on
 > another non-deployed box running pretty much the same software (courier
 > imap, clamav, spamassassin, etc.) after rsync is activated:
 > 43.48% for one rsync process
 > 1.96% for another
 > 0.00% for everything else.
 > 
 > On Tue, 5 Aug 2008 16:50:04 GMT
 > Kris Kennaway <kris@FreeBSD.org> wrote:
 > 
 >>  Jakub Jasinski wrote:
 >>  > I recompiled the kernel with the ULE scheduler and the only
 >>  > difference now is that every 'working' process has higher WCPU
 >>  > _after_ rsync is activated. Before rsync is activated all process
 >>  > have 0% WCPU.
 >>  > 
 >>  > BTW, thanks for the help.
 >>  > 
 >>  > [root@mail2 ~]# vmstat -i
 >>  > interrupt                          total       rate
 >>  > irq1: atkbd0                          78          0
 >>  > irq4: sio0                          2023          5
 >>  > irq14: ata0                           58          0
 >>  > irq16: uhci0 ehci0                     1          0
 >>  > irq18: ciss0 uhci2                  2618          6
 >>  > irq22: uhci4                          19          0
 >>  > cpu0: timer                       751994       1994
 >>  > irq257: bce1                        2271          6
 >>  > cpu1: timer                       744491       1974
 >>  > cpu2: timer                       744494       1974
 >>  > cpu3: timer                       744513       1974
 >>  > Total                            2992560       7937
 >>  
 >>  Your system is sharing an interrupt between the ciss and one of the
 >> USB controllers.  This means that every time there is an interrupt
 >> from disk I/O the USB drivers will also wake up to process it.  This
 >> could be causing things like moused and cups to run.  What devices
 >> are attached to uhci2?
 >>  
 > 
 > 
 

From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: Kris Kennaway <kris@FreeBSD.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load
 average
Date: Tue, 5 Aug 2008 15:27:51 -0400

 The devices detached.  But as you suggested, I recompiled the kernel w/o
 usb.
 
 mail2# vmstat -i
 interrupt                          total       rate
 irq1: atkbd0                           6          0
 irq4: sio0                          2281          4
 irq14: ata0                           58          0
 irq18: ciss0                       18638         37
 cpu0: timer                       985022       1998
 irq257: bce1                      804740       1632
 cpu1: timer                       976993       1981
 cpu2: timer                       976992       1981
 cpu3: timer                       976991       1981
 Total                            4741721       9618
 
 Same thing is happening:
 
 (Maybe something is wrong with 'select' or some other socket
 thing... ???)
 
 Here's a shot of top with '22 running' processes (after rsync is
 activated)--what is sendmail doing eating up 8.89%, apcupsd 15.09%,
 syslogd 5.19%?:
 
 59 processes:  22 running, 37 sle 4.62,  1.53,  up 0+00:04:07  15:08:30
 CPU:     % user6 running, 50 sleeping, 3 lock
 Mem:  0.7 Active, 0.0M Inact,38.8M Wired, 5 3.0ache, 214M Bu57.5976M
 Free                  87                                            31
 Swap: 8192M Total, 8192M Free
 
   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COM
   590 root          2   4    0 14112K  2516K CPU0   0   0:15 15.09% apc
   930 root          1   4    0 17196K 12132K RUN    2   0:12 12.16% rsy
   767 root          1   4    0 30108K  5208K select 3   0:09  8.89% sen
   684 root          1   4    0  9940K  1260K select 3   0:08  7.67% aut
   835 mailnull      2  20    0  4888K  1536K sigwai 2   0:06  5.76% mim
   710 root          1   4    0  9940K  1260K select 3   0:06  5.57% aut
   711 root          1   4    0  9940K  1260K select 0   0:06  5.47% aut
   712 root          1   4    0  9940K  1260K select 0   0:06  5.47% aut
   908 admin         1   4    0 32936K  4624K RUN    0   0:06  5.47% ssh
   647 root          1   4    0 27284K 10392K select 3   0:06  5.37% per
   752 root          1   4    0 20960K  3956K select 0   0:06  5.37% ssh
   928 root          1   4    0 17196K 11844K RUN    3   0:06  5.27% rsy
   818 mailnull      1   4    0  9252K  2024K select 0   0:06  5.27% mim
   713 root          1   4    0  9940K  1260K select 2   0:06  5.27% aut
   727 root          1   4    0  3604K  1224K select 0   0:06  5.27% cou
   541 root          1   4    0  4684K  1352K *Giant 3   0:06  5.18% sys
   702 root          1   4    0  3604K  1220K select 2   0:06  5.18% cou
   709 root          1   4    0  9940K  1260K select 3   0:06  5.18% aut
   474 root          1   4    0  1616K   628K *Giant 2   0:06  5.08% dev
   717 root          1   4    0  3604K  1224K select 2   0:06  4.98% cou
   892 admin         1   4    0 32936K  4620K CPU0   1   0:06  4.88% ssh
 
 Without rsync:
 
 57 processes:  1 running, 56 sleeping
 CPU:     % user,     % nice,     % system,     % interrupt,     % idle
 Mem: 132M Active, 643M Inact, 247M Wired, 52K Cache, 214M Buf, 6891M
 Free Swap: 8192M Total, 8192M Free
 
   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COM
   590 root          2   4    0 14112K  2516K select 3   0:16  0.00% apc
   767 root          1  44    0 30108K  5208K select 0   0:10  0.00% sen
   684 root          1  44    0  9940K  1260K select 0   0:09  0.00% aut
   711 root          1  44    0  9940K  1260K select 2   0:06  0.00% aut
   835 mailnull      2  20    0  4888K  1536K sigwai 2   0:06  0.00% mim
   818 mailnull      1  44    0  9252K  2024K select 3   0:06  0.00% mim
   713 root          1  44    0  9940K  1260K select 1   0:06  0.00% aut
   712 root          1  44    0  9940K  1260K select 0   0:06  0.00% aut
   710 root          1  44    0  9940K  1260K select 3   0:06  0.00% aut
   908 admin         1  46    0 32936K  4624K select 2   0:06  0.00% ssh
   647 root          1  44    0 27284K 10392K select 2   0:06  0.00% per
   541 root          1  44    0  4684K  1352K select 1   0:06  0.00% sys
   702 root          1   4    0  3604K  1220K select 1   0:06  0.00% cou
   892 admin         1  44    0 32936K  4620K select 0   0:06  0.00% ssh
   727 root          1   4    0  3604K  1224K select 3   0:06  0.00% cou
   709 root          1  44    0  9940K  1260K select 1   0:06  0.00% aut
   693 root          1   4    0  3604K  1220K select 2   0:06  0.00% cou
   717 root          1   4    0  3604K  1224K select 2   0:06  0.00% cou
   752 root          1   4    0 20960K  3956K select 2   0:06  0.00% ssh
   474 root          1   4    0  1616K   628K select 1   0:06  0.00% dev
   732 mailnull      1   4    0 83628K 74432K accept 3   0:03  0.00% cla
 
 
 
 On Tue, 05 Aug 2008 20:08:36 +0200
 Kris Kennaway <kris@FreeBSD.org> wrote:
 
 > Jakub Jasinski wrote:
 > > uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
 > > 0x1040-0x105f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED]
 > > uhci2: [ITHREAD]
 > > usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
 > > usb2: USB revision 1.0
 > > uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on
 > > usb2 uhub2: 2 ports with 2 removable, self powered
 > > ukbd0: <No brand KVM A1, class 0/0, rev 1.10/0.00, addr 2> on uhub2
 > > kbd2 at ukbd0
 > > ums0: <No brand KVM A1, class 0/0, rev 1.10/0.00, addr 2> on uhub2
 > > ums0: 5 buttons and Z dir.
 > > 
 > > Ok, so the keyboard and mouse are there.  I detached them, killed
 > > moused, rerun rsync and I got the same weird top output:
 > 
 > Are the devices still attached (i.e. did they detach)?  What if you 
 > compile a kernel with no USB support at all (obviously you won't be
 > able to log in with a USB keyboard, but you can still log in
 > remotely).
 > 
 > Kris
 > 
 > 
 > > 
 > > FreeBSD 7.0-RELEASE (CUSTOM) #2: Mon Aug  4 12:44:23 EDT 2008
 > > last pid:  1258;  load averages:  0.00,  0.00,  0.00   up 0+01:09:53
 > > 13:20:17 last pid:  1266;  load averages:  1.98,  0.55,  0.20   up 0
 > > +01:11:33  13:21:57 58 processes:  1 running, 57 sleeping
 > > CPU states:  0.9% user,  0.0% nice, 36.0% system,  3.5% interrupt,
 > > 59.6% idle Mem: 142M Active, 59M Inact, 278M Wired, 48K Cache, 214M
 > > Buf, 7433M Free Swap: 8192M Total, 8192M Free
 > > 
 > >   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
 > > COMMAND
 > > 662 root 2   4    0 14112K  2504K select 3   0:15 12.65% apcupsd
 > > 1266 root 1  54    0 17196K 12112K select 0 0:03 10.50% rsync
 > > 756 root 1   4    0  9940K  1260K select 2 0:08  6.40% authdaemon
 > > 1259 root 1  48    0  7656K  2328K CPU1   1   0:02  5.27% top
 > > 981 root 1   4    0 30108K  5292K select 2   0:06  4.59% sendmail
 > > 1265 root 1   4    0 17196K 11824K select 3   0:02  4.59% rsync
 > > 790 root 1   4    0 9940K  1260K select 1   0:06  4.49% authdaemon
 > > 774 root 1 4    0  3604K  1220K select 2   0:06  4.49% couriertcp
 > > 791 root 1   4    0  9940K  1260K select 0   0:06  4.49% authdaemon
 > > 
 > > etc.
 > > 
 > > ...and why are apcupsd (connected via serial port),  syslogd,
 > > couriertcpd, sendmail, etc. working a the _exact_ same time for the
 > > same duration at around the same WPCU%?
 > > 
 > > Just for comparison, this is a summary of top on FreeBSD 6.2 (32
 > > bit) on another non-deployed box running pretty much the same
 > > software (courier imap, clamav, spamassassin, etc.) after rsync is
 > > activated: 43.48% for one rsync process
 > > 1.96% for another
 > > 0.00% for everything else.
 > > 
 > > On Tue, 5 Aug 2008 16:50:04 GMT
 > > Kris Kennaway <kris@FreeBSD.org> wrote:
 > > 
 > >>  Jakub Jasinski wrote:
 > >>  > I recompiled the kernel with the ULE scheduler and the only
 > >>  > difference now is that every 'working' process has higher WCPU
 > >>  > _after_ rsync is activated. Before rsync is activated all
 > >>  > process have 0% WCPU.
 > >>  > 
 > >>  > BTW, thanks for the help.
 > >>  > 
 > >>  > [root@mail2 ~]# vmstat -i
 > >>  > interrupt                          total       rate
 > >>  > irq1: atkbd0                          78          0
 > >>  > irq4: sio0                          2023          5
 > >>  > irq14: ata0                           58          0
 > >>  > irq16: uhci0 ehci0                     1          0
 > >>  > irq18: ciss0 uhci2                  2618          6
 > >>  > irq22: uhci4                          19          0
 > >>  > cpu0: timer                       751994       1994
 > >>  > irq257: bce1                        2271          6
 > >>  > cpu1: timer                       744491       1974
 > >>  > cpu2: timer                       744494       1974
 > >>  > cpu3: timer                       744513       1974
 > >>  > Total                            2992560       7937
 > >>  
 > >>  Your system is sharing an interrupt between the ciss and one of
 > >> the USB controllers.  This means that every time there is an
 > >> interrupt from disk I/O the USB drivers will also wake up to
 > >> process it.  This could be causing things like moused and cups to
 > >> run.  What devices are attached to uhci2?
 > >>  
 > > 
 > > 
 > 

From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: Kris Kennaway <kris@FreeBSD.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load
 average
Date: Tue, 5 Aug 2008 17:05:11 -0400

 It looks like a select-related bug.
 
 I compiled the sample select program (except that it waits for 120
 seconds):
 
 #include <stdio.h>
 #include <sys/time.h>
 #include <sys/types.h>
 #include <unistd.h>
 
 int
 main(void)
 {
         fd_set          rfds;
         struct timeval  tv;
         int             retval;
 
         /* Watch stdin (fd 0) to see when it has input. */
         FD_ZERO(&rfds);
         FD_SET(0, &rfds);
         /* Wait up to five seconds. */
         tv.tv_sec = 120;
         tv.tv_usec = 0;
 
         retval = select(1, &rfds, NULL, NULL, &tv);
         /* Don't rely on the value of tv now! */
 
         if (retval == -1)
                 perror("select()");
         else if (retval)
                 printf("Data is available now.\n");
         /* FD_ISSET(0, &rfds) will be true. */
         else
                 printf("No data within 120 seconds.\n");
 
         return 0;
 }
 
 And this is what I get after running rsync:
 
 last pid:  1193;  load averages:  0.00,  0.00,  0.00          up 0
 +01:47:18  16:51:41 63 processes:  1 running, 62 sleeping
 CPU states:  0.3% user,  0.0% nice, 19.7% system,  1.0% interrupt,
 78.9% idle Mem: 146M Active, 236M Inact, 247M Wired, 116K Cache, 214M
 Buf, 7284M Free Swap: 8192M Total, 8192M Free
 
   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
 COMMAN
 590 root 2   4    0 14112K  2516K select 2   0:20 9.42% apcupsd
 1193 root 1 -16    0 17196K 12176K wdrain 0 0:03  8.15% rsync
 767 root 1   4    0 30108K  5208K select 0 0:12  5.96% sendmail
 684 root 1   4    0  9940K  1260K select 3   0:11  5.08% authdaemond
 1192 root          1  46    0  7656K  2192K CPU0   0   0:01  3.66% top
 1188 admin         1   4    0  2532K   660K select 3   0:02  3.56% a.out
 
 [snipped]
 
 a.out is consuming 3.56% while select-ing.  That's funny. :)
 
 On Tue, 5 Aug 2008 18:10:03 GMT
 Kris Kennaway <kris@FreeBSD.org> wrote:
 
 > The following reply was made to PR misc/126230; it has been noted by
 > GNATS.
 > 
 > From: Kris Kennaway <kris@FreeBSD.org>
 > To: Jakub Jasinski <jakub.jasinski@utoronto.ca>
 > Cc: freebsd-gnats-submit@FreeBSD.org
 > Subject: Re: misc/126230: weird process cpu usage stats and weird
 > load average Date: Tue, 05 Aug 2008 20:08:36 +0200
 > 
 >  Jakub Jasinski wrote:
 >  > uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
 >  > 0x1040-0x105f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED]
 >  > uhci2: [ITHREAD]
 >  > usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
 >  > usb2: USB revision 1.0
 >  > uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on
 >  > usb2 uhub2: 2 ports with 2 removable, self powered
 >  > ukbd0: <No brand KVM A1, class 0/0, rev 1.10/0.00, addr 2> on uhub2
 >  > kbd2 at ukbd0
 >  > ums0: <No brand KVM A1, class 0/0, rev 1.10/0.00, addr 2> on uhub2
 >  > ums0: 5 buttons and Z dir.
 >  > 
 >  > Ok, so the keyboard and mouse are there.  I detached them, killed
 >  > moused, rerun rsync and I got the same weird top output:
 >  
 >  Are the devices still attached (i.e. did they detach)?  What if you 
 >  compile a kernel with no USB support at all (obviously you won't be
 > able to log in with a USB keyboard, but you can still log in
 > remotely). 
 >  Kris
 >  
 >  
 >  > 
 >  > FreeBSD 7.0-RELEASE (CUSTOM) #2: Mon Aug  4 12:44:23 EDT 2008
 >  > last pid:  1258;  load averages:  0.00,  0.00,  0.00   up 0
 >  > +01:09:53 13:20:17 last pid:  1266;  load averages:  1.98,  0.55,
 >  > 0.20   up 0 +01:11:33  13:21:57 58 processes:  1 running, 57
 >  > sleeping CPU states:  0.9% user,  0.0% nice, 36.0% system,  3.5%
 >  > interrupt, 59.6% idle Mem: 142M Active, 59M Inact, 278M Wired, 48K
 >  > Cache, 214M Buf, 7433M Free Swap: 8192M Total, 8192M Free
 >  > 
 >  >   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
 >  > COMMAND
 >  > 662 root 2   4    0 14112K  2504K select 3   0:15 12.65% apcupsd
 >  > 1266 root 1  54    0 17196K 12112K select 0 0:03 10.50% rsync
 >  > 756 root 1   4    0  9940K  1260K select 2 0:08  6.40% authdaemon
 >  > 1259 root 1  48    0  7656K  2328K CPU1   1   0:02  5.27% top
 >  > 981 root 1   4    0 30108K  5292K select 2   0:06  4.59% sendmail
 >  > 1265 root 1   4    0 17196K 11824K select 3   0:02  4.59% rsync
 >  > 790 root 1   4    0 9940K  1260K select 1   0:06  4.49% authdaemon
 >  > 774 root 1 4    0  3604K  1220K select 2   0:06  4.49% couriertcp
 >  > 791 root 1   4    0  9940K  1260K select 0   0:06  4.49% authdaemon
 >  > 
 >  > etc.
 >  > 
 >  > ...and why are apcupsd (connected via serial port),  syslogd,
 >  > couriertcpd, sendmail, etc. working a the _exact_ same time for the
 >  > same duration at around the same WPCU%?
 >  > 
 >  > Just for comparison, this is a summary of top on FreeBSD 6.2 (32
 >  > bit) on another non-deployed box running pretty much the same
 >  > software (courier imap, clamav, spamassassin, etc.) after rsync is
 >  > activated: 43.48% for one rsync process
 >  > 1.96% for another
 >  > 0.00% for everything else.
 >  > 
 >  > On Tue, 5 Aug 2008 16:50:04 GMT
 >  > Kris Kennaway <kris@FreeBSD.org> wrote:
 >  > 
 >  >>  Jakub Jasinski wrote:
 >  >>  > I recompiled the kernel with the ULE scheduler and the only
 >  >>  > difference now is that every 'working' process has higher WCPU
 >  >>  > _after_ rsync is activated. Before rsync is activated all
 >  >>  > process have 0% WCPU.
 >  >>  > 
 >  >>  > BTW, thanks for the help.
 >  >>  > 
 >  >>  > [root@mail2 ~]# vmstat -i
 >  >>  > interrupt                          total       rate
 >  >>  > irq1: atkbd0                          78          0
 >  >>  > irq4: sio0                          2023          5
 >  >>  > irq14: ata0                           58          0
 >  >>  > irq16: uhci0 ehci0                     1          0
 >  >>  > irq18: ciss0 uhci2                  2618          6
 >  >>  > irq22: uhci4                          19          0
 >  >>  > cpu0: timer                       751994       1994
 >  >>  > irq257: bce1                        2271          6
 >  >>  > cpu1: timer                       744491       1974
 >  >>  > cpu2: timer                       744494       1974
 >  >>  > cpu3: timer                       744513       1974
 >  >>  > Total                            2992560       7937
 >  >>  
 >  >>  Your system is sharing an interrupt between the ciss and one of
 >  >> the USB controllers.  This means that every time there is an
 >  >> interrupt from disk I/O the USB drivers will also wake up to
 >  >> process it.  This could be causing things like moused and cups to
 >  >> run.  What devices are attached to uhci2?
 >  >>  
 >  > 
 >  > 
 >  
 > _______________________________________________
 > freebsd-bugs@freebsd.org mailing list
 > http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 > To unsubscribe, send any mail to
 > "freebsd-bugs-unsubscribe@freebsd.org"

From: Gavin Atkinson <gavin@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: misc/126230: weird process cpu usage stats and weird load
Date: Thu, 7 Aug 2008 21:35:17 +0100 (BST)

 Just a thought: what is the output of "sysctl dev.cpu.0" both before, 
 and after running rsync?  If for some reason your CPU is ending up in a 
 very low frequency mode, then it's not inconceivable that daemons would 
 appear to take a much greater share of the CPU than usual.  Also, I'm 
 assuming the clock maintains time normally?

From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: Gavin Atkinson <gavin@freebsd.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load
Date: Thu, 7 Aug 2008 21:58:55 -0400

 Please note that all of the daemons that are consuming CPU and a.out
 (the compiled sample select() program) are select()-ing.  For
 example, look at the top output in the bug description and my last
 post with sample code.
 
 Also, please refer to (I filed a new bug report--this one more
 specific than "weird something or other"):
 http://www.freebsd.org/cgi/query-pr.cgi?pr=126289 (and refer to my
 previous post in this bug report.)
 
 I don't think a.out should be consuming ANY cpu while it's select()-ing,
 even on a 386.
 
 When comparing kern_select() in 6.2 and 7.0, the one slight difference
 is how thread locking is invoked...
 
 In any event, before and after:
 
 mail2# sysctl dev.cpu.0
 dev.cpu.0.%desc: ACPI CPU
 dev.cpu.0.%driver: cpu
 dev.cpu.0.%location: handle=\_PR_.CPU0
 dev.cpu.0.%pnpinfo: _HID=none _UID=0
 dev.cpu.0.%parent: acpi0
 dev.cpu.0.freq: 2322
 dev.cpu.0.freq_levels: 2322/-1 2031/-1 1741/-1 1451/-1 1161/-1 870/-1
 580/-1 290/-1 dev.cpu.0.cx_supported: C1/0
 dev.cpu.0.cx_lowest: C1
 dev.cpu.0.cx_usage: 100.00%
 
 mail2# sysctl dev.cpu.0
 dev.cpu.0.%desc: ACPI CPU
 dev.cpu.0.%driver: cpu
 dev.cpu.0.%location: handle=\_PR_.CPU0
 dev.cpu.0.%pnpinfo: _HID=none _UID=0
 dev.cpu.0.%parent: acpi0
 dev.cpu.0.freq: 2322
 dev.cpu.0.freq_levels: 2322/-1 2031/-1 1741/-1 1451/-1 1161/-1 870/-1
 580/-1 290/-1 dev.cpu.0.cx_supported: C1/0
 dev.cpu.0.cx_lowest: C1
 dev.cpu.0.cx_usage: 100.00%
 
 Thanks anyway.
 
 In the meantime, I'm going to try FreeBSD 6.3.
 
 On Thu, 7 Aug 2008 20:40:06 GMT
 Gavin Atkinson <gavin@freebsd.org> wrote:
 
 > The following reply was made to PR misc/126230; it has been noted by
 > GNATS.
 > 
 > From: Gavin Atkinson <gavin@FreeBSD.org>
 > To: bug-followup@FreeBSD.org
 > Cc:  
 > Subject: Re: misc/126230: weird process cpu usage stats and weird load
 > Date: Thu, 7 Aug 2008 21:35:17 +0100 (BST)
 > 
 >  Just a thought: what is the output of "sysctl dev.cpu.0" both
 > before, and after running rsync?  If for some reason your CPU is
 > ending up in a very low frequency mode, then it's not inconceivable
 > that daemons would appear to take a much greater share of the CPU
 > than usual.  Also, I'm assuming the clock maintains time normally?
 > _______________________________________________
 > freebsd-bugs@freebsd.org mailing list
 > http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 > To unsubscribe, send any mail to
 > "freebsd-bugs-unsubscribe@freebsd.org"

From: Jakub Jasinski <jakub.jasinski@utoronto.ca>
To: Gavin Atkinson <gavin@freebsd.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/126230: weird process cpu usage stats and weird load
Date: Thu, 7 Aug 2008 22:21:42 -0400

 I killed most daemons to make the problem more obvious:
 
 It seems rather peculiar how a.out (which is select()-ing; see the code
 in my previous post or the other bug report) is using 7.23% cpu.
 
 last pid: 17533;  load averages:  0.00,  0.00,  0.00          up 2
 +02:41:29  22:00:50 32 processes:  1 running, 31 sleeping
 CPU states:  0.0% user,  0.0% nice,  0.1% system,  0.0% interrupt,
 99.9% idle last pid: 17536;  load averages:  0.99,  0.24,
 0.08          up 2+02:43:02  22:02:23 34 processes:  6 running, 28
 sleeping CPU states:  0.7% user,  0.0% nice, 19.3% system,  2.6%
 interrupt, 77.4% idle Mem: 29M Active, 723M Inact, 374M Wired, 88K
 Cache, 214M Buf, 6787M Free Swap: 8192M Total, 8192M Free
 
   PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
 COMMAND
 17536 root 1   4    0 18220K 12548K RUN    3   0:07 11.10% rsync
 17532 root 1   4    0  2532K   660K select  3 0:05  7.23% a.out
 832 root 1   4    0 20960K  4000K select 2 0:14  7.23% sshd
 17300 admin 1   4    0 32936K  4624K select 2   0:06  6.79% sshd
 17535 root 1  96    0  7656K  2152K CPU1   1   0:05  6.67% top
 612 root 1   4    0  4684K  1352K CPU0   0   0:14  6.64% syslogd
 17534 root 1   4    0 17196K 12256K RUN    2   0:05  6.62% rsync
 545 root 1   4    0 1616K   628K select 0   0:14  6.25% devd
 1004 admin 1   4    0 32936K  4624K CPU2   0   0:14  6.05% sshd
 17261 admin 1   4 0 32936K  4632K CPU3   3   0:06  5.66% sshd
 848 root 1   8 0  5736K  1396K nanslp 2   0:00  0.00% cron
 17304 root 1 20    0  9616K  2824K pause  0   0:00  0.00% csh
 913 smmsp 1 20    0 11720K  3972K pause  0   0:00  0.00% sendmail
 1022 root 1   5    0  8692K  2760K ttyin  2   0:00  0.00% bash
 17256 root 1   4    0 32936K  4568K sbwait 1   0:00  0.00% sshd
 17297 root 1   4    0 32936K  4560K sbwait 1   0:00 0.00% sshd
 988 root 1   4    0 32936K  4560K sbwait 1   0:00 0.00% sshd
 17274 root 1   8    0  8692K  2768K wait  3 0:00  0.00% bash
 17269 root 1  20    0  9616K  2788K pause 0   0:00  0.00% csh
 1018 root 1  20    0  9616K  2748K pause 0   0:00  0.00% csh
 1006 admin 1   8    0  8692K  2740K wait 2   0:00  0.00% bash
 17263 admin 1   8    0  8692K  2752K wait   3   0:00  0.00% bash
 17303 admin 1   8    0 19400K 1808K wait 3   0:00  0.00% su
 17268 admin 1   8    0 19400K 1808K wait   2   0:00 0.00% su
 17302 admin 1   8    0  8692K 2752K wait   3   0:00  0.00% bash
 1017 admin 1   8    0 19400K 1808K wait   3   0:00  0.00% su
 953 root 1   5    0  4668K 1200K ttyin  1   0:00  0.00% getty
 954 root 1   5    0  4668K 1200K ttyin  1   0:00  0.00% getty
 955 root 1   5    0  4668K 1200K ttyin  2   0:00  0.00% getty
 
 Now, get cracking and fix select(). ;)
 
 On Thu, 7 Aug 2008 20:40:06 GMT
 Gavin Atkinson <gavin@freebsd.org> wrote:
 
 > The following reply was made to PR misc/126230; it has been noted by
 > GNATS.
 > 
 > From: Gavin Atkinson <gavin@FreeBSD.org>
 > To: bug-followup@FreeBSD.org
 > Cc:  
 > Subject: Re: misc/126230: weird process cpu usage stats and weird load
 > Date: Thu, 7 Aug 2008 21:35:17 +0100 (BST)
 > 
 >  Just a thought: what is the output of "sysctl dev.cpu.0" both
 > before, and after running rsync?  If for some reason your CPU is
 > ending up in a very low frequency mode, then it's not inconceivable
 > that daemons would appear to take a much greater share of the CPU
 > than usual.  Also, I'm assuming the clock maintains time normally?
 > _______________________________________________
 > freebsd-bugs@freebsd.org mailing list
 > http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 > To unsubscribe, send any mail to
 > "freebsd-bugs-unsubscribe@freebsd.org"
>Unformatted:
