From nobody@FreeBSD.org  Fri Apr 22 10:14:08 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 1FA641065676
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 22 Apr 2011 10:14:08 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 0F6368FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 22 Apr 2011 10:14:08 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p3MAE7s3017273
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 22 Apr 2011 10:14:07 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p3MAE7RA017272;
	Fri, 22 Apr 2011 10:14:07 GMT
	(envelope-from nobody)
Message-Id: <201104221014.p3MAE7RA017272@red.freebsd.org>
Date: Fri, 22 Apr 2011 10:14:07 GMT
From: Igor Soumenkov <igor@soumenkov.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Add EV_CLEAR to AIO events in kqueue
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         156567
>Category:       kern
>Synopsis:       [kqueue] [patch] Add EV_CLEAR to AIO events in kqueue
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Fri Apr 22 10:20:01 UTC 2011
>Closed-Date:    Thu Mar 01 06:53:36 UTC 2012
>Last-Modified:  Thu Mar 01 06:53:36 UTC 2012
>Originator:     Igor Soumenkov
>Release:        8.2-STABLE
>Organization:
>Environment:
FreeBSD igosha2 8.2-STABLE FreeBSD 8.2-STABLE #6: Fri Apr 22 13:53:04 MSD 2011     root@igosha2:/usr/obj/usr/src/sys/IGOSHA-WORK-8  amd64

>Description:
Current implementation of AIO does not set the EV_CLEAR flag when adding new AIO events in the kqueue. This causes problems while fetching kevents from several threads. I can't imagine the situation when the AIO event should be read many times, so I suggest adding EV_CLEAR to all AIO events.
>How-To-Repeat:
Schedule aio operation, i.e. aio_read or aio_write with kqueue notification. If you use kevent in several threads, several copies of the EVFILT_AIO events will be fetched. 
>Fix:
I suggest adding EV_CLEAR to the flags of the AIO event, since there is no way to control it with aio_ operations directly. 
If it is not acceptable maybe there should be a sysctl or other tunable option.

Patch attached with submission follows:

--- sys/kern/vfs_aio.c.old	2011-04-22 13:49:37.000000000 +0400
+++ sys/kern/vfs_aio.c	2011-04-22 13:49:45.000000000 +0400
@@ -1622,7 +1622,7 @@
 	kqfd = aiocbe->uaiocb.aio_sigevent.sigev_notify_kqueue;
 	kev.ident = (uintptr_t)aiocbe->uuaiocb;
 	kev.filter = EVFILT_AIO;
-	kev.flags = EV_ADD | EV_ENABLE | EV_FLAG1;
+	kev.flags = EV_ADD | EV_ENABLE | EV_CLEAR | EV_FLAG1;
 	kev.data = (intptr_t)aiocbe;
 	kev.udata = aiocbe->uaiocb.aio_sigevent.sigev_value.sival_ptr;
 	error = kqfd_register(kqfd, &kev, td, 1);


>Release-Note:
>Audit-Trail:

From: igor <igor@soumenkov.com>
To: <bug-followup@FreeBSD.org>, <igor@soumenkov.com>
Cc:  
Subject: Re: kern/156567: [kqueue] [patch] Add =?UTF-8?Q?EV=5FCLEAR=20to?=
 =?UTF-8?Q?=20AIO=20events=20in=20kqueue?=
Date: Mon, 08 Aug 2011 15:09:12 +0400

 Hello? Anyone out there? There's a ready patch and nobody even comments 
 it...
 
 -- 
 Igor Soumenkov

From: igor <igor@soumenkov.com>
To: <bug-followup@FreeBSD.org>, <freebsd-bugs@freebsd.org>
Cc:  
Subject: Re: kern/156567: [kqueue] [patch] Add =?UTF-8?Q?EV=5FCLEAR=20to?=
 =?UTF-8?Q?=20AIO=20events=20in=20kqueue?=
Date: Tue, 31 Jan 2012 00:47:04 +0400

 --=_264fe93c4c9b89e33a0046ba700be42a
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain; charset=UTF-8
 
   
 
 Dear all, 
 
 What can I do to get my patch finally applied or
 rejected (hope not :) ? My PR is not even assigned to anyone. Should I
 do my best to become a committer (how? :-) ? 
 
 The solution I propose
 finally allows to use kqueue with aio in multi-threaded applications.
 The only reason I created it is because I wanted to use it for myself
 and it worked flawlessly. 
 
 Please, anyone? 
 
 -- 
 Igor Soumenkov
   
 --=_264fe93c4c9b89e33a0046ba700be42a
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/html; charset=UTF-8
 
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
 <html><body>
 <p>Dear all,</p>
 <p>What can I do to get my patch finally applied or rejected (hope not :) ?=
  My PR is not even assigned to anyone. Should I do my best to become a comm=
 itter (how? :-) ?</p>
 <p>The solution I propose finally allows to use kqueue with aio in multi-th=
 readed applications. The only reason I created it is because I wanted to us=
 e it for myself and it worked flawlessly.</p>
 <p>Please, anyone?</p>
 <div>
 <pre>--=20
 Igor Soumenkov</pre>
 </div>
 </body></html>
 
 --=_264fe93c4c9b89e33a0046ba700be42a--
 

From: David Xu <listlog2011@gmail.com>
To: igor <igor@soumenkov.com>
Cc: bug-followup@FreeBSD.org, freebsd-bugs@FreeBSD.org
Subject: Re: kern/156567: [kqueue] [patch] Add EV_CLEAR to AIO events in kqueue
Date: Tue, 31 Jan 2012 16:04:06 +0800

 On 2012/1/31 4:47, igor wrote:
 >
 >
 > Dear all,
 >
 > What can I do to get my patch finally applied or
 > rejected (hope not :) ? My PR is not even assigned to anyone. Should I
 > do my best to become a committer (how? :-) ?
 >
 > The solution I propose
 > finally allows to use kqueue with aio in multi-threaded applications.
 > The only reason I created it is because I wanted to use it for myself
 > and it worked flawlessly.
 >
 > Please, anyone?
 >
 It is better to allocate a member field from struct sigevent for kevent
 flags, we have spare fields available there, please check the union
 _sigev_un.
 
 Regards,
 David Xu
 

From: igor <igor@soumenkov.com>
To: <davidxu@FreeBSD.org>
Cc: <bug-followup@FreeBSD.org>, <freebsd-bugs@FreeBSD.org>
Subject: Re: kern/156567: [kqueue] [patch] Add =?UTF-8?Q?EV=5FCLEAR=20to?=
 =?UTF-8?Q?=20AIO=20events=20in=20kqueue?=
Date: Tue, 31 Jan 2012 14:52:10 +0400

 --=_3349fc38fa29cf5858e92f7cc8080a49
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain; charset=UTF-8
 
   
 
 Hello David, 
 
 The only thing I changed is the value of the flag,
 this is the only meaning of the patch. I do not know who decided to
 place the value there. 
 
 On Tue, 31 Jan 2012 16:04:06 +0800, David Xu
 wrote: 
 
 > On 2012/1/31 4:47, igor wrote:
 > 
 >> Dear all, What can I do
 to get my patch finally applied or rejected (hope not :) ? My PR is not
 even assigned to anyone. Should I do my best to become a committer (how?
 :-) ? The solution I propose finally allows to use kqueue with aio in
 multi-threaded applications. The only reason I created it is because I
 wanted to use it for myself and it worked flawlessly. Please, anyone?
 >
 
 > It is better to allocate a member field from struct sigevent for
 kevent
 > flags, we have spare fields available there, please check the
 union
 > _sigev_un.
 > 
 > Regards,
 > David Xu
 
 -- 
 Igor Soumenkov
   
 --=_3349fc38fa29cf5858e92f7cc8080a49
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/html; charset=UTF-8
 
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN">
 <html><body>
 <p>Hello David,</p>
 <p>The only thing I changed is the value of the flag, this is the only mean=
 ing of the patch. I do not know who decided to place the value there.</p>
 <p>On Tue, 31 Jan 2012 16:04:06 +0800, David Xu wrote:</p>
 <blockquote type=3D"cite" style=3D"padding-left:5px; border-left:#1010ff 2p=
 x solid; margin-left:5px; width:100%"><!-- html ignored --><!-- head ignore=
 d --><!-- meta ignored -->
 <pre>On 2012/1/31 4:47, igor wrote:</pre>
 <blockquote type=3D"cite" style=3D"padding-left:5px; border-left:#1010ff 2p=
 x solid; margin-left:5px; width:100%">Dear all, What can I do to get my pat=
 ch finally applied or rejected (hope not :) ? My PR is not even assigned to=
  anyone. Should I do my best to become a committer (how? :-) ? The solution=
  I propose finally allows to use kqueue with aio in multi-threaded applicat=
 ions. The only reason I created it is because I wanted to use it for myself=
  and it worked flawlessly. Please, anyone?</blockquote>
 <pre>It is better to allocate a member field from struct sigevent for keven=
 t
 flags, we have spare fields available there, please check the union
 _sigev_un.
 
 Regards,
 David Xu
 
 </pre>
 </blockquote>
 <p>&nbsp;</p>
 <div>
 <pre>--=20
 Igor Soumenkov</pre>
 </div>
 </body></html>
 
 --=_3349fc38fa29cf5858e92f7cc8080a49--
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/156567: commit references a PR
Date: Wed,  1 Feb 2012 02:53:15 +0000 (UTC)

 Author: davidxu
 Date: Wed Feb  1 02:53:06 2012
 New Revision: 230857
 URL: http://svn.freebsd.org/changeset/base/230857
 
 Log:
   If multiple threads call kevent() to get AIO events on same kqueue fd,
   it is possible that a single AIO event will be reported to multiple
   threads, it is not threading friendly, and the existing API can not
   control this behavior.
   Allocate a kevent flags field sigev_notify_kevent_flags for AIO event
   notification in sigevent, and allow user to pass EV_CLEAR, EV_DISPATCH
   or EV_ONESHOT to AIO kernel code, user can control whether the event
   should be cleared once it is retrieved by a thread. This change should
   be comptaible with existing application, because the field should have
   already been zero-filled, and no additional action will be taken by
   kernel.
   
   PR:	kern/156567
 
 Modified:
   head/sys/kern/vfs_aio.c
   head/sys/sys/signal.h
 
 Modified: head/sys/kern/vfs_aio.c
 ==============================================================================
 --- head/sys/kern/vfs_aio.c	Wed Feb  1 02:16:15 2012	(r230856)
 +++ head/sys/kern/vfs_aio.c	Wed Feb  1 02:53:06 2012	(r230857)
 @@ -1524,6 +1524,7 @@ aio_aqueue(struct thread *td, struct aio
  	int error;
  	int fd, kqfd;
  	int jid;
 +	u_short evflags;
  
  	if (p->p_aioinfo == NULL)
  		aio_init_aioinfo(p);
 @@ -1646,10 +1647,15 @@ aio_aqueue(struct thread *td, struct aio
  
  	if (aiocbe->uaiocb.aio_sigevent.sigev_notify != SIGEV_KEVENT)
  		goto no_kqueue;
 +	evflags = aiocbe->uaiocb.aio_sigevent.sigev_notify_kevent_flags;
 +	if ((evflags & ~(EV_CLEAR | EV_DISPATCH | EV_ONESHOT)) != 0) {
 +		error = EINVAL;
 +		goto aqueue_fail;
 +	}
  	kqfd = aiocbe->uaiocb.aio_sigevent.sigev_notify_kqueue;
  	kev.ident = (uintptr_t)aiocbe->uuaiocb;
  	kev.filter = EVFILT_AIO;
 -	kev.flags = EV_ADD | EV_ENABLE | EV_FLAG1;
 +	kev.flags = EV_ADD | EV_ENABLE | EV_FLAG1 | evflags;
  	kev.data = (intptr_t)aiocbe;
  	kev.udata = aiocbe->uaiocb.aio_sigevent.sigev_value.sival_ptr;
  	error = kqfd_register(kqfd, &kev, td, 1);
 
 Modified: head/sys/sys/signal.h
 ==============================================================================
 --- head/sys/sys/signal.h	Wed Feb  1 02:16:15 2012	(r230856)
 +++ head/sys/sys/signal.h	Wed Feb  1 02:53:06 2012	(r230857)
 @@ -169,12 +169,14 @@ struct sigevent {
  			void (*_function)(union sigval);
  			void *_attribute; /* pthread_attr_t * */
  		} _sigev_thread;
 +		unsigned short _kevent_flags;
  		long __spare__[8];
  	} _sigev_un;
  };
  
  #if __BSD_VISIBLE
  #define	sigev_notify_kqueue		sigev_signo
 +#define	sigev_notify_kevent_flags	_sigev_un._kevent_flags
  #define	sigev_notify_thread_id		_sigev_un._threadid
  #endif
  #define	sigev_notify_function		_sigev_un._sigev_thread._function
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: davidxu 
State-Changed-When: Thu Feb 2 01:22:27 UTC 2012 
State-Changed-Why:  
Committed with slight change. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=156567 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/156567: commit references a PR
Date: Wed, 29 Feb 2012 06:19:16 +0000 (UTC)

 Author: davidxu
 Date: Wed Feb 29 06:19:00 2012
 New Revision: 232290
 URL: http://svn.freebsd.org/changeset/base/232290
 
 Log:
   MFC 230857:
   
   If multiple threads call kevent() to get AIO events on same kqueue fd,
   it is possible that a single AIO event will be reported to multiple
   threads, it is not threading friendly, and the existing API can not
   control this behavior.
   Allocate a kevent flags field sigev_notify_kevent_flags for AIO event
   notification in sigevent, and allow user to pass EV_CLEAR, EV_DISPATCH
   or EV_ONESHOT to AIO kernel code, user can control whether the event
   should be cleared once it is retrieved by a thread. This change should
   be comptaible with existing application, because the field should have
   already been zero-filled, and no additional action will be taken by
   kernel.
   
   PR:	kern/156567
   
   MFC 231006:
   
   Add 32-bit compat code for AIO kevent flags introduced in revision 230857.
   
   MFC 231724:
   
   Add notes about sigev_notify_kevent_flags introduced in revision 230857
   which enables thread-friendly polling on same fd for AIO events.
   
   Reviewed by:	delphij
   
   MFC 231777:
   
   Bump .Dd date for previous revision.
 
 Modified:
   stable/9/lib/libc/sys/kqueue.2
   stable/9/sys/compat/freebsd32/freebsd32_signal.h
   stable/9/sys/kern/vfs_aio.c
   stable/9/sys/sys/signal.h
 Directory Properties:
   stable/9/lib/libc/   (props changed)
   stable/9/lib/libc/stdtime/   (props changed)
   stable/9/lib/libc/sys/   (props changed)
   stable/9/sys/   (props changed)
   stable/9/sys/amd64/include/xen/   (props changed)
   stable/9/sys/boot/   (props changed)
   stable/9/sys/boot/i386/efi/   (props changed)
   stable/9/sys/boot/ia64/efi/   (props changed)
   stable/9/sys/boot/ia64/ski/   (props changed)
   stable/9/sys/boot/powerpc/boot1.chrp/   (props changed)
   stable/9/sys/boot/powerpc/ofw/   (props changed)
   stable/9/sys/cddl/contrib/opensolaris/   (props changed)
   stable/9/sys/conf/   (props changed)
   stable/9/sys/contrib/dev/acpica/   (props changed)
   stable/9/sys/contrib/octeon-sdk/   (props changed)
   stable/9/sys/contrib/pf/   (props changed)
   stable/9/sys/contrib/x86emu/   (props changed)
   stable/9/sys/i386/conf/XENHVM   (props changed)
 
 Modified: stable/9/lib/libc/sys/kqueue.2
 ==============================================================================
 --- stable/9/lib/libc/sys/kqueue.2	Wed Feb 29 05:48:29 2012	(r232289)
 +++ stable/9/lib/libc/sys/kqueue.2	Wed Feb 29 06:19:00 2012	(r232290)
 @@ -24,7 +24,7 @@
  .\"
  .\" $FreeBSD$
  .\"
 -.Dd September 15, 2009
 +.Dd February 15, 2012
  .Dt KQUEUE 2
  .Os
  .Sh NAME
 @@ -322,6 +322,9 @@ The sigevent portion of the AIO request 
  .Va sigev_notify_kqueue
  containing the descriptor of the kqueue that the event should
  be attached to,
 +.Va sigev_notify_kevent_flags
 +containing the kevent flags which should be EV_ONESHOT, EV_CLEAR or
 +EV_DISPATCH,
  .Va sigev_value
  containing the udata value, and
  .Va sigev_notify
 
 Modified: stable/9/sys/compat/freebsd32/freebsd32_signal.h
 ==============================================================================
 --- stable/9/sys/compat/freebsd32/freebsd32_signal.h	Wed Feb 29 05:48:29 2012	(r232289)
 +++ stable/9/sys/compat/freebsd32/freebsd32_signal.h	Wed Feb 29 06:19:00 2012	(r232290)
 @@ -92,6 +92,7 @@ struct sigevent32 {
  			uint32_t _function;
  			uint32_t _attribute;
  		} _sigev_thread;
 +		unsigned short	_kevent_flags;
  		uint32_t __spare__[8];
  	} _sigev_un;
  };
 
 Modified: stable/9/sys/kern/vfs_aio.c
 ==============================================================================
 --- stable/9/sys/kern/vfs_aio.c	Wed Feb 29 05:48:29 2012	(r232289)
 +++ stable/9/sys/kern/vfs_aio.c	Wed Feb 29 06:19:00 2012	(r232290)
 @@ -1524,6 +1524,7 @@ aio_aqueue(struct thread *td, struct aio
  	int error;
  	int fd, kqfd;
  	int jid;
 +	u_short evflags;
  
  	if (p->p_aioinfo == NULL)
  		aio_init_aioinfo(p);
 @@ -1640,10 +1641,15 @@ aio_aqueue(struct thread *td, struct aio
  
  	if (aiocbe->uaiocb.aio_sigevent.sigev_notify != SIGEV_KEVENT)
  		goto no_kqueue;
 +	evflags = aiocbe->uaiocb.aio_sigevent.sigev_notify_kevent_flags;
 +	if ((evflags & ~(EV_CLEAR | EV_DISPATCH | EV_ONESHOT)) != 0) {
 +		error = EINVAL;
 +		goto aqueue_fail;
 +	}
  	kqfd = aiocbe->uaiocb.aio_sigevent.sigev_notify_kqueue;
  	kev.ident = (uintptr_t)aiocbe->uuaiocb;
  	kev.filter = EVFILT_AIO;
 -	kev.flags = EV_ADD | EV_ENABLE | EV_FLAG1;
 +	kev.flags = EV_ADD | EV_ENABLE | EV_FLAG1 | evflags;
  	kev.data = (intptr_t)aiocbe;
  	kev.udata = aiocbe->uaiocb.aio_sigevent.sigev_value.sival_ptr;
  	error = kqfd_register(kqfd, &kev, td, 1);
 @@ -2688,6 +2694,7 @@ convert_sigevent32(struct sigevent32 *si
  		break;
  	case SIGEV_KEVENT:
  		CP(*sig32, *sig, sigev_notify_kqueue);
 +		CP(*sig32, *sig, sigev_notify_kevent_flags);
  		PTRIN_CP(*sig32, *sig, sigev_value.sival_ptr);
  		break;
  	default:
 
 Modified: stable/9/sys/sys/signal.h
 ==============================================================================
 --- stable/9/sys/sys/signal.h	Wed Feb 29 05:48:29 2012	(r232289)
 +++ stable/9/sys/sys/signal.h	Wed Feb 29 06:19:00 2012	(r232290)
 @@ -169,12 +169,14 @@ struct sigevent {
  			void (*_function)(union sigval);
  			void *_attribute; /* pthread_attr_t * */
  		} _sigev_thread;
 +		unsigned short _kevent_flags;
  		long __spare__[8];
  	} _sigev_un;
  };
  
  #if __BSD_VISIBLE
  #define	sigev_notify_kqueue		sigev_signo
 +#define	sigev_notify_kevent_flags	_sigev_un._kevent_flags
  #define	sigev_notify_thread_id		_sigev_un._threadid
  #endif
  #define	sigev_notify_function		_sigev_un._sigev_thread._function
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: davidxu 
State-Changed-When: Thu Mar 1 06:53:14 UTC 2012 
State-Changed-Why:  
Fixed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=156567 
>Unformatted:
