From nobody@FreeBSD.org  Mon Jun  4 01:16:49 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 0328F106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  4 Jun 2012 01:16:49 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id E366A8FC0A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  4 Jun 2012 01:16:48 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q541Gms1066188
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 4 Jun 2012 01:16:48 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q541GmBw066187;
	Mon, 4 Jun 2012 01:16:48 GMT
	(envelope-from nobody)
Message-Id: <201206040116.q541GmBw066187@red.freebsd.org>
Date: Mon, 4 Jun 2012 01:16:48 GMT
From: Adrian Chadd <adrian@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [ath] TX frames sometimes occur out of order
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         168649
>Category:       kern
>Synopsis:       [ath] TX frames sometimes occur out of order
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jun 04 01:20:12 UTC 2012
>Closed-Date:    
>Last-Modified:  Tue Jun  5 06:10:13 UTC 2012
>Originator:     Adrian Chadd
>Release:        9.0-STABLE i386, with -HEAD net80211/ath
>Organization:
>Environment:
FreeBSD marilyn 9.0-STABLE FreeBSD 9.0-STABLE #3 r235930M: Thu May 31 07:18:55 PDT 2012     adrian@marilyn:/home/adrian/work/freebsd/stable/9/sys/i386/compile/MARILYN-9-STABLE  i386

>Description:
Under heavy enough load (250MBit TX), I'm seeing out of order frames being transmitted from the station.

This is with iperf, using a 250MBit UDP stream from a FreeBSD STA (SMP) to a FreeBSD AP, to a FreeBSD iperf server.

Doing a 250Mbit UDP stream from a macosx laptop doesn't cause the iperf server to warn about out of order frames.

After some code inspection, I discovered that there are multiple overlapping ath_start() entries.  For multiple traffic sources, there are already multiple ath_start() entries occuring - potentially one per TX'ing process. but for the single TX process, there's two;

* iperf -> sosend() -> net80211_output -> if_transmit -> ath_start();
* ath0 taskqueue -> {rx | tx completion } -> ath_start();

(and the reset, channel change paths..)

What I'm guessing is happening here is that the ath_start() processing is being preempted or racing with another, between where the frame dequeue from the ifnet and subsequent ath_tx_start() call occurs.

Since there's no lock being held across this, and no serialisation being imposed, it's possible they're overlapping slightly.

When I threw ath_start() into a taskqueue and just made ath_start() queue the taskqueue, the out of order frames went away.

I haven't yet tried this with just adding a lock around the whole of ath_start(), but that should also work.

I had issues with the taskqueue method when acpi is enabled, as there seems to be some much greater scheduling latency. I'll create a new PR for that.

>How-To-Repeat:

>Fix:
Not yet..

>Release-Note:
>Audit-Trail:

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/168649: commit references a PR
Date: Mon,  4 Jun 2012 22:01:27 +0000 (UTC)

 Author: adrian
 Date: Mon Jun  4 22:01:12 2012
 New Revision: 236583
 URL: http://svn.freebsd.org/changeset/base/236583
 
 Log:
   Migrate the TX path to a taskqueue for now, until a better way of
   implementing parallel TX and TX/RX completion can be done without
   simply abusing long-held locks.
   
   Right now, multiple concurrent ath_start() entries can result in
   frames being dequeued out of order.  Well, they're dequeued in order
   fine, but if there's any preemption or race between CPUs between:
   
   * removing the frame from the ifnet, and
   * calling and runningath_tx_start(), until the frame is placed on a
     software or hardware TXQ
   
   Then although dequeueing the frame is in-order, queueing it to the hardware
   may be out of order.
   
   This is solved in a lot of other drivers by just holding a TX lock over
   a rather long period of time.  This lets them continue to direct dispatch
   without races between dequeue and hardware queue.
   
   Note to observers: if_transmit() doesn't necessarily solve this.
   It removes the ifnet from the main path, but the same issue exists if
   there's some intermediary queue (eg a bufring, which as an aside also
   may pull in ifnet when you're using ALTQ.)
   
   So, until I can sit down and code up a much better way of doing parallel
   TX, I'm going to leave the TX path using a deferred taskqueue task.
   What I will likely head towards is doing a direct dispatch to hardware
   or software via if_transmit(), but it'll require some driver changes to
   allow queues to be made without using the really large ath_buf / ath_desc
   entries.
   
   TODO:
   
   * Look at how feasible it'll be to just do direct dispatch to
     ath_tx_start() from if_transmit(), avoiding doing _any_ intermediary
     serialisation into a global queue.  This may break ALTQ for example,
     so I have to be delicate.
   
   * It's quite likely that I should break up ath_tx_start() so it
     deposits frames onto the software queues first, and then only fill
     in the 802.11 fields when it's being queued to the hardware.
     That will make the if_transmit() -> software queue path very
     quick and lightweight.
   
   * This has some very bad behaviour when using ACPI and Cx states.
     I'll do some subsequent analysis using KTR and schedgraph and file
     a follow-up PR or two.
   
   PR:		kern/168649
 
 Modified:
   head/sys/dev/ath/if_ath.c
   head/sys/dev/ath/if_ath_misc.h
   head/sys/dev/ath/if_ath_rx.c
   head/sys/dev/ath/if_athvar.h
 
 Modified: head/sys/dev/ath/if_ath.c
 ==============================================================================
 --- head/sys/dev/ath/if_ath.c	Mon Jun  4 21:34:49 2012	(r236582)
 +++ head/sys/dev/ath/if_ath.c	Mon Jun  4 22:01:12 2012	(r236583)
 @@ -373,6 +373,7 @@ ath_attach(u_int16_t devid, struct ath_s
  		"%s taskq", ifp->if_xname);
  
  	TASK_INIT(&sc->sc_rxtask, 0, ath_rx_tasklet, sc);
 +	TASK_INIT(&sc->sc_txstarttask, 0, ath_tx_tasklet, sc);
  	TASK_INIT(&sc->sc_bmisstask, 0, ath_bmiss_proc, sc);
  	TASK_INIT(&sc->sc_bstucktask,0, ath_bstuck_proc, sc);
  	TASK_INIT(&sc->sc_resettask,0, ath_reset_proc, sc);
 @@ -2325,6 +2326,15 @@ void
  ath_start(struct ifnet *ifp)
  {
  	struct ath_softc *sc = ifp->if_softc;
 +
 +	taskqueue_enqueue(sc->sc_tq, &sc->sc_txstarttask);
 +}
 +
 +void
 +ath_tx_tasklet(void *arg, int npending)
 +{
 +	struct ath_softc *sc = (struct ath_softc *) arg;
 +	struct ifnet *ifp = sc->sc_ifp;
  	struct ieee80211_node *ni;
  	struct ath_buf *bf;
  	struct mbuf *m, *next;
 @@ -3499,7 +3509,8 @@ ath_tx_proc_q0(void *arg, int npending)
  	sc->sc_txproc_cnt--;
  	ATH_PCU_UNLOCK(sc);
  
 -	ath_start(ifp);
 +	// ath_start(ifp);
 +	ath_tx_tasklet(sc, 1);
  }
  
  /*
 @@ -3549,7 +3560,8 @@ ath_tx_proc_q0123(void *arg, int npendin
  	sc->sc_txproc_cnt--;
  	ATH_PCU_UNLOCK(sc);
  
 -	ath_start(ifp);
 +	//ath_start(ifp);
 +	ath_tx_tasklet(sc, 1);
  }
  
  /*
 @@ -3592,7 +3604,8 @@ ath_tx_proc(void *arg, int npending)
  	sc->sc_txproc_cnt--;
  	ATH_PCU_UNLOCK(sc);
  
 -	ath_start(ifp);
 +	//ath_start(ifp);
 +	ath_tx_tasklet(sc, 1);
  }
  #undef	TXQACTIVE
  
 
 Modified: head/sys/dev/ath/if_ath_misc.h
 ==============================================================================
 --- head/sys/dev/ath/if_ath_misc.h	Mon Jun  4 21:34:49 2012	(r236582)
 +++ head/sys/dev/ath/if_ath_misc.h	Mon Jun  4 22:01:12 2012	(r236583)
 @@ -83,6 +83,7 @@ extern void ath_setslottime(struct ath_s
   * we can kill this.
   */
  extern void ath_start(struct ifnet *ifp);
 +extern void ath_tx_tasklet(void *arg, int npending);
  
  
  #endif
 
 Modified: head/sys/dev/ath/if_ath_rx.c
 ==============================================================================
 --- head/sys/dev/ath/if_ath_rx.c	Mon Jun  4 21:34:49 2012	(r236582)
 +++ head/sys/dev/ath/if_ath_rx.c	Mon Jun  4 22:01:12 2012	(r236583)
 @@ -899,7 +899,8 @@ rx_proc_next:
  		ieee80211_ff_age_all(ic, 100);
  #endif
  		if (!IFQ_IS_EMPTY(&ifp->if_snd))
 -			ath_start(ifp);
 +			ath_tx_tasklet(sc, 1);
 +			//ath_start(ifp);
  	}
  #undef PA2DESC
  
 
 Modified: head/sys/dev/ath/if_athvar.h
 ==============================================================================
 --- head/sys/dev/ath/if_athvar.h	Mon Jun  4 21:34:49 2012	(r236582)
 +++ head/sys/dev/ath/if_athvar.h	Mon Jun  4 22:01:12 2012	(r236583)
 @@ -476,6 +476,7 @@ struct ath_softc {
  	struct mbuf		*sc_rxpending;	/* pending receive data */
  	u_int32_t		*sc_rxlink;	/* link ptr in last RX desc */
  	struct task		sc_rxtask;	/* rx int processing */
 +	struct task		sc_txstarttask;	/* ath_start() processing */
  	u_int8_t		sc_defant;	/* current default antenna */
  	u_int8_t		sc_rxotherant;	/* rx's on non-default antenna*/
  	u_int64_t		sc_lastrx;	/* tsf at last rx'd frame */
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/168649: commit references a PR
Date: Tue,  5 Jun 2012 03:15:00 +0000 (UTC)

 Author: adrian
 Date: Tue Jun  5 03:14:49 2012
 New Revision: 236597
 URL: http://svn.freebsd.org/changeset/base/236597
 
 Log:
   Create a function - ath_tx_kick() - which is called where ath_start() is
   called to "kick" along TX.
   
   For now, schedule a taskqueue call.
   
   Later on I may go back to the direct call of ath_rx_tasklet() - but for
   now, this will do.
   
   I've tested UDP and TCP TX. UDP TX still achieves 240MBit, but TCP
   TX gets stuck at around 100MBit or so, instead of the 150MBit it should
   be at.  I'll re-test with no ACPI/power/sleep states enabled at startup
   and see what effect it has.
   
   This is in preparation for supporting an if_transmit() path, which will
   turn ath_tx_kick() into a NUL operation (as there won't be an ifnet
   queue to service.)
   
   Tested:
   	* AR9280 STA
   
   TODO:
   	* test on AR5416, AR9160, AR928x STA/AP modes
   
   PR:		kern/168649
 
 Modified:
   head/sys/dev/ath/if_ath.c
   head/sys/dev/ath/if_ath_misc.h
   head/sys/dev/ath/if_ath_rx.c
 
 Modified: head/sys/dev/ath/if_ath.c
 ==============================================================================
 --- head/sys/dev/ath/if_ath.c	Tue Jun  5 03:14:39 2012	(r236596)
 +++ head/sys/dev/ath/if_ath.c	Tue Jun  5 03:14:49 2012	(r236597)
 @@ -142,6 +142,7 @@ static void	ath_vap_delete(struct ieee80
  static void	ath_init(void *);
  static void	ath_stop_locked(struct ifnet *);
  static void	ath_stop(struct ifnet *);
 +static void	ath_tx_tasklet(void *arg, int npending);
  static int	ath_reset_vap(struct ieee80211vap *, u_long);
  static int	ath_media_change(struct ifnet *);
  static void	ath_watchdog(void *);
 @@ -2330,7 +2331,7 @@ ath_start(struct ifnet *ifp)
  	taskqueue_enqueue(sc->sc_tq, &sc->sc_txstarttask);
  }
  
 -void
 +static void
  ath_tx_tasklet(void *arg, int npending)
  {
  	struct ath_softc *sc = (struct ath_softc *) arg;
 @@ -3509,8 +3510,7 @@ ath_tx_proc_q0(void *arg, int npending)
  	sc->sc_txproc_cnt--;
  	ATH_PCU_UNLOCK(sc);
  
 -	// ath_start(ifp);
 -	ath_tx_tasklet(sc, 1);
 +	ath_tx_kick(sc);
  }
  
  /*
 @@ -3560,8 +3560,7 @@ ath_tx_proc_q0123(void *arg, int npendin
  	sc->sc_txproc_cnt--;
  	ATH_PCU_UNLOCK(sc);
  
 -	//ath_start(ifp);
 -	ath_tx_tasklet(sc, 1);
 +	ath_tx_kick(sc);
  }
  
  /*
 @@ -3604,8 +3603,7 @@ ath_tx_proc(void *arg, int npending)
  	sc->sc_txproc_cnt--;
  	ATH_PCU_UNLOCK(sc);
  
 -	//ath_start(ifp);
 -	ath_tx_tasklet(sc, 1);
 +	ath_tx_kick(sc);
  }
  #undef	TXQACTIVE
  
 
 Modified: head/sys/dev/ath/if_ath_misc.h
 ==============================================================================
 --- head/sys/dev/ath/if_ath_misc.h	Tue Jun  5 03:14:39 2012	(r236596)
 +++ head/sys/dev/ath/if_ath_misc.h	Tue Jun  5 03:14:49 2012	(r236597)
 @@ -83,7 +83,17 @@ extern void ath_setslottime(struct ath_s
   * we can kill this.
   */
  extern void ath_start(struct ifnet *ifp);
 -extern void ath_tx_tasklet(void *arg, int npending);
  
 +static inline void
 +ath_tx_kick(struct ath_softc *sc)
 +{
 +
 +	/*
 +	 * Use a taskqueue to schedule a TX completion task,
 +	 * even if we're in taskqueue context.  That way this can
 +	 * be called from any context.
 +	 */
 +	taskqueue_enqueue(sc->sc_tq, &sc->sc_txstarttask);
 +}
  
  #endif
 
 Modified: head/sys/dev/ath/if_ath_rx.c
 ==============================================================================
 --- head/sys/dev/ath/if_ath_rx.c	Tue Jun  5 03:14:39 2012	(r236596)
 +++ head/sys/dev/ath/if_ath_rx.c	Tue Jun  5 03:14:49 2012	(r236597)
 @@ -899,8 +899,7 @@ rx_proc_next:
  		ieee80211_ff_age_all(ic, 100);
  #endif
  		if (!IFQ_IS_EMPTY(&ifp->if_snd))
 -			ath_tx_tasklet(sc, 1);
 -			//ath_start(ifp);
 +			ath_tx_kick(sc);
  	}
  #undef PA2DESC
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/168649: commit references a PR
Date: Tue,  5 Jun 2012 06:04:18 +0000 (UTC)

 Author: adrian
 Date: Tue Jun  5 06:03:55 2012
 New Revision: 236599
 URL: http://svn.freebsd.org/changeset/base/236599
 
 Log:
   Mostly revert previous commit(s).  After doing a bunch of local testing,
   it turns out that it negatively affects performance.  I'm stil investigating
   exactly why deferring the IO causes such negative TCP performance but
   doesn't affect UDP preformance.
   
   Leave the ath_tx_kick() change in there however; it's going to be useful
   to have that there for if_transmit() work.
   
   PR:		kern/168649
 
 Modified:
   head/sys/dev/ath/if_ath.c
   head/sys/dev/ath/if_ath_misc.h
   head/sys/dev/ath/if_athvar.h
 
 Modified: head/sys/dev/ath/if_ath.c
 ==============================================================================
 --- head/sys/dev/ath/if_ath.c	Tue Jun  5 05:16:04 2012	(r236598)
 +++ head/sys/dev/ath/if_ath.c	Tue Jun  5 06:03:55 2012	(r236599)
 @@ -142,7 +142,6 @@ static void	ath_vap_delete(struct ieee80
  static void	ath_init(void *);
  static void	ath_stop_locked(struct ifnet *);
  static void	ath_stop(struct ifnet *);
 -static void	ath_tx_tasklet(void *arg, int npending);
  static int	ath_reset_vap(struct ieee80211vap *, u_long);
  static int	ath_media_change(struct ifnet *);
  static void	ath_watchdog(void *);
 @@ -374,7 +373,6 @@ ath_attach(u_int16_t devid, struct ath_s
  		"%s taskq", ifp->if_xname);
  
  	TASK_INIT(&sc->sc_rxtask, 0, ath_rx_tasklet, sc);
 -	TASK_INIT(&sc->sc_txstarttask, 0, ath_tx_tasklet, sc);
  	TASK_INIT(&sc->sc_bmisstask, 0, ath_bmiss_proc, sc);
  	TASK_INIT(&sc->sc_bstucktask,0, ath_bstuck_proc, sc);
  	TASK_INIT(&sc->sc_resettask,0, ath_reset_proc, sc);
 @@ -2327,15 +2325,6 @@ void
  ath_start(struct ifnet *ifp)
  {
  	struct ath_softc *sc = ifp->if_softc;
 -
 -	taskqueue_enqueue(sc->sc_tq, &sc->sc_txstarttask);
 -}
 -
 -static void
 -ath_tx_tasklet(void *arg, int npending)
 -{
 -	struct ath_softc *sc = (struct ath_softc *) arg;
 -	struct ifnet *ifp = sc->sc_ifp;
  	struct ieee80211_node *ni;
  	struct ath_buf *bf;
  	struct mbuf *m, *next;
 
 Modified: head/sys/dev/ath/if_ath_misc.h
 ==============================================================================
 --- head/sys/dev/ath/if_ath_misc.h	Tue Jun  5 05:16:04 2012	(r236598)
 +++ head/sys/dev/ath/if_ath_misc.h	Tue Jun  5 06:03:55 2012	(r236599)
 @@ -88,12 +88,7 @@ static inline void
  ath_tx_kick(struct ath_softc *sc)
  {
  
 -	/*
 -	 * Use a taskqueue to schedule a TX completion task,
 -	 * even if we're in taskqueue context.  That way this can
 -	 * be called from any context.
 -	 */
 -	taskqueue_enqueue(sc->sc_tq, &sc->sc_txstarttask);
 +	ath_start(sc->sc_ifp);
  }
  
  #endif
 
 Modified: head/sys/dev/ath/if_athvar.h
 ==============================================================================
 --- head/sys/dev/ath/if_athvar.h	Tue Jun  5 05:16:04 2012	(r236598)
 +++ head/sys/dev/ath/if_athvar.h	Tue Jun  5 06:03:55 2012	(r236599)
 @@ -476,7 +476,6 @@ struct ath_softc {
  	struct mbuf		*sc_rxpending;	/* pending receive data */
  	u_int32_t		*sc_rxlink;	/* link ptr in last RX desc */
  	struct task		sc_rxtask;	/* rx int processing */
 -	struct task		sc_txstarttask;	/* ath_start() processing */
  	u_int8_t		sc_defant;	/* current default antenna */
  	u_int8_t		sc_rxotherant;	/* rx's on non-default antenna*/
  	u_int64_t		sc_lastrx;	/* tsf at last rx'd frame */
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
>Unformatted:
