From swell.k@gmail.com  Fri Dec 17 03:53:14 2010
Return-Path: <swell.k@gmail.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 77642106564A
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 17 Dec 2010 03:53:14 +0000 (UTC)
	(envelope-from swell.k@gmail.com)
Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54])
	by mx1.freebsd.org (Postfix) with ESMTP id 307EC8FC13
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 17 Dec 2010 03:53:13 +0000 (UTC)
Received: by qwj9 with SMTP id 9so259887qwj.13
        for <FreeBSD-gnats-submit@freebsd.org>; Thu, 16 Dec 2010 19:53:13 -0800 (PST)
Received: by 10.229.215.70 with SMTP id hd6mr353934qcb.102.1292557993184;
        Thu, 16 Dec 2010 19:53:13 -0800 (PST)
Received: from localhost (anonymizer2.torservers.net [174.36.199.200])
        by mx.google.com with ESMTPS id h20sm579371qck.0.2010.12.16.19.53.09
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 16 Dec 2010 19:53:12 -0800 (PST)
Message-Id: <86oc8lginm.fsf@gmail.com>
Date: Fri, 17 Dec 2010 06:52:45 +0300
From: Anonymous <swell.k@gmail.com>
To: FreeBSD-gnats-submit@freebsd.org
Subject: fetch(1): http transfer hangs after ^T (status) on tty

>Number:         153240
>Category:       bin
>Synopsis:       fetch(1): http transfer hangs after ^T (status) on tty
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    des
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Dec 17 04:00:24 UTC 2010
>Closed-Date:    Sun Feb 17 04:35:13 UTC 2013
>Last-Modified:  Sun Feb 17 04:35:13 UTC 2013
>Originator:     Anonymous
>Release:        FreeBSD 9.0-CURRENT amd64
>Organization:
>Environment:
- running under qemu with minimum configuration
- host system doesn't filter traffic

$ uname -vm
FreeBSD 9.0-HEAD-20101216-JPSNAP #0: Thu Dec 16 07:13:24 UTC 2010     root@build-amd64-fbsd.allbsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
After sending ^T from tty to active http transfer fetch(1) starts to
loop forever without reading any data.

  (from foreground process)
  [...]
  1941 fetch    CALL  gettimeofday(0x7fffffffe7c0,0)
  1941 fetch    RET   gettimeofday 0
  1941 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1941 fetch    RET   ioctl 0
  1941 fetch    CALL  gettimeofday(0x7fffffffe7c0,0)
  1941 fetch    RET   gettimeofday 0
  1941 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1941 fetch    RET   ioctl 0
  1941 fetch    CALL  gettimeofday(0x7fffffffe7c0,0)
  1941 fetch    RET   gettimeofday 0
  1941 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1941 fetch    RET   ioctl 0
  1941 fetch    CALL  gettimeofday(0x7fffffffe7c0,0)
  1941 fetch    RET   gettimeofday 0
  1941 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1941 fetch    RET   ioctl 0
  1941 fetch    CALL  gettimeofday(0x7fffffffe7c0,0)
  1941 fetch    RET   gettimeofday 0
  1941 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1941 fetch    RET   ioctl 0
  [...]

  (from background process)
  [...]
  1902 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1902 fetch    RET   ioctl 0
  1902 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1902 fetch    RET   ioctl 0
  1902 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1902 fetch    RET   ioctl 0
  1902 fetch    CALL  ioctl(0x2,TIOCGPGRP,0x7fffffffe7dc)
  1902 fetch    RET   ioctl 0
  [...]

>How-To-Repeat:
Look how ftp transfer continues to increase file size and how http
transfer is stalled after ^T.

$ fetch http://pub.allbsd.org/FreeBSD-snapshots/amd64-amd64/9.0-HEAD-20101216-JPSNAP/cdrom/FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.iso
FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.i  0% of  412 MB   57 kBps 02h02m
load: 0.11  cmd: fetch 1731 [select] 14.62r 0.07u 0.43s 0% 2980k
FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.i  0% of  412 MB   57 kBps 02h04m
^Z[1] + Suspended               fetch ...
$ bg %
[1] fetch ...
$ (sleep 15; ls -l *.iso; sleep 15; ls -l *.iso)
-rw-r--r--  1 luser  luser  802816 Dec 17 03:14 FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.iso
-rw-r--r--  1 luser  luser  802816 Dec 17 03:14 FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.iso

$ fetch ftp://ftp.allbsd.org/pub/FreeBSD-snapshots/amd64-amd64/9.0-HEAD-20101216-JPSNAP/cdrom/FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.iso
FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.i  0% of  412 MB   99 kBps
load: 0.06  cmd: fetch 1711 [sbwait] 14.90r 0.04u 0.54s 0% 3040k
FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.i  0% of  412 MB  106 kBps 01h08m
FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.i  0% of  412 MB  113 kBps 01h03m
^Z[1] + Suspended               fetch ...
$ bg %
[1] fetch ...
$ (sleep 15; ls -l *.iso; sleep 15; ls -l *.iso)
-rw-r--r--  1 luser  luser  4915200 Dec 17 03:11 FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.iso
-rw-r--r--  1 luser  luser  6914048 Dec 17 03:11 FreeBSD-9.0-HEAD-20101216-JPSNAP-amd64-disc1.iso
>Fix:
>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->analyzed 
State-Changed-By: des 
State-Changed-When: Fri May 13 07:52:06 UTC 2011 
State-Changed-Why:  
Confirmed, no fix yet. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153240 

From: Mark <markjdb@gmail.com>
To: bug-followup@FreeBSD.org, swell.k@gmail.com
Cc:  
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on tty
Date: Thu, 4 Aug 2011 17:38:31 -0400

 There are two ways to fix this: the easy way is to not set
 fetchRestartCalls to 0 in fetch(1) (it's set to 1 by default in
 libfetch). This prevents fetch_read() from returning -1 when a ^T
 interrupts select(2) or read(2), and I don't think this change will
 have any other consequences, though des@ may know more.
 
 The hard way is to fix libfetch's http_readfn() and http_fillbuf() to
 be restartable. http_readfn() calls http_fillbuf() in a loop, and
 http_fillbuf() calls fetch_read(). Handling the case that the >1st
 invocation of fetch_read() is interrupted is tricky, since some
 partial state is left behind in the struct httpio. I will post a patch
 once I'm satisfied that it works properly.
 
 -- 
 Mark Johnston
 markjdb@gmail.com

From: Mark <markjdb@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on tty
Date: Sat, 6 Aug 2011 01:15:47 -0400

 --20cf3071ca847dfc6b04a9cf4eea
 Content-Type: text/plain; charset=ISO-8859-1
 
 Here's the easy patch for fetch(1). This will fix fetch itself, but
 other programs that set fetchRestartCalls to 0 will still have the
 problem.
 
 Tested by downloading a large file over HTTP while sending many
 SIGINFOs and cmp'ing it with another copy of the file.
 
 -- 
 Mark Johnston
 markjdb@gmail.com
 
 --20cf3071ca847dfc6b04a9cf4eea
 Content-Type: text/plain; charset=US-ASCII; name="libfetch_siginfo_easy.patch.txt"
 Content-Disposition: attachment; filename="libfetch_siginfo_easy.patch.txt"
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_gr04liiq0
 
 ZGlmZiAtLWdpdCBhL3Vzci5iaW4vZmV0Y2gvZmV0Y2guYyBiL3Vzci5iaW4vZmV0Y2gvZmV0Y2gu
 YwppbmRleCA3NTUzYmQ4Li5jMDI1MzNjIDEwMDY0NAotLS0gYS91c3IuYmluL2ZldGNoL2ZldGNo
 LmMKKysrIGIvdXNyLmJpbi9mZXRjaC9mZXRjaC5jCkBAIC02MjIsMjAgKzYyMiwxMiBAQCBmZXRj
 aChjaGFyICpVUkwsIGNvbnN0IGNoYXIgKnBhdGgpCiAJCQlzdGF0X2VuZCgmeHMpOwogCQkJc2ln
 aW5mbyA9IDA7CiAJCX0KLQkJaWYgKChzaXplID0gZnJlYWQoYnVmLCAxLCBzaXplLCBmKSkgPT0g
 MCkgewotCQkJaWYgKGZlcnJvcihmKSAmJiBlcnJubyA9PSBFSU5UUiAmJiAhc2lnaW50KQotCQkJ
 CWNsZWFyZXJyKGYpOwotCQkJZWxzZQotCQkJCWJyZWFrOwotCQl9CisJCWlmICgoc2l6ZSA9IGZy
 ZWFkKGJ1ZiwgMSwgc2l6ZSwgZikpID09IDApCisJCQlicmVhazsKIAkJc3RhdF91cGRhdGUoJnhz
 LCBjb3VudCArPSBzaXplKTsKIAkJZm9yIChwdHIgPSBidWY7IHNpemUgPiAwOyBwdHIgKz0gd3Is
 IHNpemUgLT0gd3IpCi0JCQlpZiAoKHdyID0gZndyaXRlKHB0ciwgMSwgc2l6ZSwgb2YpKSA8IHNp
 emUpIHsKLQkJCQlpZiAoZmVycm9yKG9mKSAmJiBlcnJubyA9PSBFSU5UUiAmJiAhc2lnaW50KQot
 CQkJCQljbGVhcmVycihvZik7Ci0JCQkJZWxzZQotCQkJCQlicmVhazsKLQkJCX0KKwkJCWlmICgo
 d3IgPSBmd3JpdGUocHRyLCAxLCBzaXplLCBvZikpIDwgc2l6ZSkKKwkJCQlicmVhazsKIAkJaWYg
 KHNpemUgIT0gMCkKIAkJCWJyZWFrOwogCX0KQEAgLTkyNCw3ICs5MTYsNiBAQCBtYWluKGludCBh
 cmdjLCBjaGFyICphcmd2W10pCiAJc2lnYWN0aW9uKFNJR0FMUk0sICZzYSwgTlVMTCk7CiAJc2Eu
 c2FfZmxhZ3MgPSBTQV9SRVNFVEhBTkQ7CiAJc2lnYWN0aW9uKFNJR0lOVCwgJnNhLCBOVUxMKTsK
 LQlmZXRjaFJlc3RhcnRDYWxscyA9IDA7CiAKIAkvKiBvdXRwdXQgZmlsZSAqLwogCWlmIChvX2Zs
 YWcpIHsK
 --20cf3071ca847dfc6b04a9cf4eea--

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: Mark <markjdb@gmail.com>
Cc: freebsd-gnats-submit@freebsd.org 
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on tty
Date: Wed, 10 Aug 2011 02:27:16 +0200

 Mark <markjdb@gmail.com> writes:
 > There are two ways to fix this [...]  The hard way is to fix
 > libfetch's http_readfn() and http_fillbuf() to be
 > restartable. http_readfn() calls http_fillbuf() in a loop, and
 > http_fillbuf() calls fetch_read().
 
 I prefer the hard way :)
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: Mark <markjdb@gmail.com>
To: bug-followup@freebsd.org
Cc: des@freebsd.org
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on tty
Date: Tue, 6 Sep 2011 10:29:34 -0400

 --20cf3071ca9c13a1c304ac46a8b3
 Content-Type: text/plain; charset=ISO-8859-1
 
 Ok, here's an actual fix. =)
 
 I broke this up into two patches:
 
 The first fixes the actual infinite loop problem: when fetch_read() is
 interrupted by a signal, io->error gets set, and then http_readfn()
 returns immediately the next time it's called. This problem doesn't
 occur with ftp_readfn() because it explicitly checks for EINTR and
 unsets io->err; thus we fix http_readfn() by doing the same thing. We
 also use a temporary variable for the return value of fetch_read to
 avoid clobbering io->buflen if an error is returned.
 
 The second patch fixes fetch_read() to avoid losing data if select()
 is interrupted by SIGINFO. The problem is that some data may have
 already been read from the socket before fetch_read() returns -1. In
 this case, it has to be saved somewhere so that it can be picked up in
 the next fread. This patch adds a few fields to conn_t for this
 purpose - conn->buf can't be used, since fetch_getln uses it when
 calling into fetch_read(). Thus when read() or select() return an
 error with errno == EINTR, we copy any already-read data into
 conn->cache.buf and return -1. On the next call to fetch_read(), the
 cached data is copied back out.
 
 It's also necessary to modify fetch(1) a bit - fread() may get
 interrupted but still return a non-zero value and set an error in the
 FILE pointer. We have to add an explicit "if (size == 0) break;" since
 when the transfer is completed, size gets set to 0.
 
 These patches don't fix fetch_writev(); I'm pretty sure it would have
 the same problem, but it's not immediately clear to me how it should
 be fixed. I will look at doing it. However, I think these patches
 should go in first, since the problem only occurs if fetchRestartCalls
 == 0 (which is not the default) and since fetch(1) doesn't give a way
 to upload files, so fetch_writev() is only called a few times during a
 transfer.
 
 Thanks,
 -Mark
 
 --20cf3071ca9c13a1c304ac46a8b3
 Content-Type: text/plain; charset=US-ASCII; name="libfetch_intr_1of2.patch.txt"
 Content-Disposition: attachment; filename="libfetch_intr_1of2.patch.txt"
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_gs8z52cu0
 
 ZGlmZiAtLWdpdCBhL2xpYi9saWJmZXRjaC9odHRwLmMgYi9saWIvbGliZmV0Y2gvaHR0cC5jCmlu
 ZGV4IDMwOWNiZjIuLjg2OThlY2YgMTAwNjQ0Ci0tLSBhL2xpYi9saWJmZXRjaC9odHRwLmMKKysr
 IGIvbGliL2xpYmZldGNoL2h0dHAuYwpAQCAtMTk2LDYgKzE5Niw4IEBAIGh0dHBfZ3Jvd2J1Zihz
 dHJ1Y3QgaHR0cGlvICppbywgc2l6ZV90IGxlbikKIHN0YXRpYyBpbnQKIGh0dHBfZmlsbGJ1Zihz
 dHJ1Y3QgaHR0cGlvICppbywgc2l6ZV90IGxlbikKIHsKKwlzc2l6ZV90IG5ieXRlczsKKwogCWlm
 IChpby0+ZXJyb3IpCiAJCXJldHVybiAoLTEpOwogCWlmIChpby0+ZW9mKQpAQCAtMjA0LDEwICsy
 MDYsMTEgQEAgaHR0cF9maWxsYnVmKHN0cnVjdCBodHRwaW8gKmlvLCBzaXplX3QgbGVuKQogCWlm
 IChpby0+Y2h1bmtlZCA9PSAwKSB7CiAJCWlmIChodHRwX2dyb3didWYoaW8sIGxlbikgPT0gLTEp
 CiAJCQlyZXR1cm4gKC0xKTsKLQkJaWYgKChpby0+YnVmbGVuID0gZmV0Y2hfcmVhZChpby0+Y29u
 biwgaW8tPmJ1ZiwgbGVuKSkgPT0gLTEpIHsKLQkJCWlvLT5lcnJvciA9IDE7CisJCWlmICgobmJ5
 dGVzID0gZmV0Y2hfcmVhZChpby0+Y29ubiwgaW8tPmJ1ZiwgbGVuKSkgPT0gLTEpIHsKKwkJCWlv
 LT5lcnJvciA9IGVycm5vOwogCQkJcmV0dXJuICgtMSk7CiAJCX0KKwkJaW8tPmJ1ZmxlbiA9IG5i
 eXRlczsKIAkJaW8tPmJ1ZnBvcyA9IDA7CiAJCXJldHVybiAoaW8tPmJ1Zmxlbik7CiAJfQpAQCAt
 MjI3LDEwICsyMzAsMTEgQEAgaHR0cF9maWxsYnVmKHN0cnVjdCBodHRwaW8gKmlvLCBzaXplX3Qg
 bGVuKQogCQlsZW4gPSBpby0+Y2h1bmtzaXplOwogCWlmIChodHRwX2dyb3didWYoaW8sIGxlbikg
 PT0gLTEpCiAJCXJldHVybiAoLTEpOwotCWlmICgoaW8tPmJ1ZmxlbiA9IGZldGNoX3JlYWQoaW8t
 PmNvbm4sIGlvLT5idWYsIGxlbikpID09IC0xKSB7Ci0JCWlvLT5lcnJvciA9IDE7CisJaWYgKChu
 Ynl0ZXMgPSBmZXRjaF9yZWFkKGlvLT5jb25uLCBpby0+YnVmLCBsZW4pKSA9PSAtMSkgeworCQlp
 by0+ZXJyb3IgPSBlcnJubzsKIAkJcmV0dXJuICgtMSk7CiAJfQorCWlvLT5idWZsZW4gPSBuYnl0
 ZXM7CiAJaW8tPmNodW5rc2l6ZSAtPSBpby0+YnVmbGVuOwogCiAJaWYgKGlvLT5jaHVua3NpemUg
 PT0gMCkgewpAQCAtMjcyLDggKzI3NiwxMSBAQCBodHRwX3JlYWRmbih2b2lkICp2LCBjaGFyICpi
 dWYsIGludCBsZW4pCiAJCWlvLT5idWZwb3MgKz0gbDsKIAl9CiAKLQlpZiAoIXBvcyAmJiBpby0+
 ZXJyb3IpCisJaWYgKCFwb3MgJiYgaW8tPmVycm9yKSB7CisJCWlmIChpby0+ZXJyb3IgPT0gRUlO
 VFIpCisJCQlpby0+ZXJyb3IgPSAwOwogCQlyZXR1cm4gKC0xKTsKKwl9CiAJcmV0dXJuIChwb3Mp
 OwogfQogCg==
 --20cf3071ca9c13a1c304ac46a8b3--

From: Mark <markjdb@gmail.com>
To: bug-followup@FreeBSD.org, swell.k@gmail.com
Cc:  
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on tty
Date: Tue, 6 Sep 2011 10:30:42 -0400

 --bcaec5485fca23348004ac46ac36
 Content-Type: text/plain; charset=ISO-8859-1
 
 Here's the second patch.
 
 Thanks,
 -Mark
 
 --bcaec5485fca23348004ac46ac36
 Content-Type: text/plain; charset=US-ASCII; name="libfetch_intr_2of2.patch.txt"
 Content-Disposition: attachment; filename="libfetch_intr_2of2.patch.txt"
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_gs8z7sq30
 
 ZGlmZiAtLWdpdCBhL2xpYi9saWJmZXRjaC9jb21tb24uYyBiL2xpYi9saWJmZXRjaC9jb21tb24u
 YwppbmRleCBmZWZjNGM1Li44YTM1ODU4IDEwMDY0NAotLS0gYS9saWIvbGliZmV0Y2gvY29tbW9u
 LmMKKysrIGIvbGliL2xpYmZldGNoL2NvbW1vbi5jCkBAIC00MDQsNiArNDA0LDM0IEBAIGZldGNo
 X3NzbF9yZWFkKFNTTCAqc3NsLCBjaGFyICpidWYsIHNpemVfdCBsZW4pCiB9CiAjZW5kaWYKIAor
 LyoKKyAqIENhY2hlIHNvbWUgZGF0YSB0aGF0IHdhcyByZWFkIGZyb20gYSBzb2NrZXQgYnV0IGNh
 bm5vdCBiZSBpbW1lZGlhdGVseQorICogcmV0dXJuZWQgYmVjYXVzZSBvZiBhbiBpbnRlcnJ1cHRl
 ZCBzeXN0ZW0gY2FsbC4KKyAqLworc3RhdGljIGludAorZmV0Y2hfY2FjaGVfZGF0YShjb25uX3Qg
 KmNvbm4sIGNoYXIgKnNyYywgc2l6ZV90IG5ieXRlcykKK3sKKwljaGFyICp0bXA7CisKKwlpZiAo
 Y29ubi0+Y2FjaGUuc2l6ZSA8IG5ieXRlcykgeworCQl0bXAgPSByZWFsbG9jKGNvbm4tPmNhY2hl
 LmJ1ZiwgbmJ5dGVzKTsKKwkJaWYgKHRtcCA9PSBOVUxMKSB7CisJCQllcnJubyA9IEVOT01FTTsK
 KwkJCWZldGNoX3N5c2VycigpOworCQkJcmV0dXJuICgtMSk7CisJCX0KKwkJY29ubi0+Y2FjaGUu
 YnVmID0gdG1wOworCQljb25uLT5jYWNoZS5zaXplID0gbmJ5dGVzOworCX0KKworCW1lbWNweShj
 b25uLT5jYWNoZS5idWYsIHNyYywgbmJ5dGVzKTsKKwljb25uLT5jYWNoZS5sZW4gPSBuYnl0ZXM7
 CisJY29ubi0+Y2FjaGUucG9zID0gMDsKKworCXJldHVybiAoMCk7Cit9CisKKwogc3RhdGljIHNz
 aXplX3QKIGZldGNoX3NvY2tldF9yZWFkKGludCBzZCwgY2hhciAqYnVmLCBzaXplX3QgbGVuKQog
 ewpAQCAtNDI5LDYgKzQ1Nyw3IEBAIGZldGNoX3JlYWQoY29ubl90ICpjb25uLCBjaGFyICpidWYs
 IHNpemVfdCBsZW4pCiAJZmRfc2V0IHJlYWRmZHM7CiAJc3NpemVfdCBybGVuLCB0b3RhbDsKIAlp
 bnQgcjsKKwljaGFyICpzdGFydDsKIAogCWlmIChmZXRjaFRpbWVvdXQpIHsKIAkJRkRfWkVSTygm
 cmVhZGZkcyk7CkBAIC00MzcsNiArNDY2LDI0IEBAIGZldGNoX3JlYWQoY29ubl90ICpjb25uLCBj
 aGFyICpidWYsIHNpemVfdCBsZW4pCiAJfQogCiAJdG90YWwgPSAwOworCXN0YXJ0ID0gYnVmOwor
 CisJaWYgKGNvbm4tPmNhY2hlLmxlbiA+IDApIHsKKwkJLyoKKwkJICogVGhlIGxhc3QgaW52b2Nh
 dGlvbiBvZiBmZXRjaF9yZWFkIHdhcyBpbnRlcnJ1cHRlZCBieSBhCisJCSAqIHNpZ25hbCBhZnRl
 ciBzb21lIGRhdGEgaGFkIGJlZW4gcmVhZCBmcm9tIHRoZSBzb2NrZXQuIENvcHkKKwkJICogdGhl
 IGNhY2hlZCBkYXRhIGludG8gdGhlIHN1cHBsaWVkIGJ1ZmZlciBiZWZvcmUgdHJ5aW5nIHRvCisJ
 CSAqIHJlYWQgZnJvbSB0aGUgc29ja2V0IGFnYWluLgorCQkgKi8KKwkJdG90YWwgPSAoY29ubi0+
 Y2FjaGUubGVuIDwgbGVuKSA/IGNvbm4tPmNhY2hlLmxlbiA6IGxlbjsKKwkJbWVtY3B5KGJ1Ziwg
 Y29ubi0+Y2FjaGUuYnVmLCB0b3RhbCk7CisKKwkJY29ubi0+Y2FjaGUubGVuIC09IHRvdGFsOwor
 CQljb25uLT5jYWNoZS5wb3MgKz0gdG90YWw7CisJCWxlbiAtPSB0b3RhbDsKKwkJYnVmKz0gdG90
 YWw7CisJfQorCiAJd2hpbGUgKGxlbiA+IDApIHsKIAkJLyoKIAkJICogVGhlIHNvY2tldCBpcyBu
 b24tYmxvY2tpbmcuICBJbnN0ZWFkIG9mIHRoZSBjYW5vbmljYWwKQEAgLTQ3Miw2ICs1MTksOCBA
 QCBmZXRjaF9yZWFkKGNvbm5fdCAqY29ubiwgY2hhciAqYnVmLCBzaXplX3QgbGVuKQogCQkJdG90
 YWwgKz0gcmxlbjsKIAkJCWNvbnRpbnVlOwogCQl9IGVsc2UgaWYgKHJsZW4gPT0gRkVUQ0hfUkVB
 RF9FUlJPUikgeworCQkJaWYgKGVycm5vID09IEVJTlRSKQorCQkJCWZldGNoX2NhY2hlX2RhdGEo
 Y29ubiwgc3RhcnQsIHRvdGFsKTsKIAkJCXJldHVybiAoLTEpOwogCQl9CiAJCS8vIGFzc2VydChy
 bGVuID09IEZFVENIX1JFQURfV0FJVCk7CkBAIC00OTIsOCArNTQxLDEyIEBAIGZldGNoX3JlYWQo
 Y29ubl90ICpjb25uLCBjaGFyICpidWYsIHNpemVfdCBsZW4pCiAJCQllcnJubyA9IDA7CiAJCQly
 ID0gc2VsZWN0KGNvbm4tPnNkICsgMSwgJnJlYWRmZHMsIE5VTEwsIE5VTEwsICZkZWx0YSk7CiAJ
 CQlpZiAociA9PSAtMSkgewotCQkJCWlmIChlcnJubyA9PSBFSU5UUiAmJiBmZXRjaFJlc3RhcnRD
 YWxscykKLQkJCQkJY29udGludWU7CisJCQkJaWYgKGVycm5vID09IEVJTlRSKSB7CisJCQkJCWlm
 IChmZXRjaFJlc3RhcnRDYWxscykKKwkJCQkJCWNvbnRpbnVlOworCQkJCQkvKiBTYXZlIGFueXRo
 aW5nIHRoYXQgd2FzIHJlYWQuICovCisJCQkJCWZldGNoX2NhY2hlX2RhdGEoY29ubiwgc3RhcnQs
 IHRvdGFsKTsKKwkJCQl9CiAJCQkJZmV0Y2hfc3lzZXJyKCk7CiAJCQkJcmV0dXJuICgtMSk7CiAJ
 CQl9CkBAIC02NzcsNiArNzMwLDcgQEAgZmV0Y2hfY2xvc2UoY29ubl90ICpjb25uKQogCWlmICgt
 LWNvbm4tPnJlZiA+IDApCiAJCXJldHVybiAoMCk7CiAJcmV0ID0gY2xvc2UoY29ubi0+c2QpOwor
 CWZyZWUoY29ubi0+Y2FjaGUuYnVmKTsKIAlmcmVlKGNvbm4tPmJ1Zik7CiAJZnJlZShjb25uKTsK
 IAlyZXR1cm4gKHJldCk7CmRpZmYgLS1naXQgYS9saWIvbGliZmV0Y2gvY29tbW9uLmggYi9saWIv
 bGliZmV0Y2gvY29tbW9uLmgKaW5kZXggMDViZWUwMy4uNDE1ZGM0OCAxMDA2NDQKLS0tIGEvbGli
 L2xpYmZldGNoL2NvbW1vbi5oCisrKyBiL2xpYi9saWJmZXRjaC9jb21tb24uaApAQCAtNTIsNiAr
 NTIsMTMgQEAgc3RydWN0IGZldGNoY29ubiB7CiAJc2l6ZV90CQkgYnVmc2l6ZTsJLyogYnVmZmVy
 IHNpemUgKi8KIAlzaXplX3QJCSBidWZsZW47CS8qIGxlbmd0aCBvZiBidWZmZXIgY29udGVudHMg
 Ki8KIAlpbnQJCSBlcnI7CQkvKiBsYXN0IHByb3RvY29sIHJlcGx5IGNvZGUgKi8KKwlzdHJ1Y3Qg
 ewkJCS8qIGRhdGEgY2FjaGVkIGFmdGVyIGFuIGludGVycnVwdGVkCisJCQkJCSAgIHJlYWQgKi8K
 KwkJY2hhcgkqYnVmOworCQlzaXplX3QJIHNpemU7CisJCXNpemVfdAkgcG9zOworCQlzaXplX3QJ
 IGxlbjsKKwl9IGNhY2hlOwogI2lmZGVmIFdJVEhfU1NMCiAJU1NMCQkqc3NsOwkJLyogU1NMIGhh
 bmRsZSAqLwogCVNTTF9DVFgJCSpzc2xfY3R4OwkvKiBTU0wgY29udGV4dCAqLwpkaWZmIC0tZ2l0
 IGEvdXNyLmJpbi9mZXRjaC9mZXRjaC5jIGIvdXNyLmJpbi9mZXRjaC9mZXRjaC5jCmluZGV4IDc1
 NTNiZDguLmI3ZWI1NGQgMTAwNjQ0Ci0tLSBhL3Vzci5iaW4vZmV0Y2gvZmV0Y2guYworKysgYi91
 c3IuYmluL2ZldGNoL2ZldGNoLmMKQEAgLTMxNyw3ICszMTcsNyBAQCBmZXRjaChjaGFyICpVUkws
 IGNvbnN0IGNoYXIgKnBhdGgpCiAJc3RydWN0IHN0YXQgc2IsIG5zYjsKIAlzdHJ1Y3QgeGZlcnN0
 YXQgeHM7CiAJRklMRSAqZiwgKm9mOwotCXNpemVfdCBzaXplLCB3cjsKKwlzaXplX3Qgc2l6ZSwg
 cmVhZGNudCwgd3I7CiAJb2ZmX3QgY291bnQ7CiAJY2hhciBmbGFnc1s4XTsKIAljb25zdCBjaGFy
 ICpzbGFzaDsKQEAgLTYyMiwyMSArNjIyLDI2IEBAIGZldGNoKGNoYXIgKlVSTCwgY29uc3QgY2hh
 ciAqcGF0aCkKIAkJCXN0YXRfZW5kKCZ4cyk7CiAJCQlzaWdpbmZvID0gMDsKIAkJfQotCQlpZiAo
 KHNpemUgPSBmcmVhZChidWYsIDEsIHNpemUsIGYpKSA9PSAwKSB7CisKKwkJaWYgKHNpemUgPT0g
 MCkKKwkJCWJyZWFrOworCisJCWlmICgocmVhZGNudCA9IGZyZWFkKGJ1ZiwgMSwgc2l6ZSwgZikp
 IDwgc2l6ZSkgewogCQkJaWYgKGZlcnJvcihmKSAmJiBlcnJubyA9PSBFSU5UUiAmJiAhc2lnaW50
 KQogCQkJCWNsZWFyZXJyKGYpOwotCQkJZWxzZQorCQkJZWxzZSBpZiAocmVhZGNudCA9PSAwKQog
 CQkJCWJyZWFrOwogCQl9Ci0JCXN0YXRfdXBkYXRlKCZ4cywgY291bnQgKz0gc2l6ZSk7Ci0JCWZv
 ciAocHRyID0gYnVmOyBzaXplID4gMDsgcHRyICs9IHdyLCBzaXplIC09IHdyKQotCQkJaWYgKCh3
 ciA9IGZ3cml0ZShwdHIsIDEsIHNpemUsIG9mKSkgPCBzaXplKSB7CisKKwkJc3RhdF91cGRhdGUo
 JnhzLCBjb3VudCArPSByZWFkY250KTsKKwkJZm9yIChwdHIgPSBidWY7IHJlYWRjbnQgPiAwOyBw
 dHIgKz0gd3IsIHJlYWRjbnQgLT0gd3IpCisJCQlpZiAoKHdyID0gZndyaXRlKHB0ciwgMSwgcmVh
 ZGNudCwgb2YpKSA8IHJlYWRjbnQpIHsKIAkJCQlpZiAoZmVycm9yKG9mKSAmJiBlcnJubyA9PSBF
 SU5UUiAmJiAhc2lnaW50KQogCQkJCQljbGVhcmVycihvZik7CiAJCQkJZWxzZQogCQkJCQlicmVh
 azsKIAkJCX0KLQkJaWYgKHNpemUgIT0gMCkKKwkJaWYgKHJlYWRjbnQgIT0gMCkKIAkJCWJyZWFr
 OwogCX0KIAlpZiAoIXNpZ2Fscm0pCg==
 --bcaec5485fca23348004ac46ac36--

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: Mark <markjdb@gmail.com>
Cc: bug-followup@freebsd.org
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on tty
Date: Fri, 16 Sep 2011 03:12:31 +0200

 Mark <markjdb@gmail.com> writes:
 > The second patch fixes fetch_read() to avoid losing data if select()
 > is interrupted by SIGINFO. The problem is that some data may have
 > already been read from the socket before fetch_read() returns -1. In
 > this case, it has to be saved somewhere so that it can be picked up in
 > the next fread.
 
 I really wish there was a way to fix this without adding *yet* another
 layer of buffering :(
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: Mark Johnston <markjdb@gmail.com>
To: Dag-Erling Sm??rgrav <des@des.no>
Cc: bug-followup@freebsd.org
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on
 tty
Date: Sun, 18 Sep 2011 14:46:49 -0400

 On Fri, Sep 16, 2011 at 03:12:31AM +0200, Dag-Erling Sm??rgrav wrote:
 > Mark <markjdb@gmail.com> writes:
 > > The second patch fixes fetch_read() to avoid losing data if select()
 > > is interrupted by SIGINFO. The problem is that some data may have
 > > already been read from the socket before fetch_read() returns -1. In
 > > this case, it has to be saved somewhere so that it can be picked up in
 > > the next fread.
 > 
 > I really wish there was a way to fix this without adding *yet* another
 > layer of buffering :(
 
 Yeah... I spent a while trying to come up with a better way to do it,
 but didn't really get anywhere. The thing is that if we allow
 fetch_read() to be interrupted, it has to return -1, and we have to buffer
 the partially-read data _somewhere_. We can't put it in conn_t's buf,
 since it'll interfere with fetch_getln()'s use of it. We can't really put it in
 struct httpio's buf since we can't record the number of bytes that were
 buffered. We _could_ change fetch_read()'s parameters to include a
 "unsigned int *buflen", and then pass in &(io->buflen) or something, but
 that would break the nice parallel between fetch_read() and read(2).
 Maybe that isn't the end of the world though. =)
 Also, the corresponding structs for ftp and file transfers would have to
 be changed, which kinda sucks.
 
 IMHO, the protocol-specific code shouldn't know about the interrupted
 syscall buffering - that really has more to do with the connection
 itself. So I think my solution makes the most sense, even if it's kind
 of ugly.
 
 -Mark

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: Mark Johnston <markjdb@gmail.com>
Cc: bug-followup@freebsd.org
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on tty
Date: Sun, 18 Sep 2011 22:35:56 +0200

 Mark Johnston <markjdb@gmail.com> writes:
 > IMHO, the protocol-specific code shouldn't know about the interrupted
 > syscall buffering - that really has more to do with the connection
 > itself. So I think my solution makes the most sense, even if it's kind
 > of ugly.
 
 I reluctantly agree...
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: Mark <markjdb@gmail.com>
To: bug-followup@FreeBSD.org, =?ISO-8859-1?Q?Dag=2DErling_Sm=F8rgrav?= <des@freebsd.org>
Cc:  
Subject: Re: bin/153240: fetch(1): http transfer hangs after ^T (status) on tty
Date: Tue, 17 Jan 2012 15:05:29 -0500

 Just thought I'd ping you about this now that 9-RELEASE is out.
 
 Thanks,
 -Mark

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/153240: commit references a PR
Date: Wed, 18 Jan 2012 15:13:51 +0000 (UTC)

 Author: des
 Date: Wed Jan 18 15:13:21 2012
 New Revision: 230307
 URL: http://svn.freebsd.org/changeset/base/230307
 
 Log:
   Fix two issues related to the use of SIGINFO in fetch(1) to display
   progress information.  The first is that fetch_read() (used in the HTTP
   code but not the FTP code) can enter an infinite loop if it has previously
   been interrupted by a signal.  The second is that when it is interrupted,
   fetch_read() will discard any data it may have read up to that point.
   Luckily, both bugs are extremely timing-sensitive and therefore difficult
   to trigger.
   
   PR:		bin/153240
   Submitted by:	Mark <markjdb@gmail.com>
   MFC after:	3 weeks
 
 Modified:
   head/lib/libfetch/common.c
   head/lib/libfetch/common.h
   head/lib/libfetch/http.c
   head/usr.bin/fetch/fetch.c
 
 Modified: head/lib/libfetch/common.c
 ==============================================================================
 --- head/lib/libfetch/common.c	Wed Jan 18 14:41:26 2012	(r230306)
 +++ head/lib/libfetch/common.c	Wed Jan 18 15:13:21 2012	(r230307)
 @@ -404,6 +404,34 @@ fetch_ssl_read(SSL *ssl, char *buf, size
  }
  #endif
  
 +/*
 + * Cache some data that was read from a socket but cannot be immediately
 + * returned because of an interrupted system call.
 + */
 +static int
 +fetch_cache_data(conn_t *conn, char *src, size_t nbytes)
 +{
 +	char *tmp;
 +
 +	if (conn->cache.size < nbytes) {
 +		tmp = realloc(conn->cache.buf, nbytes);
 +		if (tmp == NULL) {
 +			errno = ENOMEM;
 +			fetch_syserr();
 +			return (-1);
 +		}
 +		conn->cache.buf = tmp;
 +		conn->cache.size = nbytes;
 +	}
 +
 +	memcpy(conn->cache.buf, src, nbytes);
 +	conn->cache.len = nbytes;
 +	conn->cache.pos = 0;
 +
 +	return (0);
 +}
 +
 +
  static ssize_t
  fetch_socket_read(int sd, char *buf, size_t len)
  {
 @@ -429,6 +457,7 @@ fetch_read(conn_t *conn, char *buf, size
  	fd_set readfds;
  	ssize_t rlen, total;
  	int r;
 +	char *start;
  
  	if (fetchTimeout) {
  		FD_ZERO(&readfds);
 @@ -437,6 +466,24 @@ fetch_read(conn_t *conn, char *buf, size
  	}
  
  	total = 0;
 +	start = buf;
 +
 +	if (conn->cache.len > 0) {
 +		/*
 +		 * The last invocation of fetch_read was interrupted by a
 +		 * signal after some data had been read from the socket. Copy
 +		 * the cached data into the supplied buffer before trying to
 +		 * read from the socket again.
 +		 */
 +		total = (conn->cache.len < len) ? conn->cache.len : len;
 +		memcpy(buf, conn->cache.buf, total);
 +
 +		conn->cache.len -= total;
 +		conn->cache.pos += total;
 +		len -= total;
 +		buf+= total;
 +	}
 +
  	while (len > 0) {
  		/*
  		 * The socket is non-blocking.  Instead of the canonical
 @@ -472,6 +519,8 @@ fetch_read(conn_t *conn, char *buf, size
  			total += rlen;
  			continue;
  		} else if (rlen == FETCH_READ_ERROR) {
 +			if (errno == EINTR)
 +				fetch_cache_data(conn, start, total);
  			return (-1);
  		}
  		// assert(rlen == FETCH_READ_WAIT);
 @@ -492,8 +541,12 @@ fetch_read(conn_t *conn, char *buf, size
  			errno = 0;
  			r = select(conn->sd + 1, &readfds, NULL, NULL, &delta);
  			if (r == -1) {
 -				if (errno == EINTR && fetchRestartCalls)
 -					continue;
 +				if (errno == EINTR) {
 +					if (fetchRestartCalls)
 +						continue;
 +					/* Save anything that was read. */
 +					fetch_cache_data(conn, start, total);
 +				}
  				fetch_syserr();
  				return (-1);
  			}
 @@ -677,6 +730,7 @@ fetch_close(conn_t *conn)
  	if (--conn->ref > 0)
  		return (0);
  	ret = close(conn->sd);
 +	free(conn->cache.buf);
  	free(conn->buf);
  	free(conn);
  	return (ret);
 
 Modified: head/lib/libfetch/common.h
 ==============================================================================
 --- head/lib/libfetch/common.h	Wed Jan 18 14:41:26 2012	(r230306)
 +++ head/lib/libfetch/common.h	Wed Jan 18 15:13:21 2012	(r230307)
 @@ -52,6 +52,13 @@ struct fetchconn {
  	size_t		 bufsize;	/* buffer size */
  	size_t		 buflen;	/* length of buffer contents */
  	int		 err;		/* last protocol reply code */
 +	struct {			/* data cached after an interrupted
 +					   read */
 +		char	*buf;
 +		size_t	 size;
 +		size_t	 pos;
 +		size_t	 len;
 +	} cache;
  #ifdef WITH_SSL
  	SSL		*ssl;		/* SSL handle */
  	SSL_CTX		*ssl_ctx;	/* SSL context */
 
 Modified: head/lib/libfetch/http.c
 ==============================================================================
 --- head/lib/libfetch/http.c	Wed Jan 18 14:41:26 2012	(r230306)
 +++ head/lib/libfetch/http.c	Wed Jan 18 15:13:21 2012	(r230307)
 @@ -196,6 +196,8 @@ http_growbuf(struct httpio *io, size_t l
  static int
  http_fillbuf(struct httpio *io, size_t len)
  {
 +	ssize_t nbytes;
 +
  	if (io->error)
  		return (-1);
  	if (io->eof)
 @@ -204,10 +206,11 @@ http_fillbuf(struct httpio *io, size_t l
  	if (io->chunked == 0) {
  		if (http_growbuf(io, len) == -1)
  			return (-1);
 -		if ((io->buflen = fetch_read(io->conn, io->buf, len)) == -1) {
 -			io->error = 1;
 +		if ((nbytes = fetch_read(io->conn, io->buf, len)) == -1) {
 +			io->error = errno;
  			return (-1);
  		}
 +		io->buflen = nbytes;
  		io->bufpos = 0;
  		return (io->buflen);
  	}
 @@ -227,10 +230,11 @@ http_fillbuf(struct httpio *io, size_t l
  		len = io->chunksize;
  	if (http_growbuf(io, len) == -1)
  		return (-1);
 -	if ((io->buflen = fetch_read(io->conn, io->buf, len)) == -1) {
 -		io->error = 1;
 +	if ((nbytes = fetch_read(io->conn, io->buf, len)) == -1) {
 +		io->error = errno;
  		return (-1);
  	}
 +	io->buflen = nbytes;
  	io->chunksize -= io->buflen;
  
  	if (io->chunksize == 0) {
 @@ -272,8 +276,11 @@ http_readfn(void *v, char *buf, int len)
  		io->bufpos += l;
  	}
  
 -	if (!pos && io->error)
 +	if (!pos && io->error) {
 +		if (io->error == EINTR)
 +			io->error = 0;
  		return (-1);
 +	}
  	return (pos);
  }
  
 
 Modified: head/usr.bin/fetch/fetch.c
 ==============================================================================
 --- head/usr.bin/fetch/fetch.c	Wed Jan 18 14:41:26 2012	(r230306)
 +++ head/usr.bin/fetch/fetch.c	Wed Jan 18 15:13:21 2012	(r230307)
 @@ -317,7 +317,7 @@ fetch(char *URL, const char *path)
  	struct stat sb, nsb;
  	struct xferstat xs;
  	FILE *f, *of;
 -	size_t size, wr;
 +	size_t size, readcnt, wr;
  	off_t count;
  	char flags[8];
  	const char *slash;
 @@ -636,21 +636,26 @@ fetch(char *URL, const char *path)
  			stat_end(&xs);
  			siginfo = 0;
  		}
 -		if ((size = fread(buf, 1, size, f)) == 0) {
 +
 +		if (size == 0)
 +			break;
 +
 +		if ((readcnt = fread(buf, 1, size, f)) < size) {
  			if (ferror(f) && errno == EINTR && !sigint)
  				clearerr(f);
 -			else
 +			else if (readcnt == 0)
  				break;
  		}
 -		stat_update(&xs, count += size);
 -		for (ptr = buf; size > 0; ptr += wr, size -= wr)
 -			if ((wr = fwrite(ptr, 1, size, of)) < size) {
 +
 +		stat_update(&xs, count += readcnt);
 +		for (ptr = buf; readcnt > 0; ptr += wr, readcnt -= wr)
 +			if ((wr = fwrite(ptr, 1, readcnt, of)) < readcnt) {
  				if (ferror(of) && errno == EINTR && !sigint)
  					clearerr(of);
  				else
  					break;
  			}
 -		if (size != 0)
 +		if (readcnt != 0)
  			break;
  	}
  	if (!sigalrm)
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: analyzed->patched 
State-Changed-By: des 
State-Changed-When: Thu Jan 19 08:42:01 UTC 2012 
State-Changed-Why:  
committed to head 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153240 
State-Changed-From-To: patched->closed 
State-Changed-By: markj 
State-Changed-When: Sun Feb 17 04:35:13 UTC 2013 
State-Changed-Why:  
Fix merged to STABLE-7 in r242608, STABLE-8 in r231248 and STABLE-9 in 
r231247. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=153240 
>Unformatted:
