From nobody@FreeBSD.org  Mon Jul  9 19:21:01 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id C8DBE106575E
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  9 Jul 2012 19:21:00 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id B23D78FC0C
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  9 Jul 2012 19:21:00 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q69JL0gY062492
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 9 Jul 2012 19:21:00 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q69JL0eQ062491;
	Mon, 9 Jul 2012 19:21:00 GMT
	(envelope-from nobody)
Message-Id: <201207091921.q69JL0eQ062491@red.freebsd.org>
Date: Mon, 9 Jul 2012 19:21:00 GMT
From: Garrett Cooper <yaneurabeya@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [bsdinstall] when distfile fetch is complete on some distfiles, but not entire set, progress can go over 100%; installer gets confused with distfiles and halts
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         169748
>Category:       bin
>Synopsis:       [patch] bsdinstall(8): when distfile fetch is complete on some distfiles, but not entire set, progress can go over 100%; installer gets confused with distfiles and halts
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-sysinstall
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jul 09 19:30:17 UTC 2012
>Closed-Date:    
>Last-Modified:  Sun Feb 03 22:30:40 UTC 2013
>Originator:     Garrett Cooper
>Release:        9.0-RELEASE
>Organization:
EMC Isilon
>Environment:
(x86 install)
>Description:
The FTP download choked on base.txz (corp. firewall issue), but continued downloading the rest of the distfiles, then asked if I wanted to retry downloading the files.

2nd try crashed and burned (the progress bar went over 100% and the installer halted).
>How-To-Repeat:
1. Download the network installer.
2. Nuke or taint one of the tarballs such that it fetches but fails to extract, or corrupt one of the bytes in the file.
>Fix:
The attached patch (truncate the file) isn't optimal, but it's simple and will address multiple potential issues dealing with distfile fetch failures without introducing having to introduce the whacky state machine that was present in sysinstall (and in many cases got things wrong). An alternate, less failure prone method should be provided which can operate outside of the installer if download retrying is needed.

That and there's no guarantee that the upstream FTP/HTTP server supports download resume until the fetch is attempted.

Patch attached with submission follows:

Index: usr.sbin/bsdinstall/distfetch/distfetch.c
===================================================================
--- usr.sbin/bsdinstall/distfetch/distfetch.c	(revision 238086)
+++ usr.sbin/bsdinstall/distfetch/distfetch.c	(working copy)
@@ -151,7 +151,7 @@
 
 		items[i*2 + 1] = "In Progress";
 		fsize = 0;
-		file_out = fopen(items[i*2], "w+");
+		file_out = fopen(items[i*2], "w");
 		if (file_out == NULL) {
 			snprintf(errormsg, sizeof(errormsg),
 			    "Error while fetching %s: %s\n",


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-sysinstall 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Jul 10 01:08:15 UTC 2012 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=169748 

From: Garrett Cooper <yaneurabeya@gmail.com>
To: bug-followup@FreeBSD.org
Cc: yanegomi@gmail.com
Subject: bin/169748: [bsdinstall] when distfile fetch is complete on some
 distfiles, but not entire set, progress can go over 100%; installer gets
 confused with distfiles and halts
Date: Sun, 2 Dec 2012 03:30:10 -0800 (PST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --967339439-2074242010-1354447815=:38080
 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
 
 The attached patch goes beyond what the previous patch did in the 
 following ways:
 
 1. Catch errors and terminate whenever a failure occurs.
 2. Catch more user errors with undefined environment variables.
 3. Catch memory allocation errors.
 4. Convert fprintf+return calls to err*(3)/warnx(3) calls.
 5. Reorder error handling to avoid potentially confusing mangling via 
 dialog(3).
 
 Thanks,
 -Garrett
 --967339439-2074242010-1354447815=:38080
 Content-Type: TEXT/PLAIN; charset=US-ASCII; name=bin-169748.patch.txt
 Content-Transfer-Encoding: BASE64
 Content-ID: <alpine.BSF.2.00.1212020330100.38080@toaster.local>
 Content-Description: 
 Content-Disposition: attachment; filename=bin-169748.patch.txt
 
 SW5kZXg6IHVzci5zYmluL2JzZGluc3RhbGwvZGlzdGZldGNoL2Rpc3RmZXRj
 aC5jDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
 PT09PT09PT09PT09PT09PT09PT09PT09PT09DQotLS0gdXNyLnNiaW4vYnNk
 aW5zdGFsbC9kaXN0ZmV0Y2gvZGlzdGZldGNoLmMJKHJldmlzaW9uIDI0Mzc0
 NykNCisrKyB1c3Iuc2Jpbi9ic2RpbnN0YWxsL2Rpc3RmZXRjaC9kaXN0ZmV0
 Y2guYwkod29ya2luZyBjb3B5KQ0KQEAgLTI4LDYgKzI4LDcgQEANCiANCiAj
 aW5jbHVkZSA8c3lzL3BhcmFtLmg+DQogI2luY2x1ZGUgPHN0ZGlvLmg+DQor
 I2luY2x1ZGUgPGVyci5oPg0KICNpbmNsdWRlIDxlcnJuby5oPg0KICNpbmNs
 dWRlIDxmZXRjaC5oPg0KICNpbmNsdWRlIDxkaWFsb2cuaD4NCkBAIC00MSwz
 NCArNDIsNDEgQEANCiAJY2hhciAqKnVybHM7DQogCWludCBpLCBuZmV0Y2hl
 ZCwgbmRpc3RzID0gMDsNCiANCi0JaWYgKGdldGVudigiRElTVFJJQlVUSU9O
 UyIpID09IE5VTEwpIHsNCi0JCWZwcmludGYoc3RkZXJyLCAiRElTVFJJQlVU
 SU9OUyB2YXJpYWJsZSBpcyBub3Qgc2V0XG4iKTsNCi0JCXJldHVybiAoMSk7
 DQotCX0NCisJaWYgKGdldGVudigiQlNESU5TVEFMTF9ESVNURElSIikgPT0g
 TlVMTCkNCisJCWVycngoMSwgIkJTRElOU1RBTExfRElTVERJUiB2YXJpYWJs
 ZSBpcyBub3Qgc2V0Iik7DQogDQorCWlmIChnZXRlbnYoIkJTRElOU1RBTExf
 RElTVFNJVEUiKSA9PSBOVUxMKQ0KKwkJZXJyeCgxLCAiQlNESU5TVEFMTF9E
 SVNUU0lURSB2YXJpYWJsZSBpcyBub3Qgc2V0Iik7DQorDQorCWlmIChnZXRl
 bnYoIkRJU1RSSUJVVElPTlMiKSA9PSBOVUxMKQ0KKwkJZXJyeCgxLCAiRElT
 VFJJQlVUSU9OUyB2YXJpYWJsZSBpcyBub3Qgc2V0Iik7DQorDQogCWRpc3Rz
 dHJpbmcgPSBzdHJkdXAoZ2V0ZW52KCJESVNUUklCVVRJT05TIikpOw0KKwlp
 ZiAoZGlzdHN0cmluZyA9PSBOVUxMKQ0KKwkJZXJyeCgxLCAiT3V0IG9mIG1l
 bW9yeSEiKTsNCisNCiAJZm9yIChpID0gMDsgZGlzdHN0cmluZ1tpXSAhPSAw
 OyBpKyspDQogCQlpZiAoaXNzcGFjZShkaXN0c3RyaW5nW2ldKSAmJiAhaXNz
 cGFjZShkaXN0c3RyaW5nW2krMV0pKQ0KIAkJCW5kaXN0cysrOw0KIAluZGlz
 dHMrKzsgLyogTGFzdCBvbmUgKi8NCiANCiAJdXJscyA9IGNhbGxvYyhuZGlz
 dHMsIHNpemVvZihjb25zdCBjaGFyICopKTsNCi0JaWYgKHVybHMgPT0gTlVM
 TCkgew0KLQkJZnByaW50ZihzdGRlcnIsICJPdXQgb2YgbWVtb3J5IVxuIik7
 DQotCQlmcmVlKGRpc3RzdHJpbmcpOw0KLQkJcmV0dXJuICgxKTsNCi0JfQ0K
 KwlpZiAodXJscyA9PSBOVUxMKQ0KKwkJZXJyeCgxLCAiT3V0IG9mIG1lbW9y
 eSEiKTsNCiANCi0JaW5pdF9kaWFsb2coc3RkaW4sIHN0ZG91dCk7DQotCWRp
 YWxvZ192YXJzLmJhY2t0aXRsZSA9IF9fREVDT05TVChjaGFyICosICJGcmVl
 QlNEIEluc3RhbGxlciIpOw0KLQlkbGdfcHV0X2JhY2t0aXRsZSgpOw0KLQ0K
 IAlmb3IgKGkgPSAwOyBpIDwgbmRpc3RzOyBpKyspIHsNCiAJCXVybHNbaV0g
 PSBtYWxsb2MoUEFUSF9NQVgpOw0KKwkJaWYgKHVybHNbaV0gPT0gTlVMTCkN
 CisJCQllcnJ4KDEsICJPdXQgb2YgbWVtb3J5ISIpOw0KIAkJc3ByaW50Zih1
 cmxzW2ldLCAiJXMvJXMiLCBnZXRlbnYoIkJTRElOU1RBTExfRElTVFNJVEUi
 KSwNCiAJCSAgICBzdHJzZXAoJmRpc3RzdHJpbmcsICIgXHQiKSk7DQogCX0N
 CisJZnJlZShkaXN0c3RyaW5nKTsNCiANCisJaW5pdF9kaWFsb2coc3RkaW4s
 IHN0ZG91dCk7DQorCWRpYWxvZ192YXJzLmJhY2t0aXRsZSA9IF9fREVDT05T
 VChjaGFyICosICJGcmVlQlNEIEluc3RhbGxlciIpOw0KKwlkbGdfcHV0X2Jh
 Y2t0aXRsZSgpOw0KKw0KIAlpZiAoY2hkaXIoZ2V0ZW52KCJCU0RJTlNUQUxM
 X0RJU1RESVIiKSkgIT0gMCkgew0KIAkJY2hhciBlcnJvcls1MTJdOw0KIAkJ
 c3ByaW50ZihlcnJvciwgIkNvdWxkIGNvdWxkIGNoYW5nZSB0byBkaXJlY3Rv
 cnkgJXM6ICVzXG4iLA0KQEAgLTgyLDcgKzkwLDYgQEANCiANCiAJZW5kX2Rp
 YWxvZygpOw0KIA0KLQlmcmVlKGRpc3RzdHJpbmcpOw0KIAlmb3IgKGkgPSAw
 OyBpIDwgbmRpc3RzOyBpKyspIA0KIAkJZnJlZSh1cmxzW2ldKTsNCiAJZnJl
 ZSh1cmxzKTsNCkBAIC0xMDksNyArMTE2LDcgQEANCiAJLyogTWFrZSB0aGUg
 dHJhbnNmZXIgbGlzdCBmb3IgZGlhbG9nICovDQogCWl0ZW1zID0gY2FsbG9j
 KHNpemVvZihjaGFyICopLCBuZmlsZXMgKiAyKTsNCiAJaWYgKGl0ZW1zID09
 IE5VTEwpIHsNCi0JCWZwcmludGYoc3RkZXJyLCAiT3V0IG9mIG1lbW9yeSFc
 biIpOw0KKwkJd2FybngoIk91dCBvZiBtZW1vcnkhIik7DQogCQlyZXR1cm4g
 KC0xKTsNCiAJfQ0KIA0KQEAgLTEyNyw3ICsxMzQsMTYgQEANCiAJLyogVHJ5
 IHRvIHN0YXQgYWxsIHRoZSBmaWxlcyAqLw0KIAl0b3RhbF9ieXRlcyA9IDA7
 DQogCWZvciAoaSA9IDA7IGkgPCBuZmlsZXM7IGkrKykgew0KLQkJaWYgKGZl
 dGNoU3RhdFVSTCh1cmxzW2ldLCAmdXN0YXQsICIiKSA9PSAwICYmIHVzdGF0
 LnNpemUgPiAwKQ0KKwkJaWYgKGZldGNoU3RhdFVSTCh1cmxzW2ldLCAmdXN0
 YXQsICIiKSA9PSAtMSkgew0KKwkJCXNucHJpbnRmKGVycm9ybXNnLCBzaXpl
 b2YoZXJyb3Jtc2cpLA0KKwkJCSAgICAiRXJyb3Igd2hpbGUgZGV0ZXJtaW5p
 bmcgZmlsZSBzaXplIGZvciAlczogJXNcbiIsDQorCQkJICAgIHVybHNbaV0s
 IGZldGNoTGFzdEVyclN0cmluZyk7DQorCQkJaXRlbXNbaSoyICsgMV0gPSAi
 RmFpbGVkIjsNCisJCQlkaWFsb2dfbXNnYm94KCJGZXRjaCBFcnJvciIsIGVy
 cm9ybXNnLCAwLCAwLA0KKwkJCSAgICBUUlVFKTsNCisJCQlyZXR1cm4gKC0x
 KTsNCisJCX0NCisJCWlmICh1c3RhdC5zaXplID4gMCkNCiAJCQl0b3RhbF9i
 eXRlcyArPSB1c3RhdC5zaXplOw0KIAl9DQogDQpAQCAtMTQ2LDEyICsxNjIs
 MTIgQEANCiAJCQlpdGVtc1tpKjIgKyAxXSA9ICJGYWlsZWQiOw0KIAkJCWRp
 YWxvZ19tc2dib3goIkZldGNoIEVycm9yIiwgZXJyb3Jtc2csIDAsIDAsDQog
 CQkJICAgIFRSVUUpOw0KLQkJCWNvbnRpbnVlOw0KKwkJCWJyZWFrOw0KIAkJ
 fQ0KIA0KIAkJaXRlbXNbaSoyICsgMV0gPSAiSW4gUHJvZ3Jlc3MiOw0KIAkJ
 ZnNpemUgPSAwOw0KLQkJZmlsZV9vdXQgPSBmb3BlbihpdGVtc1tpKjJdLCAi
 dysiKTsNCisJCWZpbGVfb3V0ID0gZm9wZW4oaXRlbXNbaSoyXSwgInciKTsN
 CiAJCWlmIChmaWxlX291dCA9PSBOVUxMKSB7DQogCQkJc25wcmludGYoZXJy
 b3Jtc2csIHNpemVvZihlcnJvcm1zZyksDQogCQkJICAgICJFcnJvciB3aGls
 ZSBmZXRjaGluZyAlczogJXNcbiIsDQpAQCAtMTYwLDcgKzE3Niw3IEBADQog
 CQkJZGlhbG9nX21zZ2JveCgiRmV0Y2ggRXJyb3IiLCBlcnJvcm1zZywgMCwg
 MCwNCiAJCQkgICAgVFJVRSk7DQogCQkJZmNsb3NlKGZldGNoX291dCk7DQot
 CQkJY29udGludWU7DQorCQkJYnJlYWs7DQogCQl9DQogDQogCQl3aGlsZSAo
 KGNodW5rID0gZnJlYWQoYmxvY2ssIDEsIHNpemVvZihibG9jayksIGZldGNo
 X291dCkpDQpAQCAtMTczLDcgKzE4OSw3IEBADQogCQ0KIAkJCWlmICh0b3Rh
 bF9ieXRlcyA+IDApIHsNCiAJCQkJbGFzdF9wcm9ncmVzcyA9IHByb2dyZXNz
 Ow0KLQkJCQlwcm9ncmVzcyA9IChjdXJyZW50X2J5dGVzKjEwMCkvdG90YWxf
 Ynl0ZXM7IA0KKwkJCQlwcm9ncmVzcyA9IChjdXJyZW50X2J5dGVzKjEwMCkv
 dG90YWxfYnl0ZXM7DQogCQkJfQ0KIA0KIAkJCWlmICh1c3RhdC5zaXplID4g
 MCkgew0KQEAgLTIwMCw2ICsyMTYsNyBAQA0KIAkJCWl0ZW1zW2kqMiArIDFd
 ID0gIkZhaWxlZCI7DQogCQkJZGlhbG9nX21zZ2JveCgiRmV0Y2ggRXJyb3Ii
 LCBlcnJvcm1zZywgMCwgMCwNCiAJCQkJICAgIFRSVUUpOw0KKwkJCWJyZWFr
 Ow0KIAkJfSBlbHNlIHsNCiAJCQlpdGVtc1tpKjIgKyAxXSA9ICJEb25lIjsN
 CiAJCQluc3VjY2VzcysrOw0K
 
 --967339439-2074242010-1354447815=:38080--
>Unformatted:
