From nobody@FreeBSD.org  Mon Nov  1 15:01:50 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 7E4BE106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  1 Nov 2010 15:01:50 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 60C348FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  1 Nov 2010 15:01:50 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id oA1F1oNs094736
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 1 Nov 2010 15:01:50 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id oA1F1oCh094735;
	Mon, 1 Nov 2010 15:01:50 GMT
	(envelope-from nobody)
Message-Id: <201011011501.oA1F1oCh094735@www.freebsd.org>
Date: Mon, 1 Nov 2010 15:01:50 GMT
From: Mark Johnston <markjdb@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [libfetch] [patch] closing the cached FTP connection
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         151866
>Category:       bin
>Synopsis:       [libfetch] [patch] closing the cached FTP connection
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    des
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Mon Nov 01 15:10:07 UTC 2010
>Closed-Date:    Mon Nov 05 13:03:49 UTC 2012
>Last-Modified:  Mon Nov 05 13:03:49 UTC 2012
>Originator:     Mark Johnston
>Release:        8.1-RELEASE
>Organization:
Sandvine Inc.
>Environment:
FreeBSD mark-laptop-bsd.mark-home 8.1-RELEASE FreeBSD 8.1-RELEASE #2: Mon Oct 25 15:07:21 EDT 2010     root@mark-laptop-bsd.mark-home:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
This patch adds a function to libfetch which allows programs to close the cached socket that libfetch holds on to when performing a file transfer.

See http://lists.freebsd.org/pipermail/freebsd-current/2010-October/020694.html
>How-To-Repeat:
Running this test program and using sockstat demonstrates the issue.

#include <stdlib.h>
#include <stdio.h>
#include <sys/param.h>
#include <fetch.h>

#define URL "ftp://ftp.freebsd.org/pub/FreeBSD/releases/README.TXT"

int
main(int argc, char **argv)
{
        char c;
        FILE *f = fetchGetURL(URL, NULL);
        if (!f) {
                printf("Error: could not open connection.\n");
                return (1);
        }

        while (!feof(f)) {
                fread(&c, sizeof(char), 1, f);
                putchar(c);
        }

        printf("Transfer complete.\n");
        fclose(f);

        if (fork())
                //Close the cached FTP connection.
                //fetchCloseCachedFTP();
        else
                for ( ;; ) ;

        return (0);
}
>Fix:
My patch adds a function to libfetch which makes it possible to force close the cached connection. Included is a patch to pkg_add(1) which fixes the problem described in the email.

Patch attached with submission follows:

diff --git a/lib/libfetch/fetch.h b/lib/libfetch/fetch.h
index 11a3f77..d379e63 100644
--- a/lib/libfetch/fetch.h
+++ b/lib/libfetch/fetch.h
@@ -109,6 +109,7 @@ FILE		*fetchGetFTP(struct url *, const char *);
 FILE		*fetchPutFTP(struct url *, const char *);
 int		 fetchStatFTP(struct url *, struct url_stat *, const char *);
 struct url_ent	*fetchListFTP(struct url *, const char *);
+void		 fetchCloseCachedFTP();
 
 /* Generic functions */
 FILE		*fetchXGetURL(const char *, struct url_stat *, const char *);
diff --git a/lib/libfetch/ftp.c b/lib/libfetch/ftp.c
index 0983a76..746ad54 100644
--- a/lib/libfetch/ftp.c
+++ b/lib/libfetch/ftp.c
@@ -1204,3 +1204,12 @@ fetchListFTP(struct url *url __unused, const char *flags __unused)
 	warnx("fetchListFTP(): not implemented");
 	return (NULL);
 }
+
+/*
+ * Force close the cached connection.
+ */
+void
+fetchCloseCachedFTP()
+{
+	ftp_disconnect(cached_connection);
+}
diff --git a/usr.sbin/pkg_install/lib/url.c b/usr.sbin/pkg_install/lib/url.c
index 914ce39..68f31bb 100644
--- a/usr.sbin/pkg_install/lib/url.c
+++ b/usr.sbin/pkg_install/lib/url.c
@@ -163,5 +163,6 @@ fileGetURL(const char *base, const char *spec, int keep_package)
 	printf("tar command returns %d status\n", WEXITSTATUS(pstat));
     if (rp && (isatty(0) || Verbose))
 	printf(" Done.\n");
+    fetchCloseCachedFTP();
     return rp;
 }


>Release-Note:
>Audit-Trail:

From: Mark <markjdb@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/151866: [libfetch] [patch] closing the cached FTP connection
Date: Sun, 7 Nov 2010 19:47:08 -0500

 An alternate (and better) solution to this problem was suggested by
 Nick Hibma on freebsd-current:
 
 diff --git a/lib/libfetch/ftp.c b/lib/libfetch/ftp.c
 index f5ae7ff..9dfbf3c 100644
 --- a/lib/libfetch/ftp.c
 +++ b/lib/libfetch/ftp.c
 @@ -992,6 +992,7 @@ ftp_connect(struct url *url, struct url *purl,
 const char *flags)
  	if ((e = ftp_authenticate(conn, url, purl)) != FTP_LOGGED_IN)
  		goto fouch;
 
 +	fcntl(conn->sd, F_SETFD, FD_CLOEXEC);
  	/* TODO: Request extended features supported, if any (RFC 3659). */
 
  	/* done */

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: freebsd-gnats-submit@freebsd.org 
Cc:  
Subject: bin/151866
Date: Tue, 18 Jan 2011 11:01:28 +0100

 There's a problem with that patch, though.  If the child *does* know
 about the connection and expects to use it, libfetch now has a socket
 descriptor which it thinks points to a (possibly dead) connection but
 which may in fact have been reused.
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: Mark Johnston <markjdb@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/151866: [libfetch] [patch] closing the cached FTP
 connection
Date: Sun, 30 Jan 2011 20:28:54 -0500

 I'm not quite sure what you mean... this won't affect fork()ed
 processes, and a child that execve()s won't be able to access the
 connection through the libfetch API anyway, it would just have the
 socket descriptor itself, and it would have to know the actual value of
 the descriptor to use it.
 
 Thanks,
 -Mark
Responsible-Changed-From-To: freebsd-bugs->des 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Feb 23 22:31:17 UTC 2011 
Responsible-Changed-Why:  
des, is this still your area of interest? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=151866 

From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= <des@des.no>
To: freebsd-gnats-submit@freebsd.org 
Cc:  
Subject: Re: bin/151866: [libfetch] [patch] closing the cached FTP connection
Date: Fri, 13 May 2011 08:58:40 +0200

 Sorry, for some reason my mind translated "close on exec" to "close on
 fork".  The patch is correct.
 
 DES
 --=20
 Dag-Erling Sm=C3=B8rgrav - des@des.no

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/151866: commit references a PR
Date: Fri, 13 May 2011 07:21:57 +0000 (UTC)

 Author: des
 Date: Fri May 13 07:21:41 2011
 New Revision: 221830
 URL: http://svn.freebsd.org/changeset/base/221830
 
 Log:
   Mark all socket and file descriptors close-on-exec.
   
   PR:		bin/151866
   MFC after:	3 weeks
 
 Modified:
   head/lib/libfetch/common.c
   head/lib/libfetch/file.c
 
 Modified: head/lib/libfetch/common.c
 ==============================================================================
 --- head/lib/libfetch/common.c	Fri May 13 05:27:58 2011	(r221829)
 +++ head/lib/libfetch/common.c	Fri May 13 07:21:41 2011	(r221830)
 @@ -213,6 +213,7 @@ fetch_reopen(int sd)
  	/* allocate and fill connection structure */
  	if ((conn = calloc(1, sizeof(*conn))) == NULL)
  		return (NULL);
 +	fcntl(sd, F_SETFD, FD_CLOEXEC);
  	conn->sd = sd;
  	++conn->ref;
  	return (conn);
 
 Modified: head/lib/libfetch/file.c
 ==============================================================================
 --- head/lib/libfetch/file.c	Fri May 13 05:27:58 2011	(r221829)
 +++ head/lib/libfetch/file.c	Fri May 13 07:21:41 2011	(r221830)
 @@ -33,6 +33,7 @@ __FBSDID("$FreeBSD$");
  #include <sys/stat.h>
  
  #include <dirent.h>
 +#include <fcntl.h>
  #include <stdio.h>
  #include <string.h>
  
 @@ -57,6 +58,7 @@ fetchXGetFile(struct url *u, struct url_
  		fetch_syserr();
  	}
  
 +	fcntl(fileno(f), F_SETFD, FD_CLOEXEC);
  	return (f);
  }
  
 @@ -84,6 +86,7 @@ fetchPutFile(struct url *u, const char *
  		fetch_syserr();
  	}
  
 +	fcntl(fileno(f), F_SETFD, FD_CLOEXEC);
  	return (f);
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: des 
State-Changed-When: Fri May 13 07:33:28 UTC 2011 
State-Changed-Why:  
Fixed in head. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=151866 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/151866: commit references a PR
Date: Wed, 19 Oct 2011 11:49:23 +0000 (UTC)

 Author: des
 Date: Wed Oct 19 11:49:14 2011
 New Revision: 226539
 URL: http://svn.freebsd.org/changeset/base/226539
 
 Log:
   MFH r221830: mark all sockets and file descriptors close-on-exec
   
   PR:		bin/151866
 
 Modified:
   stable/8/lib/libfetch/common.c
   stable/8/lib/libfetch/file.c
 Directory Properties:
   stable/8/lib/libfetch/   (props changed)
 
 Modified: stable/8/lib/libfetch/common.c
 ==============================================================================
 --- stable/8/lib/libfetch/common.c	Wed Oct 19 11:48:21 2011	(r226538)
 +++ stable/8/lib/libfetch/common.c	Wed Oct 19 11:49:14 2011	(r226539)
 @@ -213,6 +213,7 @@ fetch_reopen(int sd)
  	/* allocate and fill connection structure */
  	if ((conn = calloc(1, sizeof(*conn))) == NULL)
  		return (NULL);
 +	fcntl(sd, F_SETFD, FD_CLOEXEC);
  	conn->sd = sd;
  	++conn->ref;
  	return (conn);
 
 Modified: stable/8/lib/libfetch/file.c
 ==============================================================================
 --- stable/8/lib/libfetch/file.c	Wed Oct 19 11:48:21 2011	(r226538)
 +++ stable/8/lib/libfetch/file.c	Wed Oct 19 11:49:14 2011	(r226539)
 @@ -33,6 +33,7 @@ __FBSDID("$FreeBSD$");
  #include <sys/stat.h>
  
  #include <dirent.h>
 +#include <fcntl.h>
  #include <stdio.h>
  #include <string.h>
  
 @@ -57,6 +58,7 @@ fetchXGetFile(struct url *u, struct url_
  		fetch_syserr();
  	}
  
 +	fcntl(fileno(f), F_SETFD, FD_CLOEXEC);
  	return (f);
  }
  
 @@ -84,6 +86,7 @@ fetchPutFile(struct url *u, const char *
  		fetch_syserr();
  	}
  
 +	fcntl(fileno(f), F_SETFD, FD_CLOEXEC);
  	return (f);
  }
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: des 
State-Changed-When: Mon Nov 5 13:03:49 UTC 2012 
State-Changed-Why:  
fixed 

http://www.freebsd.org/cgi/query-pr.cgi?pr=151866 
>Unformatted:
