From simsong@nitroba.com  Fri Feb  7 11:47:45 2003
Return-Path: <simsong@nitroba.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id DB04D37B401
	for <FreeBSD-gnats-submit@freebsd.org>; Fri,  7 Feb 2003 11:47:44 -0800 (PST)
Received: from nitroba.com (ip-64-7-15-234.dsl.bos.megapath.net [64.7.15.234])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 21A3343F3F
	for <FreeBSD-gnats-submit@freebsd.org>; Fri,  7 Feb 2003 11:47:38 -0800 (PST)
	(envelope-from simsong@nitroba.com)
Received: from nitroba.com (localhost [127.0.0.1])
	by nitroba.com (8.12.6/8.12.6) with ESMTP id h17JlUpB080546
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 7 Feb 2003 14:47:30 -0500 (EST)
	(envelope-from simsong@nitroba.com)
Received: (from simsong@localhost)
	by nitroba.com (8.12.6/8.12.6/Submit) id h17JlUQg080545;
	Fri, 7 Feb 2003 14:47:30 -0500 (EST)
Message-Id: <200302071947.h17JlUQg080545@nitroba.com>
Date: Fri, 7 Feb 2003 14:47:30 -0500 (EST)
From: "Simson L. Garfinkel" <simsong@nitroba.com>
Reply-To: "Simson L. Garfinkel" <simsong@nitroba.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: mount -o snapshot doesn't work on +100GB disks 
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         48062
>Category:       kern
>Synopsis:       mount -o snapshot doesn't work on +100GB disks
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    mckusick
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Feb 07 11:50:09 PST 2003
>Closed-Date:    Wed Jul 14 18:30:10 UTC 2010
>Last-Modified:  Wed Jul 14 18:30:10 UTC 2010
>Originator:     Simson L. Garfinkel
>Release:        FreeBSD 5.0 i386
>Organization:
MIT 
>Environment:
System: FreeBSD r2.nitroba.com 5.0-RC2 FreeBSD 5.0-RC2 #2: Tue Dec 31 18:15:35 EST 2002 simsong@r2.nitroba.com:/usr/src/sys/i386/compile/R2 i386


	
>Description:
Greetings. I have successfully used mount -o snapshot on volumes
less than 30Gb, but when I use it on my 150GB volume, the command hangs
and slowly every other process hangs as well. This behavior is 
observed on both a RC2 system and on a 5.0-RELEASE system.

	
>How-To-Repeat:
	

Install FreeBSD 5.0 on a system with a 200GB disk.
mkdir /usr/snapshots
mount -u -o snapshot /usr/snapshots/snap1 /usr

System slowly hangs; the mount command doesn't come back, even after
crunching for more than 2 days.

>Fix:
Fix the code?
	

>Release-Note:
>Audit-Trail:

From: Garrett Wollman <wollman@lcs.mit.edu>
To: "Simson L. Garfinkel" <simsong@nitroba.com>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: kern/48062: mount -o snapshot doesn't work on +100GB disks 
Date: Fri, 7 Feb 2003 15:30:07 -0500 (EST)

 <<On Fri, 7 Feb 2003 14:47:30 -0500 (EST), "Simson L. Garfinkel" <simsong@nitroba.com> said:
 
 > Greetings. I have successfully used mount -o snapshot on volumes
 > less than 30Gb, but when I use it on my 150GB volume, the command hangs
 > and slowly every other process hangs as well. This behavior is 
 > observed on both a RC2 system and on a 5.0-RELEASE system.
 
 Well, at least on a 73-gig disk that was convenient to hand, I can't
 replicate this problem on a quiet system.  Creating the snapshot did
 take some tens of seconds, and it took about ten seconds to delete it
 again.  The process of creating a snapshot requires that all
 filesystem activity be brought to a temporary halt, so it is not
 surprising that creating a snapshot on /usr would eventually hang
 most other processes until it is completed.  Please check the wait
 channels (by starting something like `top' in another window/VT or by
 using the `ps' command from DDB) to see what states various processes
 are in.
 
 FYI, the exact results for my experiment are:
 
 create        39.92 real         0.00 user         1.20 sys
 remove        8.72 real         0.00 user         0.31 sys
 
 The resulting snapshot contained 42221568 bytes of metadata.
 (Keeping in mind that this filesystem is empty but for the root
 directory, the directory containing the snapshot, and the snapshot
 itself.)
 
 -GAWollman
 

From: "Simson L. Garfinkel" <simsong@lcs.mit.edu>
To: Garrett Wollman <wollman@lcs.mit.edu>
Cc: "Simson L. Garfinkel" <simsong@nitroba.com>,
	FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: kern/48062: mount -o snapshot doesn't work on +100GB disks 
Date: Fri, 7 Feb 2003 15:33:46 -0500

 Thank you. The process never finished. If you are in NE43, I can leave 
 it running on my desktop here.
 
 The snapshot feature is very useful for me, but I need to be able to do 
 it on /usr.
 
 I have to catch a train in a few minutes, but I can do this on Monday, 
 if you wish.
 
 
 On Friday, February 7, 2003, at 03:30 PM, Garrett Wollman wrote:
 
 > <<On Fri, 7 Feb 2003 14:47:30 -0500 (EST), "Simson L. Garfinkel" 
 > <simsong@nitroba.com> said:
 >
 >> Greetings. I have successfully used mount -o snapshot on volumes
 >> less than 30Gb, but when I use it on my 150GB volume, the command 
 >> hangs
 >> and slowly every other process hangs as well. This behavior is
 >> observed on both a RC2 system and on a 5.0-RELEASE system.
 >
 > Well, at least on a 73-gig disk that was convenient to hand, I can't
 > replicate this problem on a quiet system.  Creating the snapshot did
 > take some tens of seconds, and it took about ten seconds to delete it
 > again.  The process of creating a snapshot requires that all
 > filesystem activity be brought to a temporary halt, so it is not
 > surprising that creating a snapshot on /usr would eventually hang
 > most other processes until it is completed.  Please check the wait
 > channels (by starting something like `top' in another window/VT or by
 > using the `ps' command from DDB) to see what states various processes
 > are in.
 >
 > FYI, the exact results for my experiment are:
 >
 > create        39.92 real         0.00 user         1.20 sys
 > remove        8.72 real         0.00 user         0.31 sys
 >
 > The resulting snapshot contained 42221568 bytes of metadata.
 > (Keeping in mind that this filesystem is empty but for the root
 > directory, the directory containing the snapshot, and the snapshot
 > itself.)
 >
 > -GAWollman
 
Responsible-Changed-From-To: freebsd-bugs->kirk 
Responsible-Changed-By: kris 
Responsible-Changed-When: Fri Oct 10 22:17:15 PDT 2003 
Responsible-Changed-Why:  
Assign to snapshot maintainer 

http://www.freebsd.org/cgi/query-pr.cgi?pr=48062 
Responsible-Changed-From-To: kirk->mckusick 
Responsible-Changed-By: kris 
Responsible-Changed-When: Sat Oct 11 01:05:49 PDT 2003 
Responsible-Changed-Why:  
Correct kirk's user id 

http://www.freebsd.org/cgi/query-pr.cgi?pr=48062 

From: Eric Ziegast <ez@z57.com>
To: freebsd-gnats-submit@FreeBSD.org, simsong@nitroba.com
Cc:  
Subject: Re: kern/48062: mount -o snapshot doesn't work on +100GB disks
Date: Tue, 14 Sep 2004 16:00:34 -0700

 The documentation can use improvement to help set people's expectations 
 on how long a snapshot is supposed to take.  The only way people have to 
 benchmark progress during a snapshot mount is from the example in  
 src/sys/ufs/ffs/README.snapshot - 8GB should take 25 seconds to prepare 
 and an additional 5 seconds to commit.  What about 14GB? 80GB? has 
 anyone tried anyhting larger?  We don't know how it scales.  I looked at 
 Kirk's Usenix paper and saw no insights about snapshot performance.  I 
 have a machine with an 18GB OS disk and a bunch of 250GB external 
 disks.  I can run the snapshot examples on a small (14GB) partition, but 
 whenever I try to run it on an external 250GB partition, the mount hangs 
 and all operations to the .snap directory block.
 
 backup# df
 Filesystem  1K-blocks      Used     Avail Capacity  Mounted on
 /dev/da0s1a   1982798    728494   1095682    40%    /
 devfs               1         1         0   100%    /dev
 /dev/da0s1d  14019342     16852  12880944     0%    /local
 /dev/da1s1  236514320 142425778  75167398    65%    /da1
 /dev/da2s1  236514320  71959978 145633198    33%    /da2
 /dev/da3s1  236514320 117888770  99704406    54%    /da3
 /dev/da4s1  236514320  80521954 137071222    37%    /da4
 
 backup# dmesg | grep da1
 GEOM: create disk da1 dp=0xc674cc50
 da1 at sym0 bus 0 target 9 lun 0
 da1: <ATA WDC     WD2500JB 15.0> Fixed Direct Access SCSI-3 device
 da1: 80.000MB/s transfers (40.000MHz, offset 14, 16bit)
 da1: 238475MB (488397167 512 byte sectors: 255H 63S/T 30401C)
 
    Note: Using SCSI-IDE bridge.  No problems with using the device at all
    for normal FS operations, so I doubt it's an issue affecting 
 snapshots, but
    I thought I'd mention it.
 
 backup# iostat 1
       tty             da0              da1              da2             cpu
  tin tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
    0    0  6.77   1  0.00  15.98   0  0.01  15.68   0  0.00   0  0  0  0 100
    0  230  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  0  0 100
    0   77  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  0  0 100
    0   77  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  0  1 99
    0   76 16.00   2  0.03   0.00   0  0.00   0.00   0  0.00   0  0  0  0 100
    0   77  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  0  0 100
    0   77  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  1  0 99
    0   76  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   0  0  0  0 100
 
 backup# /bin/ls -1f /da1/.snap
 ^C <--- hangs - can't interrupt
 
 backup # top
 last pid:  3388;  load averages:  0.00,  0.00,  0.00    up 0+21:10:03  
 15:04:29
 24 processes:  1 running, 23 sleeping
 CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% 
 idle
 Mem: 7064K Active, 176M Inact, 148M Wired, 4K Cache, 112M Buf, 670M Free
 Swap: 1024M Total, 1024M Free
 
   PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME   WCPU    CPU COMMAND
  2822 root      -4    0  1204K   644K ufs    1   0:03  0.00%  0.00% mount
   408 root      96    0  1608K  1212K select 0   0:02  0.00%  0.00% ntpd
   427 root      96    0  3488K  2508K select 0   0:01  0.00%  0.00% sshd
   445 root       8    0  1336K  1048K nanslp 1   0:00  0.00%  0.00% cron
 
 backup# ktrace -p 2822
 ...wait 60 seconds ...
 backup# kdump
 ...nothing...
 
 
 It's just plain blocked and doesn't appear to be doing anything.  
 There's nothing in /var/log/messages or elsewhere (like "dmesg") to 
 indicate a problem.  The only way I can recover is by turning off the 
 box and rebooting.  Is filesystem size still a known open issue with UFS 
 snapshots?  Have others seen similar problems?  How long would a 1TB or 
 even a 100GB snapshot normally take to rebuild?
 
 My OS/kernel config...
 
 backup# uname -a
 FreeBSD backup.z57.com 5.2.1-RELEASE FreeBSD 5.2.1-RELEASE #1: Mon Aug 
 23 13:48:08 PDT 2004     
 root@backup.z57.com:/usr/src/sys/i386/compile/BB  i386
 
 backup# cat /usr/src/sys/i386/conf/BB [edited for brevity]
 machine         i386
 cpu             I686_CPU
 ident           GENERIC
 options:
   NFSSERVER CD9660 PROCFS PSEUDOFS COMPAT_43 COMPAT_FREEBSD4
   SCSI_DELAY=5000 KTRACE SYSVSHM SYSVMSG SYSVSEM _KPOSIX_PRIORITY_SCHEDULING
   KBD_INSTALL_CDEV AHC_REG_PRETTY_PRINT AHD_REG_PRETTY_PRINT PFIL_HOOKS
   INVARIANT_SUPPORT SMP ATA_STATIC_ID
 devices:
   apic isa eisa pci fdc ata atadisk atapicd atapifd
   scbus ch da sa cd pass ses mly mlx atkbdc atkbd psm vga splash sc
   agp npx pmtimer sio em miibus fxp rl random loop ether tun pty md
   bpf uhci ohci usb ugen uhid ukbd ulpt umass ums uscanner aue axe
 
 
 My system is otherwise quiet and I can replicate the problem easily.  My 
 next step might be to rebuild my kernel with DEBUG and DIAGNOSTIC 
 enabled and setting snapdebug=1 and collectsnapstats=1 in ffs_snapshot.c.
 
 --
 Eric Ziegast
 

From: "Simson L. Garfinkel" <simsong@csail.mit.edu>
To: Eric Ziegast <ez@z57.com>
Cc: freebsd-gnats-submit@FreeBSD.org, simsong@nitroba.com
Subject: Re: kern/48062: mount -o snapshot doesn't work on +100GB disks
Date: Thu, 16 Sep 2004 09:30:07 -0400

 On Sep 14, 2004, at 7:00 PM, Eric Ziegast wrote:
 
 > The documentation can use improvement to help set people's 
 > expectations on how long a snapshot is supposed to take.  The only way 
 > people have to benchmark progress during a snapshot mount is from the 
 > example in  src/sys/ufs/ffs/README.snapshot - 8GB should take 25 
 > seconds to prepare and an additional 5 seconds to commit.  What about 
 > 14GB? 80GB? has anyone tried anyhting larger?  We don't know how it 
 > scales.  I looked at Kirk's Usenix paper and saw no insights about 
 > snapshot performance.  I have a machine with an 18GB OS disk and a 
 > bunch of 250GB external disks.  I can run the snapshot examples on a 
 > small (14GB) partition, but whenever I try to run it on an external 
 > 250GB partition, the mount hangs and all operations to the .snap 
 > directory block.
 
 This is precisely my experience. Please let me know if you have any 
 luck with this.
 

From: "Job W. J. Snijders" <job@interlab.nl>
To: bug-followup@FreeBSD.org,
 simsong@nitroba.com
Cc:  
Subject: Re: kern/48062: mount -o snapshot doesn't work on +100GB disks
Date: Mon, 19 Jun 2006 15:26:50 +0200

 This problem occurred to me on the following two machines: the  
 command hangs and slowly every other process hangs as well.
 
 #uname -a
 FreeBSD backupstorage1.pool1.interlab.nl 6.1-RELEASE FreeBSD 6.1- 
 RELEASE #0: Sun May  7 04:42:56 UTC 2006      
 root@opus.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP  i386
 
 # df -h
 Filesystem       Size    Used   Avail Capacity  Mounted on
 /dev/amrd0s1a    721G    930M    662G     0%    /
 
 #uname -a
 FreeBSD testserver.office.interlab.nl 5.4-RELEASE-p14 FreeBSD 5.4- 
 RELEASE-p14 #2: Tue Apr 25 19:18:42 CEST 2006      
 root@testserver.office.interlab.nl:/usr/obj/usr/src/sys/GENERIC  i386
 
 [root@testserver ~] df -h
 Filesystem     Size    Used   Avail Capacity  Mounted on
 /dev/da0s1a    225G     18G    189G     9%    /
 devfs          1.0K    1.0K      0B   100%    /dev
 /dev/da1s1a    451G     42G    373G    10%    /home/users
 
 when issuing the following command: "mount -u -o snapshot /.snap/ 
 root / " or similar commands.
 
 

From: Alexander Best <alexbestms@wwu.de>
To: <bug-followup@FreeBSD.org>
Cc: "Simson L. Garfinkel" <simsong@nitroba.com>,
 Garrett Wollman <wollman@lcs.mit.edu>,
 Eric Ziegast <ez@z57.com>,
 "Job W. J. Snijders" <job@interlab.nl>
Subject: Re: kern/48062: mount -o snapshot doesn't work on +100GB disks
Date: Wed, 31 Mar 2010 13:22:13 +0200 (CEST)

 i was trying to reproduce this pr under 9-CURRENT and wasn't able to. i've
 created a snapshot of my 220GB hdd and `mount` finished after < 2 minutes. is
 this pr still an issue?
 
 -- 
 Alexander Best
State-Changed-From-To: open->feedback 
State-Changed-By: jh 
State-Changed-When: Wed Jun 9 13:30:22 UTC 2010 
State-Changed-Why:  
Note that submitter has been asked for feedback. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=48062 
State-Changed-From-To: feedback->closed 
State-Changed-By: jh 
State-Changed-When: Wed Jul 14 18:30:08 UTC 2010 
State-Changed-Why:  
Feedback timeout. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=48062 
>Unformatted:
