From eugen@eg.sd.rdtc.ru  Thu Mar 21 09:29:55 2013
Return-Path: <eugen@eg.sd.rdtc.ru>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	by hub.freebsd.org (Postfix) with ESMTP id 3F8FA897;
	Thu, 21 Mar 2013 09:29:55 +0000 (UTC)
	(envelope-from eugen@eg.sd.rdtc.ru)
Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5])
	by mx1.freebsd.org (Postfix) with ESMTP id A514A789;
	Thu, 21 Mar 2013 09:29:53 +0000 (UTC)
Received: from eg.sd.rdtc.ru (localhost [127.0.0.1])
	by eg.sd.rdtc.ru (8.14.6/8.14.6) with ESMTP id r2L9TnTS040947;
	Thu, 21 Mar 2013 16:29:49 +0700 (NOVT)
	(envelope-from eugen@eg.sd.rdtc.ru)
Received: (from eugen@localhost)
	by eg.sd.rdtc.ru (8.14.6/8.14.6/Submit) id r2L9TihL040946;
	Thu, 21 Mar 2013 16:29:44 +0700 (NOVT)
	(envelope-from eugen)
Message-Id: <201303210929.r2L9TihL040946@eg.sd.rdtc.ru>
Date: Thu, 21 Mar 2013 16:29:44 +0700 (NOVT)
From: Eugene Grosbein <egrosbein@rdtc.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc: harti@freebsd.org
Subject: [bsnmpd] [patch] snmp_hostres modules does not support FS over 1TB
X-Send-Pr-Version: 3.114
X-GNATS-Notify:

>Number:         177183
>Category:       bin
>Synopsis:       [patch] bsnmpd(8): snmp_hostres modules does not support FS over 1TB
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    trociny
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Mar 21 09:40:00 UTC 2013
>Closed-Date:    Tue Apr 09 06:02:25 UTC 2013
>Last-Modified:  Tue Apr 09 06:02:25 UTC 2013
>Originator:     Eugene Grosbein
>Release:        FreeBSD 8.3-STABLE i386
>Organization:
RDTC JSC
>Environment:
System: FreeBSD eg.sd.rdtc.ru 8.3-STABLE FreeBSD 8.3-STABLE #53: Wed Feb 13 18:22:34 NOVT 2013 root@eg.sd.rdtc.ru:/usr/local/obj/usr/local/src/sys/EG i386

>Description:
	bsnmp(1) has snmp_hostres module to support HOSTRES MIB.
	This SNMP MIB's hrStorageTable limits hrStorageSize and hrStorageUsed
	parameters to 32 bit integer values, so file systems having more
	than 2^31 allocation units cannot be shown correctly.

	Let us follow net-snmp behaviour that (since 2011) translates
	hrStorageSize/hrStorageUsed/hrStorageAllocationUnits triples
	so that hrStorageAllocationUnits get bigger and other fit
	under INT_MAX by default.

>How-To-Repeat:

	This demonstrates the problem using holey file to mimic
	3TB file system (this needs 18GB free in real fs):

# truncate -s 3T f
# mdconfig -af f
md0
# newfs -O2 -b 4096 -f 512 -i $((1024*1024*1024)) /dev/md0 >/dev/null
# mount /dev/md0 /mnt/tmp
# rmdir /mnt/tmp/.snap; dd if=/dev/zero bs=2048 count=1000 of=/mnt/tmp/f2
# df -k /mnt/tmp
Filesystem 1024-blocks Used      Avail Capacity  Mounted on
/dev/md0    3217454200 2004 2960055860     0%    /mnt/tmp

	Then look at snmpwalk's output and see hrStorageAllocationUnits
	equals to 512 (fsize) and hrStorageSize is truncated to 2^31-1.

	With the following patch you get result consistent with net-snmp:

HOST-RESOURCES-MIB::hrStorageType.393 = OID: HOST-RESOURCES-TYPES::hrFSBerkeleyFFS
HOST-RESOURCES-MIB::hrStorageDescr.393 = STRING: /mnt/tmp, type: ufs, dev: /dev/md0
HOST-RESOURCES-MIB::hrStorageAllocationUnits.393 = INTEGER: 2048 Bytes
HOST-RESOURCES-MIB::hrStorageSize.393 = INTEGER: 1608727100
HOST-RESOURCES-MIB::hrStorageUsed.393 = INTEGER: 1002

	SNMP's data 2048*1608727100=3294673100800 equals to df's data:
	1024*3217454200=3294673100800
	
>Fix:

--- usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c.orig	2013-03-21 13:42:16.000000000 +0700
+++ usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c	2013-03-21 14:15:51.000000000 +0700
@@ -442,7 +442,9 @@
 storage_OS_get_fs(void)
 {
 	struct storage_entry *entry;
-	uint64_t used_blocks_count = 0;
+	uint64_t block_size = 0;
+	uint64_t free_blocks_count = 0;
+	uint64_t total_blocks_count = 0;
 	char fs_string[SE_DESC_MLEN];
 	int mounted_fs_count;
 	int i = 0;
@@ -473,6 +475,8 @@
 	fs_tbl_pre_refresh();
 
 	for (i = 0; i < mounted_fs_count; i++) {
+		int shift = 0;
+
 		snprintf(fs_string, sizeof(fs_string),
 		    "%s, type: %s, dev: %s", fs_buf[i].f_mntonname,
 		    fs_buf[i].f_fstypename, fs_buf[i].f_mntfromname);
@@ -488,23 +492,22 @@
 		entry->flags |= HR_STORAGE_FOUND;
 		entry->type = fs_get_type(&fs_buf[i]); /*XXX - This is wrong*/
 
-		if (fs_buf[i].f_bsize > INT_MAX)
-			entry->allocationUnits = INT_MAX;
-		else
-			entry->allocationUnits = fs_buf[i].f_bsize;
-
-		if (fs_buf[i].f_blocks > INT_MAX)
-			entry->size = INT_MAX;
-		else
-			entry->size = fs_buf[i].f_blocks;
-
-		used_blocks_count = fs_buf[i].f_blocks - fs_buf[i].f_bfree;
-
-		if (used_blocks_count > INT_MAX)
-			entry->used = INT_MAX;
-		else
-			entry->used = used_blocks_count;
 
+		total_blocks_count = fs_buf[i].f_blocks;
+		block_size = fs_buf[i].f_bsize;
+		free_blocks_count = fs_buf[i].f_bfree;
+
+		while (total_blocks_count > INT_MAX) {
+			total_blocks_count >>= 1;
+			shift++;
+		}
+		if (shift) {
+			block_size <<= shift;
+			free_blocks_count >>= shift;
+		}
+		entry->size = total_blocks_count;
+		entry->allocationUnits = block_size;
+		entry->used = total_blocks_count - free_blocks_count;
 		entry->allocationFailures = 0;
 
 		/* take care of hrFSTable */
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->trociny 
Responsible-Changed-By: trociny 
Responsible-Changed-When: Sun Mar 24 21:02:42 UTC 2013 
Responsible-Changed-Why:  
I'll take it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=177183 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/177183: commit references a PR
Date: Mon, 25 Mar 2013 19:12:49 +0000 (UTC)

 Author: trociny
 Date: Mon Mar 25 19:12:36 2013
 New Revision: 248707
 URL: http://svnweb.freebsd.org/changeset/base/248707
 
 Log:
   hrStorageSize and hrStorageUsed are 32 bit integers, reporting a fs
   size and usage in hrStorageAllocationUnits. If the file system has
   more than 2^31 allocations it can not be shown correctly and the
   meters are useless.
   
   In such cases follow net-snmp behaviour and increase
   hrStorageAllocationUnits so the values fit under INT_MAX.
   
   PR:		bin/177183
   Submitted by:	Eugene Grosbein egrosbein rdtc.ru
   MFC after:	2 weeks
 
 Modified:
   head/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c
 
 Modified: head/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c
 ==============================================================================
 --- head/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c	Mon Mar 25 15:40:57 2013	(r248706)
 +++ head/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c	Mon Mar 25 19:12:36 2013	(r248707)
 @@ -442,10 +442,9 @@ static void
  storage_OS_get_fs(void)
  {
  	struct storage_entry *entry;
 -	uint64_t used_blocks_count = 0;
 +	uint64_t size, used;
 +	int i, mounted_fs_count, units;
  	char fs_string[SE_DESC_MLEN];
 -	int mounted_fs_count;
 -	int i = 0;
  
  	if ((mounted_fs_count = getfsstat(NULL, 0, MNT_NOWAIT)) < 0) {
  		syslog(LOG_ERR, "hrStorageTable: getfsstat() failed: %m");
 @@ -488,22 +487,17 @@ storage_OS_get_fs(void)
  		entry->flags |= HR_STORAGE_FOUND;
  		entry->type = fs_get_type(&fs_buf[i]); /*XXX - This is wrong*/
  
 -		if (fs_buf[i].f_bsize > INT_MAX)
 -			entry->allocationUnits = INT_MAX;
 -		else
 -			entry->allocationUnits = fs_buf[i].f_bsize;
 -
 -		if (fs_buf[i].f_blocks > INT_MAX)
 -			entry->size = INT_MAX;
 -		else
 -			entry->size = fs_buf[i].f_blocks;
 -
 -		used_blocks_count = fs_buf[i].f_blocks - fs_buf[i].f_bfree;
 -
 -		if (used_blocks_count > INT_MAX)
 -			entry->used = INT_MAX;
 -		else
 -			entry->used = used_blocks_count;
 +		units = fs_buf[i].f_bsize;
 +		size = fs_buf[i].f_blocks;
 +		used = fs_buf[i].f_blocks - fs_buf[i].f_bfree;
 +		while (size > INT_MAX) {
 +			units <<= 1;
 +			size >>= 1;
 +			used >>= 1;
 +		}
 +		entry->allocationUnits = units;
 +		entry->size = size;
 +		entry->used = used;
  
  		entry->allocationFailures = 0;
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/177183: commit references a PR
Date: Mon,  8 Apr 2013 19:48:53 +0000 (UTC)

 Author: trociny
 Date: Mon Apr  8 19:48:40 2013
 New Revision: 249271
 URL: http://svnweb.freebsd.org/changeset/base/249271
 
 Log:
   MFC r248707:
   
   hrStorageSize and hrStorageUsed are 32 bit integers, reporting a fs
   size and usage in hrStorageAllocationUnits. If the file system has
   more than 2^31 allocations it can not be shown correctly and the
   meters are useless.
   
   In such cases follow net-snmp behaviour and increase
   hrStorageAllocationUnits so the values fit under INT_MAX.
   
   PR:		bin/177183
   Submitted by:	Eugene Grosbein egrosbein rdtc.ru
 
 Modified:
   stable/9/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c
 Directory Properties:
   stable/9/usr.sbin/bsnmpd/   (props changed)
   stable/9/usr.sbin/bsnmpd/modules/snmp_hostres/   (props changed)
 
 Modified: stable/9/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c
 ==============================================================================
 --- stable/9/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c	Mon Apr  8 19:44:57 2013	(r249270)
 +++ stable/9/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c	Mon Apr  8 19:48:40 2013	(r249271)
 @@ -442,10 +442,9 @@ static void
  storage_OS_get_fs(void)
  {
  	struct storage_entry *entry;
 -	uint64_t used_blocks_count = 0;
 +	uint64_t size, used;
 +	int i, mounted_fs_count, units;
  	char fs_string[SE_DESC_MLEN];
 -	int mounted_fs_count;
 -	int i = 0;
  
  	if ((mounted_fs_count = getfsstat(NULL, 0, MNT_NOWAIT)) < 0) {
  		syslog(LOG_ERR, "hrStorageTable: getfsstat() failed: %m");
 @@ -488,22 +487,17 @@ storage_OS_get_fs(void)
  		entry->flags |= HR_STORAGE_FOUND;
  		entry->type = fs_get_type(&fs_buf[i]); /*XXX - This is wrong*/
  
 -		if (fs_buf[i].f_bsize > INT_MAX)
 -			entry->allocationUnits = INT_MAX;
 -		else
 -			entry->allocationUnits = fs_buf[i].f_bsize;
 -
 -		if (fs_buf[i].f_blocks > INT_MAX)
 -			entry->size = INT_MAX;
 -		else
 -			entry->size = fs_buf[i].f_blocks;
 -
 -		used_blocks_count = fs_buf[i].f_blocks - fs_buf[i].f_bfree;
 -
 -		if (used_blocks_count > INT_MAX)
 -			entry->used = INT_MAX;
 -		else
 -			entry->used = used_blocks_count;
 +		units = fs_buf[i].f_bsize;
 +		size = fs_buf[i].f_blocks;
 +		used = fs_buf[i].f_blocks - fs_buf[i].f_bfree;
 +		while (size > INT_MAX) {
 +			units <<= 1;
 +			size >>= 1;
 +			used >>= 1;
 +		}
 +		entry->allocationUnits = units;
 +		entry->size = size;
 +		entry->used = used;
  
  		entry->allocationFailures = 0;
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/177183: commit references a PR
Date: Mon,  8 Apr 2013 19:50:30 +0000 (UTC)

 Author: trociny
 Date: Mon Apr  8 19:50:15 2013
 New Revision: 249272
 URL: http://svnweb.freebsd.org/changeset/base/249272
 
 Log:
   MFC r248707:
   
   hrStorageSize and hrStorageUsed are 32 bit integers, reporting a fs
   size and usage in hrStorageAllocationUnits. If the file system has
   more than 2^31 allocations it can not be shown correctly and the
   meters are useless.
   
   In such cases follow net-snmp behaviour and increase
   hrStorageAllocationUnits so the values fit under INT_MAX.
   
   PR:		bin/177183
   Submitted by:	Eugene Grosbein egrosbein rdtc.ru
 
 Modified:
   stable/8/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c
 Directory Properties:
   stable/8/usr.sbin/bsnmpd/   (props changed)
 
 Modified: stable/8/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c
 ==============================================================================
 --- stable/8/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c	Mon Apr  8 19:48:40 2013	(r249271)
 +++ stable/8/usr.sbin/bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c	Mon Apr  8 19:50:15 2013	(r249272)
 @@ -442,10 +442,9 @@ static void
  storage_OS_get_fs(void)
  {
  	struct storage_entry *entry;
 -	uint64_t used_blocks_count = 0;
 +	uint64_t size, used;
 +	int i, mounted_fs_count, units;
  	char fs_string[SE_DESC_MLEN];
 -	int mounted_fs_count;
 -	int i = 0;
  
  	if ((mounted_fs_count = getfsstat(NULL, 0, MNT_NOWAIT)) < 0) {
  		syslog(LOG_ERR, "hrStorageTable: getfsstat() failed: %m");
 @@ -488,22 +487,17 @@ storage_OS_get_fs(void)
  		entry->flags |= HR_STORAGE_FOUND;
  		entry->type = fs_get_type(&fs_buf[i]); /*XXX - This is wrong*/
  
 -		if (fs_buf[i].f_bsize > INT_MAX)
 -			entry->allocationUnits = INT_MAX;
 -		else
 -			entry->allocationUnits = fs_buf[i].f_bsize;
 -
 -		if (fs_buf[i].f_blocks > INT_MAX)
 -			entry->size = INT_MAX;
 -		else
 -			entry->size = fs_buf[i].f_blocks;
 -
 -		used_blocks_count = fs_buf[i].f_blocks - fs_buf[i].f_bfree;
 -
 -		if (used_blocks_count > INT_MAX)
 -			entry->used = INT_MAX;
 -		else
 -			entry->used = used_blocks_count;
 +		units = fs_buf[i].f_bsize;
 +		size = fs_buf[i].f_blocks;
 +		used = fs_buf[i].f_blocks - fs_buf[i].f_bfree;
 +		while (size > INT_MAX) {
 +			units <<= 1;
 +			size >>= 1;
 +			used >>= 1;
 +		}
 +		entry->allocationUnits = units;
 +		entry->size = size;
 +		entry->used = used;
  
  		entry->allocationFailures = 0;
  
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->closed 
State-Changed-By: trociny 
State-Changed-When: Tue Apr 9 05:59:38 UTC 2013 
State-Changed-Why:  
Committed. Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=177183 
>Unformatted:
