From nobody@FreeBSD.org  Sun Feb 26 14:20:58 2006
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 3EAC316A420
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 26 Feb 2006 14:20:58 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id EDA3843D45
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 26 Feb 2006 14:20:57 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k1QEKvpU074318
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 26 Feb 2006 14:20:57 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id k1QEKvfg074315;
	Sun, 26 Feb 2006 14:20:57 GMT
	(envelope-from nobody)
Message-Id: <200602261420.k1QEKvfg074315@www.freebsd.org>
Date: Sun, 26 Feb 2006 14:20:57 GMT
From: Dmitry Kazarov <kazarov@mcm.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Utility for loading the kernel charset translation tables on system boot
X-Send-Pr-Version: www-2.3

>Number:         93857
>Category:       bin
>Synopsis:       [iconv] [patch] new utility: kiconv_cs_preload(8): Utility for loading the kernel charset translation tables on system boot
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Sun Feb 26 14:30:03 GMT 2006
>Closed-Date:    
>Last-Modified:  Mon Aug 16 18:55:58 UTC 2010
>Originator:     Dmitry Kazarov
>Release:        6.1-PRERELEASE
>Organization:
Multicom Ltd, Moscow
>Environment:
FreeBSD CE.multicom.ru 6.1-PRERELEASE FreeBSD 6.1-PRERELEASE #0: Wed Feb 22 23:16:06 MSK 2006     root@CE.multicom.ru:/var/tmp/obj/usr/src/sys/CE  i386
>Description:
I have to allow non-root users to mount a CD-ROMs(CD9660, UDF) and
diskettes (MSDOSFS) filesystems.

I've set vfs.usermount to 1, changed permissions of devices in
/etc/devfs.conf, permissions of mount points - everything according to
FreeBSD faq #9.22.

But our users speak Russian and use CD-ROM/diskettes with Cyrillic
letters in filenames and I've added corresponding options to /etc/fstab
/dev/acd0 /cdrom cd9660  ro,noauto,-CUTF-8 0 0
(users use KDE so their locale is ru_RU.UTF-8).

And this option made impossible for users to mount cdrom!

Because it loads kiconv charset translation table that is allowed to
root only.  Searching internet showed a common solution to this problem
is creating small startup script in /usr/local/etc/rc.d with mount/umount
commands only:

  #!/bin/sh
  mount /cdrom
  umount /cdrom

Besides ugly error messages on console this script increases boot time.

So I've written this little utility that loads the charset conversion tables.
IMHO, it could be useful for many other people. And it's worth to include it in FreeBSD.

Script loadfscharsets should be placed in /etc/rc.d/.
Script expects next variables in /etc/rc.conf:

loadfscharsets_enabled={YES|NO}
loadfscharsets_fstypes={list of filesystems that use kiconv filename translation, defaults to "cd9660 udf msdosfs"}
loadfscharsets_cspairs={list of charset pairs|Russian|Japanese}

loadfscharsets_fstypes specifies list of filesystem types that use
external kernel modules for file name translation.

The kernel module <filesystem type name>_iconv have to exist in
/boot/kernel directory.

loadfscharsets_cspairs could be list of charsets pairs like UTF-8 (which
is equal to UTF-8:UTF-16BE), UTF-8:CP866, eucJP:CP932 or words like
Russian (equal to "UTF-8 UTF-8:CP866" list), Japanese (equal to
"UTF-8 UTF-8:CP92), some locale names (ru_RU.UTF-8, ru_RU.KOI-8 etc)

>How-To-Repeat:
After boot:
root# sysctl vfs.usermount=1
root# chown user /dev/acd0
root# chown user /cdrom
root# su user
user$ mount_cd9660 -CUTF-8 /dev/acd0 /cdrom
mount_cd9660: cd9660_iconv: Operation not permitted
root# mount_cd9660 -CUTF-8 /dev/acd0 /cdrom
root# umount /cdrom
user$ mount_cd9660 -CUTF-8 /dev/acd0 /cdrom
user$ mount | fgrep cdrom
/dev/acd0 on /cdrom (cd9660, local, nosuid, read-only, mounted by user)
>Fix:
# This is a shell archive.  Save it in a file, remove anything before
# this line, and then unpack it by entering "sh file".  Note, it may
# create directories; files and directories will be owned by you and
# have default permissions.
#
# This archive contains:
#
#       Makefile
#       kiconv_cs_preload.8
#       kiconv_cs_preload.c
#       loadfscharsets
#
echo x - Makefile
sed 's/^X//' >Makefile << 'END-of-Makefile'
X#      @(#)Makefile    8.3 (Berkeley) 3/27/94
X# $$
X
XPROG=  kiconv_cs_preload
XSRCS=  kiconv_cs_preload.c
XMAN=   kiconv_cs_preload.8
XDPADD= ${LIBKICONV}
XLDADD= -lkiconv
XBINDIR=        /usr/sbin
X
XWARNS?=        0
X
X# Needs to be dynamically linked for optional dlopen() access to
X# userland libiconv
XNO_SHARED?=    NO
X
X.include <bsd.prog.mk>
END-of-Makefile
echo x - kiconv_cs_preload.8
sed 's/^X//' >kiconv_cs_preload.8 << 'END-of-kiconv_cs_preload.8'
X.Dd February 19, 2006
X.Dt kiconv_cs_preload 8
X.Os
X.Sh NAME
X.Nm kiconv_cs_preload
X.Nd load kernel charset conversion tables
X.Sh SYNOPSIS
X.Nm
X.Ar locchrset Ns Op : Ns Ar dskcharset
X[
X.Ar locchrset Ns Op : Ns Ar dskcharset
X]...
X.Sh DESCRIPTION
XThe
X.Nm
Xloads specified character conversion tables into kernel for futher
Xusage with -C options of
X.Xr mount_cd9660 8 ,
X.Xr mount_udf 8 ,
X.Xr mount_ntfs 8
Xand
X-C and -D options of
X.Xr mount_msdosfs 8 .
X.Pp
XWhen sysctl vfs.usermount==1 non-root user is not capable to use -C option of
Xmount_* until root loads convertion table using -C option of mount_* or
X.Nm .
X.Pp
XOperands:
X.Pp
X.Bl -tag
X.It Ar locchrset
XCharset used by users, i.e. UTF-8, ISO8859-1, etc.
X.It Ar dskcharset
XCharset used for filenames on filesystem. Default is UTF-16BE,
Xcharset for CD9660 with Joliet extention, UDF and NTFS filesystems.
X.El
X.Sh EXAMPLES
XTo allow russian KDE users to view cyrillic symbols on Joliet CD:
X.Pp
X.Dl "root# kldload cd9660_iconv"
X.Dl "root# kiconv_cs_preload UTF-8"
X.Dl "user$ mount -t cd9660 -C UTF-8 /dev/acd0 /cdrom"
X.Pp
XTo allow those users to correctly work with MS-DOS diskettes and Joliet CD:
X.Pp
X.Dl "root# kldload msdosfs_iconv"
X.Dl "root# kiconv_cs_preload UTF-8 UTF-8:CP866"
X.Dl "user$ mount_msdosfs -DCP866 -Lru_RU.UTF-8 /dev/fd0 /mnt"
X.Pp
XKeep in mind that user should have read-write access to devices
X(/dev/acd0,/dev/fd0, etc) and mountpoints (/mnt, /cdrom, etc), and
X.Ar vfs.usermount
Xsysctl should be 1 (see
X.Xr mount 2 Ns ).
X.Sh SEE ALSO
X.Xr iconv 3 ,
X.Xr kiconv_add_xlat16_cspairs 3 ,
X.Xr mount 2 .
X.Sh HISTORY
XThe
X.Nm
Xutility appeared in
X.Fx 6.0 .
END-of-kiconv_cs_preload.8
echo x - kiconv_cs_preload.c
sed 's/^X//' >kiconv_cs_preload.c << 'END-of-kiconv_cs_preload.c'
X#include <err.h>
X#include <errno.h>
X#include <stdio.h>
X#include <string.h>
X#include <sysexits.h>
X#include <sys/types.h>
X#include <sys/iconv.h>
X
Xvoid usage(const char*);
X
Xint main(int argc, char* argv[]) {
X    int exCode = 0;
X
X    if( argc < 2 ) usage(argv[0]);
X
X    while(--argc) {
X       char* disk  = ENCODING_UNICODE;
X       char* local = *++argv;
X       char* colon = strchr(local,':');
X       if( colon ) {
X           disk = colon + 1;
X           *colon = '\0';
X       }
X
X       if( strlen(disk) >= ICONV_CSNMAXLEN ) {
X           warnx("Too long charset name: %s", disk);
X           continue;
X       }
X
X       if( strlen(local)   >= ICONV_CSNMAXLEN ) {
X           warnx("Too long charset name: %s", local);
X           continue;
X       }
X
X       warnx(
X           "Preloading converstion tables between \"%s\" and \"%s\" charsets.",
X           disk,local
X       );
X       if( kiconv_add_xlat16_cspairs(disk,local) ) {
X           warn("kiconv");
X           exCode = 1;
X       }
X    }
X    return exCode;
X}
X
Xvoid usage(const char* n) {
X    fprintf(
X       stderr,
X       "usage: %s local_charset[:disk_charset] [local_charset[:disk_charset]...]\n"
X       "\n"
X       "Options:\n"
X       "  local_charset - local charset (UTF-8, ISO8859-1 et al),\n"
X       "  disk_charset  - code page used for filenames on devices, default UTF-16BE.\n"
X       "\n"
X       "The UTF-16BE charset is used for filenames on CD9660, UDF and "
X       "NTFS filesystems.\n",
X       n
X    );
X    exit(0);
X}
END-of-kiconv_cs_preload.c
echo x - loadfscharsets
sed 's/^X//' >loadfscharsets << 'END-of-loadfscharsets'
X#!/bin/sh
X
X# PROVIDE: loadfscharsets
X# REQUIRE: DAEMON
X# BEFORE:  LOGIN
X
X. /etc/rc.subr
X
Xname=loadfscharsets
Xrcvar=`set_rcvar`
X
Xload_rc_config $name
X
Xstop_cmd=":"
Xstart_precmd=loadfscharsets_precmd
X
Xcommand=/usr/sbin/kiconv_cs_preload
X
Xcommand_args=""
Xfor cspair in ${loadfscharsets_cspairs}; do
X    case "$cspair" in
X    [rR][uU][sS][sS][iI][aA][nN] | ru_RU.UTF-8 )
X       command_args="$command_args UTF-8 UTF-8:CP866"
X       ;;
X    [jJ][aA][pP][aA][nN][eE][sS][eE] | ja_JP:UTF-8 )
X       command_args="$command_args UTF-8 UTF-8:CP932"
X       ;;
X    ru_RU.KOI8-R )
X       command_args="$command_args KOI8-R KOI8-R:CP866"
X       ;;
X    ja_JP.eucJP )
X       command_args="$command_args eucJP eucJP:CP932"
X       ;;
X    * )
X       command_args="$command_args $cspair"
X       ;;
X    esac
Xdone
X
X: ${loadfscharsets_fstypes:=cd9660 msdosfs udf}
X
Xloadfscharsets_precmd() {
X    for fst in ${loadfscharsets_fstypes}; do
X       kldstat -q -m ${fst}_iconv || {
X           info "Loading media charset conversion module for $fst filesystems"
X           kldload ${fst}_iconv || return 1
X       }
X    done
X    [ -z "$command_args" ] &&
X       err 1 "\$loadfscharsets_cspairs is not set properly"
X    return 0
X}
X
Xrun_rc_command "$1"
END-of-loadfscharsets
exit

>Release-Note:
>Audit-Trail:

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Dmitry Kazarov <kazarov@mcm.ru>
Cc: freebsd-gnats-submit@FreeBSD.org, mlaier@FreeBSD.org
Subject: Re: bin/93857: Utility for loading the kernel charset translation tables on system boot
Date: Mon, 27 Feb 2006 10:39:29 +0300

 On Sun, Feb 26, 2006 at 02:20:57PM +0000, Dmitry Kazarov wrote:
 D> >Synopsis:       Utility for loading the kernel charset translation tables on system boot
 
 I've heard opinion that current API for loading tables into kernel is
 hack. Don't remember exactly, who said this. Afaik it can be found in
 some other PR.
 
 Now in FreeBSD we have a special API to load some data blob into kernel -
 firmwire(9). Is it possible to rewrite charset translation tables
 loading code to use this API? Is it a good idea?
 
 -- 
 Totus tuus, Glebius.
 GLEBIUS-RIPN GLEB-RIPE

From: Dmitry Kazarov <kazarov@mcm.ru>
To: Gleb Smirnoff <glebius@freebsd.org>
Cc: freebsd-gnats-submit@freebsd.org, mlaier@freebsd.org
Subject: Re: bin/93857: Utility for loading the kernel charset translation tables on system boot
Date: Mon, 27 Feb 2006 13:17:19 +0300

 Hi, Gleb!
 
 I have not digged deeply into kiconv functionality - I do not know way it 
 loads data into kernel. I use /usr/lib/libkiconv.so which makes all work for 
 me.  System utils mount_{cd9660,udf,msdosfs,ntfs} works just this way.
 IMHO if kernel interface is going to be changed the libkiconv.so is the best 
 place for changes. 
 
 In other hand I can not find anything about firmwire(9) - no mans, no pages on 
 www.freebsd.org
 
 Sincerely yours
 Dmitry
 
     27  2006 10:39 Gleb Smirnoff (a):
 > On Sun, Feb 26, 2006 at 02:20:57PM +0000, Dmitry Kazarov wrote:
 > D> >Synopsis:       Utility for loading the kernel charset translation
 > tables on system boot
 >
 > I've heard opinion that current API for loading tables into kernel is
 > hack. Don't remember exactly, who said this. Afaik it can be found in
 > some other PR.
 >
 > Now in FreeBSD we have a special API to load some data blob into kernel -
 > firmwire(9). Is it possible to rewrite charset translation tables
 > loading code to use this API? Is it a good idea?

From: Yoshihiro Ota <ota@j.email.ne.jp>
To: bug-followup@FreeBSD.org
Cc: kazarov@mcm.ru
Subject: Re: bin/93857: [patch] new utility: kiconv_cs_preload(8): Utility
 for loading the kernel charset translation tables on system boot
Date: Wed, 4 Feb 2009 04:24:51 -0500

 There have been patches on http://people.freebsd.org/~imura/kiconv/ since
 5.x releases.  After 7.0, these patches do not apply cleanly.
 
 I used them back then but I am not sure its current status.
 
 I think it was not merged back into the src.
 
 Thanks,
 Hiro
>Unformatted:
