From nobody@FreeBSD.org  Tue Feb 14 16:11:43 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id C1016106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 14 Feb 2012 16:11:43 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id B01238FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 14 Feb 2012 16:11:43 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q1EGBhsU048251
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 14 Feb 2012 16:11:43 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q1EGBhff048250;
	Tue, 14 Feb 2012 16:11:43 GMT
	(envelope-from nobody)
Message-Id: <201202141611.q1EGBhff048250@red.freebsd.org>
Date: Tue, 14 Feb 2012 16:11:43 GMT
From: Anatoly Mashanov <thor@irk.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: mount_msdosfs -W koi2dos cannot read file No.
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         165140
>Category:       misc
>Synopsis:       mount_msdosfs -W koi2dos cannot read file No.
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Feb 14 16:20:10 UTC 2012
>Closed-Date:    Thu Feb 16 08:47:43 UTC 2012
>Last-Modified:  Thu Feb 16 08:50:11 UTC 2012
>Originator:     Anatoly Mashanov
>Release:        9.0-RELEASE
>Organization:
-
>Environment:
FreeBSD dream 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Thu Jan 12 00:54:02 IRKT 2012     root@dream:/usr/src/sys/i386/compile/DARKNESS  i386
>Description:
When I mount a flash drive with Windows FAT-32 filesystem containing files whose names contain a "&#8470;" (No.) symbol CP1251=0xb9,  these files cannot be accessed. File is seen in listing with "?" instead of "&#8470;". Message is "Cannot stat". Keys -W koi2dos or -L ru_RU.KOI8-R don't help.

This problem has meen met before and resolved with creating a custom /somewhere/in/filesystem/koi2dos file remapping a number sign to copyright sign, but mount_msdos doesn't seem to read such file now.
>How-To-Repeat:
Create under Windows a flash drive containing a single file named "&#1092;&#1072;&#1081;&#1083; &#8470;1" (Translated: "File No.1") and try to access it under FreeBSD
>Fix:
No idea. Maybe find a prehistoric version of FreeBSD and take a working mount_msdosfs from here?

>Release-Note:
>Audit-Trail:

From: Andriy Gapon <avg@FreeBSD.org>
To: bug-followup@FreeBSD.org, thor@irk.ru
Cc:  
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 11:58:47 +0200

 № symbol is not in KOI8-R.
 If you need to be able to nicely handle such symbols, then you have to use an
 appropriate character encoding in your localization (UTF-8, Windows-1251).
 We _could_ add a hack for this symbol, but where do we stop?  Tomorrow someone
 would request a hack for «», then something else.
 So I don't think that this is a way to go.
 Please try to handle this issue via local configuration.
 
 -- 
 Andriy Gapon

From: Thor Ablestar <thor@irk.ru>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 20:19:27 +0900

 Hello!
 
 On 02/15/12 18:58, Andriy Gapon wrote:
 > № symbol is not in KOI8-R.
 > If you need to be able to nicely handle such symbols, then you have to use an
 > appropriate character encoding in your localization (UTF-8, Windows-1251).
 > We _could_ add a hack for this symbol, but where do we stop?  Tomorrow someone
 > would request a hack for «», then something else.
 > So I don't think that this is a way to go.
 > Please try to handle this issue via local configuration.
 >
 Problem is that I have tried UTF-8, too. And the result is the same:
 
 # mount_msdosfs -L UTF-8 /dev/da0s1 /flop
 mount_msdosfs: UTF-8: No error: 0
 (didn't mount. Why? FAT-32 uses UTF, it should be transparent!)
 
 # mount_msdosfs -L ru_RU.UTF-8 /dev/da0s1 /flop
 # umount /flop
 (Did mount but file is still "Cannot stat" and not accessible)
 
 I've tried 1251 according to your advice. With some imagination, it may 
 be named "It Works".
 
 # mount_msdosfs -L ru_RU.CP1251 /dev/da0s1 /flop
 # umount /flop
 (Did mount OK and all files are accessible but names are of course 
 garbage-like. I need something special to find a file. Midnight 
 Commander, for instance)
 # mount_msdosfs -L ru_RU.CP866 /dev/da0s1 /flop
 # umount /flop
 (Same as 1251)
 
 CP-1251 is NOT the common encoding in Russian *nix world. KOI and UTF 
 are, and UTF is a perspective encoding while 1251 is not perspective 
 even under Windows (They also transition to UTF). Use of it will break 
 compatibility, and I agree to suffer compatibility loss due to UTF 
 transfer, but not 1251 transfer. CP866 is even older and it's life cycle 
 ended 17 years ago.
 
 So I strongly believe it would be necessary either to provide a 
 _correct_ name translation to UTF or allow user to load an arbitrary 
 translation table. Or both.  Neither works for now. Or at least tell me 
 where the translation tables lie. And option "transparent" will be nice 
 for emergency, too.
 
 But at least an ugly method to extract a single valuable file (or dump 
 and restore a FAT-32 volume not looking inside) has been found. Thanks.
 
             Anatoly
 

From: Andriy Gapon <avg@FreeBSD.org>
To: Thor Ablestar <thor@irk.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 13:52:10 +0200

 on 15/02/2012 13:19 Thor Ablestar said the following:
 > # mount_msdosfs -L ru_RU.UTF-8 /dev/da0s1 /flop
 > # umount /flop
 > (Did mount but file is still "Cannot stat" and not accessible)
 
 Sorry, I couldn't understand from this what exactly was the problem.
 
 -- 
 Andriy Gapon

From: Thor Ablestar <thor@irk.ru>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 21:20:57 +0900

 On 02/15/12 20:52, Andriy Gapon wrote:
 > on 15/02/2012 13:19 Thor Ablestar said the following:
 >> # mount_msdosfs -L ru_RU.UTF-8 /dev/da0s1 /flop
 >> # umount /flop
 >> (Did mount but file is still "Cannot stat" and not accessible)
 > Sorry, I couldn't understand from this what exactly was the problem.
 >
 Mount with -L ru_RU.UTF-8 locale succeeds.
 File is seen in Midnight Commander in RED color, with "?" instead of 
 "No." symbol, with length=0 and "?" in left column (where file type is 
 displayed). The Russian letters appear garbaged but it's normal for UTF-8.
 
 File is NOT seen at all in ls -l listing
 
 # cat esc/enter on this file in Midnight Commander gives
 cat: (filename here): Invalid argument
 
 F4 on this file in Midnight Commander gives a red frame:
 Cannot open (filename here) for reading
                   [Dismiss]
 
 F5 on this file in Midnight Commander gives a red frame:
 Cannot stat source file "(filename here)"
 Invalid argument (22)
     [Skip][Retry][Abort]
 
 (filename here) is exactly the same filename that MC shows in red.
 
 OpenOffice.Org Writer "Open File" does not see that file at all.
 
 

From: Andriy Gapon <avg@FreeBSD.org>
To: Thor Ablestar <thor@irk.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 14:42:41 +0200

 on 15/02/2012 14:20 Thor Ablestar said the following:
 > On 02/15/12 20:52, Andriy Gapon wrote:
 >> on 15/02/2012 13:19 Thor Ablestar said the following:
 >>> # mount_msdosfs -L ru_RU.UTF-8 /dev/da0s1 /flop
 >>> # umount /flop
 >>> (Did mount but file is still "Cannot stat" and not accessible)
 >> Sorry, I couldn't understand from this what exactly was the problem.
 >>
 > Mount with -L ru_RU.UTF-8 locale succeeds.
 > File is seen in Midnight Commander in RED color, with "?" instead of "No."
 > symbol, with length=0 and "?" in left column (where file type is displayed). The
 > Russian letters appear garbaged but it's normal for UTF-8.
 
 I'll let somebody else worry about Midnight commander.
 
 > File is NOT seen at all in ls -l listing
 
 OK, what is output of 'locale' command in your shell?
 
 -- 
 Andriy Gapon

From: Thor Ablestar <thor@irk.ru>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 21:48:34 +0900

 On 02/15/12 21:42, Andriy Gapon wrote:
 > OK, what is output of 'locale' command in your shell? 
 
 LANG=
 LC_CTYPE=ru_RU.KOI8-R
 LC_COLLATE="C"
 LC_TIME="C"
 LC_NUMERIC="C"
 LC_MONETARY="C"
 LC_MESSAGES="C"
 LC_ALL=
 
 
 

From: Andriy Gapon <avg@FreeBSD.org>
To: Thor Ablestar <thor@irk.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 14:52:08 +0200

 on 15/02/2012 14:48 Thor Ablestar said the following:
 > On 02/15/12 21:42, Andriy Gapon wrote:
 >> OK, what is output of 'locale' command in your shell? 
 > 
 > LANG=
 > LC_CTYPE=ru_RU.KOI8-R
 > LC_COLLATE="C"
 > LC_TIME="C"
 > LC_NUMERIC="C"
 > LC_MONETARY="C"
 > LC_MESSAGES="C"
 > LC_ALL=
 
 Right.  You have to use UTF-8 locale.
 
 -- 
 Andriy Gapon

From: Thor Ablestar <thor@irk.ru>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 22:09:35 +0900

 On 02/15/12 21:52, Andriy Gapon wrote:
 > on 15/02/2012 14:48 Thor Ablestar said the following:
 >> On 02/15/12 21:42, Andriy Gapon wrote:
 >>> OK, what is output of 'locale' command in your shell?
 >> LANG=
 >> LC_CTYPE=ru_RU.KOI8-R
 >> LC_COLLATE="C"
 >> LC_TIME="C"
 >> LC_NUMERIC="C"
 >> LC_MONETARY="C"
 >> LC_MESSAGES="C"
 >> LC_ALL=
 > Right.  You have to use UTF-8 locale.
 Tried ru_RU.UTF-8 locale. Result the same except the garbage letters in 
 filename are different. The file with "No." symbol in filename still 
 inaccessible with the same symptoms.
 
 
 

From: Thor Ablestar <thor@irk.ru>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 22:15:30 +0900

 On 02/15/12 21:52, Andriy Gapon wrote:
 > on 15/02/2012 14:48 Thor Ablestar said the following:
 >> On 02/15/12 21:42, Andriy Gapon wrote:
 >>> OK, what is output of 'locale' command in your shell?
 >> LANG=
 >> LC_CTYPE=ru_RU.KOI8-R
 >> LC_COLLATE="C"
 >> LC_TIME="C"
 >> LC_NUMERIC="C"
 >> LC_MONETARY="C"
 >> LC_MESSAGES="C"
 >> LC_ALL=
 > Right.  You have to use UTF-8 locale.
 >
 Tried again to mount_msdosfs -L ru_RU.CP1251
 Result: File accessible both with LC_CTYPE=ru_RU.KOI*-R and =ru_RU.UTF-8
 In both cases file name is garbaged differently.

From: Andriy Gapon <avg@FreeBSD.org>
To: Thor Ablestar <thor@irk.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 15:52:16 +0200

 on 15/02/2012 15:09 Thor Ablestar said the following:
 > On 02/15/12 21:52, Andriy Gapon wrote:
 >> on 15/02/2012 14:48 Thor Ablestar said the following:
 >>> On 02/15/12 21:42, Andriy Gapon wrote:
 >>>> OK, what is output of 'locale' command in your shell?
 >>> LANG=
 >>> LC_CTYPE=ru_RU.KOI8-R
 >>> LC_COLLATE="C"
 >>> LC_TIME="C"
 >>> LC_NUMERIC="C"
 >>> LC_MONETARY="C"
 >>> LC_MESSAGES="C"
 >>> LC_ALL=
 >> Right.  You have to use UTF-8 locale.
 > Tried ru_RU.UTF-8 locale. Result the same except the garbage letters in filename
 > are different. The file with "No." symbol in filename still inaccessible with the
 > same symptoms.
 
 You have to use a UTF-8 locale (e.g. ru_RU.UTF-8 ) and mount using the same locale
 (e.g. ru_RU.UTF-8 ), then everything should work.
 If something doesn't work then you should describe exactly what you did and
 exactly what problems you get.  When you just "something doesn't work", then my
 answer is just "you did something wrong".
 
 BTW, such things are better be discussed on the mailing list or forums.
 The PR database is not exactly a help channel.
 
 
 -- 
 Andriy Gapon

From: Thor Ablestar <thor@irk.ru>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 22:53:58 +0900

 > Right.  You have to use UTF-8 locale.
 >
 1. Do I understand correctly that 
 /usr/src/sbin/mount_msdosfs/mount_msdosfs.c takes the conversion table 
 from /usr/ports/converters/libiconv ?
 
 2. As I understand, accessibility of the file containing "No." in it's 
 name depends entirely on -L argument for mount_msdosfs command, and not 
 on LC_CTYPE. The filenames may LOOK differently but the files are still 
 accessible under their garbaged names.
 
 

From: Andriy Gapon <avg@FreeBSD.org>
To: Thor Ablestar <thor@irk.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Wed, 15 Feb 2012 17:04:36 +0200

 on 15/02/2012 15:53 Thor Ablestar said the following:
 > 
 >> Right.  You have to use UTF-8 locale.
 >>
 > 1. Do I understand correctly that /usr/src/sbin/mount_msdosfs/mount_msdosfs.c
 > takes the conversion table from /usr/ports/converters/libiconv ?
 
 Well, as unexpected as it is, the answer is yes.
 mount_msdosfs uses libkiconv(3) which dlopen()s libiconv.so, which is not provided
 by the base system and thus has to come from a thirdparty, typically the iconv port.
 
 We already have a BSD-licensed iconv in the src tree, but it is not connected to
 the build yet.
 
 > 2. As I understand, accessibility of the file containing "No." in it's name
 > depends entirely on -L argument for mount_msdosfs command, and not on LC_CTYPE.
 
 That -L controls how the OS internally represents the file names on the filesystem.
 
 > The filenames may LOOK differently but the files are still accessible under their
 > garbaged names.
 
 Locale controls output and input of characters.
 
 -- 
 Andriy Gapon

From: Thor Ablestar <thor@irk.ru>
To: Andriy Gapon <avg@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Thu, 16 Feb 2012 13:07:09 +0900

 On 02/16/12 00:04, Andriy Gapon wrote:
 > 1. Do I understand correctly that /usr/src/sbin/mount_msdosfs/mount_msdosfs.c
 > takes the conversion table from /usr/ports/converters/libiconv ?
 > Well, as unexpected as it is, the answer is yes.
 Well. Ugly hack to add koi-8 0xa9 "(c)" character as a replacement for 
 Unicode 0x2116 "No."  character to libiconf's lib/koi8r.h succeeded 
 (Reboot was needed). Thanks, problem temporarily solved but I still 
 believe that it should be solved permanently by the FreeBSD team in 
 official way.
State-Changed-From-To: open->closed 
State-Changed-By: avg 
State-Changed-When: Thu Feb 16 08:45:45 UTC 2012 
State-Changed-Why:  
User requests a hack to compensate for local configuration issues. 
Besides the hack would really belong in converters/libiconv upstream. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=165140 

From: Andriy Gapon <avg@FreeBSD.org>
To: Thor Ablestar <thor@irk.ru>
Cc: bug-followup@FreeBSD.org
Subject: Re: misc/165140: mount_msdosfs -W koi2dos cannot read file No.
Date: Thu, 16 Feb 2012 10:45:19 +0200

 on 16/02/2012 06:07 Thor Ablestar said the following:
 > On 02/16/12 00:04, Andriy Gapon wrote:
 >> 1. Do I understand correctly that /usr/src/sbin/mount_msdosfs/mount_msdosfs.c
 >> takes the conversion table from /usr/ports/converters/libiconv ?
 >> Well, as unexpected as it is, the answer is yes.
 > Well. Ugly hack to add koi-8 0xa9 "(c)" character as a replacement for Unicode
 > 0x2116 "No."  character to libiconf's lib/koi8r.h succeeded (Reboot was needed).
 > Thanks, problem temporarily solved but I still believe that it should be solved
 > permanently by the FreeBSD team in official way.
 
 GOTO 10
 
 :-)
 
 (FreeBSD doesn't need to grow hacks without necessity.  If you want some symbol,
 use a locale/charset that has it).
 
 -- 
 Andriy Gapon
>Unformatted:
