BufDataset foCaseInsensitive not working with "ñ" "Ñ"
lazarus / 16-Databases

----------------------------------------------------------------------
lainz · 2020-01-03 15:16

"Tempo" is the TEdit text.
The filter works fine for any text, except these special characters.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Productos.Filtered := False;      Productos.FilterOptions :=
[foCaseInsensitive];      //Productos.Filter   := ‘(sNombre = ’ +
QuotedStr(‘*’ + Tempo + ‘*’)      //                    + ' OR
sCodProducto = ' + QuotedStr(Tempo)      //                    + ' OR
sEan = ' + QuotedStr(Tempo)      //                    + ' OR sEan2 =
' + QuotedStr(Tempo)+‘) ’      //                    + ' AND bActivo
<> ' + QuotedStr(‘F’);       Productos.Filter :=
ArmarFiltroProductos(Tempo); // the same as the commented lines above 
    Productos.Filter := Productos.Filter + ' AND sTipoProducto <> ' +
QuotedStr(‘I’);      Productos.Filtered := True;
The lowercase of Ñ is ñ, but setting foCaseInsensitive has no effect.

If I type Ñ only uppercase elements of the DB are returned in the
DBGrid. Like "Ñandú" but not "Cañón".
If I type ñ only lowercase elements of the DB are returned in the
DBGrid. Like "Cañón" but not "Ñandú".

The DB has mixed content, some are upper case and some are lower case.

In BufDataset.pas I found this

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
DBCompareText(subValue, aValue: pointer; size: integer; options:
TLocateOptions): LargeInt; begin  if
[loCaseInsensitive,loPartialKey]=options then    Result :=
AnsiStrLIComp(pchar(subValue),pchar(aValue),length(pchar(subValue))) 
else if [loPartialKey] = options then    Result :=
AnsiStrLComp(pchar(subValue),pchar(aValue),length(pchar(subValue))) 
else if [loCaseInsensitive] = options then    Result :=
AnsiCompareText(pchar(subValue),pchar(aValue))  else    Result :=
AnsiCompareStr(pchar(subValue),pchar(aValue));end;
Anyways, the ñ and Ñ are in the ASCII table

http://ascii-table.com/ansi-codes.php

209   D1   U+00D1   Ñ   Latin Capital Letter N With Tilde
241   F1   U+00F1   ñ   Latin Small Letter N With Tilde

Any ideas on how to fix? I'm on Windows but the project works also on
Linux and macOS.

----------------------------------------------------------------------
lainz · 2020-01-03 22:55

Attached a simple test project that replicates the Issue, I'm
reporting it at the bugtracker too.

How to test:
- Click on the first button, and see the filtered records (it displays
only 1 record, must display 2)
- Click on the second button, and see the filtered records (it
displays only 1 record, must display 2)

Edit: Reported here
https://bugs.freepascal.org/view.php?id=36512

----------------------------------------------------------------------
amartitegui · 2023-02-01 15:58

Hello
No news on this right?

----------------------------------------------------------------------
LacaK · 2023-02-22 08:44

IMO problem is in AnsiCompareText function. On Windows this functions
calls Win API function CompareStringA(), which "Compares two character
strings, for a locale specified by identifier" - so
SompareStringA(LOCALE_USER_DEFAULT, NORM_IGNORECASE, ...) is called.

And there is remark "If your application is calling the ANSI version
of CompareString, the function converts parameters via the default
code page of the supplied locale. Thus, an application can never use
CompareString to handle UTF-8 text."

But in Lazarus string are by default UTF-8 encoded so probably you
supply UTF-8 strings to AnsiCompareText function, which will not work
...

In my experiments on Windows (ansi code page 1250):
- CompareStringA(LOCALE_USER_DEFAULT, NORM_IGNORECASE,
'ábc',3,'ÁBC',3)-2 == 0 ... OK
- CompareStringA(LOCALE_USER_DEFAULT, NORM_IGNORECASE,
'ábc',3,'ABC',3)-2 <> 0 ... OK
- CompareStringA(LOCALE_USER_DEFAULT,
NORM_IGNORECASE+LINGUISTIC_IGNOREDIACRITIC, 'ábc',3,'ABC',3)-2 == 0
... OK

----------------------------------------------------------------------
Thaddy · 2023-02-22 11:54

As per msdn, CompareStrEx should be called for UTF8/16. It looks like
it has no impact except for windows versions before Vista.
See the advice here:
https://learn.microsoft.com/en-us/windows/win32/api/winnls/nf-winnls-
comparestringa

So this indeed is a bug and needs to be corrected.

Also note that even CompareStrEx has some security considerations as
announced as per:
https://learn.microsoft.com/en-us/windows/win32/intl/security-
considerations--international-features