Sorting special characters lazarus / 21-General ---------------------------------------------------------------------- IM314 · 2021-10-25 07:34 Very occasional, complete amateur hobbyist here, so please be patient with my ignorance. I've been struggling to get names with special characters to sort 'right'. The following is a *very* rough and simplified example of what I am dealing with. someObject is a class with property name : String. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i constructor someObject.create(s : string);begin   name := s;end; function compare1(s1,s2 : pointer) : integer;begin   result := comparetext(someObject(s1).name,someObject(s2).name);end; procedure TForm1.Button1Click(Sender: TObject);vari : integer;list : tfpList;begin  list := tfpList.Create;  list.add(someObject.create('Rêd'));  list.add(someObject.create('Rad'));  list.add(someObject.create('Rod'));  list.add(someObject.create('Rêzd'));  list.add(someObject.create('Rêad'));  list.Sort(@compare1);  memo1.clear;  for i := 0 to list.count-1 do   memo1.append(someObject(list[i]).name);end;   In this example (and my real-life project where I read names with special characaters from a file), the code above always sorts the special characters as if they come after z, so the result is always Rad Rod Rêad Rêd Rêzd How do I cast and/or compare these strings to get a more 'natural' sort order like every other piece of software I've tried (like Excel) seems to achieve with the same list: e.g: Rad Rêad Rêd Rêzd Rod Any advice much appreciated. ---------------------------------------------------------------------- dseligo · 2021-10-25 08:10 Maybe you can use this function to convert accented letters to non- accented ones and then sort: https://forum.lazarus.freepascal.org/inde x.php/topic,46804.msg334219.html#msg334219 ---------------------------------------------------------------------- IM314 · 2021-10-25 08:19 Thanks! This certainly works, but it feels like an incredibly roundabout (if clever) way of doing this. From my admittedly amateur reading and understanding of the documentation, something like QuoteansiCompareText(string1,string2) should do the same, but it does not appear to have any effect. ---------------------------------------------------------------------- Zvoni · 2021-10-25 08:38 Sounds like Collation Maybe this: https://www.freepascal.org/docs- html/rtl/unicodedata/incrementalcomparestring.html ---------------------------------------------------------------------- IM314 · 2021-10-25 08:55 Ah, should have known there would be a word for it. Collation. Interesting, and it does look like it addresses the issue. I knew from the start different languages would sort special characters differently based on pronunciation, so this makes sense that you can define different sort schemes, as it were. I was just being lazy and hoping for a magical shortcut that takes the nearest* equivalent from the Latin alphabet and sorts ê like e and à like A, for instance. The previous commenter's suggested function does exactly that, so at least my project now works. *Poorly-defined, I know. Hence collation being a thing. ---------------------------------------------------------------------- Zvoni · 2021-10-25 09:05 Well, another workaround might be to read all your filenames into a SQLite-InMemory Database, load the collation you need, fire off a SELECT-Query with an ORDER BY according to the chosen Collation, move the returned "List" of (hopefully) correctly sorted strings to your List ---------------------------------------------------------------------- wp · 2021-10-25 09:15 There's NaturalCompareText() in unit StrUtils. When I call this in your Compare1() function this output is as expected: Rad Rêad Rêd Rêzd Rod ---------------------------------------------------------------------- winni · 2021-10-25 09:16 Hi! The Unit LazUTF8 contains this function: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function UTF8CompareStr(const S1, S2: string): PtrInt; Winni ---------------------------------------------------------------------- IM314 · 2021-10-25 12:44 Quote from: wp on October 25, 2021, 11:15:13 amThere's NaturalCompareText() in unit StrUtils. When I call this in your Compare1() function this output is as expected: Rad Rêad Rêd Rêzd Rod I was very excited to see this, but it does not have the same result for me... I still get the strange sort result from my first post.  I'm on Mac, btw. A bit of superficial reading shows all kinds of issues with character sets (UTF-8, etc) that could impact on this. ---------------------------------------------------------------------- IM314 · 2021-10-25 12:46 Quote from: winni on October 25, 2021, 11:16:05 amHi! The Unit LazUTF8 contains this function: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function UTF8CompareStr(const S1, S2: string): PtrInt; Winni Thanks. However, as with the previous poster's NaturalCompareText, this seems to have no effect on my list being sorted in a different way. Not sure if the fact I'm on Mac could be an issue. ---------------------------------------------------------------------- AlexTP · 2021-10-25 12:58 Chars A-Z have low Unicode numbers (below $FF) and chars with accents have the high numbers in Unicode. So the sorting must be as you see. ---------------------------------------------------------------------- AlexTP · 2021-10-25 13:05 From reply above, get the function which sorts ignoring the accents. Name is SimpleCompare. Then Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function compare1(s1,s2 : pointer) : integer;begin   result := SimpleCompare(someObject(s1).name,someObject(s2).name);   if result = 0 then     result := comparetext(someObject(s1).name,someObject(s2).name);end; ---------------------------------------------------------------------- wp · 2021-10-25 13:38 Quote from: IM314 on October 25, 2021, 02:44:46 pmQuote from: wp on October 25, 2021, 11:15:13 amThere's NaturalCompareText() in unit StrUtils. I was very excited to see this, but it does not have the same result for me... I still get the strange sort result from my first post.  I'm on Mac, btw. I tested on Windows, Linux/Ubuntu and macOS Mojave. Yes, it fails on mac, but it works on Win and Linux, so I think the idea is basically correct, but is not correctly implemented on cocoa. You should write an FPC bug report for NaturalCompareText on cocoa. Add your sources as a compilable project.