Convert functions to work with PChar lazarus / 21-General ---------------------------------------------------------------------- ydd · 2021-02-11 14:48 In order to optimize work with memory manager (reduce number of calls to GetMem/FreeMem) I want to work with PChar It there set of functions to convert from PChar to Integer, Double, .. without prior convert PChar to String (which calls to Memory Manager)? for example: Function TextToFloat(Buffer: PChar; Out Value: Extended): Boolean converts Buffer to String first ---------------------------------------------------------------------- marcov · 2021-02-11 14:50 There is unit strings, but that is limited. Many string routines however have shortstring versions that don't go through heapmgr either. ---------------------------------------------------------------------- ydd · 2021-02-11 15:10 Quote from: marcov on February 11, 2021, 03:50:37 pmThere is unit strings, but that is limited. I checked what is in strings - there is no PChar to Int related Quote from: marcov on February 11, 2021, 03:50:37 pmMany string routines however have shortstring versions that don't go through heapmgr either. please give me an example of ShortString to Int or to Double? ---------------------------------------------------------------------- Kays · 2021-02-11 16:00 Quote from: ydd on February 11, 2021, 04:10:17 pm[…] please give me an example of ShortString to Int or to Double?Well, it’s not a function, but there’s a quite versatile routine already in the system unit:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program shortStringConversions(input, output, stdErr);var        s: shortString;        i: integer;        d: double;begin        s := '42';        readStr(s, i);                s := '1E9';        readStr(s, d);end.Furthermore, there’s of course val, that can work with both integers and real numbers. It’ll be an easy venture to write your own wrapper that does just what you specifically need. Note: If you replace s: shortString; with s: pChar;, this will trigger fpc_pchar_to_ansistr before calling readStr. We “love” our ansiStrings, right? However, val will expect a shortString, thus fpc_pchar_to_shortstr is invoked beforehand. ---------------------------------------------------------------------- 440bx · 2021-02-11 16:55 Quote from: ydd on February 11, 2021, 04:10:17 pmplease give me an example of ShortString to Int or to Double? I can suggest a solution on Windows, IOW, it 's unlikely to be portable.  On Windows, NTDLL offers quite a few routines to go from a null terminated string to just about any number you'd like and vice versa. For instance, sprintf by itself can convert just about any number to a string within user specified text.  sprintf pretty much works as a replacement for StrFmt and works exclusively with pchar (just in case, there is a widechar version too.)  There are a good number of routines that go the other way, they take a number as a string and return it's binary form, e.g, _atoi64 among others along the more general sscanf. On Windows, NTDLL provides just about every routine commonly found in the C library and then some!.  Also, they are usually well optimized too. HTH. ---------------------------------------------------------------------- marcov · 2021-02-11 18:17 Quote from: ydd on February 11, 2021, 04:10:17 pmplease give me an example of ShortString to Int or to Double? VAL() ---------------------------------------------------------------------- avk · 2021-02-11 18:23 Quote from: ydd on February 11, 2021, 04:10:17 pmplease give me an example of ShortString to Int or to Double? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function ShortStrToInt(const s: shortstring; out aValue: Integer): Boolean;var  I: Integer;begin  Val(s, aValue, I);  Result := I = 0;end; function TextToInt(p: PAnsiChar; out aValue: Integer): Boolean;var  I: Integer;begin  Val(p, aValue, I);  Result := I = 0;end; ---------------------------------------------------------------------- ydd · 2021-02-11 21:51 Quote from: avk on February 11, 2021, 07:23:02 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function ShortStrToIntfunction TextToInt  thank you for an example. I implemented simple version and it works 10 times faster then StrToInt or Val: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function  PCharToInt(P: PChar): Integer;var  LMinusFlag: Boolean;  B: PChar;begin  Result := 0;  B := P;  LMinusFlag := B^ = '-';  if LMinusFlag then    Inc(B);  while (B^ <> #0) do  begin    Result := Result * 10 + (Ord(B^) - Ord('0'));    Inc(B);  end;  if LMinusFlag then    Result := -Result;end; ---------------------------------------------------------------------- ydd · 2021-02-11 21:52 Quote from: Kays on February 11, 2021, 05:00:27 pmreadStr(s, i); it is about 10 times slower than StrToInt ---------------------------------------------------------------------- ydd · 2021-02-11 21:53 Quote from: 440bx on February 11, 2021, 05:55:04 pmOn Windows, NTDLL offers it will not work for me, I need to support Linux & ARM. MacOS is a plus ---------------------------------------------------------------------- Kays · 2021-02-11 23:27 Quote from: ydd on February 11, 2021, 10:52:26 pmQuote from: Kays on February 11, 2021, 05:00:27 pmreadStr(s, i); it is about 10 times slower than StrToIntWell, twice as slow. Tops. I didn’t claim it was fast. strToInt just calls val. So… It’s just difficult to come up with something that’s already been programmed for you. It’s like “Not invented here”, you know? I already see: Not checking for a nil pointer.Creating a new variable B, even though P is a value parameter, i. e. PCharToInt already has its own copy.Not limiting the number of characters read (“buffer over-read”). After all, the decimal string representation of low(integer) has de facto a maximum length.Using integer.(Ignoring a positive sign or other illegal characters or numbers [like '-', which is your function will interpret as 0].) [Presumably irrelevant for you, but still.]I just wonder, maybe you want to program in C? ---------------------------------------------------------------------- ydd · 2021-02-11 23:54 Quote from: Kays on February 12, 2021, 12:27:02 amI just wonder, maybe you want to program in C? My code is working with libpq - library to access Postgre and all values returned as PChar(s) I just want to optimize my code and eliminate double calls to Memory Manager to get and free memory for each PChar -> String convertion So I asked about PChar to Integer and wrote simple version for test ---------------------------------------------------------------------- jamie · 2021-02-12 00:19 One thing you can do that condenses code space is to use   if Length(AString) = 0 Then … over if Astring ='' then because the compiler goes out and creates an empty managed string when using a '' over the Length check actually checks the string you are testing for its length..   I don't know why the compiler can't optimize the code a little better in that regard but that is something I saw, at least with 3.0.4 compiler. ---------------------------------------------------------------------- Kays · 2021-02-12 00:48 Quote from: ydd on February 12, 2021, 12:54:22 amQuote from: Kays on February 12, 2021, 12:27:02 amI just wonder, maybe you want to program in C? My code is working with libpq - library to access Postgre and all values returned as PChar(s)[…]OK, OK… But my suggestion was serious. If this kind of function is a bottleneck for you, program that part in C and export it to Pascal. Or, you can always use libc, of course. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses      cTypes; function atoll(const nptr: pChar): cLongLong;        cdecl; external 'libc';[Disclaimer: I actually have no idea whether atoll is the “right” function. I have no libc experience.] Quote from: jamie on February 12, 2021, 01:19:20 amOne thing you can do that condenses code space is to use   if Length(AString) = 0 Then … over if Astring ='' then […]I actually wondered about that, too, quite a while ago, confer bug 36060. I think since trunk revision 43272 this will be less of an issue. PS: I can’t believe libpq is returning everything as pChars. This rather smells like poor DB design. Databases usually do support storing binary data, (especially) such as integers. ---------------------------------------------------------------------- Zvoni · 2021-02-12 07:12 Isn't "libc" deprecated (and/or just for Kylix-compatibility)? btw: The c-function atoll (and its str-sisters) are not that hard to recode in pascal. I think i did something like that as an excersize some weeks ago. ---------------------------------------------------------------------- Kays · 2021-02-12 10:24 Quote from: Zvoni on February 12, 2021, 08:12:52 amIsn't "libc" deprecated (and/or just for Kylix-compatibility)?The Kylix- compatibility unit libc is deprecated: https://wiki.freepascal.org/libc_unit ---------------------------------------------------------------------- ydd · 2021-02-12 10:55 Quote from: Kays on February 12, 2021, 01:48:20 amPS: I can’t believe libpq is returning everything as pChars. This rather smells like poor DB design. Databases usually do support storing binary data, (especially) such as integers. see: PQGetValue, https://www.postgresql.org/docs/13/libpq-exec.html in my case - many calls with millions of rows with many fields ---------------------------------------------------------------------- Kays · 2021-02-12 11:41 Quote from: ydd on February 12, 2021, 11:55:05 amsee: PQGetValue, https://www.postgresql.org/docs/13/libpq-exec.htmlYeah, but it says QuoteFor data in binary format, the value is in the binary representation determined by the data type's typsend and typreceive functions.So, it’s a PChar for all text fields, but you can (store and) retrieve binary data nevertheless. I think you are either converting data that isn’t supposed to be; Textbook example: Zipcodes. Nobody actually performs arithmetic operations on zip codes, thus zip codes are stored as strings, not integers.Or, mentioning “millions of rows with many fields”, the data is originally available as an integer, but then converted to a string representation for storage in a DB, i. e. poor DB design, and now you’re reversing that operation. ---------------------------------------------------------------------- Zvoni · 2021-02-12 11:49 I'm more wondering why he uses the PG-API directly instead of using the Object-Trinity of sqldb (Connection, Transaction, Query) ---------------------------------------------------------------------- ydd · 2021-02-12 11:57 Quote from: Kays on February 12, 2021, 12:41:59 pmI think you are wrong. Integer field value is returned as PChar string, for example 1, is returned as PChar '1'. In db it is stored as Integer. Task - is to quickly convert PChar '1' to integer, and eliminate convert PChar -> String, which causes 2 calls to Memory Manager ---------------------------------------------------------------------- ydd · 2021-02-12 11:59 Quote from: Zvoni on February 12, 2021, 12:49:09 pmI'm more wondering why he uses the PG-API directly instead of using the Object-Trinity of sqldb (Connection, Transaction, Query) because it is much faster (100 times in some cases) and uses much less RAM. So my task will complete in several hours, instead of several days. ---------------------------------------------------------------------- Zvoni · 2021-02-12 12:42 Quote from: ydd on February 12, 2021, 12:59:32 pmQuote from: Zvoni on February 12, 2021, 12:49:09 pmI'm more wondering why he uses the PG- API directly instead of using the Object-Trinity of sqldb (Connection, Transaction, Query) because it is much faster (100 times in some cases) and uses much less RAM. So my task will complete in several hours, instead of several days. Fair enough. I will look up my own pascal-version of the c-function strtoi (and sisters) if you're interested (IIRC, i had it as a generic function to cover all integer types, not sure though) ---------------------------------------------------------------------- Kays · 2021-02-12 13:03 Quote from: ydd on February 12, 2021, 12:57:23 pm[…] you are wrong. Integer field value is returned as PChar string, for example 1, is returned as PChar '1'. In db it is stored as Integer. […]Yeah, OK, but a quick Duckduckgo search “PostgreSQL PQGetValue integer” tells us how/why. SO:QuotePQgetvalue() returns the string representation of the field value unless you specify a binary cursor. You must set your PQExecParams beforehand. By default, everything—as you reported—is indeed returned in text format, a sensible default, but this can be changed.Quote[…] query results can be requested in either text or binary format […] ---------------------------------------------------------------------- PascalDragon · 2021-02-12 14:17 Quote from: jamie on February 12, 2021, 01:19:20 amOne thing you can do that condenses code space is to use   if Length(AString) = 0 Then … over if Astring ='' then because the compiler goes out and creates an empty managed string when using a '' over the Length check actually checks the string you are testing for its length..   I don't know why the compiler can't optimize the code a little better in that regard but that is something I saw, at least with 3.0.4 compiler. You're wrong. A comparison against an empty managed string type is a comparison against 0 for a long time already. ---------------------------------------------------------------------- jamie · 2021-02-12 22:09 Quote from: PascalDragon on February 12, 2021, 03:17:17 pmQuote from: jamie on February 12, 2021, 01:19:20 amOne thing you can do that condenses code space is to use   if Length(AString) = 0 Then … over if Astring ='' then because the compiler goes out and creates an empty managed string when using a '' over the Length check actually checks the string you are testing for its length..   I don't know why the compiler can't optimize the code a little better in that regard but that is something I saw, at least with 3.0.4 compiler. You're wrong. A comparison against an empty managed string type is a comparison against 0 for a long time already. You are correct, I was absolutely wrong!  Its a SHORTSTRING that gets converted to a managed string and then does the zero test..  Using a Length(AshortString) directly looks at the first byte, as it should.. I just checked this with 3.2 and it still does it. I use Short strings in records so they can remain formatted for file use or directly memory overlays.