[SOLVED] strpos() Function results 2x slower than Perl's index()
Function
lazarus / 64-FPC development

----------------------------------------------------------------------
domibay_hugo · 2020-06-01 12:30

I tried to replace a Perl  Script with a FreePascal Application.
The Goal of this Application is to read a Log File and find matching
Log Entries for a specific time window by parsing the Time References.
I got 2 files as Samples:

$ ls -lah *.log
-rw-r--r--. 1 usr15 usr15 8,0M jun  1 11:20 small_firewall.log
-rw-r--r--. 1 usr15 usr15 115M jun  1 11:20 big_firewall.log

They contain Entries like:

Aug  2 14:53:43 <server_name> kernel: IN=eth0 OUT=
MAC=40:40:c9:aa:57:d3:c4:64:13:43:69:ff:08:00 SRC=193.111.152.234
DST=<server_ip> LEN=40 TOS=0x00 PREC=0x00 TTL=75 ID=39000 DF PROTO=TCP
SPT=60703 DPT=21 WINDOW=29200 RES=0x00 SYN URGP=0
Aug  2 14:53:43 <server_name> kernel: IN=eth0 OUT=
MAC=40:40:c9:aa:57:d3:c4:64:13:43:69:ff:08:00 SRC=193.111.152.234
DST=<server_ip>  LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=6107 DF PROTO=TCP
SPT=51567 DPT=21 WINDOW=29200 RES=0x00 SYN URGP=0

and

May 11 08:00:37 <server_name> kernel: Firewall:zn12zn2:DROP:IN=eth0
OUT= MAC= SRC=213.166.70.66 DST=<server_ip> LEN=40 TOS=0x00 PREC=0x00
TTL=243 ID=21562 PROTO=TCP SPT=52760 DPT=36692 WINDOW=1024 RES=0x00
SYN URGP=0
May 11 08:00:38 <server_name> kernel: Firewall:zn12zn2:DROP:IN=eth0
OUT= MAC= SRC=195.54.160.105 DST=<server_ip> LEN=40 TOS=0x00 PREC=0x00
TTL=239 ID=32820 PROTO=TCP SPT=57781 DPT=73 WINDOW=1024 RES=0x00 SYN
URGP=0


On the Small Log File it is already a bit slower than the Perl Script
The FreePascal performs like this:

$ time ./runstringsearch
file: './small_firewall.log': read do ...
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '15'
BuildPacketList - finished with [ 0 ]!

real      0m0.018s
user     0m0.016s
sys       0m0.002s

While the Perl does the same job like this:

$ time ./udp_watch.pl -a=l0:10:50 -f small_firewall.log -s vpn
--dir=./ -s drop -t 5
Check: [ ok ]
Check finished with [ 0 ]

real   0m0.013s
user   0m0.010s
sys           0m0.002s

On the bigger Log File it becomes even more drastical:
The FreePascal Application:

$ time ./runstringsearch
file: './big_firewall.log': read do ...
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '0'
BuildPacketList - finished with [ 0 ]!

real   0m0.400s
user   0m0.372s
sys          0m0.028s

While the Perl Script does the same Job:

$ time ./udp_watch.pl -a=l0:10:50 -f big_firewall.log -s vpn --dir=./
-s drop -t 5
Check: [ ok ]
Check finished with [ 0 ]

real   0m0.200s
user   0m0.181s
sys           0m0.019s


I tried to replace the strpos() Function by an own Implementation but
it did not change anything or made it even worse.
My FreePascal Prototype is this:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
runstringsearch; uses  Classes, SysUtils  , RegExpr;   //=============
=================================================================//
Auxiliary Functions  function PCharAtPos(const ssource: String;
iposition: Integer): PChar;begin  if (iposition > -1)    and
(iposition < Length(ssource)) then  begin    Result := PChar(ssource)
+ iposition;  end  else    Result := Nil;end;  function
PCharStartsWith(const ssource: PChar; isourcelength: Cardinal  ; const
ssearch: PChar; isearchlength: Cardinal): Boolean; (*inline;*)var 
pssrc, psend, pssrh: PChar;begin  pssrc := ssource;  pssrh :=
ssearch;  psend := pssrc + isourcelength;   //Shorter Srouce String
doesn't match  Result := (isourcelength >= isearchlength);   //Empty
Strings don't match  Result := Result    and ((pssrc^ <> #0)      or
(pssrc < psend)      or (pssrh^ <> #0));   //First Byte must match 
Result := Result and (pssrc^ = pssrh^);   while Result      and
(pssrh^ <> #0)      and (pssrc < psend)      and (pssrc^ <> #0) do 
begin    if pssrc^ <> pssrh^ then      Result := False    else   
begin      inc(pssrc);      inc(pssrh);    end;  end;  //while Result
doend;  function PCharSubStringRByChar(const ssource: PChar; const
ssearch: Char; const ssearchend: PChar = Nil): PChar;var  pssrh:
PChar;begin  pssrh := ssearchend;  Result := Nil;   if pssrh = Nil
then  begin    pssrh := ssource + strlen(ssource);  end;   while
(Result = Nil)    and (pssrh >= ssource) do  begin    if pssrh^ =
ssearch then      Result := pssrh    else if pssrh > ssource then    
dec(pssrh);   end;  //while (Result = Nil) and (pssrh >= ssource)
doend;  function PCharSubStringByString(const ssource: PChar;
isourcelength: Cardinal  ; const ssearch: PChar; isearchlength:
Cardinal): PChar;var  pssrc, psend: PChar;  isrhps(*, icmpln, imtch*):
Cardinal;  bsrhgo: Boolean;begin  Result := Nil;   pssrc := ssource; 
psend := pssrc + isourcelength;  isrhps := 0;  //icmpln :=
isourcelength;   //Skip on Empty Strings  bsrhgo := (pssrc < psend);  
while bsrhgo    and (pssrc < psend) do  begin    if pssrc^ = ssearch^
then    begin      (* icmpln := isourcelength - isrhps;       if
icmpln > isearchlength then        icmpln := isearchlength; *)      
//if pssrc[0..(icmpln - 1)] = ssearch[0..(isearchlength - 1)] then   
 //imtch := ;      //if CompareByte(ssearch^, pssrc^, isearchlength) =
0 then      if PCharStartsWith(pssrc, isourcelength - isrhps, ssearch,
isearchlength) then      begin        Result := pssrc;        bsrhgo
:= False;      end;      end;  //if pssrc^ = ssearch^ then    
inc(pssrc);    inc(isrhps);     //String End reached    //bsrhgo :=
;   end;  //while bsrhgo doend;  function StringRPosByChar(const
ssource: String; const ssearch: Char): Integer;var  psstrt, pssrh:
PChar;  isrhps: Integer;begin  psstrt := PChar(ssource);  isrhps :=
Length(ssource);   pssrh := psstrt + isrhps;  Result := -1;   while
(Result = -1)    and (pssrh >= psstrt) do  begin    if pssrh^ =
ssearch then      Result := isrhps    else if pssrh > psstrt then   
begin      dec(pssrh);      dec(isrhps);    end; //if pssrh^ = ssearch
then  end;  //while (Result = -1) and (pssrh >= psstrt)
doend;  function BuildPacketList: Integer;var  chkexp: TRegExpr; 
stmfl: TFileStream;  //stmchkpkg: TStringStream;  lstchklns:
TStringList;  schkpkg: String;  sflcnk: String;  pschkpkg: PChar; 
schklnsrh: String;  pslstln, pschktm, pschktmps, pschktmstrt: PChar; 
ichkpkgln, ischktmln(*, ichktmps*), ilstlnps: Integer;  icnksz:
Integer;  bchkstrt, bchkend: Boolean;  bdbg: Boolean;begin 
//------------------------------------  //Read the Firewall Log  
Result := 0;   bdbg := false;   try    //stmfl :=
TFileStream.Create('./small_firewall.log', (fmOpenRead or
fmShareDenyWrite));    stmfl :=
TFileStream.Create('./big_firewall.log', (fmOpenRead or
fmShareDenyWrite));  except    on e: Exception do    begin     
WriteLn('File ', chr(39), stmfl.FileName, chr(39), ': File Open failed
with [', e.HelpContext, ']!');      WriteLn('Message: ', chr(39),
e.Message, chr(39));       Result := 2;    end    else    begin     
WriteLn('File ', chr(39), stmfl.FileName, chr(39), ': File Open
failed!');      WriteLn('Message [-1]: ', chr(39), 'Unknown Error',
chr(39));       Result := 2;    end;  end;  //except   lstchklns :=
TStringList.Create;   icnksz := 32768;   sflcnk := '';  schkpkg :=
'';   SetLength(sflcnk, icnksz);  SetLength(schkpkg, icnksz);  
//stmchkpkg := TStringStream.Create('');   //stmchkpkg.Size :=
Self.fl.ChunkSize;   bchkstrt := False;  bchkend := not (Result =
0);   pschktm := PChar(' 11:48:');  ischktmln := Length(' 11:48:') +
1;   schklnsrh := '^.* 11:48:[0-2][0-9] .*[:''][^:'']*(udp
in|drop)[^:'']*[:''].*$';   try  //except    chkexp :=
TRegExpr.Create;     chkexp.Expression := schklnsrh;   
chkexp.ModifierI := True;    chkexp.ModifierM := True;  //allow to
work with ^$     chkexp.ModifierS:= False; //don't catch all text by
.*    chkexp.ModifierX:= False; //don't ingore spaces     //Build the
Regular Expression Control Structure    chkexp.Compile;  except    on
e: ERegExpr do    begin      WriteLn('Expression ', chr(39),
chkexp.Expression, chr(39), ': Compile failed with [', e.ErrorCode,
']!');      WriteLn('Error at Position ', chr(39), e.CompilerErrorPos,
chr(39), ': ', chr(39), e.Message, chr(39));       if Result < 1 then 
      Result := 1;       //Stop Reading      bchkend:= True;    end; 
end;  //except   if not bchkend then  begin    WriteLn('file: ',
chr(39), stmfl.FileName, chr(39), ': read do ...');  end;  //if not
bchkend then   try  //except    while not bchkend      and
(stmfl.Read(sflcnk[1], icnksz - 1) > 0) do    begin      pslstln :=
Nil;       if sflcnk <> '' then      begin        //Skip the Last
Line        ilstlnps := StringRPosByChar(sflcnk, chr(10));         if
ilstlnps <> -1 then        begin          schkpkg := schkpkg +
sflcnk[1..ilstlnps] + chr(0);          //stmchkpkg.Write(sflcnk[1],
ilstlnps);           //Terminate the String for PChar         
//stmchkpkg.WriteByte(0);           inc(ilstlnps);           pslstln
:= PCharAtPos(sflcnk, ilstlnps);        end        else  //The Chunk
doesn't have a Linebreak        begin          schkpkg := schkpkg +
sflcnk;          //stmchkpkg.WriteString(sflcnk);        end;  //if
pslstln <> nil then         pschkpkg := PChar(schkpkg);       
//pschkpkg := PChar(stmchkpkg.DataString);        ichkpkgln :=
strlen(pschkpkg);        //ichkpkgln := stmchkpkg.Position - 1;      
 if bdbg then          WriteLn('chk cnk 1: ', chr(39), pschkpkg,
chr(39));         //ichktmps := Pos(pschktm, pschkpkg);       
pschktmps := strpos(pschkpkg, pschktm);        //pschktmps :=
PCharSubStringByString(pschkpkg, ichkpkgln, pschktm, ischktmln);     
 pschktmps := Nil;         if pschktmps <> Nil then        begin     
   pschktmstrt := PCharSubStringRByChar(pschkpkg, chr(10),
pschktmps);           if pschktmstrt <> Nil then           
inc(pschktmstrt)          else            pschktmstrt := pschktmps;  
       //Start Time Check          bchkstrt := True;           if bdbg
then          begin            WriteLn('chk cnk 2: ', chr(39),
pschktmstrt, chr(39));            Write('exp: ', chr(39),
chkexp.Expression, chr(39), ' match - ');          end;           try
 //except            (*            if chkexp.Exec(pschktmstrt) then  
        begin              if bdbg then               
WriteLn('HIT');               repeat                //Add all Matches
for Checking                lstchklns.Add(chkexp.Match[0]);          
  until not chkexp.ExecNext;               if bdbg then              
WriteLn('lst chk lns (cnt: ', chr(39), lstchklns.Count, chr(39), '):
'                  , chr(39), lstchklns.CommaText, chr(39));         
  end            else  //Regular Expression does not match           
begin              if bdbg then                WriteLn('MISS');      
     end;  //if chkexp.Exec(pschktmstrt) then            *)         
except            on e: ERegExpr do            begin              if
bdbg then                WriteLn('CRASH');              
WriteLn('Expression ', chr(39), schklnsrh, chr(39), ': Match failed
with [', e.ErrorCode, ']!');              WriteLn('Error at Position
', chr(39), e.CompilerErrorPos, chr(39), ': ', chr(39), e.HelpContext,
chr(39));               if Result < 1 then                Result :=
1;               //Stop Reading              bchkend:= True;         
 end;            on e: Exception do            begin             
WriteLn('File ', chr(39), stmfl.FileName, chr(39), ': File Read failed
with [', e.HelpContext, ']!');              WriteLn('Message: ',
chr(39), e.Message, chr(39));               if Result < 1 then       
       Result := 1;               //Stop Reading             
bchkend:= True;            end;  //on e: ERegExpr do          end;
 //except         end        else if bchkstrt then        begin      
  //Stop Time Check          bchkend := True;        end;  //if
strpos(pschkpkg, pschktm) <> Nil then      end;  //if psrdcnk^ <> ''
then       //Reset the Check String      //stmchkpkg.Position := 0;  
   if pslstln <> Nil then      begin        //Readd the Last Line    
  schkpkg := pslstln;        //stmchkpkg.WriteString(pslstln);     
end      else      begin        schkpkg := '';       end;  //if
pslstln <> nil then    end;  //while not bchkend          //  and
(stmfl.Read(sflcnk[1], 32768 - 1) > 0) do  except    on e: Exception
do    begin      WriteLn('File ', chr(39), stmfl.FileName, chr(39), ':
File Read failed with [',  e.HelpContext, ']!');      WriteLn('Message
[', e.HelpContext, ']: ', e.Message);       Result := 1;    end   
else    begin      WriteLn('File ', chr(39), stmfl.FileName, chr(39),
': File Read failed!');      WriteLn('Message [-1]: ', chr(39),
'Unknown Error', chr(39));       Result := 1;    end;  //on e:
Exception do  end;  //except   WriteLn('Expression ', chr(39),
schklnsrh, chr(39), ': Match Count ', chr(39), lstchklns.Count,
chr(39));   //------------------------  //Clean-Up   chkexp.Free; 
//stmchkpkg.Free;   lstchklns.Free;   stmfl.Free;end;   //============
==================================================================//
Executing Section  const  SMETHOD = 'BuildPacketList';var  ierr:
Integer;begin    try      ierr := BuildPacketList;    except      on
e: Exception do      begin          WriteLn(SMETHOD, ' - failed!');  
      WriteLn('Message [', e.HelpContext, ']: ', e.Message);      end 
    else      begin          WriteLn(SMETHOD, ' - failed!');         
WriteLn('Message [-1]: ', chr(39), 'Unknown Error', chr(39));     
end;  //on e: Exception do       ierr := 1;    end;   WriteLn(SMETHOD,
' - finished with [', ierr, ']!'); end. 
The Regex Match does not make any difference.
Even commenting it out the Run Time stays the same:

$ time ./runstringsearch
file: './big_firewall.log': read do ...
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '0'
BuildPacketList - finished with [ 0 ]!

real   0m0.448s
user   0m0.432s
sys           0m0.015s


But the time drops drastically on disabling the strpos() Call:

$ time ./runstringsearch
file: './big_firewall.log': read do ...
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '0'
BuildPacketList - finished with [ 0 ]!

real   0m0.130s
user   0m0.109s
sys           0m0.021s


So I was looking into the Perl Source Code and found that they use the
C Programming Language Function memchr()
FreePascal has a Function that works a bit similar the CompareByte() 
but it does not really do the same and even results slower than
Function PCharStartsWith() from the Prototype.

I wonder whether there is anything similar to the memchr() Function or
if it can be developped:
https://www.man7.org/linux/man-pages/man3/memchr.3.html

----------------------------------------------------------------------
julkas · 2020-06-01 13:07

Compile your Pascal program with -O2 or -O3 option.

----------------------------------------------------------------------
domibay_hugo · 2020-06-01 14:13

Quote from: julkas on June 01, 2020, 03:07:10 pmCompile your Pascal
program with -O2 or -O3 option.
Yes, all measurements were done with the Release Version with -O3
enabled.
Unless it would not be a fair comparison.

----------------------------------------------------------------------
marcov · 2020-06-01 14:18

So how does the Perl index function work? Is it a simple loop with a
character compare too, or does it use special primitives?

FPC also has some basic primitives, so a possible improvement could be
using primitives in FPC too, the indexbyte() function that most FPC
character search routines in the RTL are based on.

Note that anything regex based will always be a tad slow, since that
is an interpretive solution. Though some regex engines optimize some
simple cases with dedicated code.

----------------------------------------------------------------------
domibay_hugo · 2020-06-01 14:25

I supposed that the TStringStream could efficiently handle the String
Data but it resulted slower than a bare String Variable.
I also observed that the TStringStream seams to drop its Memory and
than reallocate it as shown in the strace log:

read(3, "3 05:48:20 lon08 kernel: IN=eth0"..., 32767) = 32767
read(3, "3 05:48:34 lon08 kernel: IN=eth0"..., 32767) = 32767
munmap(0x7fed5b148000, 32768)           = 0
read(3, "Aug  3 05:48:48 lon08 kernel: IN"..., 32767) = 32767
read(3, "RGP=0 \nAug  3 05:49:03 lon08 ker"..., 32767) = 32767


read(3, "kernel: IN=eth0 OUT= MAC=40:40:c"..., 32767) = 32767
mmap(NULL, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7fed5b148000
read(3, "0 \nAug  3 06:14:25 lon08 kernel:"..., 32767) = 32767

Which does not occur when using the String Variable

----------------------------------------------------------------------
domibay_hugo · 2020-06-01 14:31

Quote from: marcov on June 01, 2020, 04:18:30 pmSo how does the Perl
index function work? Is it a simple loop with a character compare too,
or does it use special primitives?

FPC also has some basic primitives, so a possible improvement could be
using primitives in FPC too, the indexbyte() function that most FPC
character search routines in the RTL are based on.

As far as I understood they compare 2 bytes at once with the memchr
Function.

The IndexByte() Function seems to me very promissing.

QuoteNote that anything regex based will always be a tad slow, since
that is an interpretive solution. Though some regex engines optimize
some simple cases with dedicated code.

I am aware that Regex Search is known to be slower. Therefore I am
trying to speed up the processing with a previous substr() check.

----------------------------------------------------------------------
Thaddy · 2020-06-01 14:32

StrPos() is a C support function and indeed is much slower than Pascal
native Pos() family.
StrPos() also FAILS on true Pascal type strings. (Since length is
known and embedded #0's are supported in Pascal string types)
It is geared towards PChars. (Which is famously not the brightest
ideafrom curly bracket language designers....).
If you have the need for StrPos(), you are not using Pascal features,
just a rude version to make some Pascal interact with the above silly
design of C like "strings". Technically C as a language does not even
know strings. Do away with any of the curly brackets and you are fine
as far as "strings" are concerned.

Basically: Pascal does not need to check for buffer overruns.
Algorithmic, walking a Pascal string or a C style buffer should be the
same in O notation: O(n). But technically, true Pascal stringtypes -
should and do -  have an advantage.
And Strpos() is not really Pascal.....

----------------------------------------------------------------------
domibay_hugo · 2020-06-01 14:44

Quote from: Thaddy on June 01, 2020, 04:32:24 pmStrPos() is a C
support function and indeed is much slower than Pascal native Pos()
family.
StrPos() also FAILS on true Pascal type strings. (Since length is
known and embedded #0's are supported in Pascal string types)
It is geared towards PChars. (Which is famously not the brightest
ideafrom curly bracket language designers....).
If you have the need for StrPos(), you are not using Pascal features,
just a rude version to make some Pascal interact with the above silly
design of C like "strings". Technically C as a language does not even
know strings. Do away with any of the curly brackets and you are fine
as far as "strings" are concerned.

Obviously it would be nicer to be able work with native String
Variables. But I'm unsure if the Syntax "pascalstring[ipos] =
searchbyte" is the same as fast as "pcharstring^ = searchbyte". Or
does it not activate serveral Range Checks which slow down the
processing even more?

Further more most String has the unconvienent habit to copy strings in
Memory which know as another slow-down.

So with PChar I can access the String without copying it over and over
again.

Even nice would be the access to String Slices without copying them.

----------------------------------------------------------------------
Thaddy · 2020-06-01 14:49

Then you are fooled by similarities in syntax.
You can run similar code with {$pointermath on}, which gives results
that are closer to C-style  [] indexing, but slightly different.
A PChar buffer can simply be indexed like in C, but using Pointer
syntax.
In theory the same speed. Do not confuse language syntax with feature
support, because you can NOT assume that to be the case. You should
investigate what the syntax is for the languages that you want to
compare the same thing.... That is a common mistake to make...

----------------------------------------------------------------------
marcov · 2020-06-01 15:53

Quote from: domibay_hugo on June 01, 2020, 04:31:29 pmAs far as I
understood they compare 2 bytes at once with the memchr Function.

So Perl doesn't use a naive own implemented function, but optimized
functions. Then you must do the same with FPC.

QuoteThe IndexByte() Function seems to me very promissing.

For two byte characters there is IndexWord.

----------------------------------------------------------------------
BeniBela · 2020-06-01 16:43

The real fast functions probably use AVX

----------------------------------------------------------------------
domibay_hugo · 2020-06-02 12:42

The Perl Interpreter is just simple C Language Code. There is no Black
Magic Assembly Code in there.
https://www.mkssoftware.com/docs/perl/pod/perlapi.asp#item_fbm_instr
This Documentation states they implemented the "Boyer-Moore algorithm"
https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string-
search_algorithm
QuoteThe key features of the algorithm are to match on the tail of the
pattern rather than the head, and to skip along the text in jumps of
multiple characters rather than searching every single character in
the text.

The Black Magic consists simply in not matching every single Character
of the Search against every single Character in the Source
like Pos() and all developed approaches do.

In deed when debugging I found that the PCharStartsWith() Function was
triggered tens of thousands of times on every Text Chunk.

----------------------------------------------------------------------
domibay_hugo · 2020-06-02 12:50

Quote from: marcov on June 01, 2020, 04:18:30 pmFPC also has some
basic primitives, so a possible improvement could be using primitives
in FPC too, the indexbyte() function that most FPC character search
routines in the RTL are based on.

I implemented 2 Function to test the Performance on the given Test
Data
Comparing 2 Bytes with the IndexByte() Function
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
PCharSubStringByString_v2(const ssource: PChar; isourcelength:
Cardinal  ; const ssearch: PChar; isearchlength: Cardinal): PChar;var 
pssrc, psend: PChar;  isrhps: Cardinal;  imtchps: Integer;  bsrhgo:
Boolean;begin  Result := Nil;   pssrc := ssource;  psend := pssrc +
isourcelength;  isrhps := 0;  //icmpln := isourcelength;   //Skip on
Empty Strings  bsrhgo := (pssrc < psend);   while bsrhgo    and (pssrc
< psend) do  begin    if pssrc[1] <> #0 then    begin      //More than
1 Byte left      imtchps := IndexByte(pssrc^, 2, Byte(ssearch^));    
 if imtchps <> -1 then      begin        inc(pssrc, imtchps);       
inc(isrhps, imtchps);         if PCharStartsWith(pssrc, isourcelength
- isrhps, ssearch, isearchlength) then        begin          Result :=
pssrc;          bsrhgo := False;        end;         inc(pssrc, 2 -
imtchps);        inc(isrhps, 2 - imtchps);      end      else  //Bytes
do not match      begin        inc(pssrc, 2);        inc(isrhps, 2); 
   end;  //if imtchps <> -1 then    end    else  //Last Byte    begin 
    if pssrc^ = ssearch^ then      begin        if
PCharStartsWith(pssrc, isourcelength - isrhps, ssearch, isearchlength)
then        begin          Result := pssrc;          bsrhgo := False; 
      end;      end;  //if pssrc^ = ssearch^ then       inc(pssrc);  
  inc(isrhps);    end;  //if pssrc[1] = #0 then  end;  //while bsrhgo
doend; 
And comparing 4 Bytes with the IndexByte() Function
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
PCharSubStringByString_v4(const ssource: PChar; isourcelength:
Cardinal  ; const ssearch: PChar; isearchlength: Cardinal): PChar;var 
pssrc, psend: PChar;  icmpln: Cardinal;  imtchps: Integer;  bsrhgo:
Boolean;begin  Result := Nil;   pssrc := ssource;  psend := pssrc +
isourcelength;  icmpln := 4;   //Skip on Empty Strings  bsrhgo :=
(pssrc < psend);   while bsrhgo    and (pssrc < psend) do  begin    if
icmpln > (psend - pssrc) then      icmpln := psend - pssrc;    
imtchps := IndexByte(pssrc^, icmpln, Byte(ssearch^));     if imtchps
<> -1 then    begin      inc(pssrc, imtchps);       if
PCharStartsWith(pssrc, psend - pssrc, ssearch, isearchlength) then   
 begin        Result := pssrc;        bsrhgo := False;      end;     
inc(pssrc, icmpln - imtchps);    end    else  //Bytes do not match   
begin      inc(pssrc, icmpln);    end;  //if imtchps <> -1 then  end;
 //while bsrhgo doend; 
The Performance dropped with the PCharSubStringByString_v2() Function
and for the PCharSubStringByString_v4() Function it was just the same
as for the Initial Version.

I suppose that the Assembly Code of IndexByte() does just the same as
the PCharSubStringByString() Function.

----------------------------------------------------------------------
domibay_hugo · 2020-06-02 13:00

Quote from: Thaddy on June 01, 2020, 04:32:24 pmStrPos() is a C
support function and indeed is much slower than Pascal native Pos()
family.
StrPos() also FAILS on true Pascal type strings. (Since length is
known and embedded #0's are supported in Pascal string types)
It is geared towards PChars. (Which is famously not the brightest
ideafrom curly bracket language designers....).
If you have the need for StrPos(), you are not using Pascal features,
just a rude version to make some Pascal interact with the above silly
design of C like "strings". Technically C as a language does not even
know strings. Do away with any of the curly brackets and you are fine
as far as "strings" are concerned.

Basically: Pascal does not need to check for buffer overruns.
Algorithmic, walking a Pascal string or a C style buffer should be the
same in O notation: O(n). But technically, true Pascal stringtypes -
should and do -  have an advantage.
And Strpos() is not really Pascal.....

I did also a try with replacing the PChar Search Functions with the
native Pos() Function in:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i         if
Pos(pschktm, pschkpkg) > 0 then        begin          pschktmstrt :=
pschkpkg;                           // [...] Do the Regex Match      
 end; 
As a result the Code slowed down another 3x than the original Version:

$ time ./runstringsearch
file: './small_firewall.log': read do ...
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '15'
BuildPacketList - finished with [ 0 ]!

real   0m0.031s
user   0m0.030s
sys   0m0.001s

----------------------------------------------------------------------
Thaddy · 2020-06-02 13:46

You also did not compile everything with all compiler settings:
E.g. fpc <yourprogram or library> -Cp<whatever core you want> etc...
Those noob script languages usually rely on optimizations from an
external library written in a real compiled language. So if you want
to compare that, you need to understand the optimization levels of
what the imported code really does...

----------------------------------------------------------------------
Thaddy · 2020-06-02 13:49

Quote from: julkas on June 01, 2020, 03:07:10 pmCompile your Pascal
program with -O2 or -O3 option.
Not only that. You have to choose your bottom line CPU and FPU
settings (and -O4)
You usually want the complete RTL compiled with these settings.
Depending on application the speed difference can be moderate or
frighteningly faster.

----------------------------------------------------------------------
domibay_hugo · 2020-06-02 14:14

Quote from: Thaddy on June 02, 2020, 03:46:01 pmYou also did not
compile everything with all compiler settings:
E.g. fpc <yourprogram or library> -Cp<whatever core you want> etc...
I activated compiler settings as suggested
No Compiler Optimization:

$ hyperfine --warmup 3 -r 100 ./runstringsearch
Benchmark #1: ./runstringsearch
  Time (mean ± σ):       9.6 ms ±   0.6 ms    [User: 9.1 ms, System:
0.6 ms]
  Range (min … max):     8.8 ms …  12.4 ms    100 runs

Compiler Optimization -CfAVX2 :

$ hyperfine --warmup 3 -r 100 ./runstringsearch
Benchmark #1: ./runstringsearch
  Time (mean ± σ):       9.5 ms ±   0.4 ms    [User: 8.6 ms, System:
0.8 ms]
  Range (min … max):     9.0 ms …  10.9 ms    100 runs

Compiler Optimizations -CfAVX2 -CpCOREAVX2:

$ hyperfine --warmup 3 -r 100 ./runstringsearch
Benchmark #1: ./runstringsearch
  Time (mean ± σ):       9.5 ms ±   0.4 ms    [User: 8.8 ms, System:
0.8 ms]
  Range (min … max):     9.0 ms …  11.1 ms    100 runs


I conclude that there is no measurable improvement.

So the solution is not on the Assembly Code Level.

----------------------------------------------------------------------
Thaddy · 2020-06-02 14:18

Quote from: domibay_hugo on June 02, 2020, 04:14:49 pmI conclude that
there is no measurable improvement.
Because you did not recompile the rtl with proper settings.

----------------------------------------------------------------------
domibay_hugo · 2020-06-02 14:23

Quote from: Thaddy on June 02, 2020, 03:49:41 pmQuote from: julkas on
June 01, 2020, 03:07:10 pmCompile your Pascal program with -O2 or -O3
option.
Not only that. You have to choose your bottom line CPU and FPU
settings (and -O4)
You usually want the complete RTL compiled with these settings.
Depending on application the speed difference can be moderate or
frighteningly faster.

I activated the -O4(use with care) Setting:

$ ./runstringsearch
file: './small_firewall.log': read do ...
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '15'
BuildPacketList - finished with [ 0 ]!

$ hyperfine --warmup 3 -r 100 ./runstringsearch
Benchmark #1: ./runstringsearch
  Time (mean ± σ):       9.5 ms ±   0.4 ms    [User: 8.8 ms, System:
0.8 ms]
  Range (min … max):     9.0 ms …  10.8 ms    100 runs


Again no measurable improvement.

I think recompiling the RTL with unsafe Settings is not a good idea
since it might break my development environment

----------------------------------------------------------------------
Thaddy · 2020-06-02 14:29

Quote from: Thaddy on June 02, 2020, 04:18:22 pmQuote from:
domibay_hugo on June 02, 2020, 04:14:49 pmI conclude that there is no
measurable improvement.
Because you did not recompile the rtl with proper settings.
You need to compile the rtl with optimal settings... Again the Hives:
https://www.youtube.com/watch?v=Uz1Jwyxd4tE

I know this is not easy for most....but it is easy...

----------------------------------------------------------------------
Martin_fr · 2020-06-02 14:37

If it was just to test the performance of "strpos" under -O4 => copy
the code, and compile the copy with O4

----------------------------------------------------------------------
domibay_hugo · 2020-06-02 14:40

I rather think that the solution lies in the Search Algorithm
Comparing the common Brute Force approach:
https://en.wikipedia.org/wiki/Brute-force_search
Against the documented Algorithm of the Perl engine:
https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string-
search_algorithm

As a side Remark:
Perl performs like this right out of the Box
without recompiling the whole Interpreter Engine

----------------------------------------------------------------------
domibay_hugo · 2020-06-02 14:48

Quote from: Martin_fr on June 02, 2020, 04:37:54 pmIf it was just to
test the performance of "strpos" under -O4 => copy the code, and
compile the copy with O4
to find a real difference you need to feed enough Test Data into the
Algorithm
like explained in the documentation
https://en.wikipedia.org/wiki/Brute-force_search
QuoteWhile a brute-force search is simple to implement, and will
always find a solution if it exists, its cost is proportional to the
number of candidate solutions – which in many practical problems tends
to grow very quickly as the size of the problem increases
(§Combinatorial explosion).on little data you won't find a difference.
but as the documentation explains the solution is not scalable

The best way to feed enough data into the Algorithm like in a Real-
World Production Tool is with a file.

But still any improvements on the code are welcomed.

----------------------------------------------------------------------
Martin_fr · 2020-06-02 14:52

Quote from: domibay_hugo on June 02, 2020, 04:40:26 pmPerl performs
like this right out of the Box
without recompiling the whole Interpreter Engine
Well we do not know if it is a question of O4 or not.
The suggestion was to test that, to see if it is.

----------------------------------------------------------------------
avk · 2020-06-02 15:08

I think it makes sense to compare implementations of the Boyer-Moore
algorithm(strutils/FindMatchesBoyerMoore***).

----------------------------------------------------------------------
marcov · 2020-06-02 15:09

Quote from: domibay_hugo on June 02, 2020, 02:50:20 pmThe Performance
dropped with the PCharSubStringByString_v2() Function
and for the PCharSubStringByString_v4() Function it was just the same
as for the Initial Version.

I suppose that the Assembly Code of IndexByte() does just the same as
the PCharSubStringByString() Function.

Indexbyte will only be worthwhile for longer strings. You use it to
find the first matching char and then do a comparebyte to compare the
found ptrs.

----------------------------------------------------------------------
domibay_hugo · 2020-06-04 14:52

I got some new benchmarks for the given Test Data:

$ grep " 11:48:" small_firewall.log|wc -l
32
$ grep " 11:48:" small_firewall.log|grep -ioE "^.* 11:48:[0-2][0-9]
.*[:'][^:']*(udp in|drop)[^:']*[:'].*\$"|wc -l
15
$ time grep " 11:48:" big_firewall.log |grep -ioE "^.*
11:48:[0-2][0-9] .*[:'][^:']*(udp in|drop)[^:']*[:'].*\$"|wc -l
0

real   0m0.100s
user   0m0.079s
sys   0m0.025s

the grep is the best performant tool for the search.

Also I implemented the same logic with the Rust Programming Language.
With the big file it achieves the same speed as Perl

$ time target/release/string_search
Expression '(?m)^.* 11:48:[0-2][0-9] .*[:'][^:']*(?i:udp
in|drop)[^:']*[:'].*$': Match Count: '0'

real   0m0.205s
user   0m0.142s
sys   0m0.063s


And also in PHP
It performs not so good on the small file but becomes better with the
bigger one

$ time ./udp_watch.php
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '15'

real   0m0.053s
user   0m0.035s
sys   0m0.019s
$ time ./udp_watch.php
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '0'

real   0m0.206s
user   0m0.160s
sys   0m0.046s

----------------------------------------------------------------------
domibay_hugo · 2020-06-04 15:03

Quote from: avk on June 02, 2020, 05:08:51 pmI think it makes sense to
compare implementations of the Boyer-Moore
algorithm(strutils/FindMatchesBoyerMoore***).

I searched around and found an Unit at:
https://forum.lazarus.freepascal.org/index.php/topic,23136.0.html

And the Procedure FindMatchesBoyerMooreCaseSensitive()
https://www.freepascal.org/docs-
html/rtl/strutils/findmatchesboyermoorecasesensitive.html

The Unit from the Post performs really badly.
The Procedure from strutils did not work when I just included the
strutils Unit.
but It started to work when I copied the sourcecode into a new file.

Now the FreePascal Application catches up with the other
Implementations:
On the small file:

$ time ./runstringsearch
file: './small_firewall.log': read do ...
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '15'
BuildPacketList - finished with [ 0 ]!

real   0m0.009s
user   0m0.007s
sys   0m0.002s


And on the big file:

$ time ./runstringsearch
file: './big_firewall.log': read do ...
Expression '^.* 11:48:[0-2][0-9] .*[:'][^:']*(udp
in|drop)[^:']*[:'].*$': Match Count '0'
BuildPacketList - finished with [ 0 ]!

real   0m0.229s
user   0m0.204s
sys   0m0.026s


I see a clear advantage at the sys time consumtion.

Still looking at the Source Code I noticed that the Character Mapping
Table is recreated on each search.
On repetitive search an Search Object could give some advantage in
processing.

----------------------------------------------------------------------
domibay_hugo · 2020-06-04 15:10

Quote from: domibay_hugo on June 02, 2020, 04:40:26 pmI rather think
that the solution lies in the Search Algorithm
Comparing the common Brute Force approach:
https://en.wikipedia.org/wiki/Brute-force_search
Against the documented Algorithm of the Perl engine:
https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string-
search_algorithm

I found this explanation on this issue which confirms my impression:
https://stackoverflow.com/questions/25965841/what-algorithm-is-used-
for-pos-function-in-pascal

----------------------------------------------------------------------
trev · 2020-06-05 11:16

https://www.freepascal.org/docs-
html/rtl/strutils/findmatchesboyermoorecasesensitive.html

The last two lines contain typos:

QuoteFindMatchesBoyerMooreCaseInSensitive Find case-sensitive matches
of a string using a Boyer-Moore algorithm
SringReplace

----------------------------------------------------------------------
marcov · 2020-06-05 11:29

Quote from: trev on June 05, 2020, 01:16:45
pmhttps://www.freepascal.org/docs-
html/rtl/strutils/findmatchesboyermoorecasesensitive.html

Thanks, the stringreplace was fixed, I fixed the other one, but
probably that won't make 3.2.0 docs anymore

----------------------------------------------------------------------
domibay_hugo · 2020-06-05 11:44

Quote from: marcov on June 02, 2020, 05:09:08 pmIndexbyte will only be
worthwhile for longer strings. You use it to find the first matching
char and then do a comparebyte to compare the found ptrs.

Thinking of this Documentation:
https://www.man7.org/linux/man-pages/man3/memchr.3.html

I was wondering if the IndexByte() Function could be useful to write a
faster strlen() Function just like in this Example:
QuoteThe following call is a fast means of locating a string's
terminating null byte:

           char *p = rawmemchr(s, '\0');

So I developed this little Function:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {$IFOPT
D+}  {$NOTE debug mode is active. disabling inline}  {$DEFINE
debug_on}{$ENDIF} function PCharLength(const ssource: PChar):
Cardinal; {$IFNDEF debug_on} inline; {$ENDIF}var  pssrc: PChar; 
imtchps: Integer;  isrhrng: Cardinal;begin  Result := 0;   pssrc :=
ssource;  isrhrng := 1024;   repeat  //until imtchps <> -1;    imtchps
:= IndexByte(pssrc^, isrhrng, 0);     if imtchps = -1 then    begin  
  inc(Result, isrhrng);      inc(pssrc, isrhrng);       isrhrng :=
isrhrng * 2;    end;  //if imtchps = -1 then  until imtchps <> -1;  
inc(Result, imtchps);end; 
I made a little benchmark to see if there is any measurable
improvement:

-------
Big Source 1 (length: '33044') - strlen (count: '1000') - Start - Now
in millisecs since midnight : 44566167
Big Source 1 - strlen - End - Now in millisecs since midnight :
44566191
Big Source 1: strlen completed in '23.9996938034892' ms.
Big Source 1 (length: '33044') - PCharLength (count: '1000') - Start -
Now in millisecs since midnight : 44566191
Big Source 1 - PCharLength - End - Now in millisecs since midnight :
44566212
Big Source 1: PCharLength completed in '21.0004393011332' ms.
-------
Big Source 1 (length: '66088') - strlen (count: '1000') - Start - Now
in millisecs since midnight : 44566212
Big Source 1 - strlen - End - Now in millisecs since midnight :
44566250
Big Source 1: strlen completed in '37.9995675757527' ms.
Big Source 1 (length: '66088') - PCharLength (count: '1000') - Start -
Now in millisecs since midnight : 44566250
Big Source 1 - PCharLength - End - Now in millisecs since midnight :
44566283
Big Source 1: PCharLength completed in '32.9999718815088' ms.


In strings > 32KB you can notice an advantage of 2 - 3 ms on 1000
Operations
and on strings > 64 KB you can observe an advantage of 3 - 5 ms on
1000 Operations

----------------------------------------------------------------------
domibay_hugo · 2020-06-19 11:13

Quote from: Thaddy on June 01, 2020, 04:32:24 pmStrPos() is a C
support function and indeed is much slower than Pascal native Pos()
family.
StrPos() also FAILS on true Pascal type strings. (Since length is
known and embedded #0's are supported in Pascal string types)
It is geared towards PChars. (Which is famously not the brightest
ideafrom curly bracket language designers....).
If you have the need for StrPos(), you are not using Pascal features,
just a rude version to make some Pascal interact with the above silly
design of C like "strings". Technically C as a language does not even
know strings. Do away with any of the curly brackets and you are fine
as far as "strings" are concerned.

Basically: Pascal does not need to check for buffer overruns.
Algorithmic, walking a Pascal string or a C style buffer should be the
same in O notation: O(n). But technically, true Pascal stringtypes -
should and do -  have an advantage.
And Strpos() is not really Pascal.....

Actually in Praxis PChar in FreePascal is the missing String Byte
Iterator to get access to the Data and avoid Copying. Widely found in
the FCL.

As shown in this little Snypplet:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TestPCharFromString;var  stream: TStringStream;  sdata: String; 
psdata: PAnsiString;  pcdata: PChar;  idatalength: Integer;beginstream
:= TStringStream.Create(''); stream.WriteString('My unchangeable
Source String. Is it really
unchangeable?'); stream.WriteByte(0); pcdata :=
@stream.DataString[1];idatalength := stream.Position -
1; writeln('stream 0 (length: ', chr(39), idatalength, chr(39), '): ',
chr(39), stream.DataString, chr(39)); pcdata^ :=
'b';inc(pcdata);pcdata^ := 'l';inc(pcdata);pcdata^ :=
'a';inc(pcdata);pcdata^ := ' '; writeln('stream 1 (length: ', chr(39),
idatalength, chr(39), '): ', chr(39), stream.DataString,
chr(39)); pcdata := PChar(stream.DataString); pcdata[0] :=
'f';pcdata[1] := 'o';pcdata[2] := 'o'; writeln('stream 2 (length: ',
chr(39), idatalength, chr(39), '): ', chr(39), stream.DataString,
chr(39)); sdata := stream.DataString; sdata[1] := 'I';sdata[2] := '
';sdata[3] := 'a';sdata[4] := 'm';sdata[5] := ' ';sdata[6] :=
'a';sdata[7] := ' ';sdata[8] := 'C';sdata[9] := 'o';sdata[10] :=
'p';sdata[11] := 'y';sdata[12] := ' '; writeln('stream 3 (length: ',
chr(39), idatalength, chr(39), '): ', chr(39), stream.DataString,
chr(39));writeln('data 3 (length: ', chr(39), Length(sdata), chr(39),
'): ', chr(39), sdata, chr(39)); psdata :=
@stream.DataString; psdata^[1] := 'I';psdata^[2] := ' ';psdata^[3] :=
'a';psdata^[4] := 'm';psdata^[5] := ' ';psdata^[6] := 'o';psdata^[7]
:= 'r';psdata^[8] := 'i';psdata^[9] := 'g';psdata^[10] :=
'i';psdata^[11] := 'n';psdata^[12] := 'a';psdata^[13] :=
'l';psdata^[14] := ' '; writeln('stream 4 (length: ', chr(39),
Length(psdata^), chr(39), '): ', chr(39), stream.DataString,
chr(39)); stream.Free;stream := Nil;end; 
Which produces:

stream 0 (length: '57'): 'My unchangeable Source String. Is it really
unchangeable?'
stream 1 (length: '57'): 'bla nchangeable Source String. Is it really
unchangeable?'
stream 2 (length: '57'): 'foo nchangeable Source String. Is it really
unchangeable?'
stream 3 (length: '57'): 'foo nchangeable Source String. Is it really
unchangeable?'
data 3 (length: '58'): 'I am a Copy ble Source String. Is it really
unchangeable?'
stream 4 (length: '58'): 'I am original e Source String. Is it really
unchangeable?'


The PChar psdata is a Pointer to the Bytes in stream.DataString.
While the String sdata is a Copy of the whole String
stream.DataString.

----------------------------------------------------------------------
PascalDragon · 2020-06-19 13:46

Quote from: domibay_hugo on June 19, 2020, 01:13:41 pmActually in
Praxis PChar in FreePascal is the missing String Byte Iterator to get
access to the Data and avoid Copying. Widely found in the FCL.

Quote from: domibay_hugo on June 19, 2020, 01:13:41 pmThe PChar psdata
is a Pointer to the Bytes in stream.DataString.
While the String sdata is a Copy of the whole String
stream.DataString.

Not quite. As long as you don't change sdata it will point to the same
string data as the internal field of TStringStream. But once you
change the string it will be made unique. This is the Copy-On-Write
mechanism of managed strings. To see this in action you should adjust
your example like this:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
//...writeln('stream 2 (length: ', chr(39), idatalength, chr(39), '):
', chr(39), stream.DataString, chr(39)); sdata :=
stream.DataString;Writeln(StringRefCount(sdata)); sdata[1] :=
'I';Writeln(StringRefCount(sdata));sdata[2] := ' ';sdata[3] :=
'a';//...
Notice the two calls to StringRefCount which will return the current
reference count of the string. The output will then be:

Code: [Select]PS E:\fpc\git> .\testoutput\tstrstrm.exe
stream 0 (length: '57'): 'My unchangeable Source String. Is it really
unchangeable? '
stream 1 (length: '57'): 'bla nchangeable Source String. Is it really
unchangeable? '
stream 2 (length: '57'): 'foo nchangeable Source String. Is it really
unchangeable? '
2
1
stream 3 (length: '57'): 'foo nchangeable Source String. Is it really
unchangeable? '
data 3 (length: '58'): 'I am a Copy ble Source String. Is it really
unchangeable? '
stream 4 (length: '58'): 'I am original e Source String. Is it really
unchangeable? '
As you can see the reference count is first 2, because both the stream
and sdata reference the string data, but upon modifying the string the
reference count becomes one, because the string has been made unique.
This can also be seen in the assembly code:

Code: [Select]# [45] sdata := stream.DataString;
        movl    -4(%ebp),%eax
        movl    4(%eax),%edx
        leal    -8(%ebp),%eax
        call    fpc_ansistr_assign
# [46] Writeln(StringRefCount(sdata));
        call    fpc_get_output
        movl    %eax,%ebx
        movl    -8(%ebp),%eax
        call    SYSTEM_$$_STRINGREFCOUNT$RAWBYTESTRING$$LONGINT
        movl    %eax,%ecx
        movl    %ebx,%edx
        movl    $0,%eax
        call    fpc_write_text_sint
        call    FPC_IOCHECK
        movl    %ebx,%eax
        call    fpc_writeln_end
        call    FPC_IOCHECK
# [48] sdata[1] := 'I';
        leal    -8(%ebp),%eax
        call    fpc_ansistr_unique
        movb    $73,(%eax)
Thus, as long as you don't change the string you don't need to access
it as a PChar.

----------------------------------------------------------------------
domibay_hugo · 2020-06-19 14:26

Quote from: PascalDragon on June 19, 2020, 03:46:25 pmAs long as you
don't change sdata it will point to the same string data as the
internal field of TStringStream. But once you change the string it
will be made unique. This is the Copy-On-Write mechanism of managed
strings.
Quote from: PascalDragon on June 19, 2020, 03:46:25 pmThus, as long as
you don't change the string you don't need to access it as a PChar.

This becomes ever more intriguing because I understand to convert a
PascalString into a valid PChar you need to add the NULL Byte.

So if I change my example to:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i stream :=
TStringStream.Create(''); stream.WriteString('My unchangeable Source
String. Is it really unchangeable?'); writeln('stream 0 (length: ',
chr(39), stream.Size, chr(39), '): ', chr(39), stream.DataString,
chr(39)); pcdata := PChar(stream.DataString);idatalength :=
stream.Position; pcdata[0] := 'f';pcdata[1] := 'o';pcdata[2] :=
'o';pcdata[3] := ' '; writeln('stream 1 (length: ', chr(39),
idatalength, chr(39), '): ', chr(39), stream.DataString,
chr(39)); stream.WriteByte(0); pcdata :=
@stream.DataString[1];idatalength := stream.Position -
1; writeln('stream 2 (length: ', chr(39), idatalength, chr(39), '): ',
chr(39), stream.DataString, chr(39)); pcdata^ :=
'b';inc(pcdata);pcdata^ := 'l';inc(pcdata);pcdata^ :=
'a';inc(pcdata);pcdata^ := ' '; writeln('stream 3 (length: ', chr(39),
idatalength, chr(39), '): ', chr(39), stream.DataString, chr(39));  
I get an Output like:

stream 0 (length: '57'): 'My unchangeable Source String. Is it really
unchangeable?'
stream 1 (length: '57'): 'foo nchangeable Source String. Is it really
unchangeable?'
stream 2 (length: '57'): 'foo nchangeable Source String. Is it really
unchangeable?'
stream 3 (length: '57'): 'bla nchangeable Source String. Is it really
unchangeable?'


At the Operation:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i pcdata :=
PChar(stream.DataString); 
Where is the NULL Byte added since the PChar pcdata always points to
the internal data of TStringStream.DataString and at this point does
not have any ?

----------------------------------------------------------------------
PascalDragon · 2020-06-20 16:08

Quote from: domibay_hugo on June 19, 2020, 04:26:56 pmWhere is the
NULL Byte added since the PChar pcdata always points to the internal
data of TStringStream.DataString and at this point does not have any ?

Pascal strings always end with a NUL to simplify conversion to PChar.
Though of course that NUL won't have any effect if a previous NUL is
encountered.

Just take a look at the generated assembly for the string.

Take this code for example:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
tstrtest; {$mode objfpc}{$H+} var  s: AnsiString;begin  s :=
'Test1234';end.
The assignment will be code like this:

Code: ASM  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i # [8] s :=
'Test1234';        leaq    _$TSTRTEST$_Ld1(%rip),%rax        movq   
%rax,%rdx        leaq    U_$P$TSTRTEST_$$_S(%rip),%rcx        call   
fpc_ansistr_assign
So we'll look at the _$TSTRTEST$_Ld1 symbol:

Code: ASM  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i .section
.rodata.n__$TSTRTEST$_Ld1,"d"        .balign 8        .short  0,1    
  .long   0        .quad   -1,8.globl 
_$TSTRTEST$_Ld1_$TSTRTEST$_Ld1:        .ascii  "Test1234\000"
And there is the NUL at the end.

Please note that this applies to AnsiString, WideString and
UnicodeString, but not to ShortString.