How to form Regex expression
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
ronhud · 2022-12-30 19:57

I am just starting to look at using TRegExpr.   I want to extract a
string of characters from a web page.   I have written an  expression
to find 'Change' and capture all the characters until it finds </li. 

RE.Expression := ('Change(.*?)</li'); 

Is this correct?    Using the program RegExpr.pas.   How do I access
the result?

----------------------------------------------------------------------
Roland57 · 2022-12-30 20:09

Hello!

I suggest to take a look at my RegExpr examples.

----------------------------------------------------------------------
AlexTP · 2022-12-30 21:23

Also it's good to use this site to test regex: https://regex101.com/

----------------------------------------------------------------------
Martin_fr · 2022-12-30 22:02

Quote from: ronhud on December 30, 2022, 08:57:22 pmRE.Expression :=
('Change(.*?)</li'); 

Is this correct?

Depends... Yes it is.

But it will also match
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Change item
in list <ul><li>item 1 </li>
Do you want that to be found?

Also you might want to ensure that the regex "</li" does not match
"</literal>"
E.g. :  "</li\b"

----------------------------------------------------------------------
PierceNg · 2022-12-31 02:40

Quote from: ronhud on December 30, 2022, 08:57:22 pmI am just starting
to look at using TRegExpr.   I want to extract a string of characters
from a web page.   I have written an  expression to find 'Change' and
capture all the characters until it finds </li. 

RE.Expression := ('Change(.*?)</li'); 

If you want the content but not that tags (aka meta content), first
parse the HTML, then apply regex on the content parts to pick out what
you want.

The expedient way is to build your program in pas2js, run the program
directly on the web page, so you can use the familiar GetElementById
and friends.

Otherwise, for HTML parsing, I found this:
https://github.com/isemenkov/libpasmyhtml. It's archived but it's a
wrapper for a C library so should continue to work.

And for fun reading, http://regex.info/blog/2006-09-15/247, which
talks about below famous saying:

QuoteSome people, when confronted with a problem, think
“I know, I'll use regular expressions.”   Now they have two problems.

----------------------------------------------------------------------
Jurassic Pork · 2022-12-31 10:16

Hello,
if your html file is an xhtml file you can also try to use the Xpath
unit of the fcl-xml :
Example :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
xPathTestChange;// J.P December 2022{$mode objfpc}{$H+} uses classes,
DOM, DOM_HTML, SAX_HTML, XPath, StrUtils;var  htmlDoc: THTMLDocument; 
XPathRes: TXPathVariable;  XPathExp: DomString;  TheNodeSet :
TNodeSet;  Res : UnicodeString; begin  try   // read input html file 
 ReadHTMLFile(htmlDoc, 'd:\temp\ChangeTest.html');   // Search for li
which contains "Change" string   XPathExp :=
'//li[contains(text(),"Change")]';   XPathRes :=
EvaluateXPathExpression(XPathExp, htmlDoc.DocumentElement); 
 TheNodeSet := XPathRes.AsNodeSet;   Res :=
TDomNode(TheNodeSet[0]).TextContent; 
 Writeln(MidStr(Res,8,Length(Res)));   XPathRes.Free;   finally   
 htmlDoc.Free;  end;  Readln; end.     
for this content as html source :
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i <!DOCTYPE
html><html><body> <h1>The ol and ul elements</h1> <p>The ol element
defines an ordered list:</p><ol>  <li>Coffee</li>  <li>Tea</li> 
<li>Milk</li></ol> <p>The ul element defines an unordered
list:</p><ul>  <li>Change Coffee</li>  <li>Tea</li> 
<li>Milk</li></ul> </body></html>
Result :
QuoteCoffee
Friendly, J.P

----------------------------------------------------------------------
ronhud · 2022-12-31 10:44

Thank you for all the replies.   I am beginning to get the feel of
regular expressions and the feeling that I need to look into xhtml.

----------------------------------------------------------------------
Roland57 · 2022-12-31 23:30

Quote from: ronhud on December 30, 2022, 08:57:22 pmHow do I access
the result?
In case you still need it, here is an example. The sample is the html
document provided by Jurassic Pork.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i {$MODE
OBJFPC}{$H+} uses  RegExpr;  const  SAMPLE =    '<!DOCTYPE html>'#10
+    '<html>'#10 +    '<body>'#10 +    ''#10 +    '<h1>The ol and ul
elements</h1>'#10 +    ''#10 +    '<p>The ol element defines an
ordered list:</p>'#10 +    '<ol>'#10 +    '  <li>Coffee</li>'#10 +   
'  <li>Tea</li>'#10 +    '  <li>Milk</li>'#10 +    '</ol>'#10 +   
''#10 +    '<p>The ul element defines an unordered list:</p>'#10 +   
'<ul>'#10 +    '  <li>Change Coffee</li>'#10 +    '  <li>Tea</li>'#10
+    '  <li>Milk</li>'#10 +    '</ul>'#10 +    ''#10 +    '</body>'#10
+    '</html>'#10; var  LExpr: TRegExpr;  begin  WriteLn('==== Demo
1');    LExpr := TRegExpr.Create('<li>Change (\w+)</li>');  if
LExpr.Exec(SAMPLE) then  begin    WriteLn(LExpr.Match[0]); // The
whole string    WriteLn(LExpr.Match[1]); // Only the captured group 
end;  LExpr.Free;  { If we look for several strings matching the
expression, we use ExecNext. }   WriteLn('==== Demo 2');    LExpr :=
TRegExpr.Create('<li>([\w\s]+)</li>');  if LExpr.Exec(SAMPLE) then 
repeat    WriteLn(LExpr.Match[0]);    WriteLn(LExpr.Match[1]);  until
not LExpr.ExecNext;  LExpr.Free;end.
Regards.

Roland

P.-S. Code edited for better readability.

----------------------------------------------------------------------
MarkMLl · 2023-01-01 11:12

In any event I think it's worth reminding OP of the much-cited
https://stackoverflow.com/questions/1732348/regex-match-open-tags-
except-xhtml-self-contained-tags which is a useful read even if one
disagrees the the implied definition of "regular language" hence
"regular expression".

The issue is that given input that looks like this

Code: [Select]function foo;

begin
end foo;

(I know that's not HTML etc., I'm just using it to illustrate the
point) a strict regex can't check that the final identifier matches
the initial one. That's why Perl etc. introduced back-references.

MarkMLl

----------------------------------------------------------------------
Thaddy · 2023-01-01 14:13

Quote from: MarkMLl on January 01, 2023, 12:12:35 pm
(I know that's not HTML etc., I'm just using it to illustrate the
point) a strict regex can't check that the final identifier matches
the initial one. That's why Perl etc. introduced back-references.

MarkMLl
Well, "Perl etc" includes TRegExpr. So etc includes FreePascal. And
does back-propagation, which I believe is the correct term.

----------------------------------------------------------------------
MarkMLl · 2023-01-01 14:26

Quote from: Thaddy on January 01, 2023, 03:13:35 pmWell, "Perl etc"
includes TRegExpr. So etc includes FreePascal. And does back-
propagation, which I believe is the correct term.

Yes, but CS purists will say that those implementations are broken
since they don't comply with the description of "regular".

I'm not criticising Perl etc. What I'm trying to do is say where the
"you can't parse HTML with regexes" doctrine comes from.

MarkMLl

----------------------------------------------------------------------
Thaddy · 2023-01-01 14:52

In that case you are absolutely right.

----------------------------------------------------------------------
matjaz · 2023-01-04 12:49

You can try this:

RE.Expression := ('Change(.*)<\/li');

/ is meta character and you must use escape char \ to capture it

----------------------------------------------------------------------
Warfley · 2023-01-04 13:17

Quote from: MarkMLl on January 01, 2023, 03:26:33 pmYes, but CS
purists will say that those implementations are broken since they
don't comply with the description of "regular".
It's actually a more practical point than just being a purist. If the
expressions describe a non regular language, you can't use a DFA for
matching. DFA matching is theoretically optimal and can be done in
O(n), meaning every character of the matched string must only be
visited once, it's the fastest algorithm for matching text that you
can have (but on the flip side the memory consumption scales badly
with the size of the regex due to the powerset construction). Non-
regularity and the requirement for back tracking (not back
propagation, thats an ML term) increases the complexity to O(n^2),
meaning in the worst case it goes through every character and needs to
walk over the rest of the string from that point only to find out at
the end that there is no match and continue with the next character.

This is still good enough for inputs that are not too large, and if
you are getting compter sciency enough you start calling anything
"efficient" that isn't exponential runtime, but when matching large
text files this can make a huge difference.

Back to topic, when I have to build regular expressions, I ususally
use online tools like https://regex101.com/ as the give direct
feedback and show what matches and how it matches

----------------------------------------------------------------------
MarkMLl · 2023-01-04 13:51

Quote from: matjaz on January 04, 2023, 01:49:58 pm/ is meta character
and you must use escape char \ to capture it

That's certainly true in the case of Perl etc. where / is the default
separator for the m and s commands, but I'm not sure it's true in the
case of regex libraries in general.

Quote from: Warfley on January 04, 2023, 02:17:12 pmIt's actually a
more practical point than just being a purist.

It's certainly a practical detail that's very important to the
implementor, but not necessarily to the user.

The really important thing is that regexes extended with back-
references (the Perl term, and I believe that Perl was largely
responsible for popularising them) /can/ be useful for processing e.g.
HTML, particularly if the generator is known to have predictable
characteristics.

But doing so is not necessarily good practice, from the POV of both
performance and determinacy: far better to use a proper parser.

Which I suppose takes me back to one of my betes noires: programming
languages and data description notations which have been promulgated
despite being known to be difficult to parse. Which has to leave me
arguing that programming syntax /should/ be regular: in the strictest
"computer-sciency" sense of the word.

Actually, there's a corollary to that fairly near to home. Wirth
designed Modula-2 such that a function definition had to be closed
with the (case-sensitive) function name, making the syntax non-
regular:

Code: [Select]function foo;

begin
end foo;

ALGOL-60 on the other hand defined that everything after  end  to the
end of the source record was discarded:

Code: [Select]function foo;

begin
end Any old crap here

Which suggests that even in the late '50s there were people who
recognised the undesirability of non-regular syntaxes.

MarkMLl