Searching HTML Code lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- katronix · 2015-11-07 00:48 Is there a unit that is specifically designed for reading HTML code? specifically looking for a way to find mailto tags in it. Currently my thought to put the html code into a memo field and then search for a mailto in it. Any better way? Thanks ---------------------------------------------------------------------- christian1987 · 2015-11-07 06:04 For your example, is would use SysUtils.Pos and Copy() and LowerCase Um, then loop the resulting string, for terminators such as: > greater than " quote ------------------- Otherwise, I saw a DOM unit, and DOM means HTML also.... I don't know ---------------------------------------------------------------------- molly · 2015-11-07 06:52 Quote from: katronix on November 07, 2015, 01:48:23 amAny better way? You could try solving it with a regular expression parser, see also here. Might be that this regular expression works for your case. ---------------------------------------------------------------------- katronix · 2015-11-07 07:05 Quote from: molly on November 07, 2015, 07:52:27 am You could try solving it with a regular expression parser, see also here. Might be that this regular expression works for your case. Hey Molly, Thanks for this, will look into it. Any idea how I would scan a TMemo.Lines for it though? Kat ---------------------------------------------------------------------- christian1987 · 2015-11-07 07:13 S:=Memo1.Lines.Text; ---------------------------------------------------------------------- molly · 2015-11-07 07:23 Quote from: katronix on November 07, 2015, 08:05:35 amAny idea how I would scan a TMemo.Lines for it though? If you take the help file and travel from Tmemo property lines you'll notice that is of type TStrings. That has a property named text which allows you to retrieve the memo.lines as one big chunk of text. It might perhaps be a bit redundant in case you are working with memo's. You could also opt for a non visual way of creating a TStringList, loading your html file in there ans apply more or less the same. path of properties  It depends a little on what you want to achieve. If your only goal is extracting the mailto information, then i would opt for a non-visual approach. ---------------------------------------------------------------------- Jurassic Pork · 2015-11-07 08:59 hello, here is an example to extract mailto tags from html string with Tregexpr : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i // uses regexpr;const teststring = 'Send your answer to Jon Doe.
' +  #13#10 +'Visit us at:
'   +  #13#10 +'Example.com
'   +  #13#10 +'Box 564, Disneyland
'   +  #13#10 +'USA
'   +  #13#10 +'Send your question to Bob Doe.
'; var  regex : TRegexpr; beginwriteln('=========  Tregexpr ==========');regex := Tregexpr.Create();regex.Expression := 'mailto:([^\"]*)';regex.Exec(testString);Writeln('mail : ' + regex.Match[1]);While regex.ExecNext dobegin Writeln('mail : ' + regex.Match[1]);end;end; Result : Quote=========  Tregexpr ========== mail : webmaster@example.com mail : webguru@acme.com Friendly, J.P ---------------------------------------------------------------------- frakno · 2015-11-07 09:01 I am working with  Internet Tools   form   http://www.benibela.de/sources_en.html#internettools  I dont remember the perfect way to install, but i think: Install Package (there is no visible component in Editor Toolbar), so dont forget to insert Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses simpleinternet.... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var WebContent, WebFilterTitel, str : String; ...WebFilterTitel :=  '{titel:=text()}';   //  {titel:=text()} the important thingtry   WebContent := retrieve('http://forum.lazarus.freepascal.org/');  if WebContent <> '' then begin       process( WebContent, WebFilterTitel);      str := processedVariables.get('titel').toString;  The parser is very powerful and it can be searched more criteria at once and are read with processedVariables ---------------------------------------------------------------------- molly · 2015-11-07 09:37 thank you Jurassic Pork. I had my own test-example ready, searching a file by reading into a stringlist. I've checked with this example html, and the regular expression method unfortunately does create some false positives. Code: [Select]>FindMailto "HTML Examples - MailTo.html" mailaddress 1 = test@somewhere mailaddress 2 = test@somewhere mailaddress 3 = test@somewhere, friend@somewhere-else mailaddress 4 = test@somewhere, friend@somewhere-else mailaddress 5 = Index.html<test@somewhere> mailaddress 6 = Index.html&lt;test@somewhere&gt; mailaddress 7 = test@somewhere?subject=Test subject mailaddress 8 = test@somewhere?subject=Test subject mailaddress 9 = test@somewhere?body=Test body mailaddress 10 = test@somewhere?body=Test body mailaddress 11 = MailTo.html<clemenzi@cpcug.org> mailaddress 12 = clemenzi@cpcug.org?subject=HTML Examples - MailTo.htm Not sure how to approach. frakno's solution might proof more beneficial as i suspect it processes the html in such way that it creates 'cleaner' results. ---------------------------------------------------------------------- katronix · 2015-11-07 18:21 I suppose that \b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b would work for the regex, its at least what is recommended here: http://www.regular-expressions.info/email.html ---------------------------------------------------------------------- molly · 2015-11-07 19:04 Oh dear, regular expression and me...  When literally copied into Jurrassic pork's example-code it results in no matches at all when processing that html i linked to. But, not all RE implementations offer the same functionality/configuration. For details on TRegExpr, see here. There is even an special tool that let's you check expressions (RegExpr studio). The link i posted earlier to stackoverflow uses this solution: 'mailto:([^\?"]*)', which seems to work, but... The html file uses escaped html characters, and uptil now i've been unable to find a standard unit that fully decodes those into normal characters (the html uses combine escapes that gets missed by those implementations i've tested so-far) The best i can come up with atm is: Code: [Select]FindMailto.exe "HTML Examples - MailTo.html" mailaddress 1 = test@somewhere mailaddress 2 = test@somewhere mailaddress 3 = test@somewhere, friend@somewhere-else mailaddress 4 = test@somewhere, friend@somewhere-else mailaddress 5 = Index.html mailaddress 6 = Index.html<test@somewhere> mailaddress 7 = test@somewhere mailaddress 8 = test@somewhere mailaddress 9 = test@somewhere mailaddress 10 = test@somewhere mailaddress 11 = MailTo.html mailaddress 12 = clemenzi@cpcug.org I think you can see for yourself where it goes wrong. Would things end up consistent then we could check and extract the part we need. But... all this is a hypothetical situation only aimed at that particular html page. have you tried Jurrassic Pork's implementation to see if that works for you ? Or do you perhaps need help with loading the html from file into a stringlist and then use the regular expression on it's text ? ---------------------------------------------------------------------- Jurassic Pork · 2015-11-08 02:50 hello, i have found a pattern for tregexpr which works with your html example molly : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i MyList := T StringList.Create();MyList.LoadFromFile('MailTo.htm');writeln('======= ==  Tregexpr ==========');regex := Tregexpr.Create();regex.Expression := 'mailto:([MI]+[^;]*(;lt;|;))?([^\"\?\&]*)';regex.Exec(MyList.Text); Writeln('mail address : ' + regex.Match[3]);While regex.ExecNext dobegin Writeln('mail address : ' + regex.Match[3]);end;Result : Quote=========  Tregexpr ========== mail address : test@somewhere mail address : test@somewhere mail address : test@somewhere, friend@somewhere-else mail address : test@somewhere, friend@somewhere-else mail address : test@somewhere mail address : test@somewhere mail address : test@somewhere mail address : test@somewhere mail address : test@somewhere mail address : test@somewhere mail address : clemenzi@cpcug.org mail address : clemenzi@cpcug.orgit seems that Tregexpr has no lookahead or lookbehind , then it is harder to find an easy pattern. ---------------------------------------------------------------------- molly · 2015-11-08 03:56 Magnifique Jurassic Pork  I was still struggling, and indeed got a little (or tbh actually a lot) stuck with the restrictions. I didn't yet fully understood your used expression, but got the gist of it. Thank you for getting myself more familiar with how TRegExpr expressions works. ---------------------------------------------------------------------- katronix · 2015-11-09 17:07 Sorry for such a newb question, but how do I add the component regex to Lazarus? I've found http://wiki.freepascal.org/Regexpr, however not sure how to get Lazarus to see packages/regexpr/src/regexpr.pas Thanks all! ---------------------------------------------------------------------- Jurassic Pork · 2015-11-09 17:29 hello, regexpr is a unit from free pascal , just add : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses regexpr;in your project. ---------------------------------------------------------------------- katronix · 2015-11-10 01:34 Quote from: Jurassic Pork on November 09, 2015, 06:29:17 pmhello, regexpr is a unit from free pascal , just add : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses regexpr;in your project. Thanks Jurassic Pork! ---------------------------------------------------------------------- katronix · 2015-11-10 01:44 Quote from: katronix on November 07, 2015, 07:21:23 pmI suppose that \b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b would work for the regex, its at least what is recommended here: http://www.regular-expressions.info/email.html Can someone explain to me why this doesn't work when used as: regex.Expression := '\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b'; Would be interested to know. Thanks, Kat ---------------------------------------------------------------------- Jurassic Pork · 2015-11-10 01:51 how you use it ? in this pattern there is no submatch ->  (...) ---------------------------------------------------------------------- katronix · 2015-11-10 01:52 Quote from: Jurassic Pork on November 07, 2015, 09:59:33 amhello, here is an example to extract mailto tags from html string with Tregexpr : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i // uses regexpr;const teststring = 'Send your answer to Jon Doe.
' +  #13#10 +'Visit us at:
'   +  #13#10 +'Example.com
'   +  #13#10 +'Box 564, Disneyland
'   +  #13#10 +'USA
'   +  #13#10 +'Send your question to Bob Doe.
'; var  regex : TRegexpr; beginwriteln('=========  Tregexpr ==========');regex := Tregexpr.Create();regex.Expression := 'mailto:([^\"]*)';regex.Exec(testString);Writeln('mail : ' + regex.Match[1]);While regex.ExecNext dobegin Writeln('mail : ' + regex.Match[1]);end;end; Result : Quote=========  Tregexpr ========== mail : webmaster@example.com mail : webguru@acme.com Friendly, J.P Only issue with this one, is it also finds things such as:  ?subject=McNaughton+Brothers+Construction+Supports+Local+La+Grange+Co mmunity&body=http%3A%2F%2Fwww.onlineprnews.com%2Fnews%2F639201- 1446740858-mcnaughton-brothers-construction-supports-local-la-grange- community.html any suggestions on how not to find these? Kat ---------------------------------------------------------------------- katronix · 2015-11-10 01:54 Quote from: Jurassic Pork on November 10, 2015, 02:51:09 amhow you use it ? in this pattern there is no submatch ->  (...) Sorry not understanding, what do you mean by submatch? This regex code I found here: http://www.regular- expressions.info/email.html ---------------------------------------------------------------------- Jurassic Pork · 2015-11-10 02:17 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i // uses regexpr;const teststring = 'Send your answer to Jon Doe.
' +  #13#10 +'Visit us at:
'   +  #13#10 +'Example.com
'   +  #13#10 +'Box 564, Disneyland
'   +  #13#10 +'USA
'   +  #13#10 +'Send your question to Bob Doe.
';var  regex : TRegexpr;beginwriteln('=========  Tregexpr ==========');regex := Tregexpr.Create();writeln ('==== Method 1 no SubExpr =====');// (?i)  case insensitiveregex.Expression := '(?i)\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b';if (regex.Exec(testString)) then  begin    Writeln('mail address : ' + regex.Match[0]);    While regex.ExecNext do    begin     Writeln('mail address : ' + regex.Match[0]);    end;  end;writeln ('====  Method 2 : 1 SubExpr =====');regex.Expression := 'mailto:([^\"\?\&]*)';if (regex.Exec(testString)) then   begin    Writeln('mail address : ' + regex.Match[1]);   While regex.ExecNext do      begin        Writeln('mail address : ' + regex.Match[1]);      end;   end;writeln ('==== Method 3 :  3 SubExpr =====');regex.Expression := 'mailto:([MI]+[^;]*(;lt;|;))?([^\"\?\&]*)';if (regex.Exec(testString)) and (regex.SubExprMatchCount > 0) then  begin   Writeln('mail address : ' + regex.Match[3]);   While regex.ExecNext do   begin     Writeln('mail address : ' + regex.Match[3]);   end;  end;                         end; Results : Quote=========  Tregexpr ========== ==== Method 1 no SubExpr ===== mail address : webmaster@example.com mail address : webguru@acme.com ====  Method 2 : 1 SubExpr ===== mail address : webmaster@example.com mail address : webguru@acme.com ==== Method 3 :  3 SubExpr ===== mail address : webmaster@example.com mail address : webguru@acme.com ---------------------------------------------------------------------- katronix · 2015-11-10 02:42 Quote from: Jurassic Pork on November 10, 2015, 03:17:34 amCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i // uses regexpr;const teststring = 'Send your answer to Jon Doe.
' +  #13#10 +'Visit us at:
'   +  #13#10 +'Example.com
'   +  #13#10 +'Box 564, Disneyland
'   +  #13#10 +'USA
'   +  #13#10 +'Send your question to Bob Doe.
';var  regex : TRegexpr;beginwriteln('=========  Tregexpr ==========');regex := Tregexpr.Create();writeln ('==== Method 1 no SubExpr =====');// (?i)  case insensitiveregex.Expression := '(?i)\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b';if (regex.Exec(testString)) then  begin    Writeln('mail address : ' + regex.Match[0]);    While regex.ExecNext do    begin     Writeln('mail address : ' + regex.Match[0]);    end;  end;writeln ('====  Method 2 : 1 SubExpr =====');regex.Expression := 'mailto:([^\"]*)';if (regex.Exec(testString)) then   begin    Writeln('mail address : ' + regex.Match[1]);   While regex.ExecNext do      begin        Writeln('mail address : ' + regex.Match[1]);      end;   end;writeln ('==== Method 3 :  3 SubExpr =====');regex.Expression := 'mailto:([MI]+[^;]*(;lt;|;))?([^\"\?\&]*)';if (regex.Exec(testString)) and (regex.SubExprMatchCount > 0) then  begin   Writeln('mail address : ' + regex.Match[3]);   While regex.ExecNext do   begin     Writeln('mail address : ' + regex.Match[3]);   end;  end;                         end; I suppose there would be a way to make it so if Method 1 finds an email address its not also reported by method 2 and 3? and likewise if method 2 finds it, its not reported by method 3? The purpose for the app is to find all of the emails, without finding the same one from the same html multiple times. Thanks again for all the help, Kat ---------------------------------------------------------------------- molly · 2015-11-10 03:20 @katronix: Ah, but that's not the goal that you started out with in your first post. Other goals, other measures  Use the regular expression to find all the Mailto's. When a single mailto is found you search a StringList if the address is already inside the list, if not add it to the list. If it is already inside the list then you do not add it again. What Jurassic Pork showed with his code is that he can use 3 different regular expressions that can match Mailto addresses. You need to pick the one that works best for you. Indeed, you could use all three of them, but you already know that one of those regular expressions is not working out as expected for you. I guess in theory you could attempt to find a single RE that matches all mailto addresses in a html text, but that would push some limits, most probably by TRegExpr that can only return a max amount of matches in a single search. And afaik you would still need to compare the found matches manually. ---------------------------------------------------------------------- katronix · 2015-11-10 03:52 @molly Sorry for not giving the whole outline at first. I've seen some of the code use writelin (regex.Match[1]) where as others have used Match[0]. How do you know which to use? Or does it matter (I can't imagine it not mattering) Sorry for all the Qs, the documentation for this unit doesn't seem to answer many qs. Kat ---------------------------------------------------------------------- Jurassic Pork · 2015-11-10 05:20 katronix have a look at TRegExpr_Interface  help   here : Quoteproperty SubExprMatchCount : integer; // ReadOnly Number of subexpressions has been found in last Exec* call. If there are no subexpr. but whole expr was found (Exec* returned True), then SubExprMatchCount=0, if no subexpressions nor whole r.e. found (Exec* returned false) then SubExprMatchCount=-1. Note, that some subexpr. may be not found and for such subexpr. MathPos=MatchLen=-1 and Match=''. For example: Expression := '(1)?2(3)?'; Exec ('123'): SubExprMatchCount=2, Match[0]='123', [1]='1', [2]='3' Exec ('12'): SubExprMatchCount=1, Match[0]='12', [1]='1' Exec ('23'): SubExprMatchCount=2, Match[0]='23', [1]='', [2]='3' Exec ('2'): SubExprMatchCount=0, Match[0]='2' Exec ('7') - return False: SubExprMatchCount=-1 property MatchPos [Idx : integer] : integer; // ReadOnly pos of entrance subexpr. #Idx into tested in last Exec* string. First subexpr. have Idx=1, last - MatchCount, whole r.e. have Idx=0. Returns -1 if in r.e. no such subexpr. or this subexpr. not found in input string. property MatchLen [Idx : integer] : integer; // ReadOnly len of entrance subexpr. #Idx r.e. into tested in last Exec* string. First subexpr. have Idx=1, last - MatchCount, whole r.e. have Idx=0. Returns -1 if in r.e. no such subexpr. or this subexpr. not found in input string. property Match [Idx : integer] : string; // ReadOnly == copy (InputString, MatchPos [Idx], MatchLen [Idx]) Returns '' if in r.e. no such subexpr. or this subexpr. not found in input string. Regular expression isn't easy for a newbie