HTML to text lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- pcurtis · 2022-05-20 06:19 Does anyone know how or have some snippet on how to remove all HTML tags from a file, but leave the text? Thanks. ---------------------------------------------------------------------- speter · 2022-05-20 06:50 Included below is a quick and dirty version of what you want... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure clean(fn : string);const  magic = '!#%&';var  f : textfile;  s,t : string;  a,b : integer;begin  assignfile(f,fn);  reset(f);  s := '';  repeat    readln(f,t);    s += t+' '+magic;  until eof(f);  closefile(f);   repeat    a := pos('<',s);    b := pos('>',s,a);    if (a > 0) and (b > 0) then      delete(s,a,b-a+1);  until (a=0);   repeat    a := pos(magic,s);    if a > 0 then      begin        memo1.append(copy(s,1,a-1));        delete(s,1,a+3);      end;  until (a=0);end; Note that this code preserves line-endings and things like tab characters in the original html file. If you don't care about the line-endings you can leave that out by changing line #14 to Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i s += t+' ';and the last loop to Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i memo1.append(s);or similar. ---------------------------------------------------------------------- SymbolicFrank · 2022-05-20 07:11 A good "magic" character sequence to use is #0. It's legal and won't be in the string. ---------------------------------------------------------------------- wp · 2022-05-20 15:27 There's a ready-made function for this task in unit HTML2TextRender: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   function RenderHTML2Text(const AHTML: String): String; ---------------------------------------------------------------------- pcurtis · 2022-05-20 18:58 Thanks. How to use it? ---------------------------------------------------------------------- wp · 2022-05-20 19:44 I must apologize: this function is not yet included in Laz v2.2.2 or earlier. But the unit is self-contained and can be used also in older versions. How to apply this function? Simply load the html file into a string and pass that to the RenderHTML2Text function. The returned string contains the text only. See attached demo. Alternatively you can use the fasthtmlparser unit which comes with fpc. See my post at https://forum.lazarus.freepascal.org/index.php/top ic,43090.msg301176.html#msg301176. ---------------------------------------------------------------------- Jurassic Pork · 2022-05-21 11:20 hello, p.curtis have you an example of html file that you want to convert  ? friendly, J.P ---------------------------------------------------------------------- pcurtis · 2022-05-22 15:04 Thanks. Try this. In browser in looks like (picture), of course text is without formatting. ---------------------------------------------------------------------- Jurassic Pork · 2022-05-22 15:59 hello, try the last wp's project. Seems to be OK with your html file. Friendly, J.P ---------------------------------------------------------------------- pcurtis · 2022-05-22 16:12 More or less yes, but the are issues. Each paragraph starts with a space and there are too many blank lines. ---------------------------------------------------------------------- wp · 2022-05-22 21:00 I wonder which application writes such an ugly html file. Looks like MS Office from the identifiers... Anyway, the empty lines are caused by the file content itself. Found pieces like this at several places: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i

 

These are paragraphs containing a space, i.e. they look like empty lines. The spaces added before each paragraph are caused by the RenderHTML2Text function since it replaces linefeeds and returns by space characters which IMHO is not always correct. If you don't like this, file a bug report, maybe Juha (who wrote this function) can have a look. ---------------------------------------------------------------------- maurobio · 2023-06-23 11:20 Dear ALL, Just for completion, here are two functions that you may also find useful: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function StripHTML(S: string): string;var  TagBegin, TagEnd, TagLength: integer;begin  TagBegin := Pos( '<', S);      // search position of first <    while (TagBegin > 0) do begin  // while there is a < in S  TagEnd := Pos('>', S);              // find the matching >     TagLength := TagEnd - TagBegin + 1;    Delete(S, TagBegin, TagLength);     // delete the tag     TagBegin:= Pos( '<', S);            // search for next <  end;    Result := S;                   // give the resultend;  Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function StripTags(const S: string): string;var  Len: Integer;   function ReadUntil(const ReadFrom: Integer; const C: Char): Integer;  var    j: Integer;  begin    for j := ReadFrom to Len do      if (s[j] = C) then      begin        Result := j;        Exit;      end;    Result := Len+1;  end; var  i, APos: Integer;begin  Len := Length(S);  i := 0;  Result := '';  while (i <= Len) do  begin    Inc(i);    APos := ReadUntil(i, '<');    Result := Result + Copy(S, i, APos-i);    i := ReadUntil(APos+1, '>');  end;end;  Hope it helps! With warmest regards, ---------------------------------------------------------------------- Roland57 · 2023-06-23 15:23 Quote from: wp on May 20, 2022, 09:44:28 pmSee attached demo. Weird, but here (Linux, Lazarus 2.2.6), with the HTM sample provided by pcurtis, I get no text. With the sample included in your demo, it works fine. ---------------------------------------------------------------------- Roland57 · 2023-06-23 15:39 @pcurtis For information, I converted your sample with pandoc (because I didn't get wp's demo to work for me): $ pandoc -f html -t plain text2.htm -o text2.txt [WARNING] text2.htm is not UTF-8 encoded: falling back to latin1. After that, I opened text2.txt in a hex editor (see attached picture). We can see that the undesired lines are unbreakable spaces mixed with line endings. I would use RegExpr to replace or delete unbreakable spaces, like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i s := ReplaceRegExpr('\xC2\xA0', s, '', FALSE); ---------------------------------------------------------------------- Jurassic Pork · 2023-06-23 23:21 Hello Roland, Quote from: Roland57 on June 23, 2023, 05:23:14 pmWeird, but here (Linux, Lazarus 2.2.6), with the HTM sample provided by pcurtis, I get no text. With the sample included in your demo, it works fine. it is because the file is windows-1252 encoded : Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i if you encode the file in utf-8 it is ok to convert it (file encoded utf-8 in attachment). OK with Ubuntu 20.04  Lazarus 2.2.0 Friendly, J.P ---------------------------------------------------------------------- Roland57 · 2023-06-23 23:54 @JP Thanks! ---------------------------------------------------------------------- Roland57 · 2023-06-24 06:27 Quote from: pcurtis on May 22, 2022, 06:12:33 pmEach paragraph starts with a space and there are too many blank lines. Here is the solution I could find (using wp's project). Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     //Memo1.Lines.Text := RenderHtml2Text(s);    SynEdit1.Lines.Text := s;     s := RenderHtml2Text(s);     // Remove spaces surrounded by line endings    s := ReplaceRegExpr('(\x0A)\x20+(\x0A)', s, '$1$2', TRUE); // Linux    s := ReplaceRegExpr('(\x0D\x0A)\x20+(\x0D\x0A)', s, '$1$2', TRUE); // Windows    // Remove spaces after line endings    s := ReplaceRegExpr('(\x0A)\x20+', s, '$1', TRUE); // L.    s := ReplaceRegExpr('(\x0D\x0A)\x20+', s, '$1', TRUE); // W.    // Remove spaces at beginning of the string    s := ReplaceRegExpr('^\x20+', s, '', FALSE);    // Set maximal blank lines number to 2    s := ReplaceRegExpr('\x0A{3,}', s, #$0A#$0A, FALSE); // L.    s := ReplaceRegExpr('(\x0D\x0A){3,}', s, #$0D#$0A#$0D#$0A, FALSE); // W.   Memo1.Lines.Text := s; Maybe someone will suggest a simpler solution.