scrap html file (using DOM_HTML,DOM, SAX_HTML) lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- alaa123456789 · 2021-02-15 21:36 hey , i have a website HTML , i am trying to read it in Thtmldocument Code: HTML5 [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i

' + LE + ''+ LE + 'Download'+ LE + ''+LE+ ''+LE+ '

' + LE + ''; function GetAttrValue(ANode : TDOMNode; AAttrName : UnicodeString) : UnicodeString;var i: LongWord; Found: Boolean;begin Result := ''; if (ANode = nil) or (ANode.Attributes = nil) then exit; Found := false; i := 0; while not Found and (i < ANode.Attributes.Length) do begin if ANode.Attributes.Item[i].NodeName = AAttrName then begin Found := true; Result := ANode.Attributes.Item[i].NodeValue; end; inc(i); end;end; { TForm1 } procedure TForm1.FormCreate(Sender: TObject);begin SynEdit1.Lines.Text := html;end; procedure TForm1.Button1Click(Sender: TObject);var stream: TMemoryStream; doc: THTMLDocument; node, child: TDOMNode; dataPreview: UnicodeString;begin stream := TMemoryStream.Create; try SynEdit1.Lines.SaveToStream(stream); stream.Position := 0; ReadHtmlFile(doc, stream); try node := doc.DocumentElement.FindNode('div'); while node <> nil do begin child := node.FirstChild; while child <> nil do begin if child.NodeName = 'button' then begin dataPreview := GetAttrValue(child, '1data-preview'); if dataPreview <> '' then begin Label1.Caption := datapreview; exit; end; end; child := child.NextSibling; end; node := node.NextSibling; end; Label1.Caption := '(not found)'; finally doc.Free; end; finally stream.Free; end;end; I surrounded your html text by tags. If you don't do that you must change the code a tiny bit: After the inner "try" replace the node := doc.GetElementByname('html') by node := doc.FirstChild;. ---------------------------------------------------------------------- BeniBela · 2021-02-16 12:17 You can do that with my internet tools: Code: [Select]uses simpleinternet; process('https://www.pdfdrive.com/a-python-book-beginning-python- advanced-python-and-python-e9236005.html', 'id("previewButtonMain")/@data-preview').toString ---------------------------------------------------------------------- alaa123456789 · 2021-02-16 12:18 Quote from: wp on February 16, 2021, 11:44:15 am I surrounded your html text by tags. If you don't do that you must change the code a tiny bit: After the inner "try" replace the node := doc.GetElementByname('html') by node := doc.FirstChild;. the html code i attached is part of webpage with many other items , i tried your code always showing 'not found' , i didnt see GetElementByname('html') in your code so what do u mean by it thanks ---------------------------------------------------------------------- Jurassic Pork · 2021-02-16 12:24 hello, you can also try something like this : Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient,opensslsockets, DOM_HTML,DOM, SAX_HTML; {$R *.lfm} function GetDoc (url: String): thtmldocument;Var S : String; doc: thtmldocument;begin With TFPHttpClient.Create(Nil) do try S := Get(url); readhtmlfile(doc,tstringstream.create(S)); finally Free; end; result := doc;end; { TForm1 } procedure TForm1.Button1Click(Sender: TObject);var page1:ThtmlDocument; el: tdomnodelist; i: integer; begin page1:= GetDoc('https://www.pdfdrive.com/a-python-book-beginning-python- advanced-python-and-python-e9236005.html');// here where i need help el:= page1.GetElementsByTagName('button'); begin if tdomelement(el[i]).getattribute('id') = 'previewButtonMain' then ShowMessage(tdomelement(el[i]).getattribute('data-preview')); end; end;end. Friendly, J.P ---------------------------------------------------------------------- alaa123456789 · 2021-02-16 12:26 Quote from: BeniBela on February 16, 2021, 01:17:43 pmYou can do that with my internet tools: Code: [Select]uses simpleinternet; process('https://www.pdfdrive.com/a-python-book-beginning-python- advanced-python-and-python-e9236005.html', 'id("previewButtonMain")/@data-preview').toString it seems intersting and much easier , it is not installing please see attachment thanks ---------------------------------------------------------------------- alaa123456789 · 2021-02-16 12:45 Quote from: Jurassic Pork on February 16, 2021, 01:24:43 pmthanks Jp but your code need some modification Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient,opensslsockets, DOM_HTML,DOM, SAX_HTML; {$R *.lfm} function GetDoc (url: String): thtmldocument;Var S : String; doc: thtmldocument;begin With TFPHttpClient.Create(Nil) do try S := Get(url); readhtmlfile(doc,tstringstream.create(S)); finally Free; end; result := doc;end; { TForm1 } procedure TForm1.Button1Click(Sender: TObject);var page1:ThtmlDocument; el: tdomnodelist; i: integer; begin page1:= GetDoc('https://www.pdfdrive.com/a-python-book-beginning-python- advanced-python-and-python-e9236005.html');// here where i need help [color=yellow]el:= page1.GetElementsByTagName('button'); begin if tdomelement(el[i]).getattribute('id') = 'previewButtonMain' then ShowMessage(tdomelement(el[i]).getattribute('data-preview')); end;[/color] end;end. Friendly, J.P As following : Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i el:= page1.GetElementsByTagName('button'); i:=0; while i < el.Length do begin if tdomelement(el[i]).getattribute('id') = 'previewButtonMain' then ShowMessage(tdomelement(el[i]).getattribute('data-preview')); inc(i); end; thanks ---------------------------------------------------------------------- BeniBela · 2021-02-16 21:49 Quote from: alaa123456789 on February 16, 2021, 01:26:38 pmQuote from: BeniBela on February 16, 2021, 01:17:43 pmYou can do that with my internet tools: Code: [Select]uses simpleinternet; process('https://www.pdfdrive.com/a-python-book-beginning-python- advanced-python-and-python-e9236005.html', 'id("previewButtonMain")/@data-preview').toString it seems intersting and much easier , it is not installing please see attachment thanks You also need this: https://github.com/benibela/flre ---------------------------------------------------------------------- zamronypj · 2021-02-17 22:11 Quote from: alaa123456789 on February 15, 2021, 10:36:32 pmhey , i have a website HTML , i am trying to read it in Thtmldocument Code: HTML5 [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i

Download

i want to get what is inside data-preview i did something like this but always failing ,why there is no resources for the fpc Units and how to use them please have a look on this code and help me Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses Classes,SysUtils, Forms, Controls, Dialogs, StdCtrls, LCLIntf, synacode,fphttpclient, regexpr, DOM_HTML,DOM, SAX_HTML,strutils, variants; function TForm1.GetDoc (url: String): thtmldocument;var HTTPGetResult: Boolean; HTTPSender: THTTPSend; //outString: string; siteRoorUrl:string; doc: thtmldocument; begin HTTPSender := THTTPSend.Create; try HTTPGetResult := HTTPSender.HTTPMethod('GET', url); siteRoorUrl := HTTPSender.TargetHost; if (HTTPSender.ResultCode >= 100) and (HTTPSender.ResultCode<=299) then begin readhtmlfile(doc,HTTPSender.Document); end; finally HTTPSender.Free; end; result := doc;end; procedure TForm1.ListBox1Click(Sender: TObject);var x2,x4,x5: string; x3:Variant; page1:THTMLDocument; lis: THTMLCollection; begin page1:= GetDoc(VarToStr(url));// here where i need help lis:= page1.GetElementsByTagName('div');// here where i need help ShowMessage(lis[0].ToString); // here where i need help end. If DOM is generated or manipulated by JavaScript, then reliable way to extract it is using headless browser as suggested by Jurassic Pork. If it is static page then THTMLDocument should work. ---------------------------------------------------------------------- alaa123456789 · 2021-02-18 05:51 Quote from: zamronypj on February 17, 2021, 11:11:31 pmIf DOM is generated or manipulated by JavaScript, then reliable way to extract it is using headless browser as suggested by Jurassic Pork. If it is static page then THTMLDocument should work. yes agree with you , but how to hide the browser window , and dos window please set an example thanks ---------------------------------------------------------------------- Jurassic Pork · 2021-02-18 07:44 hello, Quote from: alaa123456789 on February 18, 2021, 06:51:34 am[ but how to hide the browser window , and dos window please set an example to hide chrome browser you can try this in the file Webdriver4L.pas : Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function TChromeDriver.NewSession: string;const Chrome_Param = '{"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome",' + ' "platformName": "any", "goog:chromeOptions": {"extensions": [],' // + ' "args": []}}},' // visible mode + ' "args": ["--headless","--disable-gpu"]}}},' // hidden mode + ' "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY",' + ' "goog:chromeOptions": {"extensions": [], "args": []}}}';varFor Firefox you can try this : Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function TFireFoxDriver.NewSession: string;const Firefox_Param = '{"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "firefox", ' + '"acceptInsecureCerts": false, "moz:firefoxOptions": ' +// '{"binary": "%s"}}},' + // visible mode '{"binary": "%s","args": ["-headless"]}}},' + // hidden mode '"desiredCapabilities": {"browserName": "firefox", "acceptInsecureCerts": false, "moz:firefoxOptions": {"binary": "%s"}}}';var be careful , in hidden mode you can't see if something is wrong in your selenium code. Friendly, J.P ---------------------------------------------------------------------- alaa123456789 · 2021-02-18 12:01 thanks JP, if i have thtmldocument and want to show its contents in memo, how could be that done thanks ---------------------------------------------------------------------- Jurassic Pork · 2021-02-18 15:31 hello, Quote from: alaa123456789 on February 18, 2021, 01:01:55 pmif i have thtmldocument and want to show its contents in memo, how could be that done contents in memo ? -> whole html page ? -> S string in getDoc function Friendly, J.P ---------------------------------------------------------------------- alaa123456789 · 2021-02-18 15:44 Quote from: Jurassic Pork on February 18, 2021, 04:31:35 pm contents in memo ? -> whole html page ? -> S string in getDoc function Friendly, J.P yes i used this Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i varpage:thtmldocument page:= readhtmlfile(doc,HTTPSender.Document);// i want to check if page not empty so i want to show it in memo to double check if the variable page is not empty i want to show its value in memo thanks ---------------------------------------------------------------------- alaa123456789 · 2021-02-20 19:17 i have always the error as per attached photo Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);var doc: thtmldocument; el: tdomnodelist; i: integer; x2: string;begin ReadHTMLFile(doc,strxhtmldecode(GetWebPage(edit1.Text))); // GetWebPage(edit1.Text):string el := doc.GetElementsByTagName('a'); i := 0; while i < el.Length do //loop through button html content begin if tdomelement(el[i]).getattribute('class') = 'btn btn- success btn-responsive' then x2 := (tdomelement(el[i]).getattribute('href')); Inc(i); end; memo1.Lines.Add(x2);end; anyone know what is wrong? ---------------------------------------------------------------------- Jurassic Pork · 2021-02-21 00:29 hello, try : Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ReadHTMLFil e(doc,tstringstream.create(strxhtmldecode(GetWebPage(edit1.Text)))); Friendly, J.P ---------------------------------------------------------------------- alaa123456789 · 2021-02-21 06:14 didnt work , the error not showing , but the output is nothing ---------------------------------------------------------------------- Jurassic Pork · 2021-02-21 07:45 hello, Quote from: alaa123456789 on February 21, 2021, 07:14:54 amdidnt work , the error not showing , but the output is nothing alaa123456789 you are a sorcerer's apprentice Don't use function that you don't master ! Friendly, J.P ---------------------------------------------------------------------- alaa123456789 · 2021-02-21 09:46 Quote from: Jurassic Pork on February 21, 2021, 08:45:38 am alaa123456789 you are a sorcerer's apprentice Don't use function that you don't master ! Friendly, J.P [/quote] no i am not , i am trying to learn and code something and if not work i should come to this forum to ask as i think it was made for this reason ? isn't? thanks