Scraping Wikipedia pages lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- maurobio · 2021-09-11 11:23 Dear ALL, Does anyone knows of example FreePascal/Delphi code to scrape a Wikipedia page? I am especially interested in: (1) getting a "snippet" from a given page (that is, the first few lines of the summary, as plain text without any tags); and (2) getting a list of up to five images (if any) available in the gallery section. For example, given the page: https://en.wikipedia.org/wiki/Vicia_faba I would like to get the text snippet: "Vicia faba, also known in the culinary sense as the broad bean, fava bean, or faba bean, is a species of flowering plant in the pea and bean family Fabaceae. It is widely cultivated as a crop for human consumption, and also as a cover crop. Varieties with smaller, harder seeds that are fed to horses or other animals are called field bean, tic bean or tick bean. Horse bean, Vicia faba var. equina Pers., is a variety recognized as an accepted name." I would also like to get a list of the images available in the gallery (up to five): https://en.wikipedia.org/wiki/File:Vicia_faba.jpg https://en.wikipedia.org/wiki/File:Crimson_BB1.jpg https://en.wikipedia.org/wiki/File:Vicia_faba,_broad_bean_seed_showing _outer_seed_coating.jpg https://en.wikipedia.org/wiki/File:Tuinboon_voor_zaad.jpg https://en.wikipedia.org/wiki/File:Aphis_fabae,_zwarte_bonenluis.jpg In Python there is a great library for parsing HTML, called 'BeautifulSoup', which makes such things easy, I wonder if something similar does exist in FPC? Thanks in advance for any assistance you can provide. With best wishes, ---------------------------------------------------------------------- MarkMLl · 2021-09-11 11:45 I'd suggest that you'd be better off going to the underlying data, which I think is already stored as a database which can be downloaded. The summary paragraph is probably marked as such. MarkMLl ---------------------------------------------------------------------- engkin · 2021-09-11 12:39 I would suggest using Internet Tools ---------------------------------------------------------------------- Kays · 2021-09-11 13:32 Quote from: maurobio on September 11, 2021, 01:23:59 pm[…] scrape a Wikipedia page? […]Don’t do that. If you want to perform heavy-duty analysis on the Wikipedia corpus, please use a database dump https://en.wikipedia.org/wiki/WP:DD Quote from: maurobio on September 11, 2021, 01:23:59 pm[…] For example, given the page: […] I would like to get the text snippet: […]For that you could use the REST API, e.g. a request to https://en.wikipedia.org/api/rest_v1/page/summary/Vicia_faba will return a JSON containing extract. Note, there are other APIs, specifically obtaining any images in a will require processing the entire page contents. ---------------------------------------------------------------------- Jurassic Pork · 2021-09-11 15:28 hello, you can also try to use selenium with my webdriver for lazarus code to get the summary and the url of images in the web page https://en.wikipedia.org/wiki/Vicia_faba Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   Try  {$IFDEF WINDOWS}  //   Robot := TChromeDriver.Create(nil);  //   Robot := TFireFoxDriver.Create(nil);       Robot := TEdgeDriver.Create(nil);  //   Robot := TIEDriver.Create(Nil);   //   Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\chromedriver.exe');  //   Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\geckodriver.exe');       Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\msedgedriver.exe');  //   Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\IEDriverServer.exe');   {$ELSE}  //Robot := TChromeDriver.Create(nil);  Robot := TFireFoxDriver.Create(nil);  // Robot := TFireFoxDriver.Create(nil);  // Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '/chromedriver');  Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '/geckodriver');  // Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '/msedgedriver');  {$ENDIF}  Sleep(2000);  Robot.NewSession;  Robot.Implicitly_Wait(2000);  Robot.Set_Window_Size(640, 640);  Robot.GetURL('https://en.wikipedia.org/wiki/Vicia_faba');  summary := Robot.FindElementByXPath('//div[@class="mw-parser-output"]/p[2]');  Memo1.Append(summary.Text);  images := Robot.FindElementsByXPath('//a[@class="image"]');   For i:=0 to images.Count - 1 do  begin   Memo1.Append(images.Items[i].AttributeValue('href'));  end;  Memo1.Append('=================================');  Sleep(30000);  Robot.Quit;  finally  Robot.Free;  end; Result on windows 10 Lazarus 2.0.12 webdriver4L 0.2  using msedge chromium as browser in attachment. the REST api solution given by Kays is more simple : to have the summay and the media list of the page in json format : https://en.wikipedia.org/api/rest_v1/page/summary/Vicia_faba https://en.wikipedia.org/api/rest_v1/page/media-list/Vicia_faba the job is to extract the informations from the json responses. Friendly, J.P ---------------------------------------------------------------------- BobDog · 2021-09-11 18:36 Windows. This gives a .txt and .html, take your choice. I have set the wiki page as default. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program WebPageToText; function  system(s:pchar):integer ; cdecl external 'msvcrt.dll' name 'system';  var g:ansistring;defaultstring:ansistring = 'https://en.wikipedia.org/wiki/Vicia_faba';     // set as default kill:integer=1;  procedure savefile(fname:string ;text:ansistr ing;killflag:integer=0); label kill;Var T:TextFile;beginif killflag<>0 then goto kill;   AssignFile(T,fname);   {$I-}   try   Rewrite(T);   Writeln(T,text);   finally   CloseFile(T);   {$I+}   end;   kill:  if killflag<>0 then erase(T);end; procedure runscript(filename:ansiString);begin  system(pchar('cscript.exe /Nologo '+ filename) );End; beging:=g+ 'Const TriStateTrue = -1 '+chr(10);g:=g+ 'URL = InputBox("Enter (or paste) the URL to extract the Code "&vbcr&vbcr&_'+chr(10);g:=g+ '"Exemple ""https://www.freebasic.net""","Extraction of Source text and html  ","'+defaultstring+'")'+chr(10);g:=g+ 'If URL = "" Then WScript.Quit'+chr(10);g:=g+ 'Titre = "Extraction du Code Source de " & URL'+chr(10);g:=g+ 'Set ie = CreateObject("InternetExplorer.Application")'+chr(10);g:=g+ 'Set objFSO = CreateObject("Scripting.FileSys]"]>Blockedbject")'+chr(10);g:=g+ 'ie.Navigate(URL)'+chr(10);g:=g+ 'ie.Visible=false'+chr(10);g:=g+ 'DO WHILE ie.busy'+chr(10);g:=g+ 'LOOP'+chr(10);g:=g+ 'DataHTML = ie.document.documentElement.innerHTML'+chr(10);g:=g+ 'DataTxt = ie.document.documentElement.innerText'+chr(10);g:=g+ 'strFileHTML = "CodeSourceHTML.txt"'+chr(10);g:=g+ 'strFileTxt = "CodeSourceTxt.txt"'+chr(10);g:=g+ 'Set objHTMLFile = objFSO.OpenTextFile(strFileHTML,2,True, TriStateTrue)'+chr(10);g:=g+ 'objHTMLFile.WriteLine(DataHTML)'+chr(10);g:=g+ 'objHTMLFile.Close'+chr(10);g:=g+ 'Set objTxtFile = objFSO.OpenTextFile(strFileTxt,2,True, TriStateTrue)'+chr(10);g:=g+ 'objTxtFile.WriteLine(DataTxt)'+chr(10);g:=g+ 'objTxtFile.Close'+chr(10);g:=g+ 'ie.Quit'+chr(10);g:=g+ 'Set ie=Nothing'+chr(10);g:=g+ ' Ouvrir(strFileHTML)'+chr(10);g:=g+ ' Ouvrir(strFileTxt)'+chr(10);g:=g+ 'wscript.Quit'+chr(10);g:=g+ 'Function Ouvrir(File)'+chr(10);g:=g+ '    Set ws=CreateObject("wscript.shell")'+chr(10);g:=g+ '    ws.run "Notepad.exe "& File,1,False'+chr(10);g:=g+ 'end Function'+chr(10);   savefile('script.vbs',g) ;runscript('script.vbs'); writeln('Press enter to end . . .');readln;savefile('script.vbs','',kill);end. ---------------------------------------------------------------------- maurobio · 2021-09-11 21:36 Dear Wizards, Thank you all very much for your insightful answers and helpful suggestions. Using a database dump is not an option to me because my application is aimed at providing a specialized search engine providing the user with information on biological species automagically compiled from several sources on the web. The possibility of using the Wikipedia API as suggested by @Kays (and @Jurassic Pork), which offers the summary extract as required, looks great; for getting the image links, I could use the internettools library as suggested by @engkin. As @JP's pointed out, it is also possible to get the image links using the API. On the other side, I did not know the "webdriver for lazarus" library used by @Jurassic Pork; I do agree with him that using the REST API is much simpler, but I would like to know more about webdriver not only out of curiosity, but because it looks great and should be useful for other applications! (I could not find it in the link provided in JP's answer, that redirects ti the main page of the forum). Unfortunately, the suggestion by @BobDog is for Windoze only and it does not suit me because my application should be a CGI application running on a LAMP server (and also because Windoze sucks). Again, thank you very much! With best wishes, ---------------------------------------------------------------------- Jurassic Pork · 2021-09-11 22:17 Quote from: maurobio on September 11, 2021, 11:36:11 pm (I could not find it in the link provided in JP's answer, that redirects ti the main page of the forum). Are you logged in when your click on the link ? you can also search in the forum for the last 99 days  browser automation and go to the last post ---------------------------------------------------------------------- maurobio · 2021-09-11 22:34 @JP. Sure, I was not logged when I first tried to access the link. I got it now. Thank you! With best wishes, ---------------------------------------------------------------------- maurobio · 2021-09-12 14:10 Dear ALL, Here is my sample code for getting the text summary and a list of (up to five) image files for a Wikipedia page. Notice that the sample code uses the really great JsonTools library available from here: https://www.getlazarus.org/json/, which makes working with JSON data much easier than suing the buiil-in FPC libraries. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program wiki_example; {$APPTYPE CONSOLE}{$mode objfpc}{$H+} uses SysUtils, fphttpclient, JsonTools; const         url1 = 'https://en.wikipedia.org/api/rest_v1/page/summary/';       url2 = 'https://en.wikipedia.org/api/rest_v1/page/media-list/';        searchName = 'Vicia faba'; var  rawJson: AnsiString;  JsonData: TJsonNode;  N: integer; begin  WriteLn('Searching Wikipedia for: ' + searchName);  // Get the page summary   JsonData := TJsonNode.Create; rawJson := TFPHTTPClient.SimpleGet(url1 + StringReplace(searchName, ' ', '_', [rfReplaceAll]));  JsonData.Parse(rawJson);  WriteLn(JsonData.Find('extract').AsString);  JsonData.Free;  WriteLn;    // Get the images in the gallery  JsonData := TJsonNode.Create;   rawJson := TFPHTTPClient.SimpleGet(url2 + StringReplace(searchName, ' ', '_', [rfReplaceAll]));  JsonData.Parse(rawJson);  for N := 1 to 5 do        WriteLn(JsonData.Find('items/' + IntToStr(N - 1) + '/title').AsString);  JsonData.Free;end.  This code works, but I am not entirely happy with it because I could not devise a way of getting all the image file names, instead of just five, because I could not see how to get the total number of items in the JSON data (in this case, it should be 15). Also, if there are less then five images in a searched page, the above code will fail. Could anyone give me a hint? Thanks in advance! With best wishes, ---------------------------------------------------------------------- BobDog · 2021-09-12 15:18 Sorry, got all the stuff, compiled the json unit, copied a myriad of units, then I get: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Microsoft Windows [Version 10.0.19042.1165](c) Microsoft Corporation. All rights reserved. C:\Users\Computer\Desktop\fb\pascal\mystuff\jsoninternet>int ernetSearching Wikipedia for: Vicia fabaAn unhandled exception occurred at $00431920:ESSLSocketError: No SSL Socket support compiled in.Please include opensslsockets unit in program and recompile it.  $00431920  $004190A8  $00419151  $0041B0F0  $0041B449  $0041BA00  $0041BC67  $0041BCC3  $0041BD88  $0040194C  C:\Users\Computer\Desktop\fb\pascal\mystuff\jsoninternet> And I don't have the opensslsockets unit. Maybe a Linux thing? ---------------------------------------------------------------------- maurobio · 2021-09-12 15:45 Dear @BobDog, Thanks for your message. As a matter of fact, I have compiled and run successfully this code in Windows 10 without any trouble. You probably have to install OpenSSL in your system. With best wishes, ---------------------------------------------------------------------- Jurassic Pork · 2021-09-12 16:04 hello, maurobio, fpson + jsonparser from  free pascal  fcl-json is also a great tool  try this code : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program wiki_example; {$APPTYPE CONSOLE}{$mode objfpc}{$H+} uses  Classes, TypInfo,SysUtils, fphttpclient, fpjson, jsonparser, opensslsockets; const        url1 = 'https://en.wikipedia.org/api/rest_v1/page/summary/';       url2 = 'https://en.wikipedia.org/api/rest_v1/page/media-list/';        searchName = 'Vicia faba'; var  jData: TJSONData;  jItem,jItems: TJSONData;  i: integer; begin  WriteLn('Searching Wikipedia for: ' + searchName);  // Get the page summary  jData := GetJSON(TFPHTTPClient.SimpleGet(url1 + StringReplace(searchName, ' ', '_', [rfReplaceAll])));  WriteLn(jData.FindPath('extract').AsString);  WriteLn;  // Get the images in the gallery  jData := GetJSON(TFPHTTPClient.SimpleGet(url2 + StringReplace(searchName, ' ', '_', [rfReplaceAll])));  jItems := jData.FindPath('items');  for i := 0 to jItems.Count - 1 do    begin       jItem := jItems.Items[i];    writeln(jItem.FindPath('title').AsString);    end;  jData.Free;  readln; end.   Ok on Windows 10 Lazarus 2.3.0 fpc 3.3.1 Friendly, J.P ---------------------------------------------------------------------- maurobio · 2021-09-12 18:40 Dear @Jurassic Pork, Thank you, as always, for your invaluable help and useful insights. I do agree that the FPC/Lazarus built-in JSON libraries are also great, but I found them more difficult to use than JsonTools, and documentation is not as extensive as I would like. On the other side, the author of JsonTools presents some convincing comparisons between his library and those of FPC/Lazarus, showing that his are usually faster than other Pascal parsers (https://www.getlazarus.org/json/tests/). Again, thanks a lot! With best wishes,