HTML files get values lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- alaa123456789 · 2020-05-23 06:42 Hey Guys, wish everyone is okay , i am suffering with lazarus to find information about units -uses there is less information i am trying to read below html file and get the value (20) -last page number from it please help me Regards Code: HTML5  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ---------------------------------------------------------------------- trev · 2020-05-23 07:25 Show us your code for this so far and tell us what happens. [You might benefit from reading How to use the Forum too.] ---------------------------------------------------------------------- alaa123456789 · 2020-05-23 07:53 here is the code , i tried many times but couldn't get any results Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses  Classes, SysUtils, SAX_HTML, DOM, DOM_HTML, fphttpclient, Forms, Controls,  Graphics, Dialogs, StdCtrls;  var    doc:THTMLDocument;    els:TDOMNodeList;    f:THTMLElement; Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  ReadHTMLFile(doc,tstringstream.create(s))//s is html content string;      //f:=THTMLElement(doc.GetElementsByTagName('Zebra_Pagination'));    //Memo1.Lines.add(f.FirstChild.NodeValue);      els:=doc.GetElementsByTagName('div');      if (els.Count) > 0 then begin       Memo1.Lines.add(tdomelement(els[0]).getattribute('class'));       Mem o1.Lines.add(tdomelement(els[0]).getattribute('Zebra_Pagination'));      Memo1.Lines.add((tdomelement(els[0]).FirstChild.TextContent)); ---------------------------------------------------------------------- kqha · 2020-05-26 12:33 I'm not really sure about TDOMNodeList, but if GetElementsByTagName works just like javascript and PHP DOM does, it seems your logic in handling GetElementsByTagName to find the "20" textcontent is wrong. Regardless, if I were you I would getElementsByTagName('a') instead and loop through it to find an element which has "navigation next" in class attribute, and return the previous element. Some pseudo code: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i els := doc.GetElementsByTagName('a');n := -1;  //index of els which has last page number, default -1 means not foundfor a:=0 to els.Count-1 dobegin  if Pos('next',els[a].getAttribute('class'))>=0 then  begin  n := a-1;    Break;  end;end; ---------------------------------------------------------------------- howardpc · 2020-05-26 13:19 When parsing text files, and only one or two values are of interest, it is sometimes easier just to brute force a hacked solution, rather than spend time trying to understand how to use a complex library you did not write and are unfamiliar with, particularly if it has limited documentation. Say, something along these lines: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1; {$mode objfpc}{$H+} const  HTMLText = '    ';   function GetLastPageNo(const anHTMLText: String): Integer;  var    p, pStart, pEnd: PChar;    arr: array of LongInt = Nil;    s: String;    idx: Integer = -1;  begin    Result := -1;    p := PChar(anHTMLText);    pEnd := p;    SetLength(arr, Length(anHTMLText) shr 4);    Inc(pEnd, Length(anHTMLText));    while p < pEnd do      begin        Inc(p);        if (p^ = 'p') and (p[1] = 'a') and (p[2] = 'g') and (p[3] = 'e') and          (p[4] = '=') and (p[5] in ['0'..'9']) then            begin              Inc(p, 5);            pStart := p;              while p[1] in ['0'..'9'] do              Inc(p);                SetString(s, pStart, Succ(p - pStart));                Inc(idx);                ReadStr(s, arr[idx]);          end;      end;     for idx in arr do      if idx > Result then      Result := idx;  end; begin  WriteLn('Last page number is ',GetLastPageNo(HTMLText));  ReadLn;end. ---------------------------------------------------------------------- rvk · 2020-05-26 14:39 Quote from: howardpc on May 26, 2020, 03:19:09 pmWhen parsing text files, and only one or two values are of interest, it is sometimes easier just to brute force a hacked solution, rather than spend time trying to understand how to use a complex library you did not write and are unfamiliar with, particularly if it has limited documentation.Indeed. And in that case using regexpr would be even more simple  (I know, I know.... never ever parse HTML with regexpr  ) And this only works if there are not any other page=x expressions on the page (at least not with a higher number you want). Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1; {$mode objfpc}{$H+}uses  regexpr, sysutils; const  HTMLText = '    ';   function GetLastPageNo(const anHTMLText: string): integer;  var    re: TRegExpr;  begin    Result := -1;    re := TRegExpr.Create('&page=(.*?)">');    if re.Exec(anHTMLText) then      while re.ExecNext do        if StrToIntDef(re.Match[1], 0) > Result then Result := StrToIntDef(re.Match[1], 0);    re.Free;  end; begin  WriteLn('Last page number is ', GetLastPageNo(HTMLText));  ReadLn;end. ---------------------------------------------------------------------- trev · 2020-05-27 05:45 Quote from: rvk on May 26, 2020, 04:39:28 pmIndeed. And in that case using regexpr would be even more simple  (I know, I know.... never ever parse HTML with regexpr  ) You beat me to it - a regex would be my suggestion too Except my regex would have been: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i re := TRegExpr.Create('&page=([0-9]*)">');. ---------------------------------------------------------------------- alaa123456789 · 2020-05-29 07:07 thanks all for reply it was useful information , i tried to call the function Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i memo1.lines.add(getlastpageno(s));i got this error getlastpageno not identifier another thing i have text like this /python-data-analytics-data-analysis-and-science-using-pandas- matplotlib-and-the-python-e158003322.html  how to get only text without '-' and without 'e158003322.html' second thing how to replace 'e' in 'e158003322.html' with 'd' thanks Alaa ---------------------------------------------------------------------- howardpc · 2020-05-29 07:12 GetLastPageNo returns an integer. You can't add that to a TStrings instance without using a conversion function. But the error you got suggests you have not implemented the function in your code, or you spelled it differently. ---------------------------------------------------------------------- alaa123456789 · 2020-05-29 07:17 i tried to us inttostr with it didnt work also , i have copied the code itself and worked when used without function Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);var  s1:String;  re:TRegExpr;  result:Integer;begin      With TFPHttpClient.Create(Nil) do    try    S1 := Get('link');    Result := -1;    re := TRegExpr.Create('&page=([0-9]*)">');    if re.Exec(s1) then      while re.ExecNext do        if StrToIntDef(re.Match[1], 0) > Result then Result := StrToIntDef(re.Match[1], 0);           re.Free;      Memo1.Lines.add(inttostr(result));    finally      Free;    end;end;  can you help me with the other question in my previous post  Quoteanother thing i have text like this /python-data-analytics-data-analysis-and-science-using-pandas- matplotlib-and-the-python-e158003322.html  how to get only text without '-' and without 'e158003322.html' second thing how to replace 'e' in 'e158003322.html' with 'd' thanks ---------------------------------------------------------------------- trev · 2020-05-29 07:53 Second question regex: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i re := TRegExpr.Create('\(.*\)-e[0-9]*.html/\1d/'); Should work, but only tested with sed. ---------------------------------------------------------------------- alaa123456789 · 2020-05-29 12:54 thanks but how i use this Quotere := TRegExpr.Create('\(.*\)-e[0-9]*.html/\1d/'); in this Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i re:=TRegExpr.Create('';   bookname : string; begin   re:=TRegExpr.Create('
Go to PDF
how to get the link of pdf file and what is the best way to understand and learn tregx please help thanks ALaa ---------------------------------------------------------------------- alaa123456789 · 2020-05-31 20:04 here is the difficult task in my program i overcome all the previous challenges with your supports (thanks to all of who supported me) but i reach to closed door , the page i want to scrape is showing loader bar for some time then it loads the HTML which has the link of the book i want to get it how i can overcome this thanks Alaa ---------------------------------------------------------------------- rvk · 2020-05-31 20:20 Quote from: alaa123456789 on May 31, 2020, 10:04:29 pmhere is the difficult task in my program i overcome all the previous challenges with your supports (thanks to all of who supported me) but i reach to closed door , the page i want to scrape is showing loader bar for some time then it loads the HTML which has the link of the book i want to get it how i can overcome this That depends... If the loader bar is javascript (or similar) and the html doesn't get loaded if the javascript isn't executed, you can't just do a gethtml() to get the page. So first check what your gethtml() function gets as text. If in that text there is nothing you want to scrape... you'll need to execute the javascript. Maybe you can use the IE engine to get the file (which does execute the javascript). For example you could use Browser := CreateOleObject('InternetExplorer.Application'); to initiate an internet explorer browser and with navigate, navigate to the correct page and read out the html. (but that's only under Windows) If there are any security measures (like Captchas), you'll still need to do some things manually. ---------------------------------------------------------------------- trev · 2020-06-01 01:34 What web site are you trying to scrape? ---------------------------------------------------------------------- Jurassic Pork · 2020-06-01 05:10 hello, Quote from: alaa123456789 on May 31, 2020, 06:58:05 pmthanks JP , i tried to install internet tools showed me this error "one or more package is required " and didn't know which package is required regards What is your O.S ? Lazarus version ? I have tried internettools on WIndows 10 and Centos 8   with Lazarus 2.0.8 : it 's OK if : In your project :  1 - menu  Project/Project Inspector    -> in the window  add internettools & internettools_utf8 to  required Packages  2 - menu   Project/project Options -> in Compiler Options/Paths   for Other Unit files (-Fu) add the paths of some folders of the internettools package  : path_to_onlinepackagemanager/packages/internettools-master/data path_to_onlinepackagemanager/packages/internettools-master/system path_to_onlinepackagemanager/packages/internettools-master/flre- master/src with internettools you can also download files. Example  : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i implementationuses xquery , simpleinternet, bbutils, internetaccess;{$R *.lfm}{ TForm1 }                                           procedure TForm1.Bt_DownloadClick(Sender: TObject);var TargetFile, SiteURL, Download: string;begin  SiteURL := 'https://www.coursef.com/more-reviews/www.davekuhlman.org?q=python%203 %20tutorial%20pdf%20download';  TargetFile:='/home/jurassic/mybook.pdf';  //set user agent (fails without it)  defaultInternetConfiguration.userAgent:='curl/7.21.0 (i686-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18';  Download := retrieve(SiteURL);  if strBeginsWith(LowerCase(Download), '') then begin    //Download page    SiteURL:=process(Download, '//@data-full- link').toString;    Download := retrieve(SiteURL);    if strBeginsWith(LowerCase(Download), '') then raise Exception.create('Multiple redirections');  end;  strSaveToFileUTF8(TargetFile, Download);end; Friendly, J.P ---------------------------------------------------------------------- alaa123456789 · 2020-06-01 16:33 Quote from: rvk on May 31, 2020, 10:20:51 pmQuote from: alaa123456789 on May 31, 2020, 10:04:29 pmhere is the difficult task in my program i overcome all the previous challenges with your supports (thanks to all of who supported me) but i reach to closed door , the page i want to scrape is showing loader bar for some time then it loads the HTML which has the link of the book i want to get it how i can overcome this That depends... If the loader bar is javascript (or similar) and the html doesn't get loaded if the javascript isn't executed, you can't just do a gethtml() to get the page. So first check what your gethtml() function gets as text. If in that text there is nothing you want to scrape... you'll need to execute the javascript. Maybe you can use the IE engine to get the file (which does execute the javascript). For example you could use Browser := CreateOleObject('InternetExplorer.Application'); to initiate an internet explorer browser and with navigate, navigate to the correct page and read out the html. (but that's only under Windows) If there are any security measures (like Captchas), you'll still need to do some things manually. i am using TFPHTTPClient.simpleget(url) there is no gethtml method in it ,i put the result in memo1 but the text didn't have what i wanted i tried to use htmlviwer tool to load  url and tried to get the text from it i didn't know how  if u could help i will be thankful to you Regards Alaa ---------------------------------------------------------------------- alaa123456789 · 2020-06-01 16:51 Quote from: Jurassic Pork on June 01, 2020, 07:10:34 amhello, Quote from: alaa123456789 on May 31, 2020, 06:58:05 pmthanks JP , i tried to install internet tools showed me this error "one or more package is required " and didn't know which package is required regards What is your O.S ? Lazarus version ? I have tried internettools on WIndows 10 and Centos 8   with Lazarus 2.0.8 : it 's OK if : In your project :  1 - menu  Project/Project Inspector    -> in the window  add internettools & internettools_utf8 to  required Packages  2 - menu   Project/project Options -> in Compiler Options/Paths   for Other Unit files (-Fu) add the paths of some folders of the internettools package  : path_to_onlinepackagemanager/packages/internettools-master/data path_to_onlinepackagemanager/packages/internettools-master/system path_to_onlinepackagemanager/packages/internettools-master/flre- master/src with internettools you can also download files. Example  : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i implementationuses xquery , simpleinternet, bbutils, internetaccess;{$R *.lfm}{ TForm1 }                                           procedure TForm1.Bt_DownloadClick(Sender: TObject);var TargetFile, SiteURL, Download: string;begin  SiteURL := 'https://www.coursef.com/more-reviews/www.davekuhlman.org?q=python%203 %20tutorial%20pdf%20download';  TargetFile:='/home/jurassic/mybook.pdf';  //set user agent (fails without it)  defaultInternetConfiguration.userAgent:='curl/7.21.0 (i686-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18';  Download := retrieve(SiteURL);  if strBeginsWith(LowerCase(Download), '') then begin    //Download page    SiteURL:=process(Download, '//@data-full- link').toString;    Download := retrieve(SiteURL);    if strBeginsWith(LowerCase(Download), '') then raise Exception.create('Multiple redirections');  end;  strSaveToFileUTF8(TargetFile, Download);end; Friendly, J.P Hi jp i am using lazarud 2.08 64bit on windows 10 i tried your instructions please check attachment thanks ---------------------------------------------------------------------- rvk · 2020-06-01 21:04 Quote from: alaa123456789 on June 01, 2020, 06:33:26 pmi am using TFPHTTPClient.simpleget(url) there is no gethtml method in it ,i put the result in memo1 but the text didn't have what i wanted i tried to use htmlviwer tool to load  url and tried to get the text from it i didn't know how  if u could help i will be thankful to you I did mean simpleget instead of gethtml. I meant a function to get the html (so get or simpleget). If you don't get what you want, probably the javascript needs to be executed. I don't think htmlviewer does that. An instance of internet explorer does. Are you on Windows and if so, what site are you trying because we can help you better if we try some code ourselves. ---------------------------------------------------------------------- Jurassic Pork · 2020-06-02 01:14 hello Quote from: alaa123456789 on June 01, 2020, 06:51:30 pmi am using lazarud 2.08 64bit on windows 10 i tried your instructions please check attachment thanks my lazarus version is the 32 bits version on windows. To see what it the broken dependency open the  package graph (menu package/package graph). May be laz_synapse : Friendly J.P ---------------------------------------------------------------------- alaa123456789 · 2020-06-02 18:05 QuoteI did mean simpleget instead of gethtml. I meant a function to get the html (so get or simpleget). If you don't get what you want, probably the javascript needs to be executed. I don't think htmlviewer does that. An instance of internet explorer does. Are you on Windows and if so, what site are you trying because we can help you better if we try some code ourselves. i added instance of internet explorer and loaded the page but it seems javascript not allowed to run ,how to run it Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i url:=UTF8Decode(x3);  onull:=NULL;   Browser.ComServer.Navigate2(url,onull,onull,onull,onull); then how to get the code in memo if this work thanks Alaa ---------------------------------------------------------------------- rvk · 2020-06-02 19:29 Quote from: alaa123456789 on June 02, 2020, 08:05:58 pmi added instance of internet explorer and loaded the page but it seems javascript not allowed to run ,how to run it It does allow javascript but the embedded version isn't capable of jquery (because it emulates IE6). So you probably shouldn't use that embedded version of Internet Explorer because IE6 is very old. You can increase the emulated version in the registry but that might be a hassle. It is better to just use a new oleobject (with the full latest IE version). Something like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses  comobj, // for ceating Browser-object  ActiveX; // CoInitialize procedure TForm1.Button1Click(Sender: TObject);const  GoUrl = 'http://www.yourwebsite.com';var  Browser: olevariant;  Document: olevariant;  Body: olevariant;begin   CoInitialize(nil);  Browser := CreateOleObject('InternetExplorer.Application');  try    try       Browser.AddressBar := false;      Browser.Menubar := false;      Browser.ToolBar := false;      Browser.StatusBar := false;      Browser.Width := 600;      Browser.Height := 600;      Browser.Left := Screen.Width div 2 - Browser.Width div 2;      Browser.Top := Screen.Height div 2 - Browser.Height div 2;      Browser.Visible := True;       Browser.Navigate(GoUrl);       while (browser.readystate < 4) do      begin        Sleep(500);        Application.ProcessMessages;      end;       Document := Browser.Document;      Body := Document.Body;      Memo1.Lines.Add(Body.InnerHTML);       Browser.Quit;     except      on E: Exception do ; // eat exception    end;   finally    Browser := Unassigned;    CoUnInitialize;  end;end; I'm not sure if the while loop is enough to make it run the javascript entirely. Otherwise you'll need to build in a delay. You might even keep the window hidden (comment out Visible := true;) ---------------------------------------------------------------------- alaa123456789 · 2020-06-03 17:57 QuoteIt is better to just use a new oleobject (with the full latest IE version). hi RvK thanks for your advise i applied it on my program i got attached capture i tried with CoInitialize and without same result please advise regards Alaa ---------------------------------------------------------------------- alaa123456789 · 2020-06-03 18:16 thank you so much rvk finally i got the link of the book  Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.ListBox1Click(Sender: TObject);var  x1,x2: string;  x3:Variant;  page1:AnsiString;  re2: TRegExpr;  Browser: olevariant;  Document: olevariant;  Body: olevariant;  begin  x1:= ListBox1.Items.Strings[ListBox1.ItemIndex];  x2:= StringReplace(x1,'-e','-d',[rfIgnoreCase,rfReplaceAll]);  x3:=UTF8Encode('yourwebsite'+x2); // url:=UTF8Decode(x3);    CoInitialize(nil);    Browser := CreateOleObject('InternetExplorer.Application');     try    try       Browser.AddressBar := false;      Browser.Menubar := false;      Browser.ToolBar := false;      Browser.StatusBar := false;      Browser.Width := 600;      Browser.Height := 600;      Browser.Left := Screen.Width div 2 - Browser.Width div 2;      Browser.Top := Screen.Height div 2 - Browser.Height div 2;      Browser.Visible := true;       Browser.Navigate(x3);       while (browser.readystate < 4) do      begin        Sleep(20000);        Application.ProcessMessages;      end;       Document := Browser.Document;      Body := Document.Body;      Memo1.Lines.Add(Body.InnerHTML);       Browser.Quit;     except      on E: Exception do ; // eat exception    end;   finally    Browser := Unassigned;    CoUnInitialize;  end; ---------------------------------------------------------------------- rvk · 2020-06-03 18:21 Quote from: alaa123456789 on June 03, 2020, 07:57:36 pm i applied it on my program i got attached capture i tried with CoInitialize and without same result On what OS version are you trying this. What version of Internet Explorer do you have installed? For example IE6 doesn't support jquery and you might get that error. If you run Internet Explorer (so NOT Chrome or other browser) and go to that site... Do you get the same error? Ah. It worked... Good. ---------------------------------------------------------------------- alaa123456789 · 2020-06-05 05:49 hi all , i found another type of links like this /download.pdf?id=158527426&h=2a2e7156d5eb07e0bb5d263b666d9052& u=cache&ext=pdf i tried to download it with simple get it didnt work could you please advise Regards Alaa ---------------------------------------------------------------------- rvk · 2020-06-05 07:38 Quote from: alaa123456789 on June 05, 2020, 07:49:35 ami tried to download it with simple get it didnt work "it didn't work" isn't really useful information and without the actual url itself we can't know what's going on. What didn't work? Does the link work in Internet Explorer? Do you need to confirm something on the webpage? ---------------------------------------------------------------------- alaa123456789 · 2020-06-05 08:00 QuoteDoes the link work in Internet Explorer? yes when you open with chrome or firefox it show save/open dialog to download it QuoteDo you need to confirm something on the webpage? no when i download it with simpleget method i got only 1 kb of it i think the previous link which i share before direct to another link Code: HTML5  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Download ( PDF ) ---------------------------------------------------------------------- rvk · 2020-06-05 08:06 Quote from: alaa123456789 on June 05, 2020, 10:00:33 amno when i download it with simpleget method i got only 1 kb of it It is very possible that the download.pdf is just a simple page which executes javascript or is even a redirect page. In that case SimpleGet won't work (on it's own). But it all depends what's in the 1kb download you do get on that simpleget. Does it have location in the header? Does it have javascript? Again, without full source of that file or complete url it's just guessing from our side. ---------------------------------------------------------------------- trev · 2020-06-05 08:59 It would help if the OP would tell us which website this is - I gave up helping when he didn't. ---------------------------------------------------------------------- alaa123456789 · 2020-06-05 14:39 https://www.pdfdrive.com/ now please help Regards Alaa ---------------------------------------------------------------------- rvk · 2020-06-05 15:59 Quote from: alaa123456789 on June 05, 2020, 04:39:45 pmhttps://www.pdfdrive.com/ now please help As I expected, the \download.pdf URL you have given, results in a 301 code, which is a redirection. If you folow that redirection (with Allowredirect := true) it still gives a 400 error code (Bad Request). There is still something that the server expects and is not given. Cookies should be saved between the redirection. I also tried a different User-agent, but that also didn't work. Will try later on. Code for now: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient, opensslsockets; procedure TForm1.Button1Click(Sender: TObject);var  HTTP: TFPHttpClient;  Stream: TMemoryStream;  URL: string;begin  HTTP := TFPHttpClient.Create(nil);  Stream := TMemoryStream.Create;  try    HTTP.AllowRedirect := true;    HTTP.AddHeader('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36');    URL := 'https://www.pdfdrive.com/download.pdf?id=1 0172273&h=84f0f3490acb0a861ce0cf97be914eed&u=cache&ext=pdf';    try    HTTP.HTTPMethod('GET', URL, Stream, [200, 301, 400]); // 400 shouldn't happen !      Stream.SaveToFile('c:\temp\test.pdf');       // HTTP.Get(URL, 'c:\temp\test.pdf'); // gives a 301 exception       Memo1.Lines.Add(HTTP.ResponseHeaders.Text);    except      on E: Exception do        Memo1.Lines.Add(HTTP.ResponseHeaders.Text);    end;  finally    HTTP.Free;    Stream.Free;  end; end; wget on linux works fine on the link so it should be too difficult. ---------------------------------------------------------------------- rvk · 2020-06-05 16:21 Ok, first I thought maybe the cookies where not saved during redirect. But it turns out the cookies don't matter. BUT... On the first redirect you get an URL with a parameter for the filename. That filename contains spaced. But if you directly use that URL you'll get a 400 error (bad request). It turns out you need to encode the filename (replace the spaces by %20). Wget on Linux, Chrome and IE etc. does this automatically. So the redirect of fphttpclient is somewhat flawed that there is no callback so we can change the redirect URL. So we need to catch the 301 ourselves and encode the redirect correctly. Something like this: (worked for me) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient, opensslsockets; procedure TForm1.Button1Click(Sender: TObject);var  HTTP: TFPHttpClient;  Stream: TMemoryStream;  URL: string;begin  URL := 'https://www.pdfdrive.com/download.pdf?id=1017227 3&h=84f0f3490acb0a861ce0cf97be914eed&u=cache&ext=pdf';  HTTP := TFPHttpClient.Create(nil);  Stream := TMemoryStream.Create;  try    HTTP.AllowRedirect := false;    HTTP.AddHeader('User-Agent', 'Wget/1.20.1 (linux-gnu)');    HTTP.HTTPMethod('GET', URL, Stream, [200, 301]);    if HTTP.ResponseStatusCode = 301 then    begin      URL := HTTP.GetHeader(HTTP.ResponseHeaders,'Location');      URL := StringReplace(URL, ' ', '%20', [rfReplaceAll]); // IMPORTANT      HTTP.HTTPMethod('GET', URL, Stream, [200]);    end;    Stream.SaveToFile('c:\temp\test.pdf');  finally    HTTP.Free;    Stream.Free;  end;end; ---------------------------------------------------------------------- alaa123456789 · 2020-06-05 16:25 does this work on windows ? ---------------------------------------------------------------------- rvk · 2020-06-05 16:32 Quote from: alaa123456789 on June 05, 2020, 06:25:55 pmdoes this work on windows ? The procedure i showed works on Windows, yes (i made it on Windows). ---------------------------------------------------------------------- alaa123456789 · 2020-06-05 17:32 thanks RVK it worked , Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i baseurl:=('https://www.pdfdrive.com/search?q=' + Edit1.Text +'&pagecount=&pubyear=&searchin=&page=');   With TFPHttpClient.Create(Nil) do    try      page :=Get(baseUrl);// Find all book urls    finally     free;    end;               i am trying to allow user to type any keyword to search for the books in it but i see the web site has different methodes like in captures so how to figure out which link i sould use to extract books names thanks ---------------------------------------------------------------------- rvk · 2020-06-05 21:06 Quote from: alaa123456789 on June 05, 2020, 07:32:45 pmso how to figure out which link i sould use to extract books names It seems that for some search-terms, the result is xxxxx-book.html. But I don't think it matters much. https://www.pdfdrive.com/search?q=python&pagecount=&pubyear=&searchin= &more=true gives about the same result as https://www.pdfdrive.com/python-books.html And if you do something like this https://www.pdfdrive.com/somethingelse-books.html you end up on this url https://www.pdfdrive.com/search?q=somethingelse&pagecount=&pubyear=&se archin=&more=true (Just make sure allowredirect := true so a redirect will be followed if it's given.) ---------------------------------------------------------------------- alaa123456789 · 2020-06-07 17:03 hi Quotehttps://www.pdfdrive.com/search?q=somethingelse&pagecount=&pubyea r=&searchin=&more=true i have done this before but i got error when you are searching for more than word as example " visual basic " or visual basic 6 "  i got error Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  ListBox1.Items.Clear;  baseurl:=('https://www.pdfdrive.com/search?q=' + Edit1.Text +'&pagecount=&pubyear=&searchin=&more=true');  http1:=TFPHttpClient.Create(Nil);  With http1 do    try      http1.AllowRedirect:=true;      page :=http1.SimpleGet(baseUrl);// Find all book urls    finally     free;    end; ---------------------------------------------------------------------- alaa123456789 · 2020-06-07 18:31 Quote from: alaa123456789 on June 05, 2020, 06:25:55 pmdoes this work on windows ? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.downloadBook(bookname: String);var  HTTP: TFPHttpClient;  Stream: TMemoryStream;begin  HTTP := TFPHttpClient.Create(nil);  Stream := TMemoryStream.Create;  try    HTTP.AllowRedirect := false;  HTTP.AddHeader('User-Agent', 'Wget/1.20.1 (linux-gnu)');    HTTP.HTTPMethod('GET', bookname, Stream, [200, 301]);    if HTTP.ResponseStatusCode = 301 then    begin      bookname  := HTTP.GetHeader(HTTP.ResponseHeaders,'Location');      bookname := StringReplace(bookname, ' ', '%20', [rfReplaceAll]); // IMPORTANT    HTTP.HTTPMethod('GET', bookname, Stream, [200]);    end;    Stream.SaveToFile(targetDirectory+'test.pdf');  finally    HTTP.Free;    Stream.Free;  end;   end;                    it is not downloading , i dont know what is the issue ---------------------------------------------------------------------- rvk · 2020-06-07 20:17 Quote from: alaa123456789 on June 07, 2020, 07:03:47 pmhi Quotehttps://www.pdfdrive.com/search?q=somethingelse&pagecount=&pubyea r=&searchin=&more=true i have done this before but i got error when you are searching for more than word as example " visual basic " or visual basic 6 "  i got error You need to urlencode the edit1.text before adding ut in the url. For example... an url may not contain spaces, slashes, ampersand (&) etc. For now you mentioned spaces. Replace those %20 (the same as I did with the redirect url). There is a routine in synapse which does this for you but I can't remember which. (I think encodeurl() in synacode unit) Edit: ah, you already found it. But it's best to use the encodeurl so other characters like / and & are also correctly encoded. Quote from: alaa123456789 link=topic=49927.msg365197#msg365197it is not downloading , i dont know what is the issue What error are you getting? Are you passing the complete url? Because the first get you only do on bookname. If bookname is just only the name of the book or search you need to put it in a complete url. ---------------------------------------------------------------------- alaa123456789 · 2020-06-08 16:43 hi QuoteYou need to urlencode the edit1.text before adding ut in the url. For example... an url may not contain spaces, slashes, ampersand (&) etc. For now you mentioned spaces. Replace those %20 (the same as I did with the redirect url). please see attached capture i got error 400 . QuoteWhat error are you getting? Are you passing the complete url? Because the first get you only do on bookname. If bookname is just only the name of the book or search you need to put it in a complete url.i am adding the same url i shared with you before i.e download.pdf?id=158527426&h=2a2e7156d5eb07e0bb5d263b666d9052&u =cache&ext=pdf Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i x4:=('https://www.pdfdrive.com/'+ re2.Match[1]);             downloadBook(x4); i got only 1kb this what i am facing ---------------------------------------------------------------------- rvk · 2020-06-08 16:51 Quote from: alaa123456789 on June 08, 2020, 06:43:58 pm please see attached capture i got error 400 . In the 1kb file you'll see a bad request. That means the server doesn't understand the request. For me it was that spaces are not interpreted correctly. Try a search without spaces and without special characters. How does the baseurl look like before the simpleget? ---------------------------------------------------------------------- TRon · 2020-06-08 17:26 For a 'download' url the site seem to want you to comply to default url standards, for a search it seems the site expects spaces (that you've put up at the search bar) to be converted to plus a sign (whether you quote your search term or not) In that respect it is not exactly rocket-science  , you can research by attempting every possible input that you can think of and, see what the server code is doing to the url, you would have to mimic that behaviour in your own code. ---------------------------------------------------------------------- alaa123456789 · 2020-06-08 17:34 QuoteIn the 1kb file you'll see a bad request.please see capture7 QuoteFor me it was that spaces are not interpreted correctly.please see capture8 QuoteIn that respect it is not exactly rocket-science  , you can research by attempting every possible input that you can think of and, see what the server code is doing to the url, you would have to mimic that behaviour in your own code.i am new in lazarus , i am trying my best , i am working on this project since a month and trying step by step but i am facing challenges as i am not familiar with all methode but i put in my mind i will overcome this challenge thanks for everyone support me ---------------------------------------------------------------------- rvk · 2020-06-08 17:37 You used utf8encode. That does nothing for spaces. I mentioned encodeurl in the synacode unit. That will convert spaces to %20 or + (not sure which). Your baseurl still has spaces in it so it fails. ---------------------------------------------------------------------- TRon · 2020-06-08 17:45 Quote from: alaa123456789 on June 08, 2020, 07:34:06 pmi am new in lazarus , i am trying my best , i am working on this project since a month and trying step by step but i am facing challenges as i am not familiar with all methode My remark was not meant as a (negative) comment on your coding skills, or lack thereof because you are unfamiliar with Lazarus/pascal. Take you second screenshot (the search). you searched for visual basic by using your edit box. Now open your browser and do that same search manually. Despite that the website 'catches' common searches you can force the website to actually perform a search (instead of showing a cached page). Then you can see that the website turns the search term into: Code: [Select]https://www.pdfdrive.com/search?q=%22visual+basic%22&pag ecount=&pubyear=&searchin= The %22 represent the quotes, so you can omit those (i manually added those to circumvent the cached page). so that the url you need to use for searching for visual basic becomes: Code: [Select]https://www.pdfdrive.com/search?q=visual+basic&pagecount =&pubyear=&searchin= I'm positive that that website has more things up it sleeves but, for now i was unable to detect them. Give that plus sign a try ---------------------------------------------------------------------- rvk · 2020-06-08 17:51 Quote from: TRon on June 08, 2020, 07:45:04 pmGive that plus sign a try  Both a plus sign and %20 will work for spaces. It's just that real spaces don't work. And even if there are spaces in an url, every browser converts them before sending the url. Using the encodeurl (or any proper url encoder function) will work for other characters too. ---------------------------------------------------------------------- TRon · 2020-06-08 17:59 @rvk, I have not claimed that it wouldn't. I just crossed your post (i was editing while you posted yours for that i apologise as that interfered with your support. It was not my intention to mingle with your business, but i probably skimmed to quickly over the thread as i was unable to locate you mentioning urlencode/decode earlier . Using that should be good enough. In fact i wouldn't want to have it any other way    (i like to see for myself what the website does itself first, as urlencode/decode is just a black box to me) ---------------------------------------------------------------------- rvk · 2020-06-08 18:09 Quote from: TRon on June 08, 2020, 07:59:28 pmIt was not my intention to mingle with your business, but i probably skimmed to quickly over the thread as i was unable to locate you mentioning urlencode/decode earlier . O, I don't consider it mingling. Everybody can pitch in  The more the better  I even didn't think about the + signs at first. Sometimes they are converted to +. Not sure when %20 is used and when +. For the redirect from the site itself, the filenames also contain spaces in the location header (which I found weird). For that one, manually replacing might be sufficuent and even needed. Because if the filename contains a special characters and that one is already encoded, you don't want to double encode. For the edit1.text you probably do want the encodeurl because there you can enter special characters like %, & etc. ---------------------------------------------------------------------- TRon · 2020-06-08 18:24 Quote from: rvk on June 08, 2020, 08:09:43 pmO, I don't consider it mingling. Everybody can pitch in  The more the better  Ok, then  QuoteI even didn't think about the + signs at first. Sometimes they are converted to +. Not sure when %20 is used and when +. tbh, i am not sure either. I am positive i once read about it somewhere in a RFC and then forgot all about it again :-) QuoteFor the redirect from the site itself, the filenames also contain spaces in the location header (which I found weird). For that one, manually replacing might be sufficuent and even needed. Because if the filename contains a special characters and that one is already encoded, you don't want to double encode. Indeed the website does some non-standard things, as an other example look how it interpreted the quotes. That is pretty uncommon. QuoteFor the edit1.text you probably do want the encodeurl because there you can enter special characters like %, & etc. I fully agree. At least this way TS now knows why he has to use it (or something similar). If you try to do that encoding manually... *beh* And for sure you never know what the user enters in there ---------------------------------------------------------------------- alaa123456789 · 2020-06-08 18:46 hey guys , this subject is open for all to input his knowledge everyone is welcomed,  at the end we need everyone to learn and share experience i tried to encodeurl() from synacode and it worked for search still we have issue with download book actually this website have a lot to learn about web scraping and we trying to learn how to resolve all these challenges thanks Alaa ---------------------------------------------------------------------- rvk · 2020-06-09 11:00 Quote from: alaa123456789 on June 08, 2020, 08:46:43 pmstill we have issue with download book What issues? I showed some code already where the download worked for me. ---------------------------------------------------------------------- alaa123456789 · 2020-06-10 16:48 QuoteWhat issues? it doesnt show any error , only file 1kb , i tried to use urlmon also same thanks alaa ---------------------------------------------------------------------- rvk · 2020-06-10 16:58 Quote from: alaa123456789 on June 10, 2020, 06:48:05 pmQuoteWhat issues?it doesnt show any error , only file 1kb , i tried to use urlmon also same Which probably means bad request (you can see that if you open the file in a texteditor). What is your code now and what url do you use? ---------------------------------------------------------------------- alaa123456789 · 2020-06-10 17:11 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.downloadBook(bookname: String);var  HTTP: TFPHttpClient;  Stream: TMemoryStream;begin  HTTP := TFPHttpClient.Create(nil);  Stream := TMemoryStream.Create;  try    HTTP.AllowRedirect := false;  HTTP.AddHeader('User-Agent', 'Wget/1.20.1 (linux-gnu)');    HTTP.HTTPMethod('GET', bookname, Stream, [200, 301]);    if HTTP.ResponseStatusCode = 301 then    begin      bookname  := HTTP.GetHeader(HTTP.ResponseHeaders,'Location');      bookname := StringReplace(bookname, ' ', '%20', [rfReplaceAll]); // IMPORTANT    HTTP.HTTPMethod('GET', bookname, Stream, [200]);    end;    Stream.SaveToFile(targetDirectory+'test.pdf');  finally    HTTP.Free;    Stream.Free;  end;end;bookname= https://www.pdfdrive.com/download.pd f?id=158527426&h=2a2e7156d5eb07e0bb5d263b666d9052&u=cache& ext=pdf regards Alaa ---------------------------------------------------------------------- rvk · 2020-06-10 17:18 That url gives" Sorry, this file is not available Go back." How did you get that url? Because I see & in the url which shouldn't be there. You probably used encodeurl which you shouldn't do on an url you got directly from the site. You only use it on the edit1.text part for a search url. ---------------------------------------------------------------------- alaa123456789 · 2020-06-10 17:23 try the one in this capture ---------------------------------------------------------------------- rvk · 2020-06-10 17:27 Quote from: alaa123456789 on June 10, 2020, 07:23:35 pmtry the one in this capture I can't right now (on mobile) but I still see & in the url. How did you get the url??? ---------------------------------------------------------------------- alaa123456789 · 2020-06-10 17:44 please see capture 10 why it is adding "amp;" while it is not there in htmlcode Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i re2:=TRegExpr.Create('(download.pdf.*?ext=pdf)');      if re2.Exec(page1) then          begin              x4:=('https://www.pdfdrive.com/'+ re2.Match[1]); maybe this the error ---------------------------------------------------------------------- TRon · 2020-06-10 17:48 @rvk/alaa123456789 I can confirm that the html ampersand encoding is the culprit. With those in place a browser is not able to 'correct' it which results in a file not found message. If you replace those with real ampersands the download works for me. ---------------------------------------------------------------------- alaa123456789 · 2020-06-10 17:56 i replaced all 'amp;' and it worked ---------------------------------------------------------------------- rvk · 2020-06-10 17:58 Quote from: alaa123456789 on June 10, 2020, 07:44:41 pmplease see capture 10 why it is adding "amp;" while it is not there in htmlcode Can you look directly in page1 variable? Is it there? Are you using x4 directly or is there an additional conversion? ---------------------------------------------------------------------- alaa123456789 · 2020-06-10 18:05 no in page itself it is not showing but once i parse it , it add 'amp;' another question ,how i get 'e158445708' form 'https://www.pdfdrive.com/hands-on-data-science-and-python-machine- learning-perform-data-mining-and-machine-learning-efficiently-using- python-and-spark-e158445708.html' ---------------------------------------------------------------------- TRon · 2020-06-10 18:34 Quote from: alaa123456789 on June 10, 2020, 08:05:01 pmanother question ,how i get 'e158445708' form 'https://www.pdfdrive.com/hands-on-data-science-and-python-machine- learning-perform-data-mining-and-machine-learning-efficiently-using- python-and-spark-e158445708.html' 1) ExtractFilename, ChangeFileExt (to empty) and then 'catch' all characters from the back till front until you reach the '-' character. 2) split the string on '-' and '.' characters, then take the second for last item from the result or In your first search page, you find the href, there you find a tag named DATA-ID ---------------------------------------------------------------------- alaa123456789 · 2020-06-10 18:50 @ tron please set an example i gave the string thanks Alaa ---------------------------------------------------------------------- TRon · 2020-06-10 19:05 Quote from: alaa123456789 on June 10, 2020, 08:50:19 pm@ tron please set an example i gave the string I have no idea how you obtained the URL for the books from your search results. I used a custom (HTML) parser for that, and before you ask, i am not at liberty to share that part of my code. Then the best/easiest option would be the split, for that see https://www.freepascal.org/docs- html/rtl/sysutils/tstringhelper.split.html Now i gave you a detailed description on how to approach that and show you the page that explains how the string helper split function works *returns-ball* ---------------------------------------------------------------------- rvk · 2020-06-10 19:40 Quote from: alaa123456789 on June 10, 2020, 08:50:19 pmplease set an example i gave the string Why don't you follow the instruction to extract data-id? It's much easier. I was trying to make a simple example but I'm stumbling on a problem with TRegExpr... Does that one not follow the non-greedy rule for (.*?) ?? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);var  URL: string;  EncodedSearch: string;  Page1: string;  RE: TRegExpr;begin  EncodedSearch := EncodeUrl('Lazarus books');  URL := Format('https://www.pdfdrive.com/s earch?q=%s&pagecount=&pubyear=&searchin=', [EncodedSearch]);  Page1 := TFPCustomHTTPClient.SimpleGet(URL);   //with TStringStream.Create(Page1) do  //begin  //  SaveToFile('c:\temp\test.txt');  //  Free;  //end;   //    RE := TRegExpr.Create('');  try    if RE.Exec(Page1) then      repeat        Memo1.Lines.Add(RE.Match[1] + ' / id = ' + RE.Match[2]);        Memo1.Lines.Add('---');      until not RE.ExecNext;  finally    RE.Free;  end;end; Gives way too much. ---------------------------------------------------------------------- TRon · 2020-06-10 19:45 @rvk, isn't regexpr greedy  by default ?, e.g. you need to set it to non greedy first ? well, it seems that it is not even possible to copy-paste some faulty code and ask for help where one get stuck  Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program splitter;{$MODE OBJFPC}{$H+}uses  sysutils;var  URL: string= 'https://www.pdfdrive.com/hands-on-data-science-and-python-machine- learning-perform-data-mining-and-machine-learning-efficiently-using- python-and-spark-e158445708.html';  TalkingWords: TStringArray;  WantedDeadOrAlive: String= '';  Meet101: boolean= true;begin  // split up the URL into individual parts/words  TalkingWords:= URL.Split(['-','.']);    // If there are enough words  if Length(TalkingWords)> 1 then     // and if we want to meet 101    if Meet101       // then take second for last word      then WantedDeadOrAlive:= TalkingWords[High(TalkingWords)-1]    // and in case we do not want to meet 101    else WantedDeadOrAlive:= TalkingWords[High(TalkingWords)-1].Remove(0,1);   WriteLn('WantedDeadOrAlive = ', WantedDeadOrAlive.QuotedString);end. ---------------------------------------------------------------------- rvk · 2020-06-10 20:01 Quote from: TRon on June 10, 2020, 09:45:09 pmisn't regexpr greedy  by default ?, e.g. you need to set it to non greedy first ?Yes, regexpr is greedy by default but when you use the ? modifier, it should stop being greedy. So . matches any character, * multiple characters, but the ? should make it so that if you encounter the next characters (i.e. " data-id=" etc) the match is checked directly. https://regexr.com/ does this correctly. But TRegExpr gives back way too much. Can anyone fix my example to just only match As a correct example: https://regexr.com/56fv5 and https://regex101.com/r/cHyiXi/1 ---------------------------------------------------------------------- TRon · 2020-06-10 20:41 @rvk: I am sooo terrible at regexpr, so it took me a while  It is in the modifiers. S is turned on by default, and if I understand correctly that means that using "." spans cross lines ? If you turn that modifier off then your expression seems to work as you had intended. edit: the proof is in the pudding (eg, I should have read the faq first): https://regex.sorokin.engineer/en/latest/faq.html#why-does- tregexpr-return-more-then-one-line ---------------------------------------------------------------------- rvk · 2020-06-10 21:17 Quote from: TRon on June 10, 2020, 10:41:36 pmIt is in the modifiers. S is turned on by default, and if I understand correctly that means that using "." spans cross lines ?Yes. That fixes it. But it is still incorrect. I should be able to do Multi-line matches and with the ? modifer it should not match beyond what I give it (even if it is in multiline- mode) I tried the same code in Delphi RegularExpression unit, and there it worked directly. Anyway... this is the code I have so far. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient, opensslsockets, synacode, RegExpr; procedure TForm1.Button1Click(Sender: TObject);var  URL: string;  EncodedSearch: string;  Page1: string;  RE: TRegExpr;begin  EncodedSearch := EncodeUrl('Lazarus books');  URL := Format('https://www.pdfdrive.com/s earch?q=%s&pagecount=&pubyear=&searchin=', [EncodedSearch]);  Page1 := TFPCustomHTTPClient.SimpleGet(URL);  / /with TStringStream.Create(Page1) do  // begin  //   SaveToFile('c:\temp\test.txt');  //   Free;  // end;  //   RE := TRegExpr.Create('');  try    RE.ModifierS := false;    if RE.Exec(Page1) then      repeat        Memo1.Lines.Add(RE.Match[1] + ' / id = ' + RE.Match[2]);      until not RE.ExecNext;  finally    RE.Free;  end;end; Result: Quote/love-vs-marriage-e30723996.html / id = 30723996 /passion-and-reason-making-sense-of-our-emotions-e184457288.html / id = 184457288 /sweet-mary-jane-75-delicious-cannabis-infused-high-end- desserts-e188174652.html / id = 188174652 /coping-with-aging-e185443483.html / id = 185443483 /the-golden-book-of-chemistry-experiments-how-to-set-up-a-home- laboratory-over-200-simple-experiments-e158547579.html / id = 158547579 /concise-guide-to-hematology-e187221987.html / id = 187221987 /stress-appraisal-and-coping-e190017304.html / id = 190017304 /false-prophet-the-peter-decker-and-rina-lazarus-series- book-05-1992-e186115133.html / id = 186115133 /stalker-the-peter-decker-and-rina-lazarus-series- book-12-2000-e164811508.html / id = 164811508 /prayers-for-the-dead-the-peter-decker-and-rina-lazarus-series- book-09-1996-e185158478.html / id = 185158478 /grievous-sin-the-peter-decker-and-rina-lazarus-series- book-06-1993-e161188770.html / id = 161188770 /the-golden-book-of-chemistry-experiments-how-to-set-up-a-home- laboratory-over-200-simple-experiments-e157315299.html / id = 157315299 /the-burnt-house-the-peter-decker-and-rina-lazarus-series- book-16-2007-e186110366.html / id = 186110366 /the-forgotten-the-peter-decker-and-rina-lazarus-series- book-13-2000-e167725307.html / id = 167725307 /sacred-and-profane-the-peter-decker-and-rina-lazarus-series- book-02-1987-e185064090.html / id = 185064090 /sanctuary-the-peter-decker-and-rina-lazarus-series- book-07-1994-e185185365.html / id = 185185365 /serpents-tooth-the-peter-decker-and-rina-lazarus-series- book-10-1997-e185572570.html / id = 185572570 /the-ritual-bath-the-peter-decker-and-rina-lazarus-series- book-01-1986-e184206485.html / id = 184206485 /black-lazarus-conjure-book-e30430807.html / id = 30430807 /the-x-files-faq-all-thats-left-to-know-about-global-conspiracy- aliens-lazarus-species-and-monsters-of-the-week-e195026287.html / id = 195026287 I found a possible way to download the books without executing javascript. For now... it's bedtime ---------------------------------------------------------------------- TRon · 2020-06-10 21:54 Quote from: rvk on June 10, 2020, 11:17:10 pmYes. That fixes it. But it is still incorrect. I should be able to do Multi-line matches and with the ? modifer it should not match beyond what I give it (even if it is in multiline- mode) That is true, but tbh i seriously lack in that department so have no idea what author had intended. QuoteAnyway... this is the code I have so far. Nice ! QuoteI found a possible way to download the books without executing javascript. For now... it's bedtime  I got stuck in the end. Although i cannot show the parsing code, i certainly can share some of my crappy outcomes: Code: [Select]Title : A Python Book: Beginning Python, Advanced Python, and Python    ID : 9236005    cc : 1412313ed6c2dc776f148dc730ad7b82 Title : Learn Python in One Day and Learn It Well: Python for Beginners with Hands-on Project. The only book you need to start coding in Python immediately    ID : 183833259    cc : 44c75150d2524b17919d1dcf02c483ba Title : Python Programming.  Python Programming for Beginners, Python Programming for Intermediates    ID : 180663309    cc : f03acfa6e2959196a6aa252445b4c6c4 Title : Gray hat Python : Python programming for hackers and reverse    ID : 9135313    cc : 1e3fb1af9b6ec585d23f37989f5d4d65 Title : automate the boring stuff with python automate the boring stuff with python    ID : 26956384    cc : 1e26b49538fa00558f742f66cbcec016 Title : Python 2.6 Graphics Cookbook - Python Programming    ID : 18274634    cc : f2b25b98e4b751b25c655ede7ef87ce5 Title : A Python Book: Beginning Python, Advanced Python, and Python Exercises    ID : 125280    cc : f9a9d4354f989d49f7b31d1ab5681421 Title : Python Tricks: A Buffet of Awesome Python Features    ID : 158320268    cc : 12bb9c8c867707bfe15a5a4e017185d7 Title : Hands-on data science and Python machine learning : perform data mining and machine learning efficiently using Python and Spark    ID : 158445708    cc : 3ed4c56ab7fb5cf0b290e1ccea16cfff Title : Python Data Analytics: Data Analysis and Science Using Pandas, matplotlib, and the Python    ID : 158003322    cc : 6b11804596aef6423d5dcb4d4f03b364 Title : The Python Book: The ultimate guide to coding with Python    ID : 175482973    cc : b88965eea6d2a7514358e16c34a3be80 Title : Mastering Machine Learning with Python in Six Steps: A Practical Implementation Guide to Predictive Data Analytics Using Python    ID : 168776616    cc : d70605c8d404b6303b9275ff2a7e2502 Title : Black hat Python : Python programming for hackers and pentesters    ID : 158527426    cc : 6883ddb073a2db8ea0acf51978e51e8a Title : Python Crash Course.pdf    ID : 33417142    cc : 6f669a84135609d35bd72209e397a700 Title : Python Programming: An Introduction to Computer Science    ID : 941324    cc : 435ebf44f65ab314be717cc2169c553a Title : The Python Book: The Ultimate Guide to Coding with Python    ID : 175899494    cc : 37008ce32466d0505306a3bd48923236 Title : Data Analysis From Scratch With Python: Beginner Guide using Python, Pandas, NumPy, Scikit-Learn, IPython, TensorFlow and Matplotlib    ID : 188610626    cc : bd4e122c2de48657b7c9e9487ae6406f Title : Data Structures and Algorithms in Python    ID : 25119593    cc : c4ad9ce27323a0988e339034f7c37330 Title : Using Python for machine learning    ID : 18769371    cc : 60ffc7794206f3514eb2c76c0ba07c60 Title : Python Tutorial - Tutorials Point    ID : 10195863    cc : d5d7ecb45f3f936b8362c533e379f478 I refrained from outputting the actual URL but it is present in the itemlist. Also I opted for the default result page that outputs 20 entries as the point was already mute for me at that time. I got stuck (avoiding js) as I wasn't able to match/combine cc with cookie to end up at the actual location. I must be overlooking something (obviously) but perhaps you have better luck  In case you do succeed then it would perhaps be worth it to implement a sax or some other parser or perhaps using internet tools from BeniBela. I have no idea if there are better options out there. Sleep well and thx for the feedback ---------------------------------------------------------------------- rvk · 2020-06-11 09:49 This is what I have now (see attached screenshot and project). (it saves the pdf's to c:\temp) It could be made much better. For example detecting if there is a epub instead of a pdf. Same books.pdf still have html-text in them (for example NOT FOUND html page) but I saw that those books give the same error on the site itself. For now it was just proof of concept. Simple scraping via TRegExpr and downloading files. ---------------------------------------------------------------------- alaa123456789 · 2020-06-11 15:37 wow , i also made my program i will share the link to download it later i tried your software and had error , how to solve please see capture ---------------------------------------------------------------------- alaa123456789 · 2020-06-11 16:01 please download from here and give your comments  https://drive.google.com/file/d/1TZRaTrxsLDY037xSFYpxwxk4fST_5XXC/vie w?usp=sharing thanks for everyone helped ---------------------------------------------------------------------- rvk · 2020-06-11 19:28 Quote from: alaa123456789 on June 11, 2020, 05:37:35 pmwow , i also made my program i will share the link to download it later i tried your software and had error , how to solve That's strange. I thought opensslsockets was part of FPC. You could try changing it to fpopenssl or openssl. Quote from: alaa123456789 on June 11, 2020, 06:01:46 pmplease download from here and give your comments I can't comment on your version. It is an .exe file and it wouldn't run in Sandboxie (it gave an error "Error with security token") and I won't run a download .exe from the internet. ---------------------------------------------------------------------- TRon · 2020-06-11 19:44 Quote from: rvk on June 11, 2020, 09:28:32 pmThat's strange. I thought opensslsockets was part of FPC. it is, at least for rc 3.2 (haven't tried 3.04) ---------------------------------------------------------------------- alaa123456789 · 2020-06-11 20:09 QuoteThat's strange. I thought opensslsockets was part of FPC. You could try changing it to fpopenssl or openssl.yes replaced and now ok ,but you have still error when i change 'lazarus books' with edit1.text or with another text it will get an error try it  and check ---------------------------------------------------------------------- rvk · 2020-06-11 20:45 Quote from: alaa123456789 on June 11, 2020, 10:09:30 pmQuoteThat's strange. I thought opensslsockets was part of FPC. You could try changing it to fpopenssl or openssl.yes replaced and now ok ,but you have still error when i change 'lazarus books' with edit1.text or with another text it will get an error Oops. Yes, that line should be (I forgot it because I put the TEdit in last) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i EncodedSearch := EncodeUrl(Edit1.Text); Quotetry it  and check WHAT error are you getting???? ALWAYS be specific !! Edit: Now I see, if I use visual basic, I get a redirect error 301 at the first scrape page. So even for that page the redirect should be implemented. (Why is AllowRedirect not true for SimpleGet() ???) Anyway, change the following 2 procedures with a HTTP which allows redirection: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.DownloadPageForBook(URL, ShortCode: string);var  Page1: string;  RE: TRegExpr;  HTTP: TFPHttpClient;begin   // Page1 := TFPCustomHTTPClient.SimpleGet(URL);  HTTP := TFPHttpClient.Create(nil);  try    HTTP.AllowRedirect := true;    Page1 := HTTP.Get(URL);  finally    HTTP.Free;  end;   // data-preview ="/ebook/preview?id=184457288&session=e700a8a1d9bbe33a0aefdc86adfbf357 " data-download-page=  RE := TRegExpr.Create('data- preview="/ebook/preview\?id=' + ShortCode + '&session=(.*?)" data');  try    RE.ModifierS := False;    if RE.Exec(Page1) then    begin      Memo1.Lines.Add('Preview ' + RE.Match[1]);      URL := 'https://www.pdfdrive.com/download.pdf?id=' + ShortCode;      URL := URL + '&h=' + RE.Match[1] + '&u=cache&ext=pdf';      DownloadBook(URL, ShortCode);    end    else      Memo1.Lines.Add('XXXXXXXX NO PREVIEW FOUND');  finally    RE.Free;  end;end;Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.btnScrapeClick(Sender: TObject);var  BaseURL, URL: string;  Page: integer;  EncodedSearch: string;  Page1: string;  RE: TRegExpr;  HTTP: TFPHttpClient;begin  Memo1.Clear;  EncodedSearch := EncodeUrl(Edit1.Text);  BaseURL := Format('https://www.pdfdrive.com/se arch?q=%s&pagecount=&pubyear=&searchin=', [EncodedSearch]);  Page := 0;  TerminateScrape := False;  Memo1.Lines.Add('Scraping for ' + EncodedSearch);   repeat    Inc(Page);    URL := BaseURL;    if Page > 1 then      URL := URL + '&page=' + IntToStr(Page);     // Page1 := TFPCustomHTTPClient.SimpleGet(URL);    HTTP := TFPHttpClient.Create(nil);    try      HTTP.AllowRedirect := true;    Page1 := HTTP.Get(URL);    finally      HTTP.Free;    end;     //     RE := TRegExpr.Create('');    try      RE.ModifierS := False;      if RE.Exec(Page1) then        repeat        Memo1.Lines.Add('----------------------------------');          Memo1.Lines.Add('Match ' + 'https://www.pdfdrive.com/' + RE.Match[1]);          DownloadPageForBook('https://www.pdfdrive.com/' + RE.Match[1], RE.Match[2]);          Application.ProcessMessages;      until not RE.ExecNext or TerminateScrape;    finally      RE.Free;    end;     if not TerminateScrape and (Pos('rel="next">Next', Page1) > 0) then    begin      Memo1.Lines.Add('To the next page ' + IntToStr(Page + 1));    end    else      TerminateScrape := True   until TerminateScrape;   Memo1.Lines.Add('----------------------------------');  Memo1.Lines.Add('Scraping stopped'); end; ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 07:01 QuoteWHAT error are you getting???? ALWAYS be specific !! i dont know how to get error number i jsut get a message breakout or continue QuoteAnyway, change the following 2 procedures with a HTTP which allows redirection i changed them as per your code still showing same error for any word you type in edit1 thanks ---------------------------------------------------------------------- rvk · 2020-06-12 07:46 Quote from: alaa123456789 on June 12, 2020, 09:01:53 amQuoteWHAT error are you getting???? ALWAYS be specific !! i dont know how to get error number i jsut get a message breakout or continue Then I still don't know what that message says exactly. It doesn't just say breakout or continue !! Click the message, press CTRL + C and then CTRL + V in a post. And when you get the message, choose break, choose View > Debug Windows > Call Stack and look for the first unit of your own program. Double click that line and you are on the exact line that causing the error. ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 07:53 yes it is taking me to line when i press break but the error number you are asking is not showing anyway the error is coming from Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Page1 := HTTP.Get(URL);       just try to run your code and try different words and you will see this error by the way how did you get out of java script and get direct file link>>> thanks ---------------------------------------------------------------------- rvk · 2020-06-12 08:46 Quote from: alaa123456789 on June 12, 2020, 09:53:27 amyes it is taking me to line when i press break but the error number you are asking is not showing anyway the error is coming from Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Page1 := HTTP.Get(URL);       just try to run your code and try different words and you will see this error by the way how did you get out of java script and get direct file link>>> thanks Your STILL not providing the exception text. I can't reproduce it here. WHAT exact exception are you getting. I already said you can click the exception message and press CTRL+C to copy the message to the clipboard. You get something like this: Quote[Debugger Exception Notification] Project simple_scrape raised exception class 'EHTTPClient' with message: Unexpected response status code: 301  In file '.\fcl-web\src\base\fphttpclient.pp' at line 1137: Raise EHTTPClient.CreateFmt(SErrUnexpectedResponse,[ResponseStatusCode]); [Ignore this exception type] [Break] [Continue] I bypassed the javascript by making the direct URL for the download. The only variable factor is the number (which is in the links itself and data-id) and a h parameter. The h parameter is the same as in the preview link on the first page. (Look at the code how I got it out of the html) ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 09:03 Exception class "EHTTPClient" at $00000001001A7FD7 with message "Invalid protocol : """ direct link of the file does that work for all books? as there are some books have direct link THanks ---------------------------------------------------------------------- rvk · 2020-06-12 09:17 Ah, that's better. If you get this error directly and none of the other words work then I still think it has something to do with openssl. Otherwise with which word does this error occur? I don't know if it works for all books. I also don't know what happens if a book is not in cache on the site (because the link I make contains chache). You can always fallback to IE window if the download fails. It wouldn't work for files not hosted on pdfdrive itself. ---------------------------------------------------------------------- TRon · 2020-06-12 09:23 I noticed something when toying around on that website, but it might be a wrong assumption on my part. (so take it with a grain of salt). Some books are hosted on their own server, others point to third party websites. It wouldn't surprise me if some of those third party's are hosting f.e. on a ftp site. ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 10:07 Quote from: rvk on June 12, 2020, 11:17:44 amAh, that's better. If you get this error directly and none of the other words work then I still think it has something to do with openssl. Otherwise with which word does this error occur? any word you type i.e " python" , " visual basic" ...etc you will get same error could you please share example of full length of book link you got with regex thanks ---------------------------------------------------------------------- rvk · 2020-06-12 10:17 Quote from: alaa123456789 on June 12, 2020, 12:07:14 pmQuote from: rvk on June 12, 2020, 11:17:44 amAh, that's better. If you get this error directly and none of the other words work then I still think it has something to do with openssl. Otherwise with which word does this error occur? any word you type i.e " python" , " visual basic" ...etc you will get same error And you get this error directly for the first download? In which procedure was that Get()? You also get it for the default "lazarus books" ? Anobody know what to include for openssl in fpc 3.0.4? ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 10:34 QuoteAnd you get this error directly for the first download? In which procedure was that Get()? in the first procedure scrape button Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i could you please share example of full length of book link you got with regexplease share answer ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 10:37 Quote from: TRon on June 10, 2020, 09:45:09 pm@rvk, isn't regexpr greedy  by default ?, e.g. you need to set it to non greedy first ? well, it seems that it is not even possible to copy-paste some faulty code and ask for help where one get stuck  Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program splitter;{$MODE OBJFPC}{$H+}uses  sysutils;var  URL: string= 'https://www.pdfdrive.com/hands-on-data-science-and-python-machine- learning-perform-data-mining-and-machine-learning-efficiently-using- python-and-spark-e158445708.html';  TalkingWords: TStringArray;  WantedDeadOrAlive: String= '';  Meet101: boolean= true;begin  // split up the URL into individual parts/words  TalkingWords:= URL.Split(['-','.']);    // If there are enough words  if Length(TalkingWords)> 1 then     // and if we want to meet 101    if Meet101       // then take second for last word      then WantedDeadOrAlive:= TalkingWords[High(TalkingWords)-1]    // and in case we do not want to meet 101    else WantedDeadOrAlive:= TalkingWords[High(TalkingWords)-1].Remove(0,1);   WriteLn('WantedDeadOrAlive = ', WantedDeadOrAlive.QuotedString);end. @tron ,how to get book name without '-' and '-e3213123.html' thanks ---------------------------------------------------------------------- rvk · 2020-06-12 10:41 Quote from: alaa123456789 on June 12, 2020, 12:34:49 pmQuoteAnd you get this error directly for the first download? In which procedure was that Get()? in the first procedure scrape button Then there is something wrong with how you used openssl. (I take it you have it with all searches, as you again did not specify) QuoteQuotecould you please share example of full length of book link you got with regexplease share answer Can't right now by because I'm only on a mobile. (And as I said... The full download link I assemble myself from the data-id and h=some part of the preview button) And the download filename I extract from the headers during the actual download. So no need to exteact it from a link. (See the first downloadbook procedure in my code.) ---------------------------------------------------------------------- TRon · 2020-06-12 11:00 Quote from: alaa123456789 on June 12, 2020, 12:37:16 pm@tron ,how to get book name without '-' and '-e3213123.html' Do you mean as result a) https://www.pdfdrive.com/handsondatascienceandpythonmachinelearning performdataminingandmachinelearningefficientlyusingpythonandspark b) https://www.pdfdrive.com/hands on data science and python machine learning perform data mining and machine learning efficiently using python and spark c) https://www.pdfdrive.com/hands-on-data-science-and-python-machine- learning-perform-data-mining-and-machine-learning-efficiently-using- python-and-spark In case you also need to get rid of the part "https://www.pdfdrive.com/" then you can use ExtractFileName (https://www.freepascal.org/docs- html/rtl/sysutils/extractfilename.html), it is a hack but it seems to work fine. If you apply that on the initial/original URL then the result should read "hands-on-data-science-and-python-machine-learning-perform-data- mining-and-machine-learning-efficiently-using-python-and- spark-e158445708.html" ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 11:01 [/Quote] Then there is something wrong with how you used openssl. (I take it you have it with all searches, as you again did not specify) Quote i didnt change anything in your code except the uses section now as following Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses  Classes, SysUtils, openssl, Forms, Controls, Graphics, Dialogs,  StdCtrls; you have to try it on  your computer with different words and you see the error , of course on mobile you can't thanks ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 11:05 QuoteIf you apply that on the initial/original URL then the result should read "hands-on-data-science-and-python-machine-learning- perform-data-mining-and-machine-learning-efficiently-using-python-and- spark-e158445708.html" i want to get rid of red color text in quote and from all '-' thanks ---------------------------------------------------------------------- rvk · 2020-06-12 12:04 Quote from: alaa123456789 on June 12, 2020, 01:01:55 pmyou have to try it on  your computer with different words and you see the error , of course on mobile you can't I did try earlier with several other search words. Visual basic gave the redirect error which I fixed with those addition for redirection. But you are getting a protocol error and that seems that openssl isn't enough (or correct) to do https requests. I'm on FPC 3.3.1 so I can't test this with 3.0.4. Quote from: alaa123456789 on June 12, 2020, 01:05:09 pmi want to get rid of red color text in quote and from all '-' Why do you want only the text from that link? If it is to do with the filename, there are better ways to determine the filename during download. Or do you only want to display this? ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 12:30 QuoteI did try earlier with several other search words. Visual basic gave the redirect error which I fixed with those addition for redirection. But you are getting a protocol error and that seems that openssl isn't enough (or correct) to do https requests. I'm on FPC 3.3.1 so I can't test this with 3.0.4.how to be on fpc 3.3.1 the latest version of lazrus come only with fpc3.0.4 QuoteWhy do you want only the text from that link? If it is to do with the filename, there are better ways to determine the filename during download. Or do you only want to display this? want to get filename as it is already list box.item ---------------------------------------------------------------------- TRon · 2020-06-12 12:55 Quote from: alaa123456789 on June 12, 2020, 01:05:09 pmQuoteIf you apply that on the initial/original URL then the result should read "hands-on-data-science-and-python-machine-learning-perform-data- mining-and-machine-learning-efficiently-using-python-and- spark-e158445708.html" i want to get rid of red color text in quote and from all '-' thanks oh, uhm.. ok  (i saw that message too late) I present to you the overcomplica]"]>Blockedltifunctionale GetBookNameFromURL function (sorry) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program splittinghairs; {$MODE OBJFPC}{$H+} uses  sysutils; function GetBookIDFromURL(URL: String; Include101: boolean = true): String;var  WantedDeadOrAlive: String absolute Result;  TalkingWords: TStringArray;begin  // Make sure result is empty by default  WantedDeadOrAlive := '';   // split up the URL into individual parts/words  TalkingWords:= URL.Split(['-','.']);   // If there are enough words  if Length(TalkingWords)> 1 then    // and if we want to meet 101    if Include101      // then take second for last word      then WantedDeadOrAlive:= TalkingWords[High(TalkingWords)-1]    // and in case we do not want to meet 101    else WantedDeadOrAlive:= TalkingWords[High(TalkingWords)-1].Remove(0,1);end; type  TGBNFU = (RemoveProtocolAndHost, RemoveExtension, RemoveBookID, ChangeMinusToNothing, ChangeMinusToSpace);  TGBNFUSet = set of TGBNFU; function GetBookNameFromURL(URL: string; Options: TGBNFUSet = [RemoveProtocolAndHost, RemoveExtension, RemoveBookID]): string;var  BookID : string;begin  Result := URL;   // Retrieve the book ID, needs to be done before removing extension.  BookID := GetBookIDFromURL(Result);   // Remove the https and website's name from the URL  if RemoveProtocolAndHost in Options then Result := ExtractFileName(Result);   // Remove the extension by replacing it with an empty string  if RemoveExtension in Options then Result := ChangeFileExt(Result, '');   // Replace the BookID with an empty string  If RemoveBookID in Options then Result := Result.Replace(BookID, '');   // Replace '-' character with empty character  if ChangeMinusToNothing in Options then Result := Result.Replace('-', '')  else  if ChangeMinusToSpace in Options then Result := Result.Replace('-', ' ');end;   var  URL : string = 'https://www.pdfdrive.com/hands-on-data-science-and-python-machine- learning-perform-data-mining-and-machine-learning-efficiently-using- python-and-spark-e158445708.html';begin  WriteLn(1,' = ',GetBookNameFromURL(URL).QuotedString);  WriteLn(2,' = ',GetBookNameFromURL(URL, []).QuotedString);  WriteLn(3,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost]).QuotedString);  WriteLn(4,' = ',GetBookNameFromURL(URL, [RemoveExtension]).QuotedString);   WriteLn( 6,' = ',GetBookNameFromURL(URL, [ChangeMinusToNothing]).QuotedString);  WriteLn( 7,' = ',GetBookNameFromURL(URL, [ChangeMinusToSpace]).QuotedString);  WriteLn( 8,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveBookID, RemoveExtension, ChangeMinusToNothing]).QuotedString);  WriteLn( 9,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveBookID, RemoveExtension, ChangeMinusToSpace]).QuotedString);  WriteLn(10,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveBookID, RemoveExtension]).QuotedString);  WriteLn(11,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveBookID, ChangeMinusToNothing]).QuotedString);  WriteLn(12,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveBookID, ChangeMinusToSpace]).QuotedString);  WriteLn(13,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveBookID]).QuotedString);  WriteLn(14,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveExtension, ChangeMinusToNothing]).QuotedString);  WriteLn(15,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveExtension, ChangeMinusToSpace]).QuotedString);  WriteLn(16,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, RemoveExtension]).QuotedString);  WriteLn(17,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, ChangeMinusToNothing]).QuotedString);  WriteLn(18,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost, ChangeMinusToSpace]).QuotedString);  WriteLn(19,' = ',GetBookNameFromURL(URL, [RemoveProtocolAndHost]).QuotedString);  WriteLn(20,' = ',GetBookNameFromURL(URL, [RemoveBookID, RemoveExtension, ChangeMinusToNothing]).QuotedString);  WriteLn(21,' = ',GetBookNameFromURL(URL, [RemoveBookID, RemoveExtension, ChangeMinusToSpace]).QuotedString);  WriteLn(22,' = ',GetBookNameFromURL(URL, [RemoveBookID, RemoveExtension]).QuotedString);  WriteLn(23,' = ',GetBookNameFromURL(URL, [RemoveBookID, ChangeMinusToNothing]).QuotedString);  WriteLn(24,' = ',GetBookNameFromURL(URL, [RemoveBookID, ChangeMinusToSpace]).QuotedString);  WriteLn(35,' = ',GetBookNameFromURL(URL, [RemoveBookID]).QuotedString);  WriteLn(26,' = ',GetBookNameFromURL(URL, [RemoveExtension, ChangeMinusToNothing]).QuotedString);  WriteLn(27,' = ',GetBookNameFromURL(URL, [RemoveExtension, ChangeMinusToSpace]).QuotedString);  WriteLn(28,' = ',GetBookNameFromURL(URL, [RemoveExtension]).QuotedString);  WriteLn(29,' = ',GetBookNameFromURL(URL, [ChangeMinusToNothing]).QuotedString);  WriteLn(30,' = ',GetBookNameFromURL(URL, [ChangeMinusToSpace]).QuotedString);end.  edit: and i just realised it isn't even doing what you asked for, sorry about that  edit2: btw what you asked for is still disambiguous  , yes or no also get rid of the minus characters ? and if so, then replace the minus characters with what exactly ? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function GetBookNameFromURL(URL: string): string;const  RemoveAllMinus = true;  ReplacementChar = '';var  n : integer;begin  Result := ChangeFileExt(ExtractFileName(URL), '');  n := Result.LastIndexOf('-');  if n <> -1 then Result := Result.Remove(Result.LastIndexOf('-'));  if RemoveAllMinus then Result := Result.Replace('-', ReplacementChar).Trim;end; ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 13:25 Quoteuses   sysutils; function GetBookIDFromURL(URL: String; Include101: boolean = true): String; var   WantedDeadOrAlive: String absolute Result;   TalkingWords: TStringArray; begin   // Make sure result is empty by default   WantedDeadOrAlive := '';   // split up the URL into individual parts/words   TalkingWords:= URL.Split(['-','.']);   // If there are enough words   if Length(TalkingWords)> 1 then     // and if we want to meet 101     if Include101       // then take second for last word       then WantedDeadOrAlive:= TalkingWords[High(TalkingWords)-1]     // and in case we do not want to meet 101     else WantedDeadOrAlive:= TalkingWords[High(TalkingWords)-1].Remove(0,1); end;@tron what is 101? could you please explain how this code work ---------------------------------------------------------------------- TRon · 2020-06-12 13:35 Quote from: alaa123456789 on June 12, 2020, 03:25:53 pm@tron what is 101? could you please explain how this code work The code is documented, so if you have any specific question then please just ask. In my opinion the comments answers everything. ... but I can understand your confusion with regards to what the mystery is behind 101. It stands for Ascii-code 101 (decimal). Do you have any idea what character that number 101 (decimal) represents ? ---------------------------------------------------------------------- TRon · 2020-06-12 15:07 Quote from: TRon on June 12, 2020, 03:35:54 pmThe code is documented, so if you have any specific question then please just ask. In my opinion the comments answers everything. And the following is why i do not like to answer such a question, simply too much to write  Keep in mind that the following is only about the function declaration itself, not even actual code. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function GetBookIDFromURL(URL: String; Include101: boolean = true): String;  For the function declaration and parameter list see also: - https://freepascal.org/docs-html/ref/refse89.html#x173-19500014.2 - https://freepascal.org/docs-html/ref/refse91.html#x175-19700014.4 function: the (default) function headerGetBookIDFromURL: identifier for the function (name of the function)(: indicates the start of a (formal) parameter listURL: String: this is a (normal) parameter declaration;: indicates that another parameter declaration will follow (parameter list separator)Include101: boolean = true: this is a parameter declaration (*)): indicates the end of a (formal) parameter list:: separator between formal parameter list and (function) result typeString: function result type(*) Supplying a default value for a function parameter is also known as an initialised variable and has a special "side-effect" when used in a function. You are allowed to omit that parameter when you call that function, in which case the default (initialised) value (as declared in the functions parameter list) will automatically be assigned to that variable. If you choose that you do want to supply that parameter when calling that function then you can do so and the then provided value will 'overwrite' the default value. So this function can be called in following 3 ways: 1) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SomeStringVariable := GetBookIDFromURL('SomeURL');   In which case the parameter Include101 has the value "true" 2) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SomeStringVariable := GetBookIDFromURL('SomeURL', true);   In which case the parameter Include101 has the value "true" (nothing actually changed) 3) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SomeStringVariable := GetBookIDFromURL('SomeURL', false);   In which case the parameter Include101 has the value "false" (the value of the parameter is changed) For variable (declarations) etc, see also https://freepascal.org/docs- html/ref/refch4.html#refse22.html var: start of a variable declaration block (see https://freepascal.org/docs-html/ref/refse109.html#x219-24100016.5) I'll refrain from going into detail how variables are (formally) declared, however there is something special here that you might not have seen before: WantedDeadOrAlive: String absolute Result; The identifier for this variable is (of course) WantedDeadOrAlive and the type for this identifier is a string. Let's take a look at the documentation of the formal declaration of a variable (https://freepascal.org/docs- html/ref/refse22.html#x53-710004.2) There you can read about a list of modifiers, one of those modifiers is named absolute. (see also here https://www.freepascal.org/docs- html/current/ref/refsu3.html#x14-130001.3.3) Unfortunately, I seem unable to find any other reference about this modifier in the official documentation (so if someone reads this and knows about one, then please feel free to help us out here). However, there is a wiki page that mentions this absolute modifier, see https://wiki.lazarus.freepascal.org/Absolute (and in case you wondering how i know about it, this modifier was present in ancient turbo pascal and was heavily used back then) Quote"The absolute modifier causes a variable to be stored at the same memory location as another variable." Which actually just means what it reads/says. The code shows that the contents of the variable is stored at the location of something called Result. Result is the identifier used by a function to indicate the returning (resulting) value and, its contents is returned to the caller when the function exits. see also https://freepascal.org/docs- html/ref/refse90.html In laymans terms: whenever you "store" something into Result it will also be accessible by using the identifier WantedDeadOrAlive and even by using the identifier GetBookIDFromURL (which is, the name of the function) Happy reading ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 15:51 anyway thanks for your explanation but it seems too advance for me (there is less information about pascal) i have studied it in my 10th class at school but i didn't continue with it , i have learned more about vb i will try it in my program then see the result thanks tron ---------------------------------------------------------------------- TRon · 2020-06-12 16:24 @alaa123456789 That is why I initially said: if you have a specific question, then please just ask  As a (small) hint, you can not learn from manuals, documentation, snippets, examples alone. It is imho much more interesting to take some code  put it into a new fresh project and play with the code. For example, did you already figured out what the ascii code 101 stands for ?, have you already tried that function with providing a second parameter that is set to false ? And do you know what exactly happens (to the result) when you do that ? Let's take another (simple) example: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i TalkingWords:= URL.Split(['-','.']); Do you know what split does ? I did not know either. Therefor i started playing with that function (documentation: https://www.freepascal.org/docs- html/rtl/sysutils/tstringhelper.split.html) . Take that part into a new project and change the contents of the URL string to contain other characters. Then you can try changing the characters of the parameter that is given to that split function. That is how I learned to use it. It is not that I just 'happen to know' about it, just as you for example have to learn about math (what is probably taught when you went to school). And in all fairness vb can be a tough cookie as well, it just depends on how much time you want to spend on (learning) things. Whatever you do, just try to avoid just copy-pasting other peoples code without knowing what that code does. It is impossible to learn anything from that (except for the fact you are perhaps becoming one of the quickest and most versatile copy-pasters that ever existed, how is that for a title: a xerox-machine  ). cheers, ---------------------------------------------------------------------- alaa123456789 · 2020-06-12 16:59 Quote from: TRon on June 12, 2020, 06:24:32 pm@alaa123456789 That is why I initially said: if you have a specific question, then please just ask  As a (small) hint, you can not learn from manuals, documentation, snippets, examples alone. It is imho much more interesting to take some code  put it into a new fresh project and play with the code. For example, did you already figured out what the ascii code 101 stands for ?, have you already tried that function with providing a second parameter that is set to false ? And do you know what exactly happens (to the result) when you do that ? Let's take another (simple) example: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i TalkingWords:= URL.Split(['-','.']); Do you know what split does ? I did not know either. Therefor i started playing with that function (documentation: https://www.freepascal.org/docs- html/rtl/sysutils/tstringhelper.split.html) . Take that part into a new project and change the contents of the URL string to contain other characters. Then you can try changing the characters of the parameter that is given to that split function. That is how I learned to use it. It is not that I just 'happen to know' about it, just as you for example have to learn about math (what is probably taught when you went to school). And in all fairness vb can be a tough cookie as well, it just depends on how much time you want to spend on (learning) things. Whatever you do, just try to avoid just copy-pasting other peoples code without knowing what that code does. It is impossible to learn anything from that (except for the fact you are perhaps becoming one of the quickest and most versatile copy-pasters that ever existed, how is that for a title: a xerox-machine  ). cheers, @tron , i never copy paste code only, and if i do that i wouldn't ask you about your code 101( e letter ) i always learn by experiment (like this subject and scraping this website) i go through it step by step ,of course when i didn't find information i used this forum to ask help and i thank for each one helped me.  thanks ---------------------------------------------------------------------- rvk · 2020-06-12 21:45 Ok, especially for you I loaded Lazarus 2.0.8. I now also get the error invalid protocol "". I think it is a bug in fphttpclient, probably in the redirection which looses the https or something. I can't fix that. I completely redid the example using THTTPSend from Synapse (Synapse was already included for the synacode unit). (I like Synapse better anyway.) The example now works. As I said, it was just a proof of concept. There could still be a lot of things done better. Think about *) moving the downloading to threads so it would be much faster. *) adding downloads for external sites (if pdfdrive has any) *) etc. Not sure what outstanding questions you still had (or additional questions about this sample) so ask them again please if you still want them answered. ---------------------------------------------------------------------- Jurassic Pork · 2020-06-13 06:40 hello, Quote from: rvk on June 12, 2020, 11:45:46 pmOk, especially for you I loaded Lazarus 2.0.8. I now also get the error invalid protocol "". I think it is a bug in fphttpclient, probably in the redirection which looses the https or something. I can't fix that. What is the bug for fphttpclient ? have you simple code to reproduce it ? to use https with fphttpclient i use these units : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient, fpopenssl,openssl; Friendly, J.P ---------------------------------------------------------------------- Thaddy · 2020-06-13 07:34 The "invalid protocol" error only should happen when you are trying to use a deprecated version of ssl, like the ssl2/3 or tls1.0 protocols. These are no longer supported in a recent standard distribution of openssl unless you compile openssl yourself with proper flags. The latter is not an option: the removed protocols are insecure. They are simply not in the binary anymore... Recent browsers also do not support them at all, not even fall-back... Can you plz check again which protocol you expect?  Make it at least tls1.1. ---------------------------------------------------------------------- rvk · 2020-06-13 15:48 Quote from: Jurassic Pork on June 13, 2020, 08:40:46 amhello, Quote from: rvk on June 12, 2020, 11:45:46 pmOk, especially for you I loaded Lazarus 2.0.8. I now also get the error invalid protocol "". I think it is a bug in fphttpclient, probably in the redirection which looses the https or something. I can't fix that. What is the bug for fphttpclient ? have you simple code to reproduce it ? to use https with fphttpclient i use these units : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient, fpopenssl,openssl;I have posted a simple_scrape which shows the error for 3.0.4. https://forum.lazarus.freepascal.org/index.php/topic,49927.msg365618.h tml#msg365618 Trunk is fine (using sslopensockets). It could also be from the redirection. One redirect gives the complete url (with https and hostname) while another redirect only gives a local redirect. Maybe something goes wrong there. But trunk is fine so it must be something that's already fixed. My second last example completely used Synapse (which works fine if I add the redirection myself because that's not standard build in Synapse). Quote from: Thaddy on June 13, 2020, 09:34:16 amThe "invalid protocol" error only should happen when you are trying to use a deprecated version of ssl, like the ssl2/3 or tls1.0 protocols. The browsers work fine on that url. Just the redirection seems to create problems. ---------------------------------------------------------------------- Jurassic Pork · 2020-06-13 16:08 Quote from: rvk on June 13, 2020, 05:48:52 pmI have posted a simple_scrape which shows the error for 3.0.4. https://forum.lazarus.freepascal.org/index.php/topic,49927.msg365618.h tml#msg365618 Trunk is fine (using sslopensockets). i can't compile the simple scrape with sslopensockets in uses with my Lazarus  2.0.8  fpc 3.0.4  windows 10 if i replace sslopensockets  with   fpopenssl, openssl     i can compile the simple scrape and i have no error when i run it.  Friendly JP ---------------------------------------------------------------------- rvk · 2020-06-13 16:11 Quote from: Jurassic Pork on June 13, 2020, 06:08:55 pmif i replace sslopensockets by   fpopenssl, openssl     i can compile the simple scrape and i have no error when i run it.  Yes, that is correct. I created it for trunk. With the change you made, can you try the search words Visual basic And see if you get the error? With that search there is a redirection. ---------------------------------------------------------------------- alaa123456789 · 2020-06-13 16:18 Quote from: rvk on June 12, 2020, 11:45:46 pmOk, especially for you I loaded Lazarus 2.0.8. thanks appreciate your support QuoteI completely redid the example using THTTPSend from Synapse (Synapse was already included for the synacode unit). (I like Synapse better anyway.) now i think it is fine Quote) moving the downloading to threads so it would be much faster. *) adding downloads for external sites (if pdfdrive has any) these are good ideas let's do it how to make it faster,  also we can make progress bar thanks ---------------------------------------------------------------------- Jurassic Pork · 2020-06-13 16:21 Quote from: rvk on June 13, 2020, 06:11:22 pmQuote from: Jurassic Pork on June 13, 2020, 06:08:55 pmif i replace sslopensockets by   fpopenssl, openssl     i can compile the simple scrape and i have no error when i run it.  Yes, that is correct. I created it for trunk. With the change you made, can you try the search words Visual basic And see if you get the error? With that search there is a redirection. No error  with    the search words Visual Basic ---------------------------------------------------------------------- rvk · 2020-06-13 16:33 Quote from: Jurassic Pork on June 13, 2020, 06:21:57 pmNo error  with    the search words Visual Basic O, that's really strange. I got the same error alaa123456789 got. But now I think of it... That could also be an older openssl dll I standard took from my archive. openssl-1.0.2h-i386-win32.zip I didn't think that one was too old. And it would be strange that trunk doesn't have the problem with the same dlls. ---------------------------------------------------------------------- alaa123456789 · 2020-06-26 08:04 hey guys please check my programm https://youtu.be/I0JAAS6N9Y4 https://drive.google.com/file/d/1nLHMHEdLcUPqYjkVATx6viMlE1XynpoN/view ?usp=sharing please give your comments thanks ---------------------------------------------------------------------- Handoko · 2020-06-26 08:08 That's look good. Currently I'm doing research and writing similar program as yours. I downloaded your program. It's not open source.  Anyway, thank you for sharing it. ---------------------------------------------------------------------- rvk · 2020-06-26 08:55 Looks good. Quote from: Handoko on June 26, 2020, 10:08:33 amIt's not open source.  Most of the code you can find in my latest example in this topic. https://forum.lazarus.freepascal.org/index.php/topic,49927.msg365816.h tml#msg365816 ---------------------------------------------------------------------- Handoko · 2020-06-26 09:12 Quote from: rvk on June 26, 2020, 10:55:06 amMost of the code you can find in my latest example in this topic. https://forum.lazarus.freepascal.org/index.php/topic,49927.msg365816.h tml#msg365816 Thank you very much. It will greatly reduce my research time. ---------------------------------------------------------------------- alaa123456789 · 2020-06-26 10:11 Quote from: rvk on June 26, 2020, 10:55:06 amLooks good. Quote from: Handoko on June 26, 2020, 10:08:33 amIt's not open source.  it is still under development , it take long time to download the book , and i am searching for better way , at the end i will make it open source for all , i just put it to get the comments from you guys about its performance thanks ---------------------------------------------------------------------- alaa123456789 · 2020-06-26 10:17 Quote from: Handoko on June 26, 2020, 10:08:33 amThat's look good. Currently I'm doing research and writing similar program as yours. I downloaded your program. It's not open source.  Anyway, thank you for sharing it. please try it and give your comments , i still feel it is not perfect , i will share the code soon thanks ---------------------------------------------------------------------- Handoko · 2020-06-27 16:07 Quote from: alaa123456789 on June 26, 2020, 12:17:32 pmplease try it and give your comments , i still feel it is not perfect Sorry, I can't. I downloaded the binary and run it on my Linux's Wine. Unfortunately I got SSL error. I remember recent versions of Ubuntu have SSL problem with binaries compiled using Lazarus recent versions. And, I've already moved away from Windows for a very long time. I still need to use Windows, that's why many years ago I purchased a WinXP license for testing purpose. Your 64-bit binary can't run on WinXP 32-bit. It is okay if your software is not perfect. No one can write perfect software. I heard the first BASIC written by Bill Gates and his friend was very buggy. ---------------------------------------------------------------------- alaa123456789 · 2020-07-15 18:15 hey guys i am trying using lazarus synapse 40.1 in my code to load photo here is the code Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  uses httpsend  response := TMemoryStream.Create; try    if HttpGetBinary('http://feelgrafix.com/data_images/out/1/737246-apple- wallpaper.jpg', response) then    begin      response.Seek(0,soFromBeginning);      Image1.Picture.LoadFromStream(response);    end; finally   response.Free;            end;   but it is not loading ( i added libeay32.dll,ssleay32.dll) in application folder first time worked fine , then it is stopped please help QuoteMessages, Hints: 14 Note: Duplicate unit "synsock" in "photosexplorer", orphaned ppu "C:\Users\alaa-laptop\Desktop\Desktop Files\lazarus classes\videos\flickr photos\lib\x86_64-win64\synsock.ppu" Note: Duplicate unit "synsock" in "laz_synapse 40.1", ppu="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanager\p ackages\synapse40.1\lib\x86_64-win64\synsock.ppu", source="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanage r\packages\synapse40.1\synsock.pas" Note: Duplicate unit "synautil" in "photosexplorer", orphaned ppu "C:\Users\alaa-laptop\Desktop\Desktop Files\lazarus classes\videos\flickr photos\lib\x86_64-win64\synautil.ppu" Note: Duplicate unit "synautil" in "laz_synapse 40.1", ppu="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanager\p ackages\synapse40.1\lib\x86_64-win64\synautil.ppu", source="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanage r\packages\synapse40.1\synautil.pas" Note: Duplicate unit "synaip" in "photosexplorer", orphaned ppu "C:\Users\alaa-laptop\Desktop\Desktop Files\lazarus classes\videos\flickr photos\lib\x86_64-win64\synaip.ppu" Note: Duplicate unit "synaip" in "laz_synapse 40.1", ppu="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanager\p ackages\synapse40.1\lib\x86_64-win64\synaip.ppu", source="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanage r\packages\synapse40.1\synaip.pas" Note: Duplicate unit "synafpc" in "photosexplorer", orphaned ppu "C:\Users\alaa-laptop\Desktop\Desktop Files\lazarus classes\videos\flickr photos\lib\x86_64-win64\synafpc.ppu" Note: Duplicate unit "synafpc" in "laz_synapse 40.1", ppu="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanager\p ackages\synapse40.1\lib\x86_64-win64\synafpc.ppu", source="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanage r\packages\synapse40.1\synafpc.pas" Note: Duplicate unit "synacode" in "photosexplorer", orphaned ppu "C:\Users\alaa-laptop\Desktop\Desktop Files\lazarus classes\videos\flickr photos\lib\x86_64-win64\synacode.ppu" Note: Duplicate unit "synacode" in "laz_synapse 40.1", ppu="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanager\p ackages\synapse40.1\lib\x86_64-win64\synacode.ppu", source="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanage r\packages\synapse40.1\synacode.pas" Note: Duplicate unit "httpsend" in "photosexplorer", orphaned ppu "C:\Users\alaa-laptop\Desktop\Desktop Files\lazarus classes\videos\flickr photos\lib\x86_64-win64\httpsend.ppu" Note: Duplicate unit "httpsend" in "laz_synapse 40.1", ppu="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanager\p ackages\synapse40.1\lib\x86_64-win64\httpsend.ppu", source="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanage r\packages\synapse40.1\httpsend.pas" Note: Duplicate unit "blcksock" in "photosexplorer", orphaned ppu "C:\Users\alaa-laptop\Desktop\Desktop Files\lazarus classes\videos\flickr photos\lib\x86_64-win64\blcksock.ppu" Note: Duplicate unit "blcksock" in "laz_synapse 40.1", ppu="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanager\p ackages\synapse40.1\lib\x86_64-win64\blcksock.ppu", source="C:\Users\alaa-laptop\AppData\Local\lazarus\onlinepackagemanage r\packages\synapse40.1\blcksock.pas" Compile Project, Mode: Default, Target: photosexplorer.exe: Success thanks ---------------------------------------------------------------------- rvk · 2020-07-15 19:43 Quote from: alaa123456789 on July 15, 2020, 08:15:32 pmhey guys i am trying using lazarus synapse 40.1 in my code to load photo ...  but it is not loading ( i added libeay32.dll,ssleay32.dll) in application folder first time worked fine , then it is stopped First of all, what DOES it give you? Does it give an error message? If so, why didn't you mention that?????? Second... openssl library should be needed because it's not a HTTPS address. Third... have you tried this link in a browser? Fourth... Put a Showmessage(pchar(response.Memory)); after the HttpGetBinary You'll see what you actually get from the server. Does that look like data from an image? And finally... the hints you get are not fatal. It means you have a duplicate path to synapse somewhere in your project. ---------------------------------------------------------------------- alaa123456789 · 2020-07-17 04:48 hi RVK i added ssl_openssl to uses and it worked  thanks Alaa ---------------------------------------------------------------------- alaa123456789 · 2020-07-17 16:29 here another example for web scraping https://www.youtube.com/watch?v=hciB_kPj3Sc enjoy ---------------------------------------------------------------------- alaa123456789 · 2021-02-28 10:21 have a look for the final test with source code https://www.youtube.com/watch?v=Nw6N8_7ZI_o thanks