Browser Automation with selenium
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
Jurassic Pork · 2020-06-04 23:58

hello,
i have found this library to do that with pascal language : Webdriver
For Delphi lazarus.
The problem is that the lazarus part isn't done.
1 - the main problem is the json library used  in the unit
Webdriver4D.pas : JsonDataObjects  . How to replace it with fcl-json
and jsonparser ?
for example how to replace the code in brackets with the equivalent 
of the fcl-json library :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TWebDriver.ExecuteScript(const Script: string;  const Args: string =
'[]'): string;var  Command: string;  Data: string;  Resp: string; 
Json: TJsonObject;begin  Json := TJsonObject.Create;  try    Command
:= Host + '/session/' + FSessionID + '/execute/sync';{  
Json.S['script'] := Script;    Json.A['args'].FromJSON(Args);    Data
:= Json.ToJSON();    }    Resp := ExecuteCommand(cPost, Command,
Data);    result := ProcResponse(Resp);  Finally    FreeAndNil(Json); 
end;end;           
Friendly, J.P

----------------------------------------------------------------------
Leledumbo · 2020-06-06 18:48

.S[] -> https://www.freepascal.org/docs-
html/current/fcl/fpjson/tjsonobject.strings.html
.A[] -> https://www.freepascal.org/docs-
html/current/fcl/fpjson/tjsonobject.arrays.html
.FromJSON -> https://www.freepascal.org/docs-
html/current/fcl/fpjson/getjson.html (not a method, but same
functionality to convert a JSON formatted string to a JSON data,
probably needs a cast as well)
.ToJSON -> either https://www.freepascal.org/docs-
html/current/fcl/fpjson/tjsondata.formatjson.html or
https://www.freepascal.org/docs-
html/current/fcl/fpjson/tjsondata.asjson.html, depending on whether
you want to format it or not

----------------------------------------------------------------------
Jurassic Pork · 2020-06-13 12:34

hello,
thanks Leledumbo, i have found how to do that with  fpjson :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TWebDriver.ExecuteScript(const Script: string;  const Args: string =
'[]'): string;var  Command: string;  Data: string;  Resp: string; 
Json: TJsonObject;begin  Json := TJsonObject.Create(['script',
Script]);  try    Command := Host + '/session/' + FSessionID +
'/execute/sync';    Json.Add('args',GETJson(Args));    Data :=
Json.AsJSON;    Resp := ExecuteCommand(cPost, Command, Data);   
result := ProcResponse(Resp);  Finally    FreeAndNil(Json);  end;end;
I started a lazarus version of WebDriver4D ( name WebDriver4L) .
replace the delphi unit httpclient with the lazarus unit 
 fphttpclient
replace the delphi unit JsonDataObjects with the lazarus units fpson,
jsonparser
version of WebDriver4L is alpha but it begins to work :
Example of code to open chrome and clear image and files cache  in 
settings :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
implementationuses  Webdriver4L;{$R *.lfm} { TForm1 } procedure
TForm1.Button2Click(Sender: TObject);const CacheDlg = 'return
document.querySelector(''settings-ui'')' +               
 '.shadowRoot.querySelector(''settings-main'')' +               
 '.shadowRoot.querySelector(''settings-basic-page'')' +              
 '.shadowRoot.querySelector(''settings-section > settings-privacy-
page'')' +                 '.shadowRoot.querySelector(''settings-
clear-browsing-data-dialog'')';const btClear  =
'.shadowRoot.querySelector(''#clearBrowsingDataDialog'')' +          
     '.querySelector(''#clearBrowsingDataConfirm'')';const chkboxH1  =
'.shadowRoot.querySelector(''iron-pages'')' +                 
'.querySelector(''#browsingCheckboxBasic'')';const chkboxH2  =
'.shadowRoot.querySelector(''iron-pages'')' +                 
'.querySelector(''#cookiesCheckboxBasic'')';const chkboxH3  =
'.shadowRoot.querySelector(''iron-pages'')' +                 
'.querySelector(''#cacheCheckboxBasic'')';var Robot2 : TWebDriver;   
Element : TWebElement;begin  Robot2 := TChromeDriver.Create(nil); 
Robot2.StartDriver('chromedriver.exe');  Robot2.NewSession; 
Robot2.Implicitly_Wait(1000);  Robot2.Set_Window_Size(640, 640); 
Sleep(5000);  Robot2.GetURL('chrome://settings/clearBrowserData'); 
Sleep(2000);  Element := Robot2.FindElementByXPath('//settings-ui'); 
Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH1 +
';');  Memo1.Append('Etiquette case à cocher 1 : ' +
Element.AttributeValue('label'));  if Element.PropertyValue('checked')
= 'True' then  Element.Click();  Sleep(1000);  Element.ElementData :=
Robot2.ExecuteScript(CacheDlg + chkboxH2 + ';'); 
Memo1.Append('Etiquette case à cocher 2 : ' +
Element.AttributeValue('label'));  if Element.PropertyValue('checked')
= 'True' then  Element.Click();  Sleep(1000);  Element.ElementData :=
Robot2.ExecuteScript(CacheDlg + chkboxH3 + ';'); 
Memo1.Append('Etiquette case à cocher 3 : ' +
Element.AttributeValue('label'));  if Element.PropertyValue('checked')
= 'False' then  Element.Click();  Sleep(1000);  Element.ElementData :=
Robot2.ExecuteScript(CacheDlg + btClear + ';');  Element.Click(); 
Sleep(30000);  Robot2.Clear;end;
Friendly, J.P

----------------------------------------------------------------------
Jurassic Pork · 2020-06-15 00:39

hello,
now my alpha version of webdriver4L is working on linux (centos 8 -
lazarus 2.0.8 ) . Screenshots work.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
implementationuses  Webdriver4L;{$R *.lfm}{ TForm1 }procedure
TForm1.Button2Click(Sender: TObject);const CacheDlg = 'return
document.querySelector(''settings-ui'')' +               
 '.shadowRoot.querySelector(''settings-main'')' +               
 '.shadowRoot.querySelector(''settings-basic-page'')' +              
 '.shadowRoot.querySelector(''settings-section > settings-privacy-
page'')' +                 '.shadowRoot.querySelector(''settings-
clear-browsing-data-dialog'')';const btClear  =
'.shadowRoot.querySelector(''#clearBrowsingDataDialog'')' +          
     '.querySelector(''#clearBrowsingDataConfirm'')';const chkboxH1  =
'.shadowRoot.querySelector(''iron-pages'')' +                 
'.querySelector(''#browsingCheckboxBasic'')';const chkboxH2  =
'.shadowRoot.querySelector(''iron-pages'')' +                 
'.querySelector(''#cookiesCheckboxBasic'')';const chkboxH3  =
'.shadowRoot.querySelector(''iron-pages'')' +                 
'.querySelector(''#cacheCheckboxBasic'')';var Robot2 : TWebDriver;   
Element : TWebElement;    png :  TPortableNetworkGraphic;begin  Robot2
:= TChromeDriver.Create(nil);  Robot2.StartDriver('chromedriver'); 
Sleep(5000);  Robot2.NewSession;  Robot2.Implicitly_Wait(1000); 
Robot2.Set_Window_Size(640, 640); 
Robot2.GetURL('chrome://settings/clearBrowserData');  Sleep(2000); 
Robot2.ScreenShot('clearBrowserData1.png');  Element :=
Robot2.FindElementByXPath('//settings-ui');  Element.ElementData :=
Robot2.ExecuteScript(CacheDlg + chkboxH1 + ';'); 
Memo1.Append('Etiquette case à cocher 1 : ' +
Element.AttributeValue('label'));  if Element.PropertyValue('checked')
= 'True' then  Element.Click();  Sleep(1000);  Element.ElementData :=
Robot2.ExecuteScript(CacheDlg + chkboxH2 + ';'); 
Memo1.Append('Etiquette case à cocher 2 : ' +
Element.AttributeValue('label'));  if Element.PropertyValue('checked')
= 'True' then  Element.Click();  Sleep(1000);  Element.ElementData :=
Robot2.ExecuteScript(CacheDlg + chkboxH3 + ';'); 
Memo1.Append('Etiquette case à cocher 3 : ' +
Element.AttributeValue('label'));  if Element.PropertyValue('checked')
= 'False' then  Element.Click();  Sleep(1000); 
Robot2.ScreenShot('clearBrowserData2.png');  Element.ElementData :=
Robot2.ExecuteScript(CacheDlg + btClear + ';');  Element.Click(); 
Application.ProcessMessages;  Sleep(10000);  Robot2.Clear;  png :=
TPortableNetworkGraphic.Create;  png.Assign(Self.GetFormImage); 
png.SaveToFile('clearBrowserDataLazForm.png');  png.free;end;
Results in Attachments (click on the first attachment to see
animation)

Friendly, J.P

----------------------------------------------------------------------
Blade · 2020-08-05 16:57

This is very useful.  Thanks for sharing.

----------------------------------------------------------------------
devEric69 · 2020-08-06 12:17

Hello @Jurassic Pork,

It's interesting to know.

Just for information, Selenium's successor is called Katalon (HMI web
automation). It can use globally (i haven't looked at how deep
compatible it is exactly) the same scripts as Selenium, which look
like the ones coded as parameters in your Robot.ExecuteScript method.

Nevertheless, what is interesting to know, is that such version of a
plugin (Selenium or Katalon) generally imposes a compatibility
constraint, with a minimal version of a browser. Your solution is a
way to get rid of it, if need be.

----------------------------------------------------------------------
systemgvp · 2020-12-09 16:07

Hi, I would like to try your library, can you post it?

----------------------------------------------------------------------
tobiastromm · 2021-01-23 05:23

Hi, is Webdriver4L available to download?

----------------------------------------------------------------------
Jurassic Pork · 2021-01-23 09:43

hello,
the problem is  that WebDriver4L is builded from EricWang 
WebDriver4D  Sources  and it seems that there is no licence on the
code of this project. I don't know if i can publish my derivated work
and how create a new project in github (not a fork) from the
webdriver4D sources.
Friendly, J.P

----------------------------------------------------------------------
Jurassic Pork · 2021-01-25 15:29

hello,
   Finally i put WebDriver4L source code on the forum. Hope to not go
to jail 

QuoteWebDriver4L   Selenium Webdriver for Lazarus   // Jurassic Pork
01/2021
Derivative work from Ericwang1104/WebDriver4D  :
https://github.com/Ericwang1104/WebDriver4D
Source code modified to use fphttpclient, fpjson,jsonparser.
The project is in alpha stage.
A demo project is included :  get a screenshot of the lazarus forum,
get all the recent topics and display then in a memo, get stats.
the demo project has been tested on windows 10 lazarus 2.0.10   with
chrome browser and on Ubuntu 20.04 Lazarus 2.0.10 with firefox
browser.
You must download the selenium Webdriver for your browser and put it
in your project folder. You must use a compatible driver version of
your browser :
For Chrome the selenium drivers are here :
https://chromedriver.chromium.org/downloads
For Firefox the selenium drivers are here :
https://github.com/mozilla/geckodriver/releases
For example if you use Chrome 87.x 64 bits on Windows  download a 87
64 bits version of the driver.

The Software is provided “as is”, without warranty of any kind,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose and noninfringement.
In no event shall the authors or copyright holders X be liable for any
claim, damages or other liability, whether in an action of contract,
tort or otherwise, arising from, out of or in connection with the
software or the use or other dealings in the Software.

Ericwang1104/WebDriver4D has no license so this project will not be
published on github. No activity since March 2019
Source code and demo project in attachments

----------------------------------------------------------------------
Jurassic Pork · 2021-01-26 02:06

hello,
WebDriver4L works also on MacOS with chrome browser.
1 - Put the MacOs chrome webDriver in the lib folder of your user home
directory
2 - Modify the demo project source code :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  {$ELSE} 
Robot := TChromeDriver.Create(nil);  // Robot :=
TFireFoxDriver.Create(nil);  Robot.StartDriver( GetUserDir +
'/lib/chromedriver');  //
Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '/geckodriver'); 
{$ENDIF}
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
 Robot.ScreenShot(GetUserDir +
'/Documents/ForumLazarusScreenShot.png'); 
Tested with chrome 88 Lazarus 2.0.10 on MacOs Catalina

Friendly, J.P

----------------------------------------------------------------------
Jurassic Pork · 2021-01-27 14:24

hello,
1 -
Quote from: Jurassic Pork on January 25, 2021, 04:29:44 pmFor example
if you use Chrome 87.x 64 bits on Windows  download a 87 64 bits
version of the driver.
    no 64 bits version of selenium webdriver for chrome on windows
only 32bits but it works !

2 - More accurate source code in unit1.pas of demo project to find the
recent topics and stats :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i //var
topics,authors,dateMessage : TWebElements;   topics :=
Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/a'); 
authors :=
Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/b'); 
dateMessage :=
Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/span'); 
For i:=0 to topics.Count - 1 do  begin 
 Memo1.Append(topics.Items[i].AttributeValue('Title') + ' by '+      
        authors.Items[i].Text + ' ' +               
dateMessage.Items[i].Text);  end;  stats :=
Robot.FindElementByClassName('inline'); 
Memo1.Append('================================='); 
Memo1.Append('Stats : ' + stats.Text);           
Friendly, J.P

----------------------------------------------------------------------
tobiastromm · 2021-06-23 16:33

Hi @Jurassic Pork.

First of all, thanks for this.

I am finally tryng that and it seens that some things are missing or I
am not finding the correct way to access then. Any chance you help me
with that?

1 - sendKeys
- sendKey is working, example:
Robot.FindElementByID('inputLogin').SendKey('userLogin');
- sendKeys, in the other hand, appear that does not exist: Error:
identifier idents no member "SendKeys".

Links: https://www.selenium.dev/documentation/en/webdriver/keyboard/
They also have a list of common keys, that are constants, that we can
pass on SendKeys. Can we have these constants also? Here is the list:
https://www.w3.org/TR/webdriver/#keyboard-actions and  https://www.sel
enium.dev/selenium/docs/api/py/webdriver/selenium.webdriver.common.key
s.html

2 - switchTo
- I can create a New tab, by using this way:
Robot.ExecuteScript('window.open()');
- The problem is that switchTo (used to switch between tabs) appear
that does not exist. I think this need to be implemented.

This https://www.selenium.dev/documentation/en/webdriver/browser_manip
ulation/#create-new-window-or-new-tab-and-switch is the right way to
open a new tab (instead of ExecuteScript) and manipulate these tabs:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i // Opens a
new tab and switches to new
tabdriver.SwitchTo().NewWindow(WindowType.Tab) // Opens a new window
and switches to new
windowdriver.SwitchTo().NewWindow(WindowType.Window) //Close the tab
or windowdriver.Close(); //Switch back to the old tab or
windowdriver.SwitchTo().Window(originalWindow); 
Here https://stackoverflow.com/questions/12729265/switch-tabs-using-
selenium-webdriver-with-java I found how to get tab IDs and switch
beetween these tabs (Java code):

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
ArrayList<String> tabs2 = new ArrayList<String> (driver.getWindowHandl
es());driver.switchTo().window(tabs2.get(1));driver.close();driver.swi
tchTo().window(tabs2.get(0)); 
Maybe this link https://blog.testproject.io/2020/06/30/selenium-4-new-
window-tab-screenshots/ can also help to better understand  how it
work, so it can be implemented.

Well, that's it for now.

----------------------------------------------------------------------
RedOctober · 2021-08-30 16:10

Hi Jurassic Pork,  thanks for WebDriver4L.  I have downloaded your
demo to my platform, which is  Laz 2.0.10  FPC 3.2 Windows Server
2016.  I have downloaded chromedriver for 92.  (which matches my
Chrome version).

When I compile and run the demo, the main window is displayed as
expected.  However, when I click either of the buttons, I get the
following error message:


Project demoWebDriver4L raised exception class 'EJSON' with message:
Unknown object member: "sessionId"

 At address 100176865

----------------------------------------------------------------------
Jurassic Pork · 2021-08-30 16:22

Quote from: RedOctober on August 30, 2021, 06:10:13 pmWhen I compile
and run the demo, the main window is displayed as expected.  However,
when I click either of the buttons, I get the following error message:

Project demoWebDriver4L raised exception class 'EJSON' with message:
Unknown object member: "sessionId"

 At address 100176865
i have this problem when i use fpdebug as debugger , with gdb or
running without debug it seems OK

----------------------------------------------------------------------
RedOctober · 2021-08-30 16:31

I run the .exe by itself, and the same error message appears.

----------------------------------------------------------------------
Jurassic Pork · 2021-08-30 16:55

have a look in the file chromedriver.log in the project folder. When
it fails i have this in the file :
Quote[1630342063.048][INFO]: Launching chrome: "C:\Program Files
(x86)\Google\Chrome\Application\chrome.exe" --allow-pre-commit-input
--disable-background-networking --disable-backgrounding-occluded-
windows --disable-client-side-phishing-detection --disable-default-
apps --disable-hang-monitor --disable-popup-blocking --disable-prompt-
on-repost --disable-sync --enable-automation --enable-blink-
features=ShadowDOMV0 --enable-logging --log-level=0 --no-first-run
--no-service-autorun --password-store=basic --remote-debugging-port=0
--test-type=webdriver --use-mock-keychain --user-data-
dir="C:\Users\jp\AppData\Local\Temp\scoped_dir2892_1394031789" data:,

[1630342063.084][INFO]: [281292df0dfd4ae3205b7269a71a9341] RESPONSE
InitSession ERROR unknown error: Failed to create Chrome process.

----------------------------------------------------------------------
RedOctober · 2021-08-30 19:35

Hi JP.  Here is the contents of my log file.  I have to have IPV6 shut
off on my server.  Would that cause this problem?  Or, maybe port 9515
is already busy?  Is there a place I can configure/reserve this port
for chromedriver if it is needs this specific port/

[1630341068.643][INFO]: Starting ChromeDriver 92.0.4515.107
(87a818b10553a07434ea9e2b6dccf3cbe7895134-refs/branch-
heads/4515@{#1634}) on port 9515
[1630341068.643][INFO]: Please see
https://chromedriver.chromium.org/security-considerations for
suggestions on keeping ChromeDriver safe.
[1630341068.646][SEVERE]: bind() returned an error: Only one usage of
each socket address (protocol/network address/port) is normally
permitted. (0x2740)
[1630341068.647][INFO]: listen on IPv6 failed with error
ERR_ADDRESS_IN_USE

----------------------------------------------------------------------
Jurassic Pork · 2021-08-30 22:25

Quote from: RedOctober on August 30, 2021, 09:35:35 pmHi JP.  Here is
the contents of my log file.  I have to have IPV6 shut off on my
server.  Would that cause this problem?  Or, maybe port 9515 is
already busy?  Is there a place I can configure/reserve this port for
chromedriver if it is needs this specific port/

[1630341068.643][INFO]: Starting ChromeDriver 92.0.4515.107
(87a818b10553a07434ea9e2b6dccf3cbe7895134-refs/branch-
heads/4515@{#1634}) on port 9515
[1630341068.643][INFO]: Please see
https://chromedriver.chromium.org/security-considerations for
suggestions on keeping ChromeDriver safe.
[1630341068.646][SEVERE]: bind() returned an error: Only one usage of
each socket address (protocol/network address/port) is normally
permitted. (0x2740)
[1630341068.647][INFO]: listen on IPv6 failed with error
ERR_ADDRESS_IN_USE

When you have this problem it's because the webdriver is already
running (see in the task manager). You must kill it before to  run
your project (use the task manager).

An for the other problem on a faster computer i have no problem with
fpdebug  :
I have found why you haven't the reason of the error but the message
Unknown object member: "sessionId"
There is an error in the TChromeDriver.NewSession function of
WebDriver4L.

With the new version of WebDriver4L (0.2) you can see why you have an
error.

See  my next message for new version of WebDriver4L

----------------------------------------------------------------------
Jurassic Pork · 2021-08-31 10:55

hello,
here is the new version of WebDriver4L :
QuoteWebDriver4L   Selenium Webdriver for Lazarus  V 0.2  // Jurassic
Pork 08/2021
Derivative work from Ericwang1104/WebDriver4D  :
https://github.com/Ericwang1104/WebDriver4D
Source code modified to use fphttpclient, fpjson,jsonparser.
The project is in beta stage.
V 0.1 improve demo project.
V 0.2 improve WebDriver4L and demo project.
A demo project is included :  get a screenshot of the lazarus forum,
get all the recent topics and display then in a memo, get stats.
the demo project has been tested on windows 10 lazarus 2.0.10   with
chrome browser and on Ubuntu 20.04 Lazarus 2.0.10 with firefox
browser.
You must download the selenium Webdriver for your browser and put it
in your project folder. You must use a compatible driver version of
your browser :
links to download webdrivers are here : https://www.selenium.dev/docum
entation/en/webdriver/driver_requirements/ (Quick reference)
For Chrome the selenium drivers are here :
https://chromedriver.chromium.org/downloads
  If you use Chrome 92.x  on Windows  download a 92 version of the
driver.
For Firefox the selenium drivers are here :
https://github.com/mozilla/geckodriver/releases
For Edge Chromium the selenium drivers are here :
https://msedgewebdriverstorage.z22.web.core.windows.net/
 go near to the last page with the next button and choose the same
version than your ms edge chromium browser.
For IE 11 the selenium drivers are here : https://selenium-
release.storage.googleapis.com/index.html?path=3.9/   (IEDriverServer
3.9)
For Safari  the selenium drivers are here : Built-in

The Software is provided “as is”, without warranty of any kind,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose and noninfringement.
In no event shall the authors or copyright holders X be liable for any
claim, damages or other liability, whether in an action of contract,
tort or otherwise, arising from, out of or in connection with the
software or the use or other dealings in the Software.

Ericwang1104/WebDriver4D has no license so this project will not be
published on github. No activity since March 2019

To choose your automation browser comment uncomment this part of the
code
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   Try 
 {$IFDEF WINDOWS}     Robot := TChromeDriver.Create(nil);  //   Robot
:= TFireFoxDriver.Create(nil);  //   Robot :=
TEdgeDriver.Create(nil);  //   Robot := TIEDriver.Create(Nil);    
 Robot.StartDriver(ExtractFileDir(Paramstr(0)) +
'\chromedriver.exe');  //  
 Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\geckodriver.exe'); 
//   Robot.StartDriver(ExtractFileDir(Paramstr(0)) +
'\msedgedriver.exe');  //  
Robot.StartDriver(ExtractFileDir(Paramstr(0)) +
'\IEDriverServer.exe');
the demo project has been tested on windows 10 lazarus 2.0.12   with
chrome browser,  Edge Chromium and internet Explorer 11.

Friendly, J.P

----------------------------------------------------------------------
Fahmy Rofiq · 2021-09-24 14:20

How to save login ?

----------------------------------------------------------------------
Jurassic Pork · 2021-09-24 14:32

Quote from: Fahmy Rofiq on September 24, 2021, 04:20:00 pmHow to save
login ?

What does it mean ?

----------------------------------------------------------------------
Fahmy Rofiq · 2021-09-24 14:38

After login to a website, how to save its cookie?
Because after application restart, it need to login again...

----------------------------------------------------------------------
Jurassic Pork · 2021-09-24 14:46

Ok ,
to save cookies , history, cache you must use a profile stored in a
directory.
To do this with chrome change the beginning of function
TChromeDriver.NewSession in WebDriver4L unit from :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i const   
Chrome_Param =    '{"capabilities": {"firstMatch": [{}],
"alwaysMatch": {"browserName": "chrome",'    + ' "platformName":
"any", "goog:chromeOptions": {"extensions": [],'        + ' "args":
[]}}},'  // visible mode//      + ' "args": ["--headless","--disable-
gpu"]}}},' // hidden mode//      + ' "args": ["--user-data-
dir=d:/temp/lazSelenium"]}}},' // userdir    + '
"desiredCapabilities": {"browserName": "chrome", "version": "",
"platform": "ANY",'    + ' "goog:chromeOptions": {"extensions": [],
"args": []}}}';  with :
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i const   
Chrome_Param =    '{"capabilities": {"firstMatch": [{}],
"alwaysMatch": {"browserName": "chrome",'    + ' "platformName":
"any", "goog:chromeOptions": {"extensions": [],'//      + ' "args":
[]}}},'  // visible mode//      + ' "args": ["--headless","--disable-
gpu"]}}},' // hidden mode         + ' "args": ["--user-data-
dir=d:/temp/lazSelenium"]}}},' // userdir    + '
"desiredCapabilities": {"browserName": "chrome", "version": "",
"platform": "ANY",'    + ' "goog:chromeOptions": {"extensions": [],
"args": []}}}'; 
and adjust the user-data-dir to a read/write directory in your system.

----------------------------------------------------------------------
Fahmy Rofiq · 2021-09-24 14:57

Thanks, thats work.

----------------------------------------------------------------------
lazzaro2023 · 2023-02-10 17:43

Quote from: RedOctober on August 30, 2021, 06:10:13 pmHi Jurassic
Pork,  thanks for WebDriver4L.  I have downloaded your demo to my
platform, which is  Laz 2.0.10  FPC 3.2 Windows Server 2016.  I have
downloaded chromedriver for 92.  (which matches my Chrome version).

When I compile and run the demo, the main window is displayed as
expected.  However, when I click either of the buttons, I get the
following error message:


Project demoWebDriver4L raised exception class 'EJSON' with message:
Unknown object member: "sessionId"

 At address 100176865
To resolve the error, the version of chromedriver.exe must be the same
as Google Chrome.

----------------------------------------------------------------------
TRon · 2023-06-02 05:29

Thank you for yet another great project Jurassic Pork.

Small issue to mention: Unit WebDriver4L function
TFireFoxDriver.NewSession const Firefox_Param currently reads:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i const 
Firefox_Param =    '{"capabilities": {"firstMatch": [{}],
"alwaysMatch": {"browserName": "firefox", '    +
'"acceptInsecureCerts": false, "moz:firefoxOptions": ' +//  
 '{"binary": "%s"}}},' +  // visible mode    '{"binary":
"%s",args:"-headless"}}},' +  // hidden mode   
'"desiredCapabilities": {"browserName": "firefox",
"acceptInsecureCerts": false, "moz:firefoxOptions": {"binary":
"%s"}}}'; 
According to https://developer.mozilla.org/en-
US/docs/Web/WebDriver/Capabilities/firefoxOptions that should read:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i const 
Firefox_Param =    '{"capabilities": {"firstMatch": [{}],
"alwaysMatch": {"browserName": "firefox", '    +
'"acceptInsecureCerts": false, "moz:firefoxOptions": ' +//  
 '{"binary": "%s"}}},' +  // visible mode    '{"binary":
"%s","args":["-headless"]}}},' +  // hidden mode   
'"desiredCapabilities": {"browserName": "firefox",
"acceptInsecureCerts": false, "moz:firefoxOptions": {"binary":
"%s"}}}';  
Note: Code: [Select]args:"-headless" vs "args":["-headless"]
When fix not applied the geckodriver will report back that the JSON
request is invalid.


When using the function FindElementsByClassName the gecko
driver/server reports:

Code: [Select]1685771451493     webdriver::server       DEBUG   ->
POST /session/91cf59aa-1503-4a17-9bbe-ffa5728fa2ee/elements { "value"
: "page-title-text", "sessionid" : "91cf59aa-1503-4a17-9bbe-
ffa5728fa2ee", "using" : "class name" }
1685771451493   webdriver::server       DEBUG   <- 400 Bad Request
{"value":{"error":"invalid argument","message":"unknown variant `class
name`, expected one of `css selector`, `link text`, `partial link
text`, `tag name`, `xpath` at line 1 column 107","stacktrace":""}}

Apparently that is not supported.

----------------------------------------------------------------------
fpcoder · 2024-08-23 00:44

How to call Robot.ExecuteScript to click a button? So far I have used
Python for web robots, but yesterday I got the WebDriver4L demo
working under Linux and can't resist trying to switch! 

Here is my working Python code

Code: Python  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
DownloadButton = WebDriverWait(browser,
20).until(EC.element_to_be_clickable((By.XPATH, '/html/body/div[2]/div
/button'))) browser.execute_script("arguments[0].click();",
DownloadButton) 
My attempt with WebDriver4L. I confess to not knowing exactly what I'm
doing here..

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var 
DownloadButton: TWebElement;begin  DownloadButton:=
Robot.FindElementByXPath('/html/body/div[2]/div/button'); 
Robot.ExecuteScript('DownloadButton', 'arguments[0].click();') 
And the error message:

QuoteUnexpected token (arguments) encountered
EDIT: Found an example that clicks the button and completes the
download, but the downloaded textfile is garbled and unreadable for
some reason?

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
Robot.ExecuteScript ( 'var xpath = "/html/body/div[2]/div/main/div[2]/
article/div[3]/div[1]/div/div/div[3]/div[2]/div[2]/div/button";' +   
                    'function getElementByXpath(path) {' +           
            'return document.evaluate(' +                         '
 path,' +                         '  document,' +                    
   '  null,' +                         '
 XPathResult.FIRST_ORDERED_NODE_TYPE,' +                         '
 null' +                         ').singleNodeValue;' +              
         '}' +                         'var element =
getElementByXpath(xpath);' +                         'if (!element) {'
+                         'throw new Error("Error: cannot find an
element with XPath(" + xpath + ")");' +                         '}' + 
                       'element.click();' ) ; 
code is borrowed from here:
http://www.tauhidslab.my.id/2024/02/scrape-data-dari-website-kpu-
dengan.html

EDIT2:

After checking the downloaded .CSV file more closely, I found that it
was just in LibreOffice the content seemed garbled. CudaText, MousePad
etc showed it just fine. It turned out to be US-ASCII encoding. I
converted the file to UTF8 instead and LibreOffice read it fine too.

----------------------------------------------------------------------
paweld · 2024-08-23 05:13

Clicking the button is relatively easy. You need to find the button on
the page and simply click it programmatically - I usually search by ID
or Name.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses 
Webdriver4L; procedure TForm1.Button1Click(Sender: TObject);var 
Robot: TWebDriver;  downbtn: TWebElement;begin  try    Robot :=
TFireFoxDriver.Create(nil);    Robot.HideWindow := True;     //hide
robot window    Robot.StartDriver(ExtractFileDir(ParamStr(0)) +
'\geckodriver.exe');  //path to firefox webdriver exe   
Robot.NewSession;   
Robot.GetURL('https://brudnopis.ovh/downloadtest.html');    downbtn :=
Robot.FindElementByID('downloadBtn'); //button id (from page source) 
 if not downbtn.IsEmpty then  //if find button      downbtn.Click;
 //click the button    Robot.Quit;  finally    Robot.Free;  end;end;

----------------------------------------------------------------------
af0815 · 2024-08-23 12:40

BTW: I have written an issue on the repro and ask, if it is allowed to
fork ( https://github.com/Ericwang1104/WebDriver4D/issues/6 ) . I have
seen, EricWang1104 is from time to time available, so it is possible
to get an answer.

I think this is the fastest and easiest way to clear the situation
(for Jurrasic Pork and all working on the project).

----------------------------------------------------------------------
gidesa · 2024-08-23 16:11

Quote from: af0815 on August 23, 2024, 02:40:28 pmBTW: I have written
an issue on the repro and ask, if it is allowed to fork (
https://github.com/Ericwang1104/WebDriver4D/issues/6 ) . I have seen,
EricWang1104 is from time to time available, so it is possible to get
an answer.

I think this is the fastest and easiest way to clear the situation
(for Jurrasic Pork and all working on the project).

Hello, I have done a fork of EricWang Delphi project, where I have
completed a large part of W3C webdriver standard.
By the way, EricWang's project seems no more updated from some years.


https://github.com/gidesa/WebDriver4D

----------------------------------------------------------------------
af0815 · 2024-08-24 06:34

Quote from: gidesa on August 23, 2024, 06:11:51 pmBy the way,
EricWang's project seems no more updated from some years.


https://github.com/gidesa/WebDriver4D
Your repro is also 3 years not updated :-) And a new fork without
clearing the license is IMHO not the best soloution. To use the code,
i have to known if it is GPL or LGPL or ...

----------------------------------------------------------------------
gidesa · 2024-08-24 16:09

Quote from: af0815 on August 24, 2024, 08:34:01 amYour repro is also 3
years not updated :-)
Aha, ah, yes! Although 3 years is better than 5-6 

Quote from: af0815 on August 24, 2024, 08:34:01 amAnd a new fork
without clearing the license is IMHO not the best soloution. To use
the code, i have to known if it is GPL or LGPL or ...

Indeed seems that  Github code in (public) repository WITHOUT explicit
license is a totally exclusive one:
https://opensource.stackexchange.com/questions/1720/what-can-i-assume-
if-a-publicly-published-project-has-no-license
One can only view the code, and fork the repository ....

----------------------------------------------------------------------
Phoenix · 2024-09-01 20:50

I recently found this tool and it opened up a world I didn't know
about.
Thank you very much for this 

Using @paweld's example as a basis, I tried to save a page:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
..Robot.GetURL('https://retrostylegames-com.translate.goog/blog/best-
looking-isometric-games/?_x_tr_sl=auto&_x_tr_tl=it&_x_tr_hl=it&_x_tr_p
to=wapp');Robot.SaveCurDocToFile('d:\temp\text.html');.. 
But is there a way to save it completely translated? I noticed that
when you scroll a page (browser), the displayed portion is updated. Is
it possible to force the entire translation or simulate the scroll
action?

I did some research but I don't know how to get the result with
firefox or chrome

https://stackoverflow.com/questions/53717431/chromedriver-how-to-
translate-a-page-using-selenium

https://stackoverflow.com/questions/68761826/running-forced-automated-
translation-in-google-chrome-and-edge-translation-not

----------------------------------------------------------------------
vfclists · 2024-09-02 00:05

Quote from: af0815 on August 24, 2024, 08:34:01 amQuote from: gidesa
on August 23, 2024, 06:11:51 pmBy the way, EricWang's project seems no
more updated from some years.


https://github.com/gidesa/WebDriver4D
Your repro is also 3 years not updated :-) And a new fork without
clearing the license is IMHO not the best soloution. To use the code,
i have to known if it is GPL or LGPL or ...

It may be covered by the Selenium license and those of other libraries
it depends on.

----------------------------------------------------------------------
fpcoder · 2024-09-02 02:36

To scroll on a page you can use ExecuteScript to call
Element.ScrollIntoView :

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i   s:=
WebDriver.ExecuteScript( 'var xpath =
"/element/of/the/xpath/that/you/want/to/use";' +                     
 'function getElementByXpath(path) {' +                        'return
document.evaluate(' +                        '  path,' +             
         '  document,' +                        '  null,' +          
            '  XPathResult.FIRST_ORDERED_NODE_TYPE,' +               
       '  null' +                        ').singleNodeValue;' +      
                '}' +                        'var element =
getElementByXpath(xpath);' +                        'if (!element) {'
+                        'throw new Error("Error: cannot find an
element with XPath(" + xpath + ")");' +                        '}' + 
                     'element.scrollIntoView(true);' ) ; 
Quote from: Phoenix on September 01, 2024, 10:50:08 pmIs it possible
to force the entire translation or simulate the scroll action?

----------------------------------------------------------------------
Phoenix · 2024-09-02 05:18

Thanks @kapibara for the script  ! Unfortunately it doesn't solve the
translation problem as I hoped. It continues to translate one piece at
the beginning and one at the end in the same way. I also tried adding
a sleep(3000) to give the Browser time before saving but it doesn't
solve the problem..

----------------------------------------------------------------------
fpcoder · 2024-09-02 07:33

I don't know. Sometimes AI can help so I asked: Is it possible to have
Googles translation extension for firefox to translate a webpage
completely even if i don't scroll to the end of a page?

ChatGPT: (https://talkai.info/chat/)

QuoteAs of my last knowledge update in October 2023, Google does not
offer a dedicated translation extension for Firefox that translates
entire web pages automatically without scrolling. However, there are
other ways to achieve webpage translation in Firefox:

    Firefox Translation Feature: Firefox has a built-in translation
feature that can translate entire web pages. You can access it by
right-clicking on the page and selecting "Translate Page."

    Third-party Extensions: There are various translation extensions
available for Firefox, such as "To Google Translate" or "Translate web
pages." These extensions can help you translate pages, but the
functionality may vary, and the translation might still require you to
load the entire content.

    Using Google Translate: You can manually copy the URL of the
webpage you want to translate, go to Google Translate
(translate.google.com), and paste the URL in the translation box.
Google Translate will then open the page translated into your desired
language.

Always check the latest extensions in the Firefox Add-ons store, as
new features and tools may have been introduced after my last update.

Is it possible to start a webdriver with extensions loaded?

ChatGPT

Yes, you can start a WebDriver session with extensions loaded. This
capability can be particularly useful when you want to automate tasks
that involve browser extensions, such as translation tools.

Here's how you can do it with popular web drivers like ChromeDriver
and Firefox's geckodriver.
Using ChromeDriver

To start Chrome with an extension, you can specify the path to the
extension directory when initializing the WebDriver. Here’s an example
in Python using Selenium:

    Install the required libraries:

    
    pip install selenium

    Download the ChromeDriver if you haven't done so already, and make
sure it matches your version of Chrome.

    Load your desired extension: Assuming you have a folder with the
unpacked extension (let's say it's called my_extension), run the
following code:

    
    from selenium import webdriver
    from selenium.webdriver.chrome.service import Service
    from selenium.webdriver.chrome.options import Options

    # Path to the ChromeDriver
    chrome_driver_path = '/path/to/chromedriver'

    # Path to the unpacked extension
    extension_path = '/path/to/my_extension'

    chrome_options = Options()
    chrome_options.add_argument(f'--load-extension={extension_path}')

    service = Service(chrome_driver_path)
    driver = webdriver.Chrome(service=service, options=chrome_options)

    # Now you can navigate to a page and the extension should be
loaded.
    driver.get('https://example.com')

    # Perform your actions...

    driver.quit()

Using GeckoDriver (Firefox)

Similarly, you can start a Firefox WebDriver session with extensions
loaded. Here's how to do it:

    Install the required libraries (if not already done):

    
    pip install selenium

    Download the GeckoDriver if you haven't done so.

    Load your desired extension: Assuming you have a .xpi extension
file, run the following code:

    
    from selenium import webdriver
    from selenium.webdriver.firefox.service import Service
    from selenium.webdriver.firefox.options import Options

    # Path to the GeckoDriver
    gecko_driver_path = '/path/to/geckodriver'

    firefox_options = Options()
    firefox_options.add_extension('/path/to/my_extension.xpi')

    service = Service(gecko_driver_path)
    driver = webdriver.Firefox(service=service,
options=firefox_options)

    # Now you can navigate to a page and the extension should be
loaded.
    driver.get('https://example.com')

    # Perform your actions...

    driver.quit()

Notes

    Make sure that the extension is either unpacked (for Chrome) or in
the proper format (e.g., .xpi for Firefox).
    Some extensions may require additional permissions or specific
configurations to work correctly when loaded via WebDriver.
    Be aware that certain behaviors of extensions might interfere with
automated testing, so always test thoroughly to ensure your scripts
run as expected.

Using these methods, you should be able to start a WebDriver session
with your desired extensions loaded and use them for automation tasks,
including translation functionalities!

----------------------------------------------------------------------
Phoenix · 2024-09-02 17:44

Thanks  , I confirm that if you use firefox and its translation
function (beta) it actually translates the entire page. While passing
a link to google translator this does not happen. To test just save
without scrolling the page and open the files offline. I noticed that
on some opened pages the possibility of using this function does not
appear because the relative icon does not appear (but this is another
story, maybe calculate how much text there is..)
So I'm happy that the function exists, the only problem remains of how
to make it run.

----------------------------------------------------------------------
Phoenix · 2024-09-02 19:05

Trying to find some useful points I did some research (but
unfortunately I don't know much about this topic  ):

Using the extension argument, I think it only works with the old add-
on. Now the feature is integrated.

https://support.mozilla.org/en-US/kb/firefox-translations-add-
on?redirectslug=firefox-translations&redirectlocale=en-US

https://addons.mozilla.org/it/firefox/addon/firefox-translations

https://github.com/mozilla/firefox-translations

Maybe scrolling should be triggered via "Element SendKeys" by sending
the arrow key?

https://developer.mozilla.org/en-US/docs/Web/WebDriver/Commands

The most interesting link I found is about a script but no idea if it
can be converted or its actual usefulness.. 

https://stackoverflow.com/questions/13030153/translate-website-to-any-
specific-language-on-page-load

----------------------------------------------------------------------
Phoenix · 2024-09-03 20:53

It seems to work with one modification
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
element.scrollIntoView({ behavior: "smooth", block: "start" });  

but maybe someone will find a more "automated" method than scrolling..

----------------------------------------------------------------------
fpcoder · 2024-09-04 00:20

I'm not so good at this, but the problem could be how the scrolling is
done. The extension translates page by page when you scroll manually
one page at a time. If we simulate this by using "continous scrolling"
it might work.

Once again, this is code generated by AI and untested and you'll need
to try to make it work. If possible. Please do tell if you succeed.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var 
Driver: TWebDriver;  ScrollScript: String;  ScrollHeight,
NewScrollHeight: Integer;  I: Integer; begin  // Initialize the
WebDriver  Driver := TFirefoxDriver.Create;   try    // Open the
desired URL    Driver.Get('http://example.com');  // Change to your
target webpage     // Allow the page to load    Sleep(3000); // Wait
for 3 seconds – adjust as necessary     // Continuous scrolling    for
I := 1 to 10 do // Change the number of iterations as needed    begin 
    // Execute JavaScript to scroll down      ScrollScript :=
'window.scrollTo(0, document.body.scrollHeight);';     
Driver.ExecuteScript(ScrollScript);            // Wait for content to
load      Sleep(1000); // Wait for 1 second – adjust as necessary    
 // Get the new scroll height      NewScrollHeight :=
Driver.ExecuteScript('return document.body.scrollHeight;').AsInteger; 
          // Break if the scroll height doesn't change, indicating the
bottom has been reached      if NewScrollHeight <= ScrollHeight then 
     Break;       ScrollHeight := NewScrollHeight; // Update the last
scroll height    end;   finally    // Clean up    Driver.Quit; 
end;end.

----------------------------------------------------------------------
Phoenix · 2024-09-04 18:47

Thank you, @kapibara try this  :

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var Driver:
TWebDriver; NewScrollHeight,tmp: Integer;begin Driver:=
TFirefoxDriver.Create(nil); try 
Driver.StartDriver(ExtractFileDir(ParamStr(0)) + '\geckodriver.exe'); 
Driver.NewSession;  Driver.GetURL(<input google link>);  
Sleep(1000);   NewScrollHeight:= 0;  while True do  begin 
 Driver.ExecuteScript('window.scrollTo(0,
window.scrollY+window.outerHeight);');   Sleep(500);   tmp:=
NewScrollHeight;   NewScrollHeight:=
StrToInt(Driver.ExecuteScript('return window.scrollY;'));   if tmp =
NewScrollHeight then    Break;  end;   Sleep(1000);  
Driver.SaveCurDocToFile(<save path>); finally  Driver.Quit; end; 
EDIT NOTE:

doing more tests, a modification is needed
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
NewScrollHeight:= Round(StrToFloat(Robot.ExecuteScript('return
window.scrollY;'))); 
in my case this was also needed
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
FormatSettings.DecimalSeparator:= '.';

----------------------------------------------------------------------
fpcoder · 2024-09-04 23:12

Looks like you made it. Well done.

----------------------------------------------------------------------
gidesa · 2024-09-05 11:03

The EricWang's library is a pure Pascal/Delphi implementation of the
W3C web driver protocol. So no Selenium license apply.
Only method/property names could remind, perhaps, Selenium ones.

Quote from: vfclists on September 02, 2024, 02:05:53 amIt may be
covered by the Selenium license and those of other libraries it
depends on.