[Solved] Need Help with program extracting data lazarus / 21-General ---------------------------------------------------------------------- madref · 2022-11-11 13:42 HowardPC helped me with some code and I still don't get it how he has done the job I asked him about. He managed to extract data from a webpage (a game information sheet). The union has changed it's website code and url so it doesn't work anymore. Is there anyone who can help me change it so it works again? The url to the game sheets is now: https://api.hockeydata.net/data/ih/GameDocument?apiKey=&referer=https: //www.ijshockeynederland.nl/×tamp=1661761801761&gameId=6033efc6-a27c- 4960-a7ee-863a3f41adf7&lang=en&docType=gameReport Where the gameID is: 6033efc6-a27c-4960-a7ee-863a3f41adf7 Because the attachment is to big for here, you can download it here: https://www.vanloononline.nl/PenaltyMinutes.zip ---------------------------------------------------------------------- dje · 2022-11-12 07:02 I don't know this project, but, I could provide some tips. If only the source URL has been changed, then just update the code in uURLtoStrings. You should be able to easily restore the download feature of your program. I can see that HowardPC (?) has included some debugging code in uParseWebPage, which should help you locate the HTML parsing issue. Ideally, you should have a copy of the previous HTML source page. You could use a visual diff tool to compare the two to find out what has changed.  Only then will you know if its a minor fix, vs a complete rewrite. You need to work out how broken the current code is first. ---------------------------------------------------------------------- madref · 2022-11-12 10:26 I already made the change to the url.... The previous was a one part url with a value behind is... (https://www.nijb.nl/nijbsheet.php?GameID=27650 The new one is a 2-part url has the value in the url. (https://api.hockeydata.net/data/ih/GameDocument?apiKey=&referer=https ://www.ijshockeynederland.nl/×tamp=1661761801761&gameId=1e6ba484- 3bbd-476e-bdc5-e29475fbba06&lang=en&docType=gameReport) Where the id is a string (1e6ba484-3bbd-476e-bdc5-e29475fbba06) of numbers and letters The problem is that the original webpage is not available anymore The next problem is that the original code uses values (intergers) and now it uses a string. And as I said this code is higher math for me @HowardPC can you help me out again? ---------------------------------------------------------------------- howardpc · 2022-11-12 13:55 Access to the data requires use of their specialised API, which in turn is restricted thus: "Who may use the API? The use of the API requires the approval of those responsible for the respective league (this is checked when an API key is requested). If you want to use the API to set up your own statistics portals or applications (mobile or desktop-based apps), please contact the relevant league or association directly. In this case, API keys may only be passed on by hockeydata after written confirmation by the league managers. It is not permitted to use the API to integrate widgets into pages for the purpose of betting or gambling. Request API key To request an API key for your website, please fill out the form below. The API key is restricted to the URL of your website. If you want to use widgets in different directories or under different domains, only enter the main domain name here. Data they ask for: Your name (required field) Your e-mail address (mandatory field)  URL of your website (required)  Title of your website (required)  Which leagues would you like to include widgets from? (required field)  annotation (optional)" I don't know if I can access the data via your website and your existing approval(s), and which leagues you want data for. Unfortunately I don't speak a word of Dutch except Sorry (which is also English), so have to use Google Translate (as above) to find my way around their website, nor do I have a website of my own to do a trial if I approached them directly myself. ---------------------------------------------------------------------- Thaddy · 2022-11-12 14:22 Quote from: howardpc on November 12, 2022, 02:55:21 pm nor do I have a website of my own to do a trial if I approached them directly myself. I can setup a subdomain for such things if you like?, Like http://howardpc.thaddy.org. I have enough space. Just send me a private mail so I can setup the subdomain and your requirements. (It will be http, not https, because subdomains do not fall under my certificate) But as a scratch book? You can even run FPC cgi and fastcgi from there, create databases etc. My dot org domain is specifically setup to run tests, not really used for other things. ---------------------------------------------------------------------- paweld · 2022-11-12 15:55 Sample in attachment.Tested on games: - 1e6ba484-3bbd-476e-bdc5-e29475fbba06 - 6033efc6-a27c-4960-a7ee-863a3f41adf7 ---------------------------------------------------------------------- howardpc · 2022-11-12 17:06 paweld has shown you the way using existing fpc json functionality. You can abandon my earlier attempt and all my complex html parsing for good. Come back if you need further help. Thaddy - thanks for your generous offer, it is probably not needed now. ---------------------------------------------------------------------- paweld · 2022-11-12 23:40 Attach a new version of the program - it has been expanded to download all games in the division/season, you just need to specify DivisionID.   Edit: fixed empty the GameID on download a division ---------------------------------------------------------------------- madref · 2022-11-13 11:14 Thanks everyone for the great help...... Now I will start to implement it in my app. If I need any help I know where to find you guys. ---------------------------------------------------------------------- Sheepdog · 2022-11-13 12:18 You may view able to find the old page at... https://archive.org/web/ For instance, if you want me to turn a bright shade of red, you can see what my page about re-using an old Lazarus project as the start of a new one, which is at... http://sheepdogguides.com/dt2c.htm ... As I was saying... you can see what that looked like in 2005 by visiting... https://web.archive.org/web/20050215223231/http://sheepdogguides.com/d t2c.htm (For a different page, just go to https://archive.org/web/ and interact with it.) ---------------------------------------------------------------------- madref · 2022-11-14 16:37 Sometimes the website is down. So I can not retrieve any data. How and where can I create an error so I can give a message? ---------------------------------------------------------------------- paweld · 2022-11-14 19:22 I added error handling when downloading the report, but unfortunately I can't fully check because I get the error "ApiKey invalid" ---------------------------------------------------------------------- madref · 2022-11-16 23:51 That's because the apices didn't get copied right. This is the correct url. https://api.hockeydata.net/data/ih/GameDocument?apiKey=a0a1258810e90e7 7a2367f403c6635d1&referer=https://www.ijshockeynederland.nl/×tamp =1661761801761&gameId=1e6ba484-3bbd-476e-bdc5- e29475fbba06&lang=en&docType=gameReport I even found out that the timestamps can be deleted from the url. ---------------------------------------------------------------------- paweld · 2022-11-17 10:32 thanks for info. ---------------------------------------------------------------------- madref · 2022-11-22 10:34 Is this also possible from a pdf? Example pdf: https://assets.beneleague- icehockey.com/game/2223/7058/esheet ---------------------------------------------------------------------- paweld · 2022-11-24 23:01 The only solution I have found that should work on Mac is to use XpdfTools. You need to download the tools for your system from here: https://www.xpdfreader.com/download.html  and then in the ESheet_ConvertPDF function modify the path to "pdftotext" to the correct one. I tested on Windows. Checked results of a few matches and it seems to download the correct data. The modified sources are in the attachment. ---------------------------------------------------------------------- madref · 2023-02-27 10:35 I am running into a problem. When there is a penalty that is being sat out by a different player. Here is the url for that form: [size=78%]https://api.hockeydata.net/da ta/ih/GameDocument?apiKey=a0a1258810e90e77a2367f403c6635d1&referer=htt ps://www.ijshockeynederland.nl/&gameId=06ff450b-a252-47ae-a83f- 6a38b91d9124&lang=en&docType=gameReport[/size] How can I resolve that? I was trying to do this...but that does not work Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TGameReport.GetPenalties(Team: String; Table: THTMLTableElement);var  row: THTMLTableRowElement;  i, j, v, w: Integer;  s, t: String;  tp: TPenalty;begin  for i := 2 to Table.Rows.Count - 1 do  begin    row := THTMLTableRowElement(Table.Rows[i]);    s := '';    for j := 0 to row.Cells.Count - 1 do      s := s + UTF8Trim(row.Cells[j].TextContent);    if UTF8Trim(s, []) <> '' then  begin      if Team = 'homestats'  then        tp := Self.TeamA.Penalties.Add      else        tp := Self.TeamB.Penalties.Add;      tp.Tijd := UTF8Trim(row.Cells[0].TextContent);      t := UTF8Trim(row.Cells[1].TextContent);        w := length (t);      v := 1;      if w > 2 then begin        while (t[v] <> '/') or (v = w) do        v := v + 1;      end;  // if      if v < w then        tp.Nr := copy (t, 1, v)      else        tp.Nr:= t;        TryStrToInt(UTF8Trim(row.Cells[2].TextContent), v);      tp.Min := v;      tp.Penalty := UTF8Trim(row.Cells[3].TextContent);      tp.Start := UTF8Trim(row.Cells[4].TextContent);      tp.Eind := UTF8Trim(row.Cells[5].TextContent);    end;  end;end; ---------------------------------------------------------------------- paweld · 2023-02-27 16:04 Since I am not familiar with hockey, can you describe how the data should look in this case.    Edit: I looked through your code again and I think I know what you want to achieve. Try this:  Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TGameReport.GetPenalties(Team: String; Table: THTMLTableElement);  var    row: THTMLTableRowElement;    i, j: Integer;    s: String;    tp: TPenalty;    nrarr: TStringArray;  begin    for i := 2 to Table.Rows.Count - 1 do    begin      row := THTMLTableRowElement(Table.Rows[i]);      s := '';      for j := 0 to row.Cells.Count - 1 do        s := s + UTF8Trim(row.Cells[j].TextContent);      if UTF8Trim(s, []) <> '' then      begin        if Team = 'homestats'  then          tp := Self.TeamA.Penalties.Add        else          tp := Self.TeamB.Penalties.Add;        tp.Tijd := UTF8Trim(row.Cells[0].TextContent);         nrarr := UTF8Trim(row.Cells[1].TextContent).Split(['/']);        tp.Nr := nrarr[0];           TryStrToInt(UTF8Trim(row.Cells[2].TextContent), v);        tp.Min := v;        tp.Penalty := UTF8Trim(row.Cells[3].TextContent);        tp.Start := UTF8Trim(row.Cells[4].TextContent);        tp.Eind := UTF8Trim(row.Cells[5].TextContent);      end;    end;  end; ---------------------------------------------------------------------- madref · 2023-02-27 22:54 If you look at 'Screen 1'  you will see [34/80] this means that number 34 has gotten the penalty but number 80 did the time. I only need number 34. ---------------------------------------------------------------------- madref · 2023-02-27 22:57 Yes that did the trick Thank you ---------------------------------------------------------------------- paweld · 2023-02-28 04:47 Quote from: madref on February 27, 2023, 11:54:59 pmIf you look at 'Screen 1'  you will see [34/80] this means that number 34 has gotten the penalty but number 80 did the time.   Thank you for the explanation ---------------------------------------------------------------------- madref · 2023-03-06 23:59 I found an error and don't know how to correct this. If you look at screen 1 you will see the goals and penalty minutes of each team. In the [TOT] column you will see the total of each team. If you look at screen 2 you will see that there are no penalty minutes in the records per period and the total goals for team B is not extracted properly. Can anyone help me? ---------------------------------------------------------------------- paweld · 2023-03-07 06:42 For penalties, I added an additional PAB value in the TPeriod class. In the attachment you have the entire project and only the changes (in diff format). sample:Code: [Select] "Periods" : [       {         "GAB" : "4:3",         "PAB" : "0:2",         "PMAB" : "",         "PPAB" : "",         "Periode" : "1",         "SHAB" : ""       },       {         "GAB" : "5:0",         "PAB" : "2:2",         "PMAB" : "",         "PPAB" : "",         "Periode" : "2",         "SHAB" : ""       },       {         "GAB" : "0:1",         "PAB" : "8:2",         "PMAB" : "",         "PPAB" : "",         "Periode" : "3",         "SHAB" : ""       },       {         "GAB" : "0:0",         "PAB" : "0:0",         "PMAB" : "",         "PPAB" : "",         "Periode" : "OT",         "SHAB" : ""       },       {         "GAB" : "0:0",         "PAB" : "0:0",         "PMAB" : "",         "PPAB" : "",         "Periode" : "PS",         "SHAB" : ""       },       {         "GAB" : "9:4",         "PAB" : "10:6",         "PMAB" : "",         "PPAB" : "",         "Periode" : "TOT",         "SHAB" : ""       }     ], ---------------------------------------------------------------------- madref · 2023-03-07 08:01 thanks for the work again. But I changed it a little back because the values PAB can also be stored in PMAB (PenaltyMinutes AB). And now the HTML_Version gives the same data as the PDF_Version ---------------------------------------------------------------------- paweld · 2023-03-07 08:12 That's how I thought it could be one of the already existing values, so I also gave the changes themselves so you can adjust.