[solved] Python and Yahoo lazarus / 21-General ---------------------------------------------------------------------- Nicole · 2025-05-29 10:49 There is a possibility to grab data (stock quotes and much more) from Yahoo, by Python. Am I right, that it is rather hard, to make Lazarus work with Python? Or is it possbile? Or is there anywhere a Lazarus way to say, "dear yahoo, get me the close-quote of exasol of the 20 of may 2025"). As the task seems quite useful, there may be a prefab solution around. I can see all those pleasant information shown by Yahoo. However, it takes some seconds to show up and I have to click two times. This makes me pessimistic. ---------------------------------------------------------------------- gues1 · 2025-05-29 10:53 Look that. I use it in Delphi and it works. It's compatible with Lazarus. https://github.com/pyscripter/python4delphi ---------------------------------------------------------------------- Warfley · 2025-05-29 16:39 The Python implementation just queries the internal Yahoo finance API, its a simple REST API you can query using HTTP. For example to download the price history of a stock, you can look St the Python code: https://python-yahoofinance.readthedocs.io/en/latest/_modules/ya hoofinance/historicaldata.html#HistoricalPrices Code: Bash  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i url = 'https://query1.finance.yahoo.com/v7/finance/download/{i}'        r= requests.get(url.format(i=instrument),            cookies={'B': cookie},            params={                "period1": start_period,              "period2": end_period,                "interval": frequency,                "event": event,                "crumb": crumb            }        )         self.prices = r.textHardly rocket science, only hurdle is that you need the Crumb and the cookie, which you can fetch by doing a "normal" request to the yahoo finance Main site (see Python function _find_cookie_crumb_pair) That said, in many countries web scraping is not allowed if the service forbids it or places technical measures to avoid scraping. The cookie and crumble was implemented in 2017/because Yahoo explicitly wanted to stop automatic querying of their API. So you should check if in your jurisdiction it is legal. Here in Germany what you wantto do would be illegal, no matter if you do it in Python or pascal ---------------------------------------------------------------------- Nicole · 2025-05-29 19:13 Thank you for the answers. Do I need to install the Python plugin for the second solution? If not, how to use it? And are you sure, it works still? I wasted so many days in implementing solutions to find out, they cannot work. I am from Austria. Here the law is less restrictive than Germany. And if it should be illegal never the less, - pls tell nobody, ---------------------------------------------------------------------- paweld · 2025-05-30 04:56 Quote from: WarfleyThat said, in many countries web scraping is not allowed if the service forbids it or places technical measures to avoid scraping. The cookie and crumble was implemented in 2017/because Yahoo explicitly wanted to stop automatic querying of their API. So you should check if in your jurisdiction it is legal. Here in Germany what you wantto do would be illegal, no matter if you do it in Python or pascalI wouldn't call using the API as “scraping.” That's what the API is exposed for, so that the user can download data for himself without any problems. ---------------------------------------------------------------------- Thaddy · 2025-05-30 05:51 @Nicole Why use python for that? ---------------------------------------------------------------------- Leledumbo · 2025-05-30 06:51 There is a better (still unofficial and probably undocumented) way to query than the v7 API, which is the v8 API. This one requires no auth and can be publicly scraped (illegally, probably): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc}{$H+} uses  opensslsockets,  fphttpclient,  fpjson,  jsonparser; const  {    period1 = start    period2 = end    both are in Unix timestamp  }  URL = 'https://query2.finance.yahoo.com/v8/finan ce/chart/EXL.DE?period1=1747699200&period2=1747785600';  var  J,M: TJSONData;begin  with TFPHTTPClient.Create(nil) do    try      RequestHeaders.Values['User-Agent'] := 'Mozilla/5.0'; // required, Yahoo will respond with HTTP 429 if this is empty      J := GetJSON(Get(URL));      M := J.FindPath('chart.result[0].meta');      WriteLn(M.AsJSON);    finally      J.Free;      Free;    end;end. I'm not sure what you're looking for so I'm just taking the meta field, parse it further to get what you need. ---------------------------------------------------------------------- gues1 · 2025-05-30 09:37 Quote from: paweld on May 30, 2025, 06:56:39 amQuote from: WarfleyThat said, in many countries web scraping is not allowed if the service forbids it or places technical measures to avoid scraping. The cookie and crumble was implemented in 2017/because Yahoo explicitly wanted to stop automatic querying of their API. So you should check if in your jurisdiction it is legal. Here in Germany what you wantto do would be illegal, no matter if you do it in Python or pascalI wouldn't call using the API as “scraping.” That's what the API is exposed for, so that the user can download data for himself without any problems. If it is done without human intervention (for example with bots or automated tools) it is called "webscraping" ---------------------------------------------------------------------- paweld · 2025-05-30 10:45 Quote from: gues1 on May 30, 2025, 11:37:54 amIf it is done without human intervention (for example with bots or automated tools) it is called "webscraping" I may be f***ed up, but in my opinion, at the core of web/rest api is process automation. ---------------------------------------------------------------------- Thaddy · 2025-05-30 11:01 It is not scraping because it is a provided API and it needs just one (1) connection to run all day. Frankly this is exactly how all banks and traders get their data. I know, because it payed my pension. Still bemused why anybody needs Python for that: it is a rest based transfer protocol, iow language agnostic. Note banks do get their data from specialist companies, guaranteeng ms (even ns resolution for local), not Yahoo, which is way too slow for traders, but the principle is the same. E.g. Reuters offers real-time ticks(trades or bids, volume may arrive out of sync) in ns resolution for NYSE and NASDAQ for companies that connect on local fiber and their specified hardware. In this context ms means not 1 ms, it means in ms range upto 100 (not 1000), ns means ns range upto but not including 1 ms. ---------------------------------------------------------------------- gues1 · 2025-05-30 12:25 Quote from: paweld on May 30, 2025, 12:45:17 pmQuote from: gues1 on May 30, 2025, 11:37:54 amIf it is done without human intervention (for example with bots or automated tools) it is called "webscraping" I may be f***ed up, but in my opinion, at the core of web/rest api is process automation. Quote from: Thaddy on May 30, 2025, 01:01:54 pmIt is not scraping because it is a provided API and it needs just one (1) connection to run all day. Frankly this is exactly how all banks and traders get their data. I know, because it payed my pension. @Thaddy, good to know that with a pubblic API you can get the pension. But I think you would need more, like a account, 2FA security, OTP and others ... or not ?  Web scraping is about "catch" free data public available without any security measures, like a public API of weather services or geolocation or scan a web site (but I'm sure not API bank  ) This in Italy (and in Europe too, I think) is permitted only for statistics purpose and without any economic scope (unless it is prohibited by the provider, but without protective measures they certainly can't take you to court) ---------------------------------------------------------------------- paweld · 2025-05-30 12:46 Quote from: gues1 on May 30, 2025, 02:25:32 pmWeb scraping is about "catch" free data public available without any security measures, like a public API of weather services or geolocation or scan a web site (but I'm sure not API bank  ) From a web site: YES, this is web scraping. From API: NO.  And it doesn't matter whether these are public resources or not. What matters is the site use rules. ---------------------------------------------------------------------- Nicole · 2025-05-30 15:46 @ Leledumbo   This works. You helped me to solve something, which kept me busy since weeks. I gave the link to ChatGPT and it solved all there was to solve. We brought the data into my interface-Var. This I sent to my DB and voila... for others who may seek it: the ticker must have in this case an ".DE" attached. And you must Yahoo tell, you are a browser and not some scrapping software. No way! btw: The abilities of ChatGPT are in a way, that it has the power to stop the world as we know it. It is able to hack the communications and computers of this world, power off the tools and interchange communications. It understands all languages of this world. And interpreters, programmers, journalists, authors, artits, musicians...? They are not necessary any more soon. ChatGPT changes the world. ---------------------------------------------------------------------- gues1 · 2025-05-30 17:21 Quote from: Nicole on May 30, 2025, 05:46:46 pmThe abilities of ChatGPT are in a way, that it has the power to stop the world as we know it. It is able to hack the communications and computers of this world, power off the tools and interchange communications. It understands all languages of this world. And interpreters, programmers, journalists, authors, artits, musicians...? They are not necessary any more soon. ChatGPT changes the world. SkyNet is coming