Forum slow for others as well ? lazarus / 8-Other ---------------------------------------------------------------------- 440bx · 2025-04-17 07:23 Hello, The forum has been quite slow for me the past 4 hours or so, even getting gateway timeouts. Anyone else experiencing this ? ---------------------------------------------------------------------- Zvoni · 2025-04-17 08:45 This morning i had a 504 TimeOut. But currently it works again.... ---------------------------------------------------------------------- 440bx · 2025-04-17 09:03 Yes, it seems to be working fine now. ---------------------------------------------------------------------- cdbc · 2025-04-17 09:15 Hi YES! It's been like that for a couple of days now...  It started with timeouts ca. 12.00 .. 15.00 GMT+1, then good again Today it started around 8.00 GMT+1 .. till about now, seems fine again...   @Marc: We need your strong Voodoo  ...again  Regards Benny ---------------------------------------------------------------------- Remy Lebeau · 2025-04-17 18:07 Right now, it's working fine. But performance has been up and down for the past couple of months at least: https://forum.lazarus.freepascal.org/index.php/topic,70194.0.html https://forum.lazarus.freepascal.org/index.php/topic,70391.0.html https://forum.lazarus.freepascal.org/index.php/topic,70517.0.html ---------------------------------------------------------------------- 440bx · 2025-04-17 19:34 Quote from: Remy Lebeau on April 17, 2025, 08:07:38 pmRight now, it's working fine. But performance has been up and down for the past couple of months at least: Yes, even in the same day, the performance seems to vary significantly.  Earlier today, performance was really good but, just a minute ago, I got a gateway timeout.  It's quite unpredictable. ---------------------------------------------------------------------- TRon · 2025-04-17 21:26 Same issues here. It comes and goes. At my age I'm getting pretty slow with typing these days but I am still able to outpace the login and posting speed (or the speed of posting alone). At least it doesn't time out as it used to. ---------------------------------------------------------------------- Thaddy · 2025-04-18 07:35 It is a recurring issue but not due to the site manager. It is more like: we need better hardware, but that costs money and that is not really an option (yet). Most bots are evil, but not all of them. There was even a request to index the forum. The result is that many of the popular AI sites produce better Freepascal code and that is noticable...So it is annoying but also gives us better tooling. I usually let these engines fight for what code I should use... Don't worry, 99% I post is by me, myself alone and otherwise I contribute to the source I used. My observation is that the AI models adapt when you post back your own corrected source code.. e.g. if you now ask for a freepascal example they often do a {$mode objfpc}...  Post back your corrected sourcecode! Fighting with those AI engines is great fun for the time being! ---------------------------------------------------------------------- PascalDragon · 2025-04-18 12:58 One could probably deploy a system like Anubis which invokes some proof-of-work and thus makes things harder for AI scrapers. Sites like git.kernel.org employ it because scrapers were repeatedly scraping endpoints like snapshot or blame which are very expensive for the server side. Disadvantage would be that it requires enabled JavaScript though specific browsers could be excluded based on their user agents. The developers are also trying to convert their proof-of-work code from pure JS to wasm which would prohibit big endian systems cause for those there are no wasm engines available as far as I know... 😕 ---------------------------------------------------------------------- dbannon · 2025-04-19 00:29 But its a Catch-22 situation, we want the forum indexed because that is how end users get good information about FPC/Lazarus. But we need the forum usable for our own purpose. Would a solution be providing a second, crawler accessible 'copy' of the forum, maybe replicated daily, rsync like to minimize load ?  The crawlers don't seem to mind how slow their target is. And protect, as best we can, the 'real' forum, maybe redirecting requests identified as crawlers to the relatively static one ? That seems a better approach that the 'trending' one of feeding bad data to a robot.txt ignoring crawler, we do actually want these crawlers to get good data. Davo ---------------------------------------------------------------------- 440bx · 2025-04-19 01:04 I'm not sure but I think there is a way of limiting the number of requests per IP.  If an IP is detected to make more than 1 request per second, it could be limited to getting no more than 1 request per second (or every 2 or 3 seconds... crawlers don't have a life to take care of... they can wait.) That way, any crawler that's hitting the server gets put in its place. A human user would be hard pressed to make more than 1 request per second, therefore there is little chance that could negatively affect forum members and, even if it did, one second isn't that much time to wait to get a page. Just for the record, Google does that.  When it detects a program using its search engine, it doesn't make the program wait, it simply bans the IP for 24 hours (the first time, I don't know what it does the second time, I didn't want to find out ) ---------------------------------------------------------------------- Thaddy · 2025-04-19 09:25 Marc already uses fail2ban - alongside other tools - as far as I know. ---------------------------------------------------------------------- PascalDragon · 2025-04-20 09:58 Quote from: dbannon on April 19, 2025, 02:29:56 amBut its a Catch-22 situation, we want the forum indexed because that is how end users get good information about FPC/Lazarus. But we need the forum usable for our own purpose. Proper behaving search engine crawlers are not a problem (and for example in Anubis they are excluded by default). The problem is AI scrapers that burden the servers of open source software and drive up both their server usage and their traffic usage for improving their AI models without even remotely respecting things like robots.txt. Quote from: 440bx on April 19, 2025, 03:04:21 amA human user would be hard pressed to make more than 1 request per second, therefore there is little chance that could negatively affect forum members and, even if it did, one second isn't that much time to wait to get a page. Well, I tend to open all the "most recently" posts in new tabs and depending on my browser's usage it might indeed do more than one request a second then 😅 ---------------------------------------------------------------------- 440bx · 2025-09-27 21:32 I doubt I'm the only one who's noticed but, the forum has been fairly slow the last 24 hours or so. The number of users seems to be within reason (which doesn't mean there aren't a few abusing the server.) Anyway, figured I'd post this to make it obvious there is a problem (at least for one person ) ---------------------------------------------------------------------- Jurassic Pork · 2025-09-28 02:24 Hello, it is also my case : 18 s to load the home page (see attachment) Friendly, J.P ---------------------------------------------------------------------- cdbc · 2025-09-28 07:07 Hi Yup, got the same problem here, although it's more like 45 seconds..1minute for loading...  (current users = 609, users = 5, guests = 604) Regards Benny    P e  r   h    a     p     s        I        s              h          o              u          l             d                w      r       i         t         e                         s           l            o              w                 e               r?!? ---------------------------------------------------------------------- Marc · 2025-09-28 12:06 Friday I tried a different way of detecting bots. It looked promising, but failed. I've reverted the change ---------------------------------------------------------------------- 440bx · 2025-09-28 12:29 Is there no way to detect the number of requests per second that come from an IP  and limit it some dynamically calculated maximum ? I know Google detects the number of requests per second.  If it's a bit on the high side they put a "gotcha" that must be solved before any additional searches can go through and, if the number of requests per second exceeds what is humanly possible they ban the IP for 24 hours (I have personally experienced both situations.) I believe that it's time to "put the brakes" on bots and/or people who generate more traffic than what is sustainably humanly possible and/or reasonable. ---------------------------------------------------------------------- Marc · 2025-09-28 12:37 The number request from a single IP is not the problem. They use random addresses from /24 or sometimes /20 networks Initially when we suffered from this, I've been playing with ratelimiting for various levels of subnets, but that didn't work well ---------------------------------------------------------------------- 440bx · 2025-09-28 13:25 @Marc, Thank you for the additional info. ---------------------------------------------------------------------- speter · 2025-09-28 22:46 Quote from: 440bx on September 28, 2025, 03:25:42 pm@Marc, Thank you for the additional info.  and MANY thanks Marc, for you ongoing efforts. ---------------------------------------------------------------------- 440bx · 2025-09-28 23:52 Quote from: speter on September 29, 2025, 12:46:01 amand MANY thanks Marc, for you ongoing efforts. Yes, that too! ---------------------------------------------------------------------- Joanna · 2025-09-29 13:53 So this is the new normal? The only place to get help is a forum that might not be usable? ---------------------------------------------------------------------- af0815 · 2025-09-29 14:34 Quote from: Joanna on September 29, 2025, 03:53:49 pmSo this is the new normal? The only place to get help is a forum that might not be usable? No, Marc wrote, he have reverted the changes. And now its normal working for me. Edit: Now its slow again ---------------------------------------------------------------------- Joanna · 2025-09-29 21:11 I don’t have any hopes that this problem is going to go away. Therefore the ability to get help with pascal is unavailable if you happen to need it on an “unlucky day or time”. If I was considering using pascal for the first time, uncertain support would not encourage me to do so. ---------------------------------------------------------------------- 440bx · 2025-09-29 23:54 Quote from: Joanna on September 29, 2025, 11:11:51 pmI don’t have any hopes that this problem is going to go away. Therefore the ability to get help with pascal is unavailable if you happen to need it on an “unlucky day or time”. If I was considering using pascal for the first time, uncertain support would not encourage me to do so. The problem with inconsiderate AI scrapers is NOT specific to Pascal. It is quite likely that every forum that is worth scraping, is or has, experienced the problem, therefore choosing a programming language based on the presence or absence of that problem is likely unwise. Eventually a reasonable, workable solution will be found for this forum, in the meantime, a little patience is required. ---------------------------------------------------------------------- Joanna · 2025-09-30 21:28 That might be true, however getting help with pascal problems did not used to rely entirely upon one place that can be frequently disabled. Now it does. In fact there were other places where many people preferred for getting help which have been destroyed. This forum is the last remaining place to get help and it is more and more frequently disabled. The decision to fully rely upon this one place especially under the current circumstances could be considered imprudent. ---------------------------------------------------------------------- Martin_fr · 2025-09-30 22:15 Moderation note Quote from: Joanna on September 30, 2025, 11:28:33 pm Don't try to change the topic. Don't embed hidden references to circumvent the rules. (For other readers: Hidden ref to IRC => see forum rules) Also "the decision" => is a misrepresentation. There was no decision to give up any alternative services. Any further circumventions of the rules will be deleted.