Post Adb32k5aVDlGWS8GRs by stefano@mastodon.bsd.cafe
(DIR) More posts by stefano@mastodon.bsd.cafe
(DIR) Post #AdXUwa1T2LELeHDPBA by stefano@mastodon.bsd.cafe
2024-01-05T15:06:54Z
2 likes, 1 repeats
This morning, a VPS hosting a small e-commerce site (powered by WooCommerce on Ubuntu 22.04) experienced another out-of-memory issue. A colleague (one of the developers) urgently called me, asking to upgrade the VPS due to excessive load.I pointed out (again, as has often happened in recent weeks) that a VPS with 32GB of RAM (!!!) and 16 dedicated cores (!!!) should not run out of memory with just five simultaneous visits to a small e-commerce site. We host much larger and busier websites on much smaller VPSs. There's likely a WordPress module with a leak, or some interaction between modules causing this issue.The response: Okay, but just add more RAM and power to solve it.The illusion of "infinite resources" in the Cloud has led to poor development habits. Sometimes I wonder how much energy and resources we waste (and pollution we generate) due to a lack of basic optimization.#Tech #Development #CloudComputing #Optimization #WebHosting #WooCommerce #Ubuntu #WordPress #Sustainability #TechRant
(DIR) Post #AdXVZaGe3o6XhYufXU by ado@furry.engineer
2024-01-05T15:13:03Z
0 likes, 0 repeats
@stefano Out of curiosity, where would you start to solve this? `vmstat?
(DIR) Post #AdXW36amxLZULyPb0K by stefano@mastodon.bsd.cafe
2024-01-05T15:19:14Z
0 likes, 0 repeats
@thomzane yesterday, another website experienced a flood of visits by the Bytedance bot. In case of self scaling, I wonder how higher the costs of that VPS could have arrived.
(DIR) Post #AdXWZbqkHGc4YyNI80 by stefano@mastodon.bsd.cafe
2024-01-05T15:25:09Z
0 likes, 0 repeats
@ado I start looking at the logs and a couple of ps. When I see the php processes that start eating ram like crazy, I understand what's happening. Then I try to limit the number of concurrent php processes but this usually kills the performance, for concurrent visits. The only way to solve is...fixing the code 😉
(DIR) Post #AdXXBB1aSe8WN2MKsC by stefano@mastodon.bsd.cafe
2024-01-05T15:31:55Z
0 likes, 0 repeats
@polishdub I think it's quite common, nowadays. Managements tend to believe to ads more than they should.
(DIR) Post #AdXXgxJ500fV5lh7dw by linus@telegrafverket.cc
2024-01-05T15:37:40Z
0 likes, 0 repeats
@stefano cloud images often come without any swap configured, sometimes it’s as easy as just configuring swap period to at least avoid OOM murder. with Ubuntu you also have the luxury of making it ZFS.
(DIR) Post #AdXXhnmHuevhpPlBOC by stefano@mastodon.bsd.cafe
2024-01-05T15:37:46Z
0 likes, 0 repeats
@sanity @polishdub or Raspberry PIs 😉
(DIR) Post #AdXXnNxwnHqLVlsNbE by stefano@mastodon.bsd.cafe
2024-01-05T15:38:51Z
0 likes, 0 repeats
@linus that's a VM I have personally installed in a Proxmox VM - swap is enabled, but gets filled up as well
(DIR) Post #AdXXyCeJD3pZbAqfeC by linus@telegrafverket.cc
2024-01-05T15:40:47Z
0 likes, 0 repeats
@stefano if swap isn’t compressed and what’s being put there is text for example it could make a big difference, not just for storage but paging throughput as well. still, obviously there’s no way to fix bad code with platform changes.
(DIR) Post #AdXYF0K8HwpWILQGrg by stefano@mastodon.bsd.cafe
2024-01-05T15:43:51Z
0 likes, 0 repeats
@linus sure, that would help. But I can't tolerate that a single visit to a simple e-commerce spins up many php processes that occupy more than 6 GB of ram, per concurrent visits. There's something absolutely wrong there.
(DIR) Post #AdXYH1mTqEmJOSeAu8 by gyptazy@mastodon.bsd.cafe
2024-01-05T15:44:14Z
0 likes, 0 repeats
@stefano unfortunately a common issue nowadays. Just throw more resources on it - infinity resources and power!
(DIR) Post #AdXYJkvAyx9PyN10QC by stefano@mastodon.bsd.cafe
2024-01-05T15:44:41Z
0 likes, 0 repeats
@sanity @polishdub I'm trying to move snac to a Raspberry PI zero running NetBSD. I think it'll collapse immediately, but I'm going to try
(DIR) Post #AdXYU9cM14LJSmAa8G by stefano@mastodon.bsd.cafe
2024-01-05T15:46:35Z
0 likes, 0 repeats
@gyptazy then, often, they run out of money and start asking for "discounts". They prefer to waste money in useless resources than in people managing their business
(DIR) Post #AdXYYoAhBCwAOgS4e0 by wtfismyip@gnu.gl
2024-01-05T15:47:24Z
0 likes, 1 repeats
@stefano @linus On https://wtfismyip.com, we're serving around 1000rps. Memory footprint is under 8GB; uses about 3 modern AMD cores. Seems many webapps using PHP aren't well thought out from a performance perspective.
(DIR) Post #AdXYfM9JlR4bVyvhrc by stefano@mastodon.bsd.cafe
2024-01-05T15:48:37Z
0 likes, 0 repeats
@wtfismyip @linus completely agree!
(DIR) Post #AdXYnFwUizxua6hNuC by gyptazy@mastodon.bsd.cafe
2024-01-05T15:50:02Z
0 likes, 1 repeats
@stefano reminds me of a customer when working in a DC where he managed his system on his own, had 64G RAM and his Java application running on Wildfly ran out of memory. I asked if I might have a look at it before upgrading and the application came bundled with a wildfly instance where the JVM was limited to 2GB XmX :)
(DIR) Post #AdXZXGkQ3yVcOpzWgC by stefano@mastodon.bsd.cafe
2024-01-05T15:58:20Z
0 likes, 0 repeats
@polishdub @sanity I'll try to switch as soon as I'll be home. Let's see 😉
(DIR) Post #AdXaDhINkWNMtm6mki by akop@mastodon.akop.online
2024-01-05T16:05:59Z
0 likes, 1 repeats
@stefano I also noticed that the goals of developer are changing.In one of mine web-project happend a OOM (~4 concurrent users). The reaction of my dev-teammates? "#Kubernetes will restart the pod. We have no time to fix that."I never had to fight so hard, that we get some time to fix a OOM. IMO a OOM is a very critical exception.
(DIR) Post #AdXb0Ls8Bp1ge21vCS by stefano@mastodon.bsd.cafe
2024-01-05T16:14:48Z
0 likes, 0 repeats
@akop I agree. An OOM is a serious problem. Auto restart is a workaround, not a fix.
(DIR) Post #AdXbN9Q0eqhogHkQgC by stethewwolf@mastodon.uno
2024-01-05T16:18:55Z
0 likes, 0 repeats
@stefano usually I deal with the next level puzzle: trying to explain that 9 engineers cannot complete the job one person can do in 9 months
(DIR) Post #AdXe52PuFcT7cEkp9M by oxyhyxo@mastodon.bsd.cafe
2024-01-05T16:49:15Z
0 likes, 0 repeats
@stefano the incentive structures from the top down demand new features as quickly as possible and above all else. So many orgs will drive off a cliff before even thinking about slowing down and stabilising
(DIR) Post #AdXeVqut7ImZlc22BU by Nerdfest@mastodon.online
2024-01-05T16:54:06Z
0 likes, 0 repeats
@stefano @ado "Hardware is cheap and programmers are expensive"
(DIR) Post #AdXftxW0KoDb8WS38a by xbezdick@masto.ai
2024-01-05T17:09:40Z
0 likes, 1 repeats
@stefano Glad you have asked - https://media.ccc.de/v/37c3-11796-energy_consumption_of_datacenters
(DIR) Post #AdXiAsIdBBtRn8OEnA by abit_kt7@mastodon.social
2024-01-05T16:18:11Z
0 likes, 0 repeats
@akop I have been ~ordered to deploy a kubernetes cluster on top of... an already workload scheduler. But I am still fighting.@stefano
(DIR) Post #AdXiAt9nzUQaS3Ghfs by stefano@mastodon.bsd.cafe
2024-01-05T17:35:08Z
0 likes, 0 repeats
@abit_kt7 @akop keep fighting!
(DIR) Post #AdXiLZyjdWW29iBiy0 by oxyhyxo@mastodon.bsd.cafe
2024-01-05T16:49:54Z
0 likes, 0 repeats
@stefano and anyone inside the org that pushes back against the constantly rising tempo will be weeded out
(DIR) Post #AdXiLaqcPBcKqpOkxE by stefano@mastodon.bsd.cafe
2024-01-05T17:37:05Z
0 likes, 0 repeats
@oxyhyxo In this specific situation, they just don't think that optimization is needed at all. When anything goes wrong, just give more resources or restore from backup.
(DIR) Post #AdXjwVWpFEuF1SJ34q by stefano@mastodon.bsd.cafe
2024-01-05T17:54:57Z
0 likes, 0 repeats
@KayOhtie @winload_exe They have much complex, busier and full e-commerce nodes with much less ram and cpu... but it seems they don't seem to understand what optimization means.
(DIR) Post #AdXkDrp7basj5n8qyu by oxyhyxo@mastodon.bsd.cafe
2024-01-05T17:58:06Z
0 likes, 0 repeats
@stefano ugh. I’ve had clients like that. At some point once the bills have gotten out of hand they might actually look but until then 🤷
(DIR) Post #AdXkIbOM7qA3Zusd4y by stefano@mastodon.bsd.cafe
2024-01-05T17:58:58Z
0 likes, 0 repeats
@oxyhyxo exactly...
(DIR) Post #AdXlgGnURmyAbqUC6i by abit_kt7@mastodon.social
2024-01-05T17:54:17Z
0 likes, 0 repeats
@xbezdick @stefano That's the wrong target, such DCs and providers are very well optimized and more energy conscious, even more since RU-UA crisis.on-prem waste is real. it doesn't mean you have to rush for the cloud, you just have to change your BIGFOUR mindset.Why do you need a freaking TMA, a remote rigid indian sweatshop to manage the freaking hypervisor you have 50m from the decision maker office? Why do you still use a mainframe?
(DIR) Post #AdXlgHlOr2tLbeW2UK by abit_kt7@mastodon.social
2024-01-05T18:08:06Z
0 likes, 1 repeats
@xbezdick @stefano The last joke I heard of before I quit that client: vSphere on Azure.just WTF?!And I prefer to not talk about the massive resources they have on-prem. No way they would get back a 1/10 of that on Azure price-to-price. Sure, products are "managed" on Azure but still....
(DIR) Post #AdXnBaKTydhq7PPjBw by rimmon1971@misskey.social
2024-01-05T18:28:36.029Z
0 likes, 1 repeats
@abit_kt7@mastodon.social @xbezdick@masto.ai @stefano@mastodon.bsd.cafe "my IT people knows vSphere, and their manager knows vSphere, and the company's department heads knows vSphere... And someone high on the food chain gets back something from VMware".In the past, I saw a multinational's Azure VNET gateways set to their on premise Forcepoint firewall, because their network/firewall people knew Forcepoint (only). #Azure #vSphere
(DIR) Post #AdXo7yp945AqsnbKZE by philip@mastodon.mallegolhansen.com
2024-01-05T18:41:49Z
0 likes, 0 repeats
@stefano At that kind of resource use, I’d start to wonder if the hosting cost isn’t significantly eating into any potential profit from the site.If “That’s insane” doesn’t change minds, maybe “That’s very expensive” might?
(DIR) Post #AdXqM0H0fksnLBRuVc by stefano@mastodon.bsd.cafe
2024-01-05T19:06:48Z
0 likes, 0 repeats
@philip For that client, we rent physical servers and partiton them into VMs, so they don't really get the impact of those choices. When I say "that VM is eating more than half the resources of the server", they just reply "Ok". Maybe they should put those services into an "autoscaling cloud" - and will start optimizing as soon as they'll see the bills. But I'm unable to give bad advices to my colleagues/clients - I'd feel really bad.
(DIR) Post #AdXr15T8GCxjKLi3W4 by philip@mastodon.mallegolhansen.com
2024-01-05T19:14:11Z
0 likes, 1 repeats
@stefano Fair enough, definitely not suggesting you should do anything unethical, I agree with you there.If they're paying for the whole physical server either way, and you're just updating the relative resources applied to the VM, I can see how it might feel "free" to just ask for more RAM. In the long run it's probably a bad decision, time will tell.
(DIR) Post #AdXrcjwajnLhZbRXTU by stefano@mastodon.bsd.cafe
2024-01-05T19:21:02Z
0 likes, 0 repeats
@philip Exactly. When they'll ask for a new VPS and I'll reply "sorry, all the resources have already been given to the other VMs", they'll understand 🙂
(DIR) Post #AdXrjnDMyZQZkBebnU by thepanz@phpc.social
2024-01-05T19:22:17Z
0 likes, 1 repeats
@stefano been there too. A PHP application suddenly misbehave and RAM skyrocketing. The "other devs" just increased 5x the nodes and 10x the RAM, so that the issue appeared less frequently.I had to fight to have a look, 1h was enough to find that the app was waiting for an API response, which was dropped by new firewall rules. Without any timeout on PHP, Ngnix was simply spinning up new threads for PHP which filled up the RAM.2 PHP lines changed, and moved back to 2 CPUs and 512MB of RAM.
(DIR) Post #AdXsLcYj2crthUgH9U by stefano@mastodon.bsd.cafe
2024-01-05T19:29:09Z
0 likes, 0 repeats
@thepanz this. All I'm asking is to spend just some minutes on the dev (yes, I created a perfect copy of that server to perform some tests) to identify the "offending" plugin or combination.The point is, they think it's a waste of time as the solution is to give more resources.The problem is also that the company owners are good people but completely incompetent when it comes to these kinds of topics. So, they see their developers insisting that debugging is useless and that the issue is that power alone isn't enough, and they somewhat start to believe it too.
(DIR) Post #AdXsW9AsdInNkdXx8y by stefano@mastodon.bsd.cafe
2024-01-05T19:31:02Z
0 likes, 0 repeats
@Nerdfest @ado "Hardware is silent and programmers are sure that optimizing is a total waste of time" 🙂
(DIR) Post #AdXsZm9aAMRrVlfarI by stefano@mastodon.bsd.cafe
2024-01-05T19:31:42Z
0 likes, 0 repeats
@polishdub @sanity I had to revert back, but I'll try again with an Orange PI PC and a Banana Pro (both armv7)
(DIR) Post #AdYkcveggsDpNFAhge by mckern@xoxo.zone
2024-01-06T05:37:19Z
0 likes, 0 repeats
@stefano FWIW I have workloads on soon to be EOL 18.04 hosts that handle anything we throw at them but as soon as we try to move the workloads to 22.04, we start OOMing under even modest load. I genuinely think something is rotten under the hood in 22.04.
(DIR) Post #AdYsOBWYCwvlsaqWYa by stefano@mastodon.bsd.cafe
2024-01-06T07:04:19Z
0 likes, 0 repeats
@mckern interesting. Is that workload in PHP?
(DIR) Post #AdYt1b0TOjSCCqdPs0 by mckern@xoxo.zone
2024-01-06T07:11:25Z
0 likes, 0 repeats
@stefano CI workloads, so Docker (and containerd), an agent written in Go, and whatever our users are funneling through their pipelines -- could be Go, could be Python, could be an appalling amount of Typescript. But it will start OOMing under modest load regardless of what's *in* the load. These are amd64 EC2 instances (cvm, not nitro I think), which is about as close to a "vanilla" VM as you'll find in a cloud environment.
(DIR) Post #AdZG8mB6YiETpK05Nw by vasper@libretooth.gr
2024-01-06T11:30:26Z
0 likes, 1 repeats
@stefano had a similar experience with a prestashop site, a month ago. The owner was willing to pay for a 128gb ram site with 24 cores to run a shop for the local market! That would cost him +200$/m. I managed to cut down the resources the site needed to run to 1/5th it used by implementing some best practices on the specific shop and I had him upgrade his system to just 16gb ram from 8 and 4 cores from 2. Responds instantly.The joke? I billed him 300$, but he thought I was too expensive!
(DIR) Post #AdZHb9wft0ULbOLgvI by stefano@mastodon.bsd.cafe
2024-01-06T11:46:45Z
0 likes, 0 repeats
@vasper that's the point. The often prefer to pay for more hardware than (much less) to optimize what they have (and that will keep working with less expenses)
(DIR) Post #AdZKnCJvBKSMIzecls by Grandalf@aus.social
2024-01-06T12:21:22Z
0 likes, 1 repeats
@vasperThat's a well-known and much used psychological effect: people are happy to pay per month while railing at a one-off payment.@stefano
(DIR) Post #AdZOPqQYoRXbYtsaoK by fedops@fosstodon.org
2024-01-06T12:39:28Z
0 likes, 1 repeats
@Grandalf @vasper @stefano I actually see that with physical equipment as well. Replacing a component with a better one, thereby increasing cost of a machine by 500 euro is a no-go. But paying 2000 euro on additional maintenance *per year per machine* instead is fine.TCO is a concept that many people don't understand.
(DIR) Post #AdZVSv72Y6g6dmA9AW by imkali@mas.to
2024-01-06T14:22:08Z
0 likes, 1 repeats
@stefano @uhuru "Guys guys, we don't need to optimise our PC port of the game! Guys guys PC players can just buy a 4090Ti! Guys guys we don't need to do that just tell the consumer they need a new PC!"
(DIR) Post #AdZirbqUcQchWrWfHk by bennes@k8s.social
2024-01-06T16:52:16Z
0 likes, 0 repeats
@stefano @gyptazy The problem, at least in my experience, lies in the fact that paying half a day of a single person optimizing the code or finding the leak and fix it has a much higher cost than just paying the cloud provider for more resources :( which is sad but still…
(DIR) Post #AdZjsGCmfAG6ejpJGS by stefano@mastodon.bsd.cafe
2024-01-06T17:03:37Z
0 likes, 0 repeats
@bennes @gyptazy they have their own employees. It's not a matter of money, but of habits
(DIR) Post #AdZt1CI7U22ei8etcm by oschonrock@mastodon.social
2024-01-06T18:45:59Z
0 likes, 0 repeats
@stefano that's not a VPS.. that's a dedicated machine!
(DIR) Post #AdZt5bpuRwHZ5zZEh6 by stefano@mastodon.bsd.cafe
2024-01-06T18:46:51Z
0 likes, 0 repeats
@oschonrock This VPS is using a big percentage of the dedicated resources...
(DIR) Post #AdZtCY4eprZtSFm5Ng by oschonrock@mastodon.social
2024-01-06T18:48:03Z
0 likes, 0 repeats
@stefano indeed
(DIR) Post #AdaCu5EXK8tLsTrrvs by glzr@mastodon.bsd.cafe
2024-01-06T22:28:52Z
0 likes, 0 repeats
@stefano i di ran SAP ECC for 200 users in less then that 🙂 mad world
(DIR) Post #AdaDutFZSlSwxKIgAS by stefano@mastodon.bsd.cafe
2024-01-06T22:40:15Z
0 likes, 0 repeats
@glzr true!
(DIR) Post #AdaIM80N7224LvR0C0 by greatquux@mastodon.social
2024-01-06T23:29:58Z
0 likes, 0 repeats
@stefano While i totally sympathize, and finding the leak is important, maybe just set a reboot or http server restart for, lets says, 4:20am. 🌿
(DIR) Post #AdaXH6zCyHsF3cxLuq by 0x00string@infosec.exchange
2024-01-07T02:17:07Z
0 likes, 0 repeats
@stefano i can tell you... that the actual real life answer would upset you.
(DIR) Post #AdaYDl1kG4Lo2Q6ifo by the_Effekt@universeodon.com
2024-01-07T02:27:44Z
0 likes, 0 repeats
@stefano I saw this at an agency as a front end developer, building Coldfusion site on MURA templates.It wasn't about optimizing functionality. It was about packing in code (or extensions/plugins) until something worked. I was even admonished for stripping out the previous failures.
(DIR) Post #AdaafNY1LbU3EbVE1o by jrefior@hachyderm.io
2024-01-07T02:55:07Z
0 likes, 0 repeats
@stefano In my work experience, the amount wasted is massive.
(DIR) Post #Adb2zp6NPPqTCU6q36 by stefano@mastodon.bsd.cafe
2024-01-07T08:12:35Z
0 likes, 0 repeats
@0x00string Go on 😉
(DIR) Post #Adb32k5aVDlGWS8GRs by stefano@mastodon.bsd.cafe
2024-01-07T08:13:07Z
0 likes, 0 repeats
@greatquux sure, but that would solve the problem for something like...5 minutes 😉
(DIR) Post #AdbKenOQMk8wzgfqOe by SenseException@phpc.social
2024-01-07T11:30:27Z
0 likes, 0 repeats
@stefano Is it still faster and cheaper accomplished that letting a developer find out one or multiple reasons for these problems?I can tell that devs weren't allowed to put time into research and fixes because of this reason.
(DIR) Post #AdbL0XEn6zObGAXBWy by stefano@mastodon.bsd.cafe
2024-01-07T11:34:25Z
0 likes, 0 repeats
@SenseException they have some servers, and they're moving from the old setup to this "new" setup. That dedicated will never be enough, especially because there's a leak so there's no upper limit to the resources requested.They have two choices: dedicate some time to fix this or just see all their servers crashing every 5 visits.
(DIR) Post #AdbzSbDLR96CUl5YIK by pies@qoto.org
2024-01-07T19:07:39Z
0 likes, 0 repeats
@stefano In a business it might make sense to raise the resources until the problem is fixed. It might also be cheaper to run a larger server instead of getting devs to optimize it/fix it.
(DIR) Post #Adc17YZ0IS5i6LJTjk by stefano@mastodon.bsd.cafe
2024-01-07T19:26:17Z
0 likes, 0 repeats
@pies There's no limit, here. There's a leak, so resources will be saturated just a few seconds later.They need to fix.