https://www.theregister.com/2023/09/18/lhc_infrastructure_monitoring/ # # Sign in / up The Register(r) -- Biting the hand that feeds IT # # # Topics Security Security All SecurityCyber-crimePatchesResearchCSO (X) Off-Prem Off-Prem All Off-PremEdge + IoTChannelPaaS + IaaSSaaS (X) On-Prem On-Prem All On-PremSystemsStorageNetworksHPCPersonal Tech (X) Software Software All SoftwareAI + MLApplicationsDatabasesDevOpsOSesVirtualization (X) Offbeat Offbeat All OffbeatDebatesColumnistsScienceGeek's GuideBOFHLegalBootnotesSite NewsAbout Us (X) Special Features Special Features All Special Features Cloud Infrastructure Week Blackhat and DEF CON Sysadmin Month The Reg in Space Emerging Clean Energy Tech Week Spotlight on RSA Energy Efficient Datacenters Vendor Voice Vendor Voice Vendor Voice All Vendor VoiceAmazon Web Services (AWS) Business TransformationDDN Google Cloud Data TransformationGoogle Cloud for StartupsGoogle Cloud InfrastructureHewlett Packard Enterprise: AI & ML solutionsHewlett Packard Enterprise: Edge-to-Cloud PlatformIntel vProVMware (X) Resources Resources Whitepapers Webinars & Events Newsletters [databases] Databases 2 comment bubble on white CERN swaps out databases to feed its petabyte-a-day habit 2 comment bubble on white Run 3 reboot provoked challenges for Europe's particle-smashing project icon Lindsay Clark Mon 18 Sep 2023 // 15:00 UTC # Europe's particle accelerator at CERN spews out around a petabyte of data daily, which means monitoring the computing infrastructure that processes the data is crucial. CERN's main activities are based on the Large Hadron Collider (LHC), which propels sub-atomic particles around a 27km circuit, 100 meters underground, then smashes them into each other under the guise of eight distinct experiments. Among them is the CMS experiment, which aims to spot the particles responsible for dark matter, among other things. Like other experiments, CMS shut down for a period of upgrades from 2018 to 2022, and restarted in July last year for the three-year Run 3 period in which scientists will increase the beam energy and sample physics data at a higher rate. [databases] In preparation, four big LHC experiments performed major upgrades to their data readout and selection systems, with new detector systems and computing infrastructure. The changes will allow them to collect significantly larger data samples of higher quality than previous runs. [databases] [databases] But Brij Kishor Jashal, a scientist in the CMS collaboration, told The Register that his team were currently aggregating 30 terabytes over a 30-day period to monitor their computing infrastructure performance. "Entering the new era for our Run 3 operation, we will see more and more scaling of the storage as well as the data. One of our main jobs is to ensure that we are able to meet all this demand and cater to the requirements of users and manage the storage," he said. [databases] "After the pandemic, we have started our Run 3 operations, which creates higher luminosity, which generates much more data. But in addition to that, the four experiments have had a major upgrade to their detectors." The back-end system monitoring the infrastructure that supports the physics data had been based on the time series database InfluxDB and the monitoring database Prometheus. Cornell University's Valentin Kuznetsov, a member of the CMS team, said in a statement: "We were searching for alternative solutions following performance issues with Prometheus and InfluxDB." [databases] Jashal said the system had problems with scalability and reliability. "As we were increasing the detail on our data points we started to experience some reliability issues as well as the performance issue, in terms of how much resources of the virtual machines, and the services being used." * CERN spots Higgs boson decay breaking the rules * CERN celebrates 30 years since releasing the web to the public domain * Galactic anti-nuclei travelers could help illuminate dark matter * CERN, Fermilab particle boffins bet on AlmaLinux for big science In search for an alternative, the CMS monitoring team came across VictoriaMetrics, a San Francisco startup built around an open source wide column time series database, via a Medium post by CTO and co-founder Aliaksandr Valialkin. Speaking to The Register, Roman Khavronenko, co-founder of VictoriaMetrics, said the previous system had experienced problems with high cardinality, which refers to the level of repeated values - and high churn data - where applications can be redeployed multiple times over new instances. Implementing VictoriaMetrics as backend storage for Prometheus, the CMS monitoring team progressed to using the solution as front-end storage to replace InfluxDB and Prometheus, helping remove cardinality issues, the company said in a statement. Jashal told The Register: "We are quite happy with how our deployment clusters and services are performing. We have not yet hit any limits in terms of scalability. We now run the services in high availability mode in our Kubernetes clusters, adding another reliability in the services." The system runs in CERN's own datacenter, an OpenStack service run on clusters of x86 machines. InfluxDB said in March this year it had solved the cardinality issue with a new IOx storage engine. "For a long time, cardinality was the proverbial 'rock-in-the-shoe' for InfluxDB. Sure, it still ran, but not as comfortably as it could. With the InfluxDB IOx engine, performance is front and center, and with cardinality no longer the problem it once was, InfluxDB can ingest and analyze large workloads in real time," it said. (r) Get our Tech Resources # Share More about * CERN * Data * Database More like these x More about * CERN * Data * Database * Storage Narrower topics * Backup * Blu-Ray * Cross-border data flow * DRAM * HDD * MongoDB * MySQL * Network Attached Storage * NoSQL * PostgreSQL * Semiconductor Memory * Snowflake Inc. * SQL * SQL Server Broader topics * Oracle * Software More about # Share 2 comment bubble on white COMMENTS More about * CERN * Data * Database More like these x More about * CERN * Data * Database * Storage Narrower topics * Backup * Blu-Ray * Cross-border data flow * DRAM * HDD * MongoDB * MySQL * Network Attached Storage * NoSQL * PostgreSQL * Semiconductor Memory * Snowflake Inc. * SQL * SQL Server Broader topics * Oracle * Software TIP US OFF Send us news --------------------------------------------------------------------- Other stories you might like Tabular's Iceberg vision goes from Netflix and chill to database thrill Promise of neutral data layer between vendors' vested interests attracts $26M Databases19 Sep 2023 | There are lots of ways to put a database in the cloud - here's what to consider Feature Choosing the right one for you means understanding the trade-off, says MySQL expert Peter Zaitsev Cloud Infrastructure Week15 Sep 2023 | 1 VCs lay $52.5M golden egg for MotherDuck's serverless analytics platform Database service vendor based on open source DuckDB fattens up to $400M valuation Databases21 Sep 2023 | 10Gbps campus networks: How to get LAN infrastructure right Huawei named Gartner Peer Insights Customers' Choice fourth year in a row Sponsored Feature [databases] Oracle cloud hardware to reside in Azure datacenters - and Microsoft's good with that Larry Ellison and Satya Nadella find a common enemy: latency that alows data moving from DBs to AIs Off-Prem15 Sep 2023 | 7 Azure SQL Database takes Saturday off on US east coast following network power failure At least it was the weekend PaaS + IaaS18 Sep 2023 | 3 Venture capital firm makes 'unsolicited' bid for MariaDB buyout Database company, which went through an IPO in December last year, was still in search of credit facility as of August Databases19 Sep 2023 | 4 Toyota servers ran out of storage, crashed production at 14 plants in Japan Oh, what a foul-up as database maintenance created a mess On-Prem7 Sep 2023 | 58 Decades-old Home Office asylum system misses EOL deadline, no new timetable in place Exclusive Replacement due more than a decade ago, Casework Information Database soldiers on as case backlog hits record highs Databases6 Sep 2023 | 38 I'll see your data loss and raise you a security policy violation On Call Engineer trumped angry user by pointing to the rulebook Storage1 Sep 2023 | 202 Google wants to takes a byte out of Oracle workloads with PostgreSQL migration service Third-party and bespoke apps most likely candidates for the switch Databases30 Aug 2023 | 15 Snowflake explains that Instacart's bills aren't melting - it's called 'optimization' Everybody chill, we're still in the cart Storage31 Aug 2023 | 3 The Register icon Biting the hand that feeds IT About Us* * Contact us * Advertise with us * Who we are Our Websites* * The Next Platform * DevClass * Blocks and Files Your Privacy* * Cookies Policy * Privacy Policy * T's & C's * Do not sell my personal information Situation Publishing Copyright. All rights reserved (c) 1998-2023 no-js