https://www.wsj.com/articles/inside-intels-secret-warehouse-in-costa-rica-11638181801 WSJ.COM SubscribeSign In Search * Home * News * Research * Newsletters * Events SubscribeSign In Search * Home * News * Research * Newsletters * Events [wsj_pro_logo] This copy is for your personal, non-commercial use only. To order presentation-ready copies for distribution to your colleagues, clients or customers visit https://www.djreprints.com. https://www.wsj.com/articles/ inside-intels-secret-warehouse-in-costa-rica-11638181801 Share * Facebook * Twitter * LinkedIn * Copy Link WSJ PRO Inside Intel's Secret Warehouse in Costa Rica Chip maker is stockpiling legacy technology for security research, plans to expand facility to house 6,000 pieces of equipment [im-441029] Aging tech equipment is stored at an Intel warehouse and laboratory in an undisclosed location in Costa Rica. Intel employees world-wide can put in requests for technicians at the facility to build machines for remote security tests. Photo: INTEL CORP. By James Rundle Close James Rundle * Biography * @JimRundle * james.rundle@wsj.com Nov. 29, 2021 5:30 am ET | WSJ Pro A few years ago, executives at Intel Corp. began to realize they had a problem. The company was making dozens of new products each year, from chips to software platforms, but it didn't have a formal method for cataloging and storing older technology so engineers could test it for security flaws. Some devices, such as Sandy Bridge microprocessors--launched in 2011 and discontinued in 2013--were so scarce that Intel's security researchers resorted to combing the internet for them. "We had to actually go on eBay and start looking for these platforms," said Mohsen Fazlian, general manager of Intel's product assurance and security unit. --------------------------------------------------------------------- Newsletter Sign-up WSJ Pro Cybersecurity Cybersecurity news, analysis and insights from WSJ's global team of reporters and editors. PREVIEW SUBSCRIBE --------------------------------------------------------------------- Intel's issue reflects a wider concern: Legacy technology can introduce cybersecurity weaknesses. Tech makers constantly improve their products to take advantage of speed and power increases, but customers don't always upgrade at the same pace. This creates a long tail of old products that remain in widespread use, vulnerable to attacks. Intel's answer to this conundrum was to create a warehouse and laboratory in Costa Rica, where the company already had a research-and-development lab, to store the breadth of its technology and make the devices available for remote testing. After planning began in mid-2018, the Long-Term Retention Lab was up and running in the second half of 2019. The warehouse stores around 3,000 pieces of hardware and software, going back about a decade. Intel plans to expand next year, nearly doubling the space to 27,000 square feet from 14,000, allowing the facility to house 6,000 pieces of computer equipment. Intel engineers can request a specific machine in a configuration of their choice. It is then assembled by a technician and accessible through cloud services. The lab runs 24 hours a day, seven days a week, typically with about 25 engineers working any given shift. [im-441027] An engineer at the Costa Rica facility assembles a device requested by security researchers for remote testing. Photo: INTEL CORP. The lab gives Intel, which is based in Santa Clara, Calif., and has more than 100,000 employees, a centralized, secure location where security tests can be run from anywhere in the world. Access to the building is strictly controlled and approved by senior managers, while surveillance cameras watch the equipment at all times. Even its location is secret--Intel representatives declined to say where exactly it is. The lab brings commercial value to Intel, Mr. Fazlian said, citing company research that shows customers are more likely to buy technology from manufacturers that proactively test their products. Establishing the lab required getting hard-to-find equipment to Costa Rica, hiring engineers and computer scientists who could work on the machines, and tapping accountants and managers to put processes in place to make it work, said Fawn Taylor, senior director of corporate remediation programs for Intel's product assurance and security unit. At times, contributions came from engineers who had long since moved on to other projects or even left the company. They helped assemble technical documentation and discussed what they knew about products from years ago, Ms. Taylor said. Marcel Cortes Beer, a manager at the lab, said it gets about 1,000 requests a month to build equipment for remote security tests, and 50 new devices come in weekly. Anders Fogh, a Germany-based senior principal engineer at Intel, said the facility quickly became an integral part of his work, particularly when trying to replicate security flaws reported to Intel by outside researchers through its bug-bounty program. Examples of recent vulnerabilities disclosed by Intel include flaws discovered in its Safestring code library that could allow hackers to gain access to sensitive systems, and errors in drivers that could let attackers give themselves credentials. [im-441052] Intel Is expanding the facility to be able to warehouse 6,000 pieces of technology. Photo: INTEL CORP. "I can make an exact replica of the submitting researcher's system. Same CPU, same operating system version, microcode, BIOS," Mr. Fogh said. "All of which increase the chance of reproducing the issue, which is often the best starting point." The lab provides isolated systems for this work, which can otherwise prove dangerous for hard-to-find machines that are still actively used in corporate environments. Testing security vulnerabilities often causes systems to crash, which can result in data loss, Mr. Fogh said. The facility's "huge library of machines is really the go-to place for doing this kind of work," he said. The lab has changed Intel's product development. All new technology is now built with the facility in mind, with technical documentation created to allow engineers to support it for up to 10 years, and units are sent to the lab before they are released, Mr. Fazlian said. "Hopefully, I will never find myself searching eBay for Intel hardware again," he said. Write to James Rundle at james.rundle@wsj.com More From WSJ Pro Cybersecurity * Norsk Hydro Probe Shows Slow Pace of International Ransomware Cases November 23, 2021 * Hackers Circle as Individual Investors Pour Cash Into Crypto November 21, 2021 * Cybersecurity Oversight of Water Sector Falls Short, Report Says November 18, 2021 Copyright (c)2021 Dow Jones & Company, Inc. All Rights Reserved. 87990cbe856818d5eddac44c7b1cdeb8 Appeared in the November 30, 2021, print edition as 'Intel Fills Lab With Old Tech to Test Security Flaws.' MUST READS FROM CYBERSECURITY * Facebook to End Facial Recognition in Photos and Videos Facebook to End Facial Recognition in Photos and Videos * Biden Administration Orders Federal Agencies to Fix Hundreds of Cyber Flaws Biden Administration Orders Federal Agencies to Fix Hundreds of Cyber Flaws * FTC's Effort to Strengthen Online Privacy Protections Faces Hurdles FTC's Effort to Strengthen Online Privacy Protections Faces Hurdles * Workers Get State Income Benefits After Cyberattack on German Manufacturer Workers Get State Income Benefits After Cyberattack on German Manufacturer * SolarWinds Hackers Target Another Weak Point in Tech Supply Chain SolarWinds Hackers Target Another Weak Point in Tech Supply Chain Close [D] Some older Intel devices were so scarce that staff had to comb eBay to test them for bugs. To fix the problem, Intel built a secret tech stash in Costa Rica. Chip maker is stockpiling legacy technology for security research, plans to expand facility to house 6,000 pieces of equipment From [ ] To [ ] Message [ ] SEND An error has occurred, please try again later. Thank you This article has been sent to BACK TO TOP Professional Resources * WSJ Conferences * Factiva * Risk & Compliance Journal * Dow Jones Risk & Compliance * Dow Jones Newswires * CFO Journal * CIO Journal * CMO * Logistics * Facebook * Twitter * Podcasts Send us your feedback:pronewsletter@dowjones.com * Subscriber Agreement & Terms of Use * Privacy Notice * Cookie Notice * Do Not Sell My Personal Information * Copyright Policy * Data Policy 2021 Dow Jones & Company, Inc.All Rights Reserved Copyright 2021 Dow Jones & Company, Inc. All Rights Reserved This copy is for your personal, non-commercial use only. Distribution and use of this material are governed by our Subscriber Agreement and by copyright law. For non-personal use or to order multiple copies, please contact Dow Jones Reprints at 1-800-843-0008 or visit www.djreprints.com.