https://www.fujitsu.com/global/about/resources/news/press-releases/2024/0510-01.html This is a skip link click here to skip to main contents Fujitsu Global * [icon-earth]Change Change Language o o World Location [global_gre] Change Location This is a list of country/region websites in the Fujitsu group. If your country/region is not in this list, please visit ' World Locations'. o Japan (HQ) o Americas o Asia Pacific o Europe o Japan o Global o United States o Canada (English) o Canada (French) o Caribbean o Brazil o South America o Other countries o APAC (English) o APAC (Indonesian) o APAC (Vietnamese) o China | Zhong Guo o Hong Kong | Xiang Gang Di Qu o Korea o Taiwan | Tai Wan Di Qu o Thailand (Thai) o Other countries o Austria o Belgium o Denmark o Estonia o Finland o France o Germany o Ireland o Luxembourg o Netherlands o Poland o Portugal o Spain o Sweden o Switzerland o United Kingdom o Other countries Close * [icon-searc] Search [ ] Search Close * Menu Back * Fujitsu Uvance + Fujitsu Uvance o Fujitsu Uvance o Sustainable Manufacturing o Consumer Experience o Healthy Living o Trusted Society o Digital Shifts o Business Applications o Hybrid IT Fujitsu Uvance Innovative solutions that address business challenges and solve societal issues Read More Fujitsu Uvance Sustainable Manufacturing Manufacturing for a harmonious coexistence between people and the planet - value chain for sustainable manufacturing Read More Sustainable Manufacturing Consumer Experience Unique consumer experiences for all - delivering personalized consumer value Read More Consumer Experience Healthy Living Maximize your life experience - unleash your potential Read More Healthy Living Trusted Society Toward a prosperous, sustainable society - creating your own life Read More Trusted Society Digital Shifts Make the shift - bringing the power of digital to business and to people Read More Digital Shifts Business Applications Enabling you to adapt with pace and intelligence - sustainable business transformation Read More Business Applications Hybrid IT Digital infrastructure for a connected society - seamlessly and securely connect the world Read More Hybrid IT Close Services % Services * Business Application Services * Managed Infrastructure Services * Work Life Shift * Hybrid IT Services * Enterprise and Cyber Security from Fujitsu * Internet of Things * Artificial Intelligence * Co-Creation * Digital Annealer Services * Customer stories Business Application Services Business Application Services help organizations to address key digital transformation challenges by leveraging a co-creation approach and our Connected Ecosystem. Whether you want to focus on a specific technology, digital transformation or solutions specific to your sector, we look forward to co-creating with you. Read More Business and Application Services + Application Transformation + Oracle + ServiceNow + SAP Services + IoT & RFID Services (GlobeRanger) + Salesforce Managed Infrastructure Services To digitalize you need a new speed of IT delivery so you can drive new value, build the right culture and transform your business. We've built our portfolio to help you achieve this with hybrid IT, end to end networking solutions and Digital Workplace Services. We are recognized worldwide for the quality of our work and have continuously improved our position in analyst rankings. Read More Managed Infrastructure Services + Data Center Services + Network and Communication Work Life Shift Create an adaptive, productive and resilient workforce The way we work and live is fast changing. Digital technology continues to accelerate and help organizations reimagine the way they operate. It is now time to shift and build a better workplace for our people, organizations and society. With FUJITSU Work Life Shift, you can empower creativity through smarter solutions, enabling collaboration and ultimately building a truly adaptive and trusted organization. Read More Work Life Shift Hybrid IT Services We are rapidly approaching a time when all things in society will be connected through digital touchpoints and services, where data will be utilized to deliver insights and benefits well beyond the boundaries of industries and companies. In this new connected world, a strong and resilient digital infrastructure foundation will be required for close collaboration. Read more hybrid-it nav Enterprise and Cyber Security from Fujitsu Safeguarding against cyber crime is vital for business in the digital world. Fujitsu's extensive portfolio helps you strengthen your resilience against cyber attacks and improve security of your data, premises and people. Read more Managed Infrastructure Services o Explore our security portfolio o Cyber Security Webinars o Latest security insights Internet of Things Hyperconnected Business and IoT Fujitsu combines the power of IoT with digital technologies, AI, & network solutions to deliver hyperconnected business transformation. We connect objects across your entire enterprise to provide a real-time view of how your business is performing at all times. Read More Internet of Things o Digital Business Solutions o Fujitsu Managed Networks Solutions o Industry Solutions o IoT White Paper Artificial Intelligence See Beyond. Think Beyond. Go Beyond. Fujitsu human-centric AI solutions focus on ethics, transparency and trust. We help you understand what AI can achieve within your organization. Using our co-creation methodology, we help you unlock value from your existing systems, as well as lay the AI foundations to overcome the challenges you face - now and in the future. Read More Artificial Intelligence o Fujitsu AI Platform and solutions Co-Creation Co-creating Program Fujitsu's co-creating Program helps you to harness the power of collaboration to deliver your unique digital transformation by driving ideation. The Co-creating Program has been developed over decades of experience in Japan and around the world; working with customers, exchanging perspectives, ideas, and information in a highly focused, purpose-driven, and innovative way. Read More Co-Creation Digital Annealer Services Some business problems have a vast number of potential solutions which are just too challenging to calculate with standard computing technology. From financial model stress testing in banks and process optimization in manufacturing, through to pharmaceutical drug research and development, conventional optimization methods cannot handle the complexity of some of today's most challenging business problems. This is where our Quantum-Inspired Optimization Services come in. Services that leverage our Digital Annealer Platform to solve complex optimization problems using quantum logic, using tomorrow's technology today. Read More Quantum-Inspired Computing Solutions Customer stories IT Services & Solutions Case Studies At Fujitsu we create strong partnerships with our customers, enabling us to work together to find innovative IT services & solutions. Our case studies reveal how we help your businesses sector wide. Read More Customer stories Close Products # Products @ Data-Driven Transformation @ Hybrid Cloud @ SAP Landscape Transformation @ Data Center Products @ Consumption-based IT @ Services and Support @ Air Conditioning @ Network Solutions @ Sustainability Data-Driven Transformation There is a huge value in data and understandably a drive towards digital transformation initiated in every organization. More enterprises than ever are assessing the opportunities hidden in their treasure troves of data to supercharge their business and take the lead in their field. Read more Data-Driven Transformation - CX Lab - Data Transformation Experience - DX Innovation Platform - Assessment and Consultative Services - Take an AI Test Drive Hybrid Cloud Drive business resilience and sustainability by choosing the 'right cloud' for the right workload. Enabled by hybrid cloud, digital resilience - the ability for organizations to rapidly adapt to business disruptions - is a core enabler of modern digital business. It's seen as the key to a successful, agile, scalable and sustainable business future. Make Fujitsu hybrid cloud the next step in your business evolution and build a digitally resilient enterprise that's protected against any uncertainty. Read more Hybrid Cloud - Hybrid IT services - AWS connect SAP Landscape Transformation Based on 50 years partnership with SAP Fujitsu has developed excellence in building ready-to run, private and hybrid-cloud enabled IT infrastructure solutions to support customers in their journey to SAP S/4 HANA. Utilizing unique assessment and consulting methodologies to gather and interpret real-life workload data of existing SAP landscapes, Fujitsu tailors your next SAP IT refresh in order to achieve perfect service levels whilst optimizing IT spending. Read more SAP Landscape Transformation Data Center Products Fujitsu is your single point of contact for setting up a distributed IT infrastructure that stretches from edge to core to cloud. We combine own server and storage technologies with networking and software products from strategic partners to build complete IT solutions for a hybrid cloud world. Comprehensive consultative, implementation and support services ensure that customers complete this transformation smoothly and successfully. Pay-per-use options enable a cloud-like payment scheme also for on-premises and private cloud environments. Data Center = Integrated Systems = Storage Solutions = Server = Network Switches = Infrastructure Management = = Product Finder = = Configurator for customers = Configurator for partners = = Pay-per-Use = Sustainability Consumption-based IT Fujitsu uSCALE Fujitsu uSCALE delivers flexible, on-premises IT infrastructures "as-a-service" solution via monthly consumption-based billing based on actual usage. Benefit from an IT solution that precisely focuses on your specific needs, saves investment costs, enables dynamic growth, and realize faster time to value. Read More Fujitsu uSCALE = uSCALE price estimator Services and Support Fujitsu offers a wide range of services which accompany customers in all phases of their IT infrastructure journey. We start our engagement with you with assessment and consulting services to make sure that your business objectives and IT purchase decisions go hand in hand. Once products or solutions have been purchased, we provide expert implementation, installation and integration services enabling a smooth go-live phase. Finally end-to-end support offerings help to fix any potential issues in the operations and maintenance phase. A great end-to-end customer experience is our mission. Services and Support = Assessment and Consultative Services = Installation and Implementation Services = = Product Related Support x Hardware Support x Software Support x Infrastructure Support = Contact Product Support = My Support Portal Air Conditioning Read More Air conditioning Network Solutions Read more Network solutions Sustainability Given the requirement to enhance sustainability-focused corporate management, fulfilling corporate social responsibility from a global perspective has become an increasingly important issue. The Fujitsu Platform Business promotes its Sustainability activities based on the Fujitsu Way. Fujitsu takes care to operate responsibly at every stage of the product's lifecycle. Read more Close Industries = Industries x Automotive x Manufacturing x Retail x Financial Services x Transport x Public Sector x Energy and Utilities x Customer stories Automotive Automotive Technology Solutions Fujitsu combines connected and autonomous vehicle technology with world-leading IT services, infrastructure & integration skills to deliver end-to-end automotive IT solutions that increase efficiency, reduce costs & lower environmental impact. Read More Automotive Manufacturing Smart Manufacturing Manufacturing is a continuously evolving industry. Yet in 2020, a global disruptor brought production to a standstill: COVID-19. Now the industry has powered up again and COVID-19 is forcing companies to accelerate their digitalization strategy. Manufacturers are striving to meet the demands of a changed consumer by developing agility, resilience and security, responsiveness and innovation. Fast. Read More Manufacturing * Empower your people * Transform the shopfloor * Rethink the supply chain * Evolve your ecosystem * Knowledge Hub Retail Retail Technology and Hospitality Solutions Fujitsu's innovative retail technology and hospitality solutions increase efficiency and future proof the retail customer experience in an omni-channel world. Read More Retail * Forgotten Shop Floor * Fujitsu and the future of retail * Workplace 2025 * PAC report: what AI can bring to business applications * Fujitsu Market Place Financial Services Driving a trusted future in financial services Fujitsu's digital finance technology & solutions increase business efficiency & lower costs. Our agile financial IT services empower you to enhance your customer experience to aid retention. Read More Financial Services * Transforming your customer experience * Deepening your employee engagement * Accelerating your digital ambition * Insights & Events Transport Digital solutions for transport Fujitsu has been working with transport operators for over 50 years, providing innovative transport IT solutions that provide real business value. Our urban mobility IT solution transform operations, increase efficiency, improve security & reduce cost across road, rail, aviation and maritime. Read More Transport * Rail * Road * Aviation * Maritime * Urban Mobility Public Sector Public Sector Transformation Fujitsu is a world leading Public Sector IT Service provider. We help public sector organizations harness the power of technology to improve citizens' lives. Our Digital Government Solutions ensure autonomy, secure sharing & data protection. Read More Public Sector * Central Government * Local Government Energy and Utilities Digital Solutions for Energy and Utilities By implementing innovative new digital solutions - from advanced sensors/devices driving smart grids, machine learning predicting asset availability to AI enabling better fault prediction and smart devices giving consumers power over their consumption - providers can overcome the challenges they face in guaranteeing quality, availability and reliability. Read More [energy-uti] * Connected Assets * Intelligent Operations * Intelligence-led Security Customer stories IT Services & Solutions Case Studies At Fujitsu we create strong partnerships with our customers, enabling us to work together to find innovative IT services & solutions. Our case studies reveal how we help your businesses sector wide. Read More Customer Stories Close About Fujitsu % About Fujitsu * Who we are * What we do * How we work with you * News and Trends * Investor Relations Who we are Our Story Leadership Message from CEO Fujitsu Technology and Service Vision Fujitsu Facts Locations Global Fujitsu Distinguished Engineer Who we are What we do Fujitsu Uvance Customer Stories Our Business Key Technologies Research and Development What we do How we work with you Fujitsu Way Sustainability and Responsible Business Careers Partners how News and Trends Global Newsroom Fujitsu Blog Events PR Investor Relations Investor Relations Integrated Report Investor Relations Close Careers Some content may not be displaying correctly because JavaScript is turned off. Please turn it on to view this content. 1. Home 2. Press Releases 3. Release of "Fugaku-LLM" - a large language model trained on the supercomputer "Fugaku" Share x Facebook x LinkedIn x X [top-r_tcm1] Release of "Fugaku-LLM" - a large language model trained on the supercomputer "Fugaku" Enhanced Japanese language ability, for use in research and business Tokyo Institute of Technology, Tohoku University, Fujitsu Limited, RIKEN, Nagoya University, CyberAgent Inc., Kotoba Technologies Inc. Kawasaki, May 10, 2024 Summary x Large language model with enhanced Japanese language ability was developed using Japanese supercomputing technology x Distributed parallel learning by maximizing the performance of the supercomputer "Fugaku" x Commercial use is permitted, which will lead to innovative research and business applications such as AI for Science Abstract A team of researchers in Japan released Fugaku-LLM, a large language model ^(1) with enhanced Japanese language capability, using the RIKEN supercomputer Fugaku. The team is led by Professor Rio Yokota of Tokyo Institute of Technology, Associate Professor Keisuke Sakaguchi of Tohoku University, Koichi Shirahata of Fujitsu Limited, Team Leader Mohamed Wahib of RIKEN, Associate Professor Koji Nishiguchi of Nagoya University, Shota Sasaki of CyberAgent, Inc, and Noriyuki Kojima of Kotoba Technologies Inc. To train large language models on Fugaku, the researchers developed distributed training methods, including porting the deep learning framework Megatron-DeepSpeed to Fugaku in order to optimize the performance of Transformers on Fugaku. They accelerated the dense matrix multiplication library for Transformers, and optimized communication performance for Fugaku by combining three types of parallelization techniques and accelerated the collective communication library on the Tofu interconnect D. Fugaku-LLM has 13 billion parameters ^(2) and is larger than the 7-billion-parameter models that have been developed widely in Japan. Fugaku-LLM has enhanced Japanese capabilities, with an average score of 5.5 on the Japanese MT-Bench ^(3), the highest performance among open models that are trained using original data produced in Japan. In particular, the benchmark performance for humanities and social sciences tasks reached a remarkably high score of 9.18. Fugaku-LLM was trained on proprietary Japanese data collected by CyberAgent, along with English data, and other data. The source code of Fugaku-LLM is available on GitHub ^(4) and the model is available on Hugging Face ^(5 ). Fugaku-LLM can be used for research and commercial purposes as long as users comply with the license. In the future, as more researchers and engineers participate in improving the models and their applications, the efficiency of training will be improved, leading to next-generation innovative research and business applications, such as the linkage of scientific simulation and generative AI, and social simulation of virtual communities with thousands of AIs. Background In recent years, the development of large language models (LLMs) has been active, especially in the United States. In particular, the rapid spread of ChatGPT ^ (6), developed by OpenAI, has profoundly impacted research and development, economic systems, and national security. Countries other than the U.S. are also investing enormous human and computational resources to develop LLMs in their own countries. Japan, too, needs to secure computational resources for AI research so as not to fall behind in this global race. There are high expectations for Fugaku, the flagship supercomputer system in Japan, and it is necessary to improve the computational environment for large-scale distributed training on Fugaku to meet these expectations. Therefore, Tokyo Institute of Technology, Tohoku University, Fujitsu, RIKEN, Nagoya University, CyberAgent, and Kotoba Technologies have started a joint research project on the development of large language models. Role of each institution/ company Tokyo Institute of Technology: General oversight, parallelization and communication acceleration of large language models (optimization of communication performance by combining three types of parallelization, acceleration of collective communication on the Tofu interconnect D) Tohoku University: Collection of training data and model selection Fujitsu: Acceleration of computation and communication (acceleration of collective communication on Tofu interconnect D, performance optimization of pipeline parallelization) and implementation of pre-training and fine-tuning after training RIKEN: Distributed parallelization and communication acceleration of large-scale language models (acceleration of collective communication on Tofu interconnect D) Nagoya University: Study on application methods of Fugaku-LLM to 3D generative AI CyberAgent: Provision of training data Kotoba Technologies: Porting of deep learning framework to Fugaku Figure 1. RIKEN's supercomputer Fugaku (c)RIKEN Figure 1. RIKEN's supercomputer Fugaku (c)RIKEN Research outcome 1. Significantly improved the computational performance of training large language models on the supercomputer Fugaku GPUs ^(7) are the common choice of hardware for training large language models. However, there is a global shortage of GPUs due to the large investment from many countries to train LLMs. Under such circumstances, it is important to show that large language models can be trained using Fugaku, which uses CPUs instead of GPUs. The CPUs used in Fugaku are Japanese CPUs manufactured by Fujitsu, and play an important role in terms of revitalizing Japanese semiconductor technology. By extracting the full potential of Fugaku, this study succeeded in increasing the computation speed of the matrix multiplication by a factor of 6, and the communication speed by a factor of 3. To maximize the distributed training performance on Fugaku, the deep learning framework Megatron-DeepSpeed was ported to Fugaku, and the dense matrix multiplication library was accelerated for Transformer. For communication acceleration, the researchers optimized communication performance for Fugaku by combining three types of parallelization techniques and accelerated the collective communication on the Tofu interconnect D. The knowledge gained from these efforts can be utilized in the design of the next-generation computing infrastructure after Fugaku and will greatly enhance Japan's future advantage in the field of AI. 2. An easy-to-use, open, and secure, large language model with 13 billion parameters In 2023, many large language models were developed by Japanese companies, but most of them have less than 7 billion parameters. Since the performance of large-scale language models generally improves as the number of parameters increases, the 13-billion-parameter model the research team developed is likely to be more powerful than other Japanese models. Although larger models have been developed outside of Japan, large language models also require large computational resources, making it difficult to use models with too many parameters. Fugaku-LLM is both high performance and well-balanced. In addition, most models developed by Japanese companies employ continual learning ^(8), in which open models developed outside of Japan are continually trained on Japanese data. In contrast, Fugaku-LLM is trained from scratch using the team's own data, so the entire learning process can be understood, which is superior in terms of transparency and safety. Fugaku-LLM was trained on 380 billion tokens using 13,824 nodes of Fugaku, with about 60% of the training data being Japanese, combined with English, mathematics, and code. Compared to models that continually train on Japanese, Fugaku-LLM learned much of its information in Japanese. Fugaku-LLM is the best model among open models that are produced in Japan and trained with original data. In particular, it was confirmed that the model shows a high benchmark score of 9.18 in the humanities and social sciences tasks. It is expected that the model will be able to perform natural dialogue based on keigo (honorific speech) and other features of the Japanese language. Future Development The results from this research are being made public through GitHub and Hugging Face so that other researchers and engineers can use them to further develop large language models. Fugaku-LLM can be used for research and commercial purposes as long as users comply with the license. Fugaku-LLM will be also offered to users via the Fujitsu Research Portal from May 10th, 2024. In the future, as more researchers and engineers participate in improving the models and their applications, the efficiency of training will be improved, leading to next-generation innovative research and business applications, such as the linkage of scientific simulation and generative AI, and social simulation of virtual communities with thousands of AIs. Acknowledgement This research was supported by the Fugaku policy-supporting proposal "Development of Distributed Parallel Training for Large Language Models Using Fugaku" (proposal number: hp230254). ----------------------------- x [1] Large language model : Models the probability with which text appears and can predict the text (response) that follows a given context (query). x [2] Parameter : A measure of the size of a neural network. The more parameters, the higher the performance of the model, but the more data is required for training. x [3] Japanese MT-Bench : Benchmark test provided by Stability AI x [4] GitHub : Platform used to publish open source software x [5] Hugging Face : Platforms used to publish AI datasets x [6] ChatGPT : A large language model developed by OpenAI, which has brought about a major social change, surpassing 100 million users in about two months after its release. x [7] GPU : Originally produced as an accelerator for graphics, but has recently been used to accelerate deep learning x [8] Continual learning : A method for performing additional training on a large language model that has already been trained. Used for training language models in different languages or domains. About Fujitsu Fujitsu's purpose is to make the world more sustainable by building trust in society through innovation. As the digital transformation partner of choice for customers in over 100 countries, our 124,000 employees work to resolve some of the greatest challenges facing humanity. Our range of services and solutions draw on five key technologies: Computing, Networks, AI, Data & Security, and Converging Technologies, which we bring together to deliver sustainability transformation. Fujitsu Limited (TSE:6702) reported consolidated revenues of 3.7 trillion yen (US$26 billion) for the fiscal year ended March 31, 2024 and remains the top digital services company in Japan by market share. Find out more: www.fujitsu.com. Press Contacts Fujitsu Limited Public and Investor Relations Division Inquiries ----------------------------- All company or product names mentioned herein are trademarks or registered trademarks of their respective owners. Information provided in this press release is accurate at time of publication and is subject to change without advance notice. Date: 10 May, 2024 City: Kawasaki, Japan Company: Tokyo Institute of Technology, Tohoku University, Fujitsu Limited, RIKEN, Nagoya University, CyberAgent Inc., Kotoba Technologies Inc. ogp fugaku Top of Page x Fujitsu Uvance % Sustainable Manufacturing % Consumer Experience % Healthy Living % Trusted Society % Digital Shifts % Business Applications % Hybrid IT x Services & Products % Multi-Cloud % Business Application Services % Managed Infrastructure Services % Work Life Shift % Cyber Security % Internet of Things % Artificial Intelligence % Co-creation % Computing Products % Infrastructure Management % Network % Support % Pay-Per-Use x Industries % Automotive % Manufacturing % Retail % Financial Services % Transport % Public Sector % Energy & Utilities % Customer Stories x About Fujitsu % Our Story % Fujitsu Technology and Service Vision % Fujitsu Facts % Our Business % Research & Development % Fujitsu Way % Sustainability and Responsible Business % Careers % Global News Room % Investor Relations x Terms of use x Privacy x Contact x Sitemap Official Social Media Accounts % Facebook % Instagram % X % YouTube % LinkedIn Copyright 1995 - 2024 Fujitsu