Kyoto University (Kyoto U) is a world-class research and education institution with campuses in Japan and extended schools around the world. The university has a broad research community that relies on High Performance Computing (HPC) to contribute to worldwide knowledge.
The Yukawa Institute for Theoretical Physics (YITP) at Kyoto U is one of the world’s great centers for the study of theoretical physics. YITP researchers study all fields in theoretical physics, including particle physics, nuclear physics, cosmology, astrophysics, solid-state physics, statistical physics, quantum information, and others. They rely on supercomputing for simulations that bring new insights and discoveries, from the smallest quantum scales to the astronomical. In early 2021, they upgraded their computing resources with a 1.3 petaFLOPS cluster built on Dell EMC PowerEdge 4-socket servers with 2nd Gen Intel® Xeon® Scalable processors. The new system delivers 3.8 times the performance of their previous system.
Challenge
YITP was established in 1953, after Hideki Yukawa was awarded in 1949 the first Nobel Prize for a Japanese citizen. Yukawa was YITP’s first director and led the institute until his retirement in 1970. At its founding, the institute was considered a new type of national research center for theoretical physics. Its facilities were opened to the entire theoretical physics research community in Japan for collaboration on projects.
Although YITP is formally an institute of Kyoto University, physicists from all over the country are elected to join YITP’s academic staff to discuss and establish policies and projects. YITP also hosts many international and domestic workshops and conferences each year to broaden knowledge in physics. Being an open institution, YITP’s supercomputer resources are offered to the entire community of theoretical physics researchers in Japan without charge.
Computational resources are central to the work of YITP researchers. Every year, scientists require higher resolution simulations for deeper understanding across many theoretical physics areas of research. These demands place tougher and tougher requirements on the institute’s HPC systems. The existing system, installed in 2016 and operated through 2020, was a Cray XC40 cluster, consisting of 292 nodes with a total of 9,344 cores and 36.5 TB of memory. It provided peak performance of 343 teraFLOPS.
“We needed a system that meets the purpose of many kinds of simulations,” stated Naoyuki Itagaki, researcher at YITP. “Some of the users need to perform large-scale calculations using many nodes, including research and development of their applications designed to run on Fugaku.”
Fugaku is Japan’s largest supercomputer sited at the RIKEN Center for Computational Science in Kobe, Japan. Some large-scale simulations targeted for Fugaku begin work at YITP.
“Many other users run on a small number of nodes that need high-performance cores. For Yukawa-21, we had to consider these different needs and design accordingly,” concluded Itagaki.
The Yukawa-21 system features Dell EMC PowerEdge R840 platforms with 28-core Intel® Xeon® Platinum 8280 processors. (Photo courtesy YITP)
Solution
YITP application developers historically compiled their code with high-performance Intel Fortran/C compilers and were used to Intel® technologies. For the new Yukawa-21 supercomputer, system architects originally designed a cluster built on 2-socket servers, because of the supposed favorable ROI for these smaller nodes. But, the most important point of the new system is the actual performance of the simulations. Dell Technologies and Intel presented larger, 4-socket nodes with 28-core Intel Xeon Platinum 8280 processors (112 cores per node) to deliver the performance and scalability the users needed, while leveraging the experience of the developers with Intel architecture.
The advanced Intel compilers and CPUs are essential factors leading to the achievements of our research. The benchmark calculations have proven the significant improvements in the performance compared with the previous system.”—Naoyuki Itagaki, YITP researcher
To assess the performance of the design, scientists prepared benchmark programs. Three benchmark applications developed by physics researchers provided simulation codes for nuclear structure, the structure of the universe, and quantum spin systems. Three other more general benchmarks included the High Performance Conjugate Gradients (HPCG), Interleaved or Random (IOR), and STREAM OpenMP.
Yukawa-21 facilitates the work of YITP in theoretical physics. One example is simulations that calculate the density distribution of neutron-rich nucleus. (Image courtesy of YITP)
“Simulation codes developed by users are written in Fortran and/or C,” added Itagaki. “The advanced Intel compilers and CPUs are essential factors leading to the achievements of our research. The benchmark calculations have proven the significant improvements in the performance compared with the previous system.”
Dell Technologies was awarded the project using 135 PowerEdge R840 servers with a total of 202.5 TB of memory. The high-performance CPU nodes are complemented by two nodes with four GPUs per node. The cluster is rated at 1.3 petaFLOPS peak, which is 3.8 times faster than their previous system.1
Result
The system reached production status in January of 2021. Since then, more than 100 researchers have run various projects on it.
Not only does the new system reduce simulation times, it also enables us to engage novel subjects that have not been possible with the existing systems.”—Naoyuki Itagaki, YITP researcher
“Scientists have carried out simulations for physical systems at many different scales, from elementary particles to the universe,” commented Itagaki. “For instance, one simulation was for nucleon-nucleon interaction in atomic nuclei starting with the first principle. Others include neutron star mergers, synthesis of elements in the universe, various noble properties of many-body quantum systems, and others.”
According to Itagaki, most users are quite satisfied with the performance of the new system.
“Not only does the new system reduce simulation times, it also enables us to engage novel subjects that have not been possible with the existing systems,” he said.
Itagaki has been studying nuclear structure for more than 25 years, utilizing supercomputers to accomplish his work.
“The nuclei are quantum, many-body systems comprised of protons and neutrons,” explained Itagaki. “To describe microscopically, we must perform large scale calculations. My interest is to establish a unified model that combines different conventional pictures. In the field of nuclear physics, the neutron-rich nuclei, which do not exist in nature, is now extensively investigated. The neutron-rich nuclei have short lifetimes, but they are believed to have played essential roles in the synthesis of the elements in the universe. For such calculation, high-performance supercomputing is inevitable.”
This structure formation of the universe image is a result of the astronomical simulations that researchers are able to conduct on Yakawa-21. (Image courtesy YITP)
Solution Summary
At the Yukawa Institute for Theoretical Physics, it was time to upgrade their four-year-old supercomputer in order to address more demanding computational challenges in physics research. Instead of 2-socket servers, they deployed 4-socket Dell EMC PowerEdge R840 platforms with 28-core Intel Xeon Platinum 8280 processors. The larger systems with 202.5 TB of memory, along with the support of Intel C and FORTRAN compilers, allow YITP researchers to run a wide scale of workloads, from quantum to astronomical simulations in shorter time and to explore new subjects not possible before. See the Yukawa-21 system components.
Solution Ingredients
- 135 Dell EMC PowerEdge R840 4-socket server nodes
- 15,120 cores of Intel Xeon Platinum 8280 processors (28 cores each)
- Open resource for all physics researchers across Japan