My dear reader, how are you? السلام عليكم
No two things have been combined better than Knowledge and Patience – Prophet Muhammad (S.A.W)
In this post, I explain the role of benchmarks in performance evaluation in the modern multicore computing era for embedded and HPC platforms. I also share some most popular benchmarks that are widely trusted and respected in the computer processors’ domain.
A benchmark or a computer benchmark is a standard software program, a set of programs, or other operations, that is used to assess the relative performance of a machine, normally by running a number of standard tests and trials against it. Benchmarking has become an important part of the design process for performance analysis of design-based systems and is generally used as a tool for comparative analysis of various processor architectures.
Categories of Computing Benchmarks
Benchmarks can be classified into two types on the basis of the level of performance they measure:
- Component level or Synthetic benchmarks
- System-level or application benchmarks
Synthetic benchmarks are component-level benchmarks as they evaluate the particular capability of a system such as a cache subsystem performance, I/O bandwidth, floating-point processing capabilities, etc. Whereas application benchmarks are system-level benchmarks as they are built to measure the overall performance of a system for a typical workload such as office automation, encryption, etc.
The concept of using benchmarks for performance characterization of the system is a common practice and some processor manufacturers have proposed their own benchmarks such as Msp430 competitive benchmarking DirectMe. However, such benchmarks usually show better performance on the manufacturer’s own platform, and maybe biased in design to outperform a contender. Therefore third party benchmarks are a good way to compare the performance amongst various architectures impartially and transparently.
Popular benchmarks for Multicore CPUs
The following are some of the most used benchmark suites developed using highly optimized parallel programming interfaces including OpenMP and MPI.
- SPEC (Standard Performance Evaluation Corporation) CPU Benchmarks DirectMe.
- EEMBC (Embedded Microprocessor Benchmark Consortium) CPU Benchmarking suite DirectMe
- AutoBench DirectMe
- Futuremark (3DMark, Powermark) DirectMe
- NASA NAS Parallel Benchmarks DirectMe
- SPLASH 2 DirectMe and SPLASH 3 DirectMe benchmarking suite (Stanford Parallel Applications for Shared Memory)
- PARSEC Benchmarking suite DirectMe
Popular benchmarks for Accelerators and Heterogeneous HPC platforms
The following are some of the most trusted and widely used benchmarks for evaluating the performance of HPC systems.
- LINPACK: Linear algebra benchmark DirectMe. A list of top performant supercomputers in the world is evaluated for their efficiency in terms of performance using LINPACK benchmarks. It is available for a variety of OS distributions and kernel releases.
- Intel Math Kernel Library (MKL) DirectMe. This is a library of highly optimized numerical routines for various scientific applications. Fundamental numerical functions include BLAS, LAPACK, ScaLAPACK, sparse solvers, fast Fourier transforms, and vector math.
- SPEC HPC benchmark suite DirectMe. It consists of a wide and real application set in scientific computing using industry-standard parallel APIs such as OpenMP and MPI.
A real-time (RT) or mission-critical system is one that must process information and produce a response within a specified time, else risk severe consequences, including failure. RT systems are of two types, i.e.,
- Soft RT systems
- Hard RT systems
Soft RT systems are not subject to strict deadlines. Whereas, HRT systems are periodic activities that receive, process, and send messages. Each instance of a task is characterized by a hard deadline, within which it has to be completed. Real-time multiprocessors are used in numerous fields such as military equipment, avionics, automotive industry, digital signal processing, network routers and many more. The importance of using multiprocessor systems in such applications is very high, mainly because the execution time must be the same along with the process execution, i.e. no delays being accepted. Benchmarks are realistic models of real applications so there exists a need for real-time embedded systems benchmark. Our Benchmark suite is targeting Hard real-time systems, i.e., jet engines, embedded microprocessors, digital media, automotive, and other application areas. This suite can also be used for general-purpose performance analysis.
A few real-time benchmarks for NASA jet engines’ thermodynamic calculations are as follows.
- JetBench (NASA Jet Engines Thermodynamic Calculations) DirectMe
- MPBench DirectMe
- XenoJetBench  (Hard Real-Time Version of JetBench) DirectMe
 Shahid, A., Qadri, M. Y., Nawaz, N., & Ahmed, J. (2016, August). XenoJetBench: An open-source hard-real-time multiprocessor benchmark. In 2016 6th International Conference on Intelligent and Advanced Systems (ICIAS) (pp. 1-6). IEEE.
I hope you find this post useful. If you find any errors or feel any need for improvement, let me know in your comments below.
Signing off for today. Stay tuned and I will see you next week! Happy learning.