Advanced Computing (Supercomputing; Edge computing; Cloud computing and services; High-performance data storage and data centers; Computing architectures; Advanced modeling and simulation; Data processing and analysis techniques), The Science and Engineering of Evaluation (Evaluatology and Benchmarkology), Artificial Intelligence (AI system frameworks, AI assurance and assessment techniques; Generative AI systems, multimodal and large language models; Synthetic data approaches for training, tuning, and testing; Technologies for improving AI safety, trust, security, and responsible use), and Operating Systems.

Evaluation is a crucial aspect of human existence and plays a vital role in various fields[1]. However, it is often approached in an empirical and ad-hoc manner, lacking consensus on universal concepts, terminologies, theories, and methodologies. I formally introduce the discipline of evaluatology, which encompasses the science and engineering of evaluation. Derived from the essence of evaluation, I propose five axioms focusing on key aspects of evaluation outcomes as the foundational evaluation theory. These axioms serve as the bedrock upon which a universal framework for evaluation is proposed, encompassing concepts, terminologies, theories, and methodologies that can be applied across various disciplines. Building upon the science of evaluation, I propose a formal definition of a benchmark as a simplified and sampled evaluation condition that guarantees different levels of equivalency. This concept serves as the cornerstone for a universal benchmark-based engineering approach to evaluation across various disciplines, which I refer to as benchmarkology.

Under my leadership, the big data and AI benchmarking projects, BigDataBench [2,3] and AIBench [4,5], have had significant impacts. BigDataBench is widely adopted in academia and industry worldwide, with the article published at HPCA'14 receiving 726 citations.

I am the founding chair of the International Open Benchmark Council (BenchCouncil) (https://www.benchcouncil.org ). Renowned scholars such as Matsuoka, Satoshi, Andrew A. Chien, Jeffrey S. Vetter, and John Shalf have ranked BenchCouncil's contributions alongside SPEC, founded in 1988, and before MLCommons, which was founded by Google and other industry and academic leaders such as Stanford and Facebook [6]. In their citation, they state, "Other commonly used benchmarking sets have been curated by organizations such as the Standard Performance Evaluation Corporation (SPEC), the International Open Benchmark Council (BenchCouncil), or MLCommons—with proxies for CPUs, distributed systems, accelerators, artificial intelligence—or have been developed by companies such as Intel (HiBench), Baidu (DeepBench), and Google (PerfKit)."

[1] Jianfeng Zhan, Lei Wang, Wanling Gao et al. Evaluatology: The Science and Engineering of Evaluation. arXiv preprint arXiv:2404.00021, 2024.
[2] Wang, Lei, Jianfeng Zhan, et al. "Bigdatabench: A big data benchmark suite from internet services." In 2014 IEEE 20th international symposium on high-performance computer architecture (HPCA), pp. 488-499. IEEE, 2014. Google Citations: 711. The first author, Lei Wang, is my Ph. D student.
[3] Jia, Zhen, Lei Wang, Jianfeng Zhan, et al. "Characterizing data analysis workloads in data centers." IISWC 2013 Best Paper Award.
[4]Hao, Tianshu, …Jianfeng Zhan. "Edge AIBench: Towards comprehensive end-to-end edge computing benchmarking." In Benchmarking, Measuring, and Optimizing: First BenchCouncil International Symposium, Bench 2018, Seattle, WA, USA, December 10-13, 2018. Google Citations: 81. The first author, Tianshu Hao, is my Ph. D student.
[5] Tang, Fei, Wanling Gao, Jianfeng Zhan, et al. "AIBench training: Balanced industry-standard AI training benchmarking." In 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 24-35. IEEE, 2021. Google Citations: 21. The first author, Fei Tang, is my Ph. D student.
[6] Matsuoka, Satoshi, Jens Domke, Mohamed Wahib, Aleksandr Drozd, Andrew A. Chien, Raymond Bair, Jeffrey S. Vetter, and John Shalf. "Preparing for the Future—Rethinking Proxy Applications." Computing in Science & Engineering 24, no. 2 (2022): 85-90.

I spearheaded the research and development of the cluster operating system [7] for the Dawning 4000A, which achieved the 10th position on the Top 500 supercomputer list in June 2004 (source: https://www.top500.org/system/173564/). As mentioned in Sugon's initial public offering prospectus, my team's patent titled "A Method for Autonomously Constructing Cluster Operating System Kernels and an Intelligent Constructor," along with two other patents and one software IP, were transferred to Sugon, accounting for 34.44% of its total equity share [8]. Sugon is China's leading supercomputer company. In recognition of my contributions, I received the Outstanding Science and Technology Achievement Prize from the Chinese Academy of Sciences in 2005 and the Second Prize of The State Scientific and Technological Progress Award from the Central People's Government of the People's Republic of China in 2006.

Furthermore, I played a pivotal role in the design and implementation of cutting-edge cloud computing systems, namely PhoenixCloud [9,11] and DawningCloud [10], on the Dawning 5000 and 6000 Supercomputers. These innovative solutions revolutionized the support for high-performance computing and the consolidation of heterogeneous workloads on cloud systems. Additionally, I transferred 34 OS patents to a world-class communication and IT company.

[7] Jianfeng Zhan, and Ninghui Sun. "Fire Phoenix cluster operating system kernel and its evaluation." In 2005 IEEE International Conference on Cluster Computing, pp. 1-9. IEEE, 2005.
[8] Sugon' initial public offering prospectus. Page 1-1-62. https://pdf.dfcfw.com/pdf/H2_AN201410230007390902_1.pdf .Accessed at September 10, 2023
[9] Jianfeng Zhan, Lei Wang, Bibo Tu, Yong Li, Peng Wang, Wei Zhou, and Dan Meng. "Phoenix cloud: Consolidating heterogeneous workloads of large organizations on cloud computing platforms." The first workshop on Cloud Computing and its Applications (CCA'08), 2008. https://www.alcf.anl.gov/events/cca-cloud-computing-and-its-applications
[10] Wang, Lei, Jianfeng Zhan, Weisong Shi, and Yi Liang. "In cloud, can scientific communities benefit from the economies of scale?" IEEE Transactions on Parallel and Distributed Systems 23, no. 2 (2011): 296-303. Google Citations: 214. The first author Lei Wang is my Ph. D student at the publication.
[11] Jianfeng Zhan, et al. "Cost-aware cooperative resource provisioning for heterogeneous workloads in data centers." IEEE Transactions on Computers 62, no. 11 (2012): 2155-2168. Google Citations: 69.

Evaluation is one of the fundamental activities of humanity. Advanced computing systems will help unleash human creativity and create a better world.  (1) Establishing a unified discipline of Evaluation Science and Engineering; (2) Researching foundational theories, models, and methods to construct and manage simple, secure, and fair advanced computing-based social infrastructures.

In the next five years (2025-2030), I plan to engage in the following research topics:

Evaluatology and Benchmarkology. I have been fortunate enough to coin these terms (with contributions from my doctoral student, Hongxiao Li to the term of Evaluatology). Evaluation is a fundamental issue across all disciplines. Unfortunately, evaluation methods in most disciplines are often ad-hoc and empirical. Developing unified principles and methods for evaluation is a challenging problem. I have proposed the fundamental concepts, unified terminology, axioms, theories, methodologies, and foundational issues of Evaluatology. I will further advance Evaluatology and Benchmarkology in disciplines such as safety-critical, mission-critical and business-critical AI systems, intelligence, energy efficiency, science and technology, social infrastructure, computers, medicine, and pharmaceuticals and safety (especially AI safety). I plan to publish five volumes English monograph on Evaluatology and Benchmarkology.

Research on advanced computing-based social infrastructures. I will introduce new abstract simplifications for advanced computing-based social infrastructures, ensuring their security and fairness based on Evaluatology. I will further refine the Open Computer Systems Initiative (OpenCS) that I proposed earlier. The OpenCS initiative will facilitate research on advanced computing-based social infrastructures, without reinventing the wheel. Instead, I will propose new abstractions and methods to address these significant challenges.

   

2022-up to now, Director, Advanced computer systems research center, ICT, Chinese Academy of Sciences

 Sep 2014-2022, Deputy Director, Advanced computer systems research center, ICT, Chinese Academy of Sciences

October 2012-up to now,  Full Professor, ICT, Chinese Academy of Sciences, Beijing, P.R.China 
Professor
Mar 2004- October 2012, Associate Professor, ICT Chinese Academy of Sciences, Beijing, P.R.China

Aug 2002- Mar 2004
Assistant Professor, ICT, Chinese Academy of Sciences, Beijing, P.R.China

Second Class of National Science and Technology Promotion Prize, The Central People’s Government of the People’s Republic of China, 2006.

Outstanding Science and Technology Achievement Prize of the Chinese Academy of Sciences, Chinese Academy of Science, P. R. China, 2005.

   

Computation Abstractions

·       Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Daoyi Zheng, Fei Tang, Biwei Xie, Chen Zheng, Xu Wen, Xiwen He and Hainan Ye: Data Motifs: A Lens Towards Fully Understanding Big Data and AI Workloads. Parallel Architectures and Compilation Techniques (PACT) 2018.

AI Benchmarking

·       Zihan Jiang, Lei Wang, Xingwang Xiong, Wanling Gao, Chunjie Luo, Fei Tang, Chuanxin Lan, Hongxiao Li, Jianfeng Zhan. HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems. arXiv preprint arXiv:2007.00279

·       Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong Wu, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, Hainan Ye. AIBench: an industry standard internet service AI benchmark suite. arXiv preprint arXiv:1908.08998

·       Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, Jianfeng Zhan. Edge AIBench: towards comprehensive end-to-end edge computing benchmarking. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimization

Big Data Benchmarking

·       Lei Wang, Jianfeng Zhan, Chunjie Luo, Yuqing Zhu, Qiang Yang, Yongqiang He, Wanling Gao, Zhen Jia, Yingjie Shi, Shujie Zhang, Chen Zheng, Gang Lu, Kent Zhan, Xiaona Li, Bizhu Qiu: BigDataBench: A big data benchmark suite from internet services. HPCA 2014: 488-499.

·       Zijian Ming, Chunjie Luo, Wanling Gao, Rui Han, Qiang Yang, Lei Wang, Jianfeng Zhan: BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking. WBDB 2013: 138-154.

·       Zhen Jia, Lei Wang, Jianfeng Zhan, Lixin Zhang, Chunjie Luo: Characterizing data analysis workloads in data centers. IISWC 2013: 66-76.

Cloud Benchmarking

·       Chuniie Luo, Jianfeng Zhan, Lei Wang, Gang Lu, Lixin Zhang, Chengzhong Xu, and Ninghui Sun, Cloudrank-d: benchmarking and ranking cloud computing systems for data processing applications, Frontiers of Computer Science 6 (4), 347-362

Computing Paradigms

·       Gang Lu, Jianfeng Zhan, Tianshu Hao, Lei Wang: 10-millisecond Computing. CoRR abs/1610.01267 (2016).

·       Jianfeng Zhan, Lixin Zhang, Ninghui Sun, Lei Wang, Zhen Jia, Chunjie Luo, High volume throughput computing: Identifying and characterizing throughput oriented workloads in data centers, 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum

·       Xinhui Tian, Rui Han, Lei Wang, Gang Lu, Jianfeng Zhan, Latency critical big data computing in finance, The Journal of Finance and Data Science, 2015

Operating Systems

·       Gang Lu, Jianfeng Zhan, Chongkang Tan, Xinlong Lin, Defei Kong, Tianshu Hao, Lei Wang, Fei Tang, Chen Zheng: Isolate First, Then Share: A New OS Architecture for Datacenter Computing. arXiv:1604.01378.

·       Chen Zheng, Lei Wang, Sally A McKee, Lixin Zhang, Hainan Ye, Jianfeng Zhan. XOS: An Application-Defined Operating System for Datacenter Computing. 2018 IEEE International Conference on Big Data (Big Data)

Cluster and Cloud Resource management

·       Jianfeng Zhan, Lei Wang, Xiaona Li, Weisong Shi, Chuliang Weng, Wenyao Zhang, Xiutao Zang: Cost-Aware Cooperative Resource Provisioning for Heterogeneous Workloads in Data Centers. IEEE Trans. Computers 62(11): 2155-2168 (2013).

·       Lei Wang, Jianfeng Zhan, Weisong Shi, Yi Liang: In Cloud, Can Scientific Communities Benefit from the Economies of Scale? IEEE Trans. Parallel Distrib. Syst. 23(2): 296-303 (2012).

·       Jianfeng Zhan, L Wang, B Tu, Y Li, P Wang, W Zhou, D Meng, Phoenix cloud: Consolidating heterogeneous workloads of large organizations on cloud computing platforms, Proceedins of CCA 08, 2008

·       Jianfeng Zhan, Ninghui Sun: Fire Phoenix Cluster Operating System Kernel and its Evaluation. CLUSTER 2005: 1-9.

Programming Systems

·       Peng Wang, Dan Meng, Jizhong Han, Jianfeng Zhan, Bibo Tu, Xiaofeng Shi, Le Wan: Transformer: A New Paradigm for Building Data-Parallel Programming Models. IEEE Micro 30(4): 55-64 (2010).

·       Xinhui Tian, Biwei Xie, Jianfeng Zhan. Cymbalo: An Efficient Graph Processing Framework for Machine Learning. 2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications

Performance Analytics, Management, and Optimization

·       Biwei Xie, Jianfeng Zhan, Xu Liu, Wanling Gao, Zhen Jia, Xiwen He, Lixin Zhang. CVR: efficient vectorization of SpMV on x86 processors. Proceedings of the 2018 International Symposium on Code Generation and Optimization

·       Zhen Jia, Chao Xue, Guancheng Chen, Jianfeng Zhan, Lixin Zhang, Yonghua Lin, Peter Hofstee. Auto-tuning Spark big data workloads on POWER8: Prediction-based dynamic SMT threading. 2016 International Conference on Parallel Architecture and Compilation Techniques (PACT)

·       Xiaoyu Fu, Rui Ren, Jianfeng Zhan, Wei Zhou, Zhen Jia, Gang Lu: LogMaster: Mining Event Correlations in Logs of Large-Scale Cluster Systems. SRDS 2012: 71-80.

·       Zhihong Zhang, Jianfeng Zhan, Yong Li, Lei Wang, Dan Meng, Bo Sang: Precise request tracing and performance debugging for multi-tier services of black boxes. DSN 2009: 337-346.

Deep learning and Its applications

·       Chunjie Luo, Jianfeng Zhan, X Xue, Lei Wang, Rui Ren, Qiang Yang, Cosine normalization: Using cosine similarity instead of dot product in neural networks, 2018 International Conference on Artificial Neural Networks, 382-391

·       Xiexuan Zhou, Wenfeng Zeng, H Chi, C Luo, C Liu, Jianfeng Zhan, SM He, Z Zhang, pdeep: Predicting MS/MS spectra of peptides with deep learning, Analytical chemistry 89 (23), 12690-12697

已指导学生

邱必祝  硕士研究生  081202-计算机软件与理论  

桑波  硕士研究生  081202-计算机软件与理论  

臧秀涛  硕士研究生  081201-计算机系统结构  

袁琳  硕士研究生  081201-计算机系统结构  

黄雅斌  硕士研究生  081201-计算机系统结构  

刘雪娅  硕士研究生  081280-软件工程  

宫士敏  硕士研究生  081201-计算机系统结构  

任睿  硕士研究生  081201-计算机系统结构  

罗纯杰  硕士研究生  081203-计算机应用技术  

周俊平  硕士研究生  081280-软件工程  

杨强  硕士研究生  081202-计算机软件与理论  

谭崇康  硕士研究生  081202-计算机软件与理论  

明子鉴  硕士研究生  081202-计算机软件与理论  

王磊  博士研究生  081202-计算机软件与理论  

陆钢  博士研究生  081202-计算机软件与理论  

田昕晖  博士研究生  081201-计算机系统结构  

周撷璇  博士研究生  081202-计算机软件与理论  

何习文  硕士研究生  081201-计算机系统结构  

郭远晴  硕士研究生  085211-计算机技术  

王标  硕士研究生  081202-计算机软件与理论  

唐昊宁  硕士研究生  085212-软件工程  

高婉铃  博士研究生  081202-计算机软件与理论  

孔德飞  硕士研究生  085212-软件工程  

任睿  博士研究生  081202-计算机软件与理论  

黄运有  博士研究生  081202-计算机软件与理论  

杜梦佳  硕士研究生  081202-计算机软件与理论  

戴绍鹏  硕士研究生  081202-计算机软件与理论  

现指导学生

周润林  博士研究生  085271-电子与信息  

江梓涵  博士研究生  081201-计算机系统结构  

王娜娜  博士研究生  081201-计算机系统结构  

罗纯杰  博士研究生  081202-计算机软件与理论  

康国新  博士研究生  081201-计算机系统结构  

李亚韬  博士研究生  081202-计算机软件与理论  

郝天舒  博士研究生  081202-计算机软件与理论  

温旭  硕士研究生  081201-计算机系统结构  

陈家楠  硕士研究生  081201-计算机系统结构  

黄成  硕士研究生  081201-计算机系统结构  

汤飞  博士研究生  081201-计算机系统结构  

李鸿霄  博士研究生  081201-计算机系统结构  

刘可  硕士研究生  085211-计算机技术  

范帆达  博士研究生  081201-计算机系统结构  

2011, CSE 2011, GCC 2011, Cloud 2011, External Reviewer of ICDCS 2011 

2012, PC Member of AINA 2012 ( The 26th IEEE International Conference on Advanced Information Networking and Applications, Tokyo, Japan, March 26-29, 2012.) , PC Member of ICDCS 2012 (The 32nd International Conference on Distributed Computing Systems), Publicity Chair for China of ICAC 2012 (The 9th International Conference on Autonomic Computing).  PC Member of IDPDS 2012 Ph.D Forum,  Guest editor of Cloud computing special issue of Frontier of Computer Sciences, 2012.  PC member of NPC 2012.