情感计算、语音与音频处理、人机交互

   

081104-模式识别与智能系统
081203-计算机应用技术

情感计算，语音与音频处理，人机交互

2011-09--2015-07   中科院自动化所   工学博士
2007-09--2009-07   北京理工大学   工学硕士
2003-09--2007-07   北京理工大学   工学学士

   

2018-11~现在, 中科院自动化所, 副研究员
2015-07~2018-10,中科院自动化所, 助理研究员

2022-02-14-今,中国指挥学会虚拟现实与人机交互专委会, 委员
2021-05-30-今,中国图象图形学会情感计算与理解专委会, 委员
2021-04-02-今,中国中文信息学会情感计算专委会, 委员
2019-11-27-今,中国图象图形学学会人机交互专委会, 委员
2018-12-25-今,中国人工智能学会情感智能专委会, 委员

情感计算

   

（1） SVFAP_Self-supervised_Video_Facial_Affect_Perceiver, IEEE Transactions on Affective Computing, 2025, 第 7 作者  通讯作者
（2） Enhancing Multimodal Personality Assessment with LLM-Augmented Hierarchical Fusion, ACM multimedia, 2025, 第 9 作者  通讯作者
（3） Listen, Watch, and Learn to Feel: Retrieval-Augmented Emotion Reasoning for Compound Emotion Generation, The 63rd Annual Meeting of the Association for Computational Linguistics, 2025, 第 6 作者  通讯作者
（4） DPP: ADual-Phase Processing Method for Cross-Cultural Humor Detection, ACM Multimedia Muse workshop, 2024, 第 8 作者
（5） Pseudo Labels Regularization for Imbalanced Partial-label Learning, The 49th IEEE International Conference on Acoustics, Speech, & Signal Processing, 2024, 第 3 作者  通讯作者
（6） GPT-4V with emotion A zero shot benchmark for Generalized Emotion Recognition, Information Fusion, 2024, 第 7 作者  通讯作者
（7） Social Perception Prediction for MuSe 2024: Joint Learning of Multiple Perceptions, ACM Multimedia Muse Workshop, 2024, 第 8 作者
（8） HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion Recognition, INFORMATION FUSION, 2024, 第 3 作者  通讯作者
（9） PSEUDO LABELS REGULARIZATION FOR IMBALANCED PARTIAL-LABEL LEARNING, 2024 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING, ICASSP 2024, 2024, 第 3 作者  通讯作者
（10） Efficient Multimodal Transformer With Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis, IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, 2024, 第 3 作者  通讯作者
（11） PIRNet: Personality-Enhanced Iterative Refinement Network for Emotion Recognition in Conversation, IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2024, 第 2 作者  通讯作者
（12） Social Perception Prediction for MuSe 2024: Joint Learning of Multiple Perceptions, PROCEEDINGS OF THE 5TH MULTIMODAL SENTIMENT ANALYSIS CHALLENGE AND WORKSHOP: SOCIAL PERCEPTION AND HUMOR, MUSE 2024, 2024, 第 8 作者
（13） DPP: A Dual-Phase Processing Method for Cross-Cultural Humor Detection, PROCEEDINGS OF THE 5TH MULTIMODAL SENTIMENT ANALYSIS CHALLENGE AND WORKSHOP: SOCIAL PERCEPTION AND HUMOR, MUSE 2024, 2024, 第 8 作者
（14） Dense Modality Interaction Network for Audio-Visual Event Localization, IEEE TRANSACTIONS ON MULTIMEDIA, 2023, 第 5 作者
（15） SMIN: Semi-Supervised Multi-Modal Interaction Network for Conversational Emotion Recognition, IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, 2023, 第 2 作者  通讯作者
（16） Multimodal Spatiotemporal Representation for Automatic Depression Level Detection, IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, 2023, 第 3 作者
（17） GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2023, 第 4 作者  通讯作者
（18） MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition, PROCEEDINGS OF THE 31ST ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA, MM 2023, 2023, 第 3 作者
（19） 多模态人机交互综述, A survey on multi-modal human-computer interaction, 中国图象图形学报, 2022, 第 8 作者
（20） End-to-End Network Based on Transformer for Automatic Detection of Covid-19, 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, 第 2 作者
（21） PIRNet: Personality-Enhanced Iterative Refinement Network for Emotion Recognition in Conversation, IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2022, 第 2 作者  通讯作者
（22） Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis, 2022, 第 3 作者
（23） Multimodal Temporal Attention in Sentiment Analysis, Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge（ACMMM2022), 2022, 第 4 作者
（24） SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition, IEEE Transactions on Affective Computing, 2022, 第 2 作者  通讯作者
（25） Dense Modality Interaction Network for Audio-Visual Event Localization, IEEE TRANSACTIONS ON MULTIMEDIA, 2022, 第 5 作者
（26） A time-frequency channel attention and vectorization network for automatic depression level prediction, NEUROCOMPUTING, 2021, 第 2 作者  通讯作者
（27） MULTI-SCALE AND MULTI-REGION FACIAL DISCRIMINATIVE REPRESENTATION FOR AUTOMATIC DEPRESSION LEVEL PREDICTION, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, 第 3 作者
（28） Gated Recurrent Fusion With Joint Training Framework for Robust End-to-End Speech Recognition, IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2021, 第 5 作者
（29） Multimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model, The 2nd Multimodal Sentiment Analysis Challenge, 2021, 第 4 作者
（30） F-0-Noise-Robust Glottal Source and Vocal Tract Analysis Based on ARX-LF Model, IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2021, 第 4 作者
（31） Multimodal Sentiment Analysis based on Recurrent Neural Network and Multimodal Attention, The 2nd Multimodal Sentiment Analysis Challenge, 2021, 第 5 作者
（32） 多通道运动特征融合的微表情识别方法, Aggregation of Motion Features of Multiple Paths for Micro-Expression Recognition, 计算机辅助设计与图形学学报, 2021, 第 2 作者
（33） CTNet: Conversational Transformer Network for Emotion Recognition, IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2021, 第 2 作者
（34） MULTIMODAL CROSS- AND SELF-ATTENTION NETWORK FOR SPEECH EMOTION RECOGNITION, 2021 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP 2021), 2021, 第 2 作者
（35） TDCA-Net Time-Domain Channel Attention Network for Depression Detection, InterSpeech 2021, 2021, 第 3 作者
（36） Learning long-term temporal contexts using skip RNN for continuous emotion recognition, Learning long-term temporal contexts using skip RNN for continuous emotion recognition, 虚拟现实与智能硬件(中英文), 2021, 第 2 作者
（37） Multi-Scale and Multi-Region Facial Discriminative Representation for Automatic Depression Level Detection, ICASSP, 2021, 第 3 作者
（38） DECN: Dialogical Emotion Correction Network for Conversational Emotion Recognition, NEUROCOMPUTING, 2021, 第 2 作者  通讯作者
（39） Review of micro-expression spotting and recognition in video sequences, Review of micro-expression spotting and recognition in video sequences, 虚拟现实与智能硬件(中英文), 2021, 第 4 作者
（40） 基于多头注意力机制的模型层融合维度情感识别方法, Model Level Fusion Dimension Emotion Recognition Method Based on Transformer, 信号处理, 2021, 第 3 作者
（41） F 0 -Noise-Robust Glottal Source and Vocal Tract Analysis Based on ARX-LF Model, IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2021, 第 4 作者
（42） AMINN: Attention-Based Multi-Information Neural Network for Emotion Recognition, ICCPR, 2020, 第 2 作者
（43） End-to-End Post-Filter for Speech Separation With Deep Attention Fusion Features, IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2020, 第 3 作者  通讯作者
（44） Hybrid Network Feature Extraction for Depression Assessment from Speech, Interspeech, 2020, 第 4 作者
（45） Comparison of glottal source parameter values in emotional vowels, Interspeech, 2020, 第 3 作者
（46） MULTIMODAL TRANSFORMER FUSION FOR CONTINUOUS EMOTION RECOGNITION, ICASSP 2020, 2020, 第 3 作者
（47） Conversational Emotion Recognition Using Self-Attention Mechanisms and Graph Neural Networks, Interspeech, 2020, 第 3 作者
（48） Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism, The 1st Multimodal Sentiment Analysis Challenge, 2020, 第 2 作者
（49） Learning Utterance-level Representationswith Label Smoothing for Speech Emotion Recognition, Interspeech, 2020, 第 3 作者
（50） Multimodal Spatiotemporal Representation for Automatic Depression Level Detection, IEEE Transactions on Affective Computing, 2020, 第 3 作者
（51） Joint Training for Simultaneous Speech Denoising and Dereverberation with Deep Embedding Representations, INTERSPEECH, 2020, 第 3 作者
（52） Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition, Interspeech, 2020, 第 3 作者
（53） Gated Recurrent Fusion of Spatial and Spectral Features for Multi-channel Speech Separation with Deep Embedding Representations, Interspeech, 2020, 第 3 作者
（54） "Automatic Depression Level Detection via lp-norm Pooling", INTERSPEECH 2019, 2019, 第 3 作者
（55） Discriminative Learning for Monaural Speech Separation Using Deep Embedding Features, interspeech2019, 2019, 第 2 作者
（56） Conversational Emotion Analysis via Attention Mechanisms, Interspeech2019, 2019, 第 3 作者
（57） 智能语音识别系统噪声鲁棒性研究, Research on Noisy Robustness of Intelligent Speech Recognition System, 信息技术与标准化, 2019, 第 2 作者
（58） 一种基于卷积神经网络的端到端语音分离方法, An End-to-End Speech Separation Method Based on Convolutional Neural Network, 信号处理, 2019, 第 2 作者
（59） 无菌条件非接触式多通道自然交互手术环境, Non Contact Multi-channel Natural Interactive Surgical Environment under Sterile Conditions, 软件学报, 2019, 第 10 作者
（60） Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition, interspeeh2019, 2019, 第 3 作者
（61） 基于迁移学习的噪声鲁棒语音识别声学建模, Transfer learning for acoustic modeling of noise robust speech recognition, 清华大学学报：自然科学版, 2018, 第 3 作者
（62） CTC Regularized Model Adaptation for Improving LSTM RNN Based Multi-Accent Mandarin Speech Recognition, JOURNAL OF SIGNAL PROCESSING SYSTEMS FOR SIGNAL IMAGE AND VIDEO TECHNOLOGY, 2018, 第 5 作者
（63） Investigating Deep Neural Network Adaptation for Generating Exclamatory and Interrogative Speech in Mandarin, JOURNAL OF SIGNAL PROCESSING SYSTEMS FOR SIGNAL IMAGE AND VIDEO TECHNOLOGY, 2018, 第 4 作者
（64） 联合长短时记忆递归神经网络和非负矩阵分解的语音混响消除方法, 信号处理, 2017, 第 1 作者
（65） 基于注意力的端到端韵律结构和重音联合预测方法, 第十四届全国人机语音通讯学术会议 (NCMMSC 2017), 2017, 第 5 作者
（66） Investigating Efficient Feature Representation Method and Training Object Function for BLSTM-based Phone Duration Prediction, Interspeech2017, 2017, 第 5 作者
（67） A NOVEL PITCH EXTRACTION BASED ON JOINTLY TRAINED DEEP BLSTM RECURRENT NEURAL NETWORKS WITH BOTTLENECK FEATURES, 2017年IEEE声学，语音和信号处理国际会议(ICASSP 2017), 2017, 第 1 作者
（68） 联合长短时记忆递归神经网络和非负矩阵分解的语音混响消除方法, 信号处理, 2017, 第 1 作者
（69） Investigating Deep Neural Network Adaptation for Generating Exclamatory and Interrogative Speech in Mandarin, 2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP), 2016, 第 5 作者
（70） EXTRACTION OF TONGUE CONTOUR IN REAL-TIME MAGNETIC RESONANCE IMAGING SEQUENCES, ICASSP2016, 2016, 第 5 作者
（71） A Novel Research to Artificial Bandwidth Extension Based on Deep BLSTM Recurrent Neural Networks and Exemplar-based Sparse Representation, interspeech2016, 2016, 第 1 作者
（72） EXTRACTION OF TONGUE CONTOUR IN REAL-TIME MAGNETIC RESONANCE IMAGING SEQUENCES, 2016 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING PROCEEDINGS, 2016, 第 5 作者
（73） A BLSTM Guided Unit Selection Synthesis System for Blizzard Challenge 2016, Blizzard2016, 2016, 第 5 作者
（74） Speech Enhancement Based on Analysis-Synthesis Framework with Improved Parameter Domain Enhancement, JOURNAL OF SIGNAL PROCESSING SYSTEMS FOR SIGNAL IMAGE AND VIDEO TECHNOLOGY, 2016, 第 1 作者  通讯作者
（75） CTC Regularized Model Adaptation for Improving LSTM RNN Based MultiAccent Mandarin Speech Recognition, 2016 10TH INTERNATIONAL SYMPOSIUM ON CHINESE SPOKEN LANGUAGE PROCESSING (ISCSLP), 2016, 第 4 作者
（76） End-to-end Keywords Spotting Based on Connectionist Temporal Classification for Mandarin, 2016 10TH INTERNATIONAL SYMPOSIUM ON CHINESE SPOKEN LANGUAGE PROCESSING (ISCSLP), 2016, 第 5 作者
（77） Text-based sentential stress prediction using continuous lexical embedding for Mandarin speech synthesis, 2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP), 2016, 第 4 作者
（78） A Novel Method of Artificial Bandwidth Extension Using Deep Architecture, 16TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION (INTERSPEECH 2015), VOLS 1-5, 2015, 第 1 作者  通讯作者
（79） User behavior fusion in dialog management with multi-modal history cues, MULTIMEDIA TOOLS AND APPLICATIONS, 2015, 第 8 作者
（80） ESTIMATE ARTICULATORY MRI SERIES FROM ACOUSTIC SIGNAL USING DEEP ARCHITECTURE, 2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 2015, 第 4 作者
（81） 一种改进的基于分析合成框架的语音增强算法, 2015年第十三届全国人机语音通讯会议（NCMMSC2015） (NCMMSC2015), 2015, 第 1 作者
（82） Efficient voice activity detection algorithm based on sub-band temporal envelope and sub-band long-term signal variability, 第九届中文口语语言处理国际会议 (ISCSLP 2014), 2014, 第 1 作者
（83） Speech Enhancement Based on Analysis–Synthesis Framework With Improved Pitch Estimation and Spectral Envelope Enhancement, ICSP2014, 2014, 第 1 作者
（84） Context features based pre-selection and weight prediction in concatenation speech synthesis system, 第九届中文口语语言处理国际会议 (ISCSLP 2014), 2014, 第 5 作者
（85） 面向窄带通信的极低速率语音编码算法研究, Research on Speech Coding Algorithm at Very Low Bit Rate for Narrow-Band Communication, 信号处理, 2013, 第 1 作者
（86） MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning, ARXIV, 第 6 作者

（1） 情感计算理论与方法, 清华大学出版社, 2024-06, 第 2 作者

   

（ 1 ） 基于多模态大模型的心理状态自动评估与调控技术, 负责人, 国家任务, 2026-01--2026-12
（ 2 ） 大模型专业知识问答技术研究, 负责人, 国家任务, 2025-05--2027-04
（ 3 ） 基于大模型对齐增强的数据清洗治理, 负责人, 国家任务, 2025-05--2026-10
（ 4 ） 态势认知原型系统开发, 负责人, 境内委托项目, 2024-10--2025-03
（ 5 ） 适配国产异构系统架构的跨体系结构开发框架, 负责人, 中国科学院计划, 2023-09--2026-08
（ 6 ） 面向小团体的多模态连续情感识别技术研究, 负责人, 国家任务, 2023-01--2026-12
（ 7 ） 情感识别技术, 参与, 境内委托项目, 2020-06--2025-05

已指导学生

凡佳辉  硕士研究生  085410-人工智能  

柳晗  硕士研究生  085410-人工智能  

李世渺  硕士研究生  085211-计算机技术  

徐名宇  硕士研究生  081104-模式识别与智能系统  

王于华  硕士研究生  085410-人工智能  

徐珂  硕士研究生  085410-人工智能  

张思源  硕士研究生  085404-计算机技术  

陈顺  硕士研究生  085410-人工智能  

温卓凡  硕士研究生  081203-计算机应用技术  

周俊佐  硕士研究生  085410-人工智能  

现指导学生

张凤羽  硕士研究生  085410-人工智能  

余聪  硕士研究生  081203-计算机应用技术  

刘嘉煜  硕士研究生  085410-人工智能  

吕皓  硕士研究生  085410-人工智能  

王蓉  硕士研究生  081104-模式识别与智能系统  

张祺  硕士研究生  085410-人工智能  

黄嘉文  硕士研究生  085410-人工智能  

邹鑫  硕士研究生  085410-人工智能