Lingwei Meng

About

I am a Ph.D. candidate at Human-Computer Communications Laboratory (HCCL), The Chinese University of Hong Kong (CUHK), supervised by Prof. Helen Meng. Previously, I received my M.Phil. degree from the Institute of Automation, Chinese Academy of Sciences (CASIA), where I was supervised by Prof. Jie Tian. And I received my Bachelor's degree from Harbin Institute of Technology (HIT).

I am currently a research intern at Microsoft Research Asia. My research interests focus on language modeling for speech synthesis and the integration of speech with large language models. I am also working on speech processing and recognition.

🟢   I am currently on the job market. Please drop me emails!

Highlights

[Sep 2024]  Four papers was submitted to ICASSP 2025, including one first-authored [paper]. Thank you to all collaborators!
[Jul 2024]  We present [MELLE], a concise, continuous-value token-based language model for TTS, forgoing cumbersome NAR steps.
[Jun 2024]  We propose [WavLLM], a robust and adaptive speech LLM achieving SOTA performance on various speech-related tasks.
[Jun 2024]  Three papers has been accepted to ISCA INTERSPEECH 2024, including one first-authored [paper].
[Jan 2024]  Two papers has been accepted to IEEE ICASSP 2024.
[Nov 2023]  Started an internship at Microsoft Research Asia.
[May 2023]  One first-authored [paper] has been accepted to ISCA INTERSPEECH 2023.
[Feb 2023]  One first-authored [paper] has been accepted to IEEE ICASSP 2023 (Oral).
[Jun 2022]  Two papers have been accepted to ISCA INTERSPEECH 2022.
[Mar 2022]  Our [paper] was selected as an ⭐️ ESI Highly Cited Paper by Web of Science.
[Jan 2022]  We won the 2nd place on the diarization track of the ICASSP 2022 - M2Met Chanllenge. This is our [paper].

Education

Since 2021, Ph.D. student, The Chinese University of Hong Kong (CUHK)
2018 - 2021, M.Phil., Pattern Recognition and Intelligent Systems, Institute of Automation, Chinese Academy of Sciences (CASIA)
2014 - 2018, B.Eng., Electrical Engineering, Harbin Institute of Technology (HIT)

Experiences

• Since Nov. 2023Research Intern, Natural Language Computing Group, Microsoft Research Asia
  Studying language modeling for speech synthesis; integrating speech with large language models
  Working closely with Dr. Long Zhou, Dr. Shujie Liu, and Dr. Furu Wei

Selected Publications

[Google Scholar]

• MELLE: Autoregressive Speech Synthesis without Vector Quantization   |   [demo]
  Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu,
  Helen Meng, Furu Wei
  arXiv, 2024

• Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
  Lingwei Meng, Shujie Hu, Jiawen Kang, Zhaoqing Li, Yuejiang Wang, Wenxuan Wu, Xixin Wu, Xunying Liu, Helen Meng
  submitted to IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2025

• Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System   |   [code]
  Lingwei Meng, Jiawen Kang, Yuejiao Wang, Zengrui Jin, Xixin Wu, Xunying Liu, Helen Meng
  ISCA INTERSPEECH, Kos Island, Greece, 2024

• A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One   |   [slides] [video]
  Lingwei Meng, Jiawen Kang, Mingyu Cui, Yuejiao Wang, Xixin Wu, Helen Meng
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Rhodes Island, Greece, 2023   (Oral)

• Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator
  Lingwei Meng, Jiawen Kang, Mingyu Cui, Haibin Wu, Xixin Wu, Helen Meng
  ISCA INTERSPEECH, Dublin, Ireland, 2023

• Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTC
  Jiawen Kang, Lingwei Meng, Mingyu Cui, Yuejiao Wang, Xixin Wu, Xunying Liu, Helen Meng
  arXiv, 2024

• WavLLM: Towards Robust and Adaptive Speech Large Language Model   |   [code]
  Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
  arXiv, 2024

• VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment   |   [demo]
  Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
  arXiv, 2024

• Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder
  Yuejiao Wang, Xianmin Gong, Lingwei Meng, Xixin Wu, Helen Meng Sheng Zhao, Jinyu Li, Furu Wei
  ISCA INTERSPEECH, Kos Island, Greece, 2024

• Cross-Speaker Encoding Network for Multi-Talker Speech Recognition   |   [code]
  Jiawen Kang, Lingwei Meng, Mingyu Cui, Haohan Guo, Xixin Wu, Xunying Liu, Helen Meng
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Seoul, Korea, 2024

• UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization
  Yuejiao Wang, Xixin Wu, Disong Wang, Lingwei Meng, Helen Meng
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Seoul, Korea, 2024

• The defender's perspective on automatic speaker verification: An overview
  Haibin Wu, Jiawen Kang, Lingwei Meng, Helen Meng, Hung-yi Lee
  International Joint Conferences on Artificial Intelligence (IJCAI) Workshop, Macao SAR, China, 2023

• Spoofing-Aware Speaker Verification by Multi-Level Fusion
  Haibin Wu, Lingwei Meng, Jiawen Kang, Jinchao Li, Xu Li, Xixin Wu, Hung-yi Lee, Helen Meng
  ISCA INTERSPEECH, Incheon, Korea, 2022

• Exploring Linguistic Feature and Model Combination for Speech Recognition Based Automatic AD Detection
  Yi Wang, Tianzi Wang, Zi Ye, Lingwei Meng, Shoukang Hu, Xixin Wu, Xunying Liu, Helen Meng
  ISCA INTERSPEECH, Incheon, Korea, 2022

• The CUHK-Tencent Speaker Diarization System for the ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge
  Naijun Zheng, Na Li, Xixin Wu, Lingwei Meng, Jiawen Kang, Haibin Wu, Chao Weng, Dan Su, Helen Meng
  IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Singapore, 2022

• Tackling Spoofing-Aware Speaker Verification with Multi-Model Fusion
  Haibin Wu, Jiawen Kang, Lingwei Meng, Yang Zhang, Xixin Wu, Zhiyong Wu, Hung-yi Lee, Helen Meng
  ISCA The Speaker and Language Recognition Workshop (Odyssey), Beijing, China, 2022

- Medical Image Analysis related journal papers:

• 2D and 3D CT Radiomic Features Performance Comparison in Characterization of Gastric Cancer: a Multi-Center Study
  Lingwei Meng, Di Dong, Xin Chen, Mengjie Fang, Rongpin Wang, Jing Li, Zaiyi Liu, Jie Tian
  IEEE Journal of Biomedical and Health Informatics (IEEE JBHI), 2021  (Impact Factor: 6.7)
  (ESI Highly Cited Paper)

• A Deep Learning Prognosis Model Help Alert for COVID-19 Patients at High-Risk of Death: a Multi-Center Study
  Lingwei Meng, Di Dong, Liang Li, Meng Niu, Yan Bai, Meiyun Wang, Xiaoming Qiu, Yunfei Zha, Jie Tian
  IEEE Journal of Biomedical and Health Informatics (IEEE JBHI), 2020  (Impact Factor: 6.7)

• Noninvasive Model for Predicting Future Ischemic Strokes in Patients with Silent Lacunar Infarction Using Radiomics
  Lingwei Meng#, Jiehua Su#, Di Dong#, Wenyan Zhuo, Jianming Wang, Libin Liu, Yi Qin, Ye Tian, Jie Tian, Zhaohui Li
  BMC Medical Imaging, 2020  (# denotes the co-first authorship.)  (Impact Factor: 2.9)

Challenges

• Rank 2/14IEEE ICASSP 2022 Multi-channel Multi-party Meeting Transcription Challenge (M2MeT) - Track 1

Activities

• Reviewer of International Conference on Learning Representations (ICLR)
• Reviewer of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)
• Reviewer of ISCA INTERSPEECH
• Reviewer of IEEE Journal of Biomedical and Health Informatics

Teaching Assistance

SEEM 3440, Operation Research II
AIST 3510 / SEEM 3510, Human-Computer Interaction

Miscellaneous

Some awards on control algorithm and circuit design competitions:

• First Prize (main contributor) Aug 2018, the National Marine Vehicle Design Competition, China
• Second Prize (team leader) Jun 2017, ABU Asia-Pacific Robot Contest (RoboCon), China Division
• Second Prize (team leader) Jul 2016, NXP Cup National University Students Intelligent Car Race, China
• Second Prize (team leader) Sep 2015, the National Undergraduate Electronics Design Contest, China