Prof. Jingjing Liu is Professor, Principal Investigator at Institute for AI Industry Research, Tsinghua University. Her current research interests span from Multimodal LLM to AI for Science and Reinforcement Learning. Prof. Liu has published 100+ papers at top AI conferences and journals (CVPR, NeurIPS, ICML, ACL, etc.) with 15,000+ citations, and has received the Best Student Paper Honorable Mention Awards at CVPR and WACV. Prior to joining Tsinghua University in 2021, Prof. Liu was Senior Principal Research Manager at Microsoft in Redmond, US, leading a Multimodal AI research group that pioneered in Vision+Language Multimodal Pre-training. Before joining Microsoft Research in 2014, Prof. Liu was Research Scientist at MIT CSAIL in Cambridge, US, with the research focus on Spoken Dialogue Systems. Prof. Liu received the PhD degree in Computer Science from MIT Department of EECS. She also holds an MBA degree from Judge Business School (JBS) at University of Cambridge in the UK.

Homepage：

LinkedIn

Google Scholar

Research Fields:

Multimodal LLM, AI for Science, Reinforcement Learning

Selected Publications:

[1]Jinliang Zheng, Jianxiong Li, Sijie Cheng, Yinan Zheng, Jiaming Li, Jihao Liu, Yu Liu, Jingjing Liu, and Xianyuan Zhan, Instruction-Guided Visual Masking, NeurIPS 2024. (arxiv/2405.19783)

[2]Xin Ma, Yang Liu, Jingjing Liu, and Xiaoxu Ma, Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs, NeurIPS 2024.

[3]Tianyuan Zou, Yang Liu, Peng Li, Jianqing Zhang, Jingjing Liu, and Ya-Qin Zhang, FuseGen: PLM Fusion for Data-generation based Zero-shot Learning, EMNLP 2024. (arxiv/2406.12527)

[4]Jianxiong Li, Jinliang Zheng, Yinan Zheng, Liyuan Mao, Xiao Hu, Sijie Cheng, Haoyi Niu, Jihao Liu, Yu Liu, Jingjing Liu, Ya-Qin Zhang, Xianyuan Zhan, DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning, ICML 2024. (arxiv/2402.18137)

[5]Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, Jingjing Liu, CapsFusion: Rethinking Image-Text Data at Scale, CVPR 2024. (arxiv/2310.20550)

[6]Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, Xinlong Wang, Generative Multimodal Models are In-Context Learners, CVPR 2024. (arxiv/2312.13286)

[7]Qiying Yu, Yudi Zhang, Yuyan Ni, Shikun Feng, Yanyan Lan, Hao Zhou, and Jingjing Liu, Multimodal Molecular Modeling via Modality Blending, ICLR 2024. (arxiv/2307.06235)

[8]Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu, Safe Offline Reinforcement Learning with Feasibility-guided Diffusion Model, ICLR 2024. (arxiv/2401.10700)

[9]Yuxuan Song, Jingjing Gong, Hao Zhou, Mingyue Zheng, Jingjing Liu, and Wei-Ying Ma, Unified Generative Modeling of 3D Molecules with Bayesian Flow Networks, ICLR 2024.(arxiv/2403.15441)

[10]Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, Xinlong Wang, Emu: Generative Pretraining in Multimodality, ICLR 2024. (arxiv/2307.05222)

[11]Tongda Xu, Dailan He, Ziran Zhu, Yanghao Li, Lina Guo, Yuanyuan Wang, Zhe Wang, Hongwei Qin, Yan Wang, Jingjing Liu, Ya-Qin Zhang, Idempotence and Perceptual Image Compression, ICLR 2024. (arxiv/2401.08920)

[12]Shanzhi Yin, Tongda Xu, Yongsheng Liang, Yuanyuan Wang, Yanghao Li, Yan Wang, and Jingjing Liu, Bandwidth-efficient Inference for Neural Image Compression, ICASSP 2024. (arxiv/2309.02855)

[13]Yanghao Li, Tongda Xu, Yan Wang, Jingjing Liu, and Ya-Qin Zhang, Idempotent Learned Image Compression with Right-Inverse, NeurIPS 2023.(proceedings.neurips)

[14]Bowen Gao, Bo Qiang, Haichuan Tan, Yinjun Jia, Minsi Ren, Minsi Lu, Jingjing Liu, Wei-Ying Ma, and Yanyan Lan, DrugCLIP: Contrasive Protein-Molecule Representation Learning for Virtual Screening, NeurIPS 2023.(arxiv/2310.06367)

[15] Hideaki Takahashi, Jingjing Liu, Yang Liu, Breaching FedMD: Image Recovery via Paired-Logits Inversion Attack, CVPR 2023. (arxiv/2304.11436)

[16] Tongda Xu, Han Gao, Yuanyuan Wang, Hongwei Qin, Yan Wang, Jingjing Liu, Ya-Qin Zhang, Correcting the Sub-optimal Bit Allocation, ICML 2023. (arxiv/2209.14575)

[17] Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu, Multimodal Federated Learning via Contrastive Representation Ensemble, ICLR 2023. (arxiv/2302.08888)

[18] Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang, When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning, ICLR 2023. (arxiv/2205.11027)

[19] Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shan Jia, Ya-Qin Zhang, Mind the Gap: Offline Policy Optimization for Imperfect Rewards, ICLR 2023. (arxiv/2302.01667)

[20] Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, and Jingjing Liu, ADAPT: Action-aware Driving Caption Transformer, ICRA 2023. (arxiv:2302.00673)

[21] Qiying Yu, Jieming Lou, Xianyuan Zhan, Qizhang Li, Wangmeng Zuo, Yang Liu, and Jingjing Liu, Adversarial Contrastive Learning via Asymmetric InfoNCE, ECCV 2022. (arxiv:2207.08374)

[22] Tianlong Chen, Yu Cheng, Zhe Gan, Jianfeng Wang, Lijuan Wang, Jingjing Liu, and Zhangyang Wang, Adversarial Feature Augmentation and Normalization for Visual Recognition, Transactions on Machine Learning Research (TMLR), 2022. (arxiv:21Z03.12171)

[23] Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, Jingjing Liu, Lijuan Wang and Zicheng Liu, Playing Lottery Tickets with Vision and Language, AAAI 2022.(arxiv:2104.11832)

[24] Jinghui Chen, Yu Cheng, Zhe Gan, Quanquan Gu, and Jingjing Liu, Efficient Robust Training via Backward Smoothing, AAAI 2022.(arxiv:2010.01278)

[25] Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu, Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models, ICCV 2021.(arxiv:2106.00245)

[26] Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang, The Elastic Lottery Ticket Hypothesis, NeurIPS 2021. (arxiv:2103.16547)

[27] Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, Zhangyang Wang, Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective, NeurIPS 2021. (arxiv:2103.00397)

[28] Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, Tamara L. Berg, Mohit Bansal, Jingjing Liu, Lijuan Wang, Zicheng Liu, VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation, NeurIPS 2021. (arxiv:2106.04632)

[29] Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, Jingjing Liu, EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets, ACL 2021 (Oral). (arXiv:2101.00063)

[30] Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, Jingjing Liu, Cluster-Former: Clustering-based Sparse Transformer for Question Answering, Findings of ACL 2021. (arxiv/2009.06097)

[31] Chen Zhu, Yu Cheng, Zhe Gan, Furong Huang, Jingjing Liu, and Tom Goldstein, Adaptive Learning Rates with Maximum Variation Averaging, ECML 2021. (arXiv:2006.11918v1)

[32] Jie Lei*, Linjie Li*, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu, Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling, CVPR 2021 (Oral). (arXiv:2102.06183)

[33] Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu, UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training, CVPR 2021. (arXiv:2104.00332)

[34] Liqun Chen*, Dong Wang*, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin, Wasserstein Contrastive Representation Distillation, CVPR 2021. (arXiv:2012.08674).

[35] Shuyang Dai, Zhe Gan, Yu Cheng, Chenyang Tao, Lawrence Carin, and Jingjing Liu, APo-VAE: Text Generation in Hyperbolic Space, NAACL 2021. (arXiv:2005.00054)

[36] Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, and Jingjing Liu, LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval, NAACL 2021. (arXiv:2103.08784)

[37] Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu, InfoBERT: Improving Robustness of Language Models from an Information Theoretic Perspective, ICLR 2021. (arXiv:2010.02329)

[38] Yuwei Fang*, Shuohang Wang*, Zhe Gan, Siqi Sun, and Jingjing Liu, FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding, AAAI 2021. (arXiv:2009.05166)

[39] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu, Large-Scale Adversarial Training for Vision-and-Language Representation Learning, NeurIPS 2020 (Spotlight) (arXiv:2006.06195)

[40] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu, HERO: Hierarchical Encoder for Video Language Omni-representation Pre-training, EMNLP 2020. (arXiv:2005.00200)

[41] Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu, Contrastive Distillation on Intermediate Representations for Language Model Compression, EMNLP 2020. (arXiv:2009.14167)

[42] Shuohang Wang, Yuwei Fang, Siqi Sun, Zhe Gan, Yu Cheng, Jiang Jing, and Jingjing Liu, Cross-Thought for Sentence Encoder Pre-training, EMNLP 2020. (arXiv:2010.03652)

[43] Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu, Hierarchical Graph Network for Multi-hop Question Answering, EMNLP 2020. (arXiv:1911.03631)

[44] Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu, Multi-Fact Correction in Abstractive Text Summarization, EMNLP 2020. (arXiv:2010.02443)

[45] Yu Cheng, Yizhe Zhang, Oussama Elachqar, Zhe Gan, and Jingjing Liu, Contextual Text Style Transfer, EMNLP 2020 (Findings of EMNLP). (arXiv:2005.00136)

[46] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu, UNITER: Learning UNiversal Image-TExt Representations, ECCV 2020. (arXiv:1909.11740)

[47] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu, Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models, ECCV 2020 (Spotlight). (arXiv:2005.07310)

[48] Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu, Graph Optimal Transport for Cross-Domain Alignment, ICML 2020. (arXiv:2006.14744)

[49] Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu, Distilling Knowledge Learned in BERT for Text Generation, ACL 2020. (arXiv:1911.03829)

[50] Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu, Discourse-Aware Neural Extractive Model for Text Summarization, ACL 2020. (arXiv:1910.14142)

[51] Yizhe Zhang Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan, DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation, ACL 2020. (arXiv:1911.00536)

[52] Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu, VIOLIN: A Large-Scale Dataset for Video-and-Language Inference, CVPR 2020. (arXiv:2003.11618)

[53] Yandong Li, Yu Cheng, Zhe Gan, Licheng Yu, Liqiang Wang, and Jingjing Liu, BachGAN: High-Resolution Image Synthesis from Salient Object Layout, CVPR 2020. (arXiv:2003.11690)

[54] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu, FreeLB: Enhanced Adversarial Training for Language Understanding, ICLR 2020. (arXiv: 1909.11764)

[55] Shuohang Wang, Yunshi Lan, Yi Tay, Jing Jiang, and Jingjing Liu, Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching, AAAI 2020. (arXiv:2001.07234)

[56] Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig, What Makes A Good Story? Designing Composite Rewards for Visual Storytelling, AAAI 2020. (arXiv: 1909.05316)

[57] Zhe Gan, Yu Cheng, Ahmed EI Kholy, Linjie Li, Jingjing Liu, and Jianfeng Gao, Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog, ACL 2019. (arXiv: 1902.00579)

[58] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu, Relation-aware Graph Attention Network for Visual Question Answering, ICCV 2019. (arXiv: 1903.12314)

[59] Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao, StoryGAN: A Sequential Conditional GAN for Story Visualization, CVPR 2019. (arXiv: 1812.02784)

[60] Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation, CVPR 2019 (Oral). (arXiv: 1903.02547)

[61] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient Knowledge Distillation for BERT Model Compression, EMNLP 2019. (arXiv: 1908.09355)