Research Areas: Video Understanding - Content Creation - Robust Deep Learning - Curriculum Learning

Preprints Google Scholar

  • Muse: Text-To-Image Generation via Masked Generative Transformers
    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, Yuanzhen Li, Dilip Krishnan
    arXiv preprint arXiv:2301.00704 [pdf], [code]

  • Auditing Gender Presentation Differences in Text-to-Image Models
    Yanzhe Zhang, Lu Jiang, Greg Turk, Diyi Yang
    arXiv preprint arXiv:2302.03675 [pdf], [code]

Selected Publications Google Scholar

  • MAGVIT: Masked Generative Video Transformer
    Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, Lu Jiang
    CVPR 2023 [project page], [code]

  • Visual Prompt Tuning for Generative Transfer Learning
    Kihyuk Sohn, Huiwen Chang, José Lezama, Luisa Polania, , Han Zhang, Yuan Hao, Irfan Essa, Lu Jiang
    CVPR 2023 [pdf], [code]

  • Discrete Predictor-Corrector Diffusion Models for Image Synthesis
    Jose Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, Irfan Essa
    ICLR 2023 [pdf], [code coming soon]

  • BLT: Bidirectional Layout Transformer for Controllable Layout Generation
    Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, Irfan Essa
    ECCV 2022 [pdf], [code]

  • Improved Masked Image Generation with Token-Critic
    Jose Lezama, Huiwen Chang, Lu Jiang, Irfan Essa
    ECCV 2022 [pdf], [code]

  • MaskGIT: Masked Generative Image Transformer
    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman
    CVPR 2022 [pdf], [code]

  • Pyramid Adversarial Training Improves ViT Performance [Oral, Best paper finalist]
    Charles Herrmann*, Kyle Sargent*, Lu Jiang, Ramin Zabih, Huiwen Chang, Ce Liu, Dilip Krishnan, Deqing Sun
    CVPR 2022 (*equal contribution) [pdf], [code coming soon]

  • Discrete Representations Strengthen Vision Transformer Robustness
    Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, Irfan Essa
    ICLR 2022 [pdf], [code coming soon]

  • ViTGAN: Training gans with vision transformers [Spotlight]
    Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, Ce Liu
    ICLR 2022 [pdf], [code]

  • Text as Neural Operator: Image Manipulation by Text Instruction
    Tianhao Zhang, Hung-Yu Tseng, Lu Jiang, Weilong Yang, Honglak Lee, Irfan Essa
    ACM Multimedia 2021 [pdf], [code], [patent]

  • Self-supervised and Supervised Joint Training for Resource-rich Machine Translation
    Yong Cheng, Wei Wang, Lu Jiang, Wolfgang Macherey
    ICML 2021 [pdf], [code]

  • Regularizing Generative Adversarial Networks under Limited Data
    Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, Weilong Yang
    CVPR 2021 [pdf], [code]

  • Faster Meta Update Strategy for Noise-Robust Deep Learning [Oral]
    Youjiang Xu, Linchao Zhu, Lu Jiang, Yi Yang
    CVPR 2021 [pdf], [code]

  • Controllable and Progressive Image Extrapolation
    Yijun Li, Lu Jiang, Ming-Hsuan Yang
    WACV 2021 [pdf]

  • Confident Learning: Estimating Uncertainty in Dataset Labels
    Curtis G. Northcutt, Lu Jiang, Isaac L. Chuang
    Journal of Artificial Intelligence Research (JAIR) 2021 [pdf], [code(cleanlab)].

  • Contrastive Adaptation Network for Single- and Multi-Source Domain Adaptation
    Guoliang Kang, Lu Jiang, Yunchao Wei, Yi Yang, Alexander G. Hauptmann
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2021 [pdf], [code].

  • Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
    Lu Jiang, Di Huang, Mason Liu, Weilong Yang
    ICML 2020 [pdf], [slides], [video], [Google AI blog], [dataset, tfds], [code]

  • SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
    Junwei Liang, Lu Jiang, Alexander Hauptmann
    ECCV 2020 [pdf], [code]

  • RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval
    Hung-Yu Tseng*, Hsin-Ying Lee*, Lu Jiang, Ming-Hsuan Yang, Weilong Yang
    ECCV 2020 (*equal contribution) [pdf]

  • Neural Design Network: Graphic Layout Generation with Constraints [Spotlight]
    Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, Weilong Yang
    ECCV 2020 [pdf]

  • The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
    Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, Alexander Hauptmann
    CVPR 2020 [pdf], [code], [blog post]

  • AdvAug: Robust Adversarial Augmentation for Neural Machine Translation
    Yong Cheng, Lu Jiang, Wolfgang Macherey, Jacob Eisenstein
    ACL 2020 [pdf], [slides]

  • Revisiting EmbodiedQA: A Simple Baseline and Beyond
    Yu Wu, Lu Jiang, Yi Yang
    IEEE Transactions on Image Processing 2020 [pdf]

  • Robust Neural Machine Translation with Doubly Adversarial Inputs [Best paper candidate]
    Yong Cheng, Lu Jiang, Wolfgang Macherey
    ACL 2019 [pdf], [Google AI blog]

  • Peeking into the future: Predicting Future Person Activities and Locations in Videos
    Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander Hauptmann, Li Fei-Fei
    CVPR 2019 [pdf], [code], [demo video]

  • Composing Text and Image for Image Retrieval - An Empirical Odyssey [Oral]
    Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, James Hays
    CVPR 2019 [pdf], [code]

  • Contrastive Adaptation Network for Unsupervised Domain Adaptation
    Guoliang Kang, Lu Jiang, Yi Yang, Alexander Hauptmann
    CVPR 2019 [pdf], [code]

  • Eidetic 3D LSTM: A Model for Video Prediction and Beyond
    Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, Li Fei-Fei
    ICLR 2019 [pdf], [code]

  • Focal Visual-Text Attention for Memex Question Answering.
    Junwei Liang, Lu Jiang, Liangliang Cao, Yannis Kalantidis, Li-Jia Li, Alexander Hauptmann
    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2019 [pdf], [dataset], [code], [demo video]

  • MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
    Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, Li Fei-Fei
    ICML 2018 [pdf], [supplementary materials], [code], [slides]

  • Focal Visual-Text Attention for Visual Question Answering [Spotlight]
    Junwei Liang, Lu Jiang, Liangliang Cao, Li-Jia Li, Alexander Hauptmann
    CVPR 2018 [pdf], [project page]

  • Graph Distillation for Action Detection with Privileged Information
    Zelun Luo, Jun-Ting Hsieh, Lu Jiang, Juan Carlos Niebles, Li Fei-Fei
    ECCV 2018 [pdf], [project page]

  • Switchable Novel Object Captioner
    Yu Wu, Lu Jiang, Yi Yang
    TPAMI 2022 [pdf] and ACM MM 2018 [pdf]

  • Delving Deep into Personal Photo and Video Search
    Lu Jiang, Yannis Kalantidis, Liangliang Cao, Sachin, Farfade, Jiliang Tang, Alex Hauptmann
    WSDM 2017 [pdf]

  • Leveraging Multi-modal Prior Knowledge for Large-scale Concept Learning in Noisy Web Data
    Junwei Liang, Lu Jiang, Deyu Meng, Alexander Hauptmann
    ICMR 2017 [pdf]

  • Revealing Event Saliency in Unconstrained Video Collection
    Dingwen Zhang, Junwei Han, Lu Jiang, Senmao Ye, Xiaojun Chang
    IEEE Transactions on Image Processing 26(4): 1746-1758 2017 [pdf]

  • Web-scale Multimedia Search for Internet Video Content
    Lu Jiang
    WWW 2016 [pdf], [full thesis]

  • Learning to Detect Concepts from Webly-Labeled Video Data.
    Junwei Liang, Lu Jiang, Deyu Meng, Alexander Hauptmann
    IJCAI 2016, [pdf], [demo video]

  • Fast and Accurate Content-based Semantic Search in 100M Internet Videos
    Lu Jiang, Shoou-I Yu, Deyu Meng, Yi Yang, Teruko Mitamura, Alexander Hauptmann
    ACM MM 2015, [pdf], [slides], [project page]

  • Bridging the Ultimate Semantic Gap: A Semantic Search Engine for Internet Videos [Best paper candidate]
    Lu Jiang, Shoou-I Yu, Deyu Meng, Teruko Mitamura, Alexander Hauptmann
    ICMR 2015, [pdf], [slides], [project page]
    Featured in Pittsburgh Supercomputer Center Science Highlights

  • Content-Based Video Search over 1 Million Videos with 1 Core in 1 Second
    Shoou-I Yu, Lu Jiang, Zhongwen Xu, Yi Yang and Alexander Hauptmann
    ICMR 2015, [pdf]

  • Self-paced Curriculum Learning. [Oral]
    Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, Alexander Hauptmann
    AAAI 2015, [pdf], [supplementary materials], [demo code]

  • Self-paced Learning for Matrix Factorization
    Qian Zhao, Deyu Meng, Lu Jiang, Qi Xie, Zongben Xu, Alexander Hauptmann
    AAAI 2015, [pdf], [supplementary materials]

  • A Self-Paced Multiple-Instance Learning Framework for Co-Saliency Detection
    Dingwen Zhang, Deyu Meng, Chao Li, Lu Jiang, Qian Zhao, Junwei Han
    ICCV 2015, [pdf]

  • Self-paced Learning with Diversity
    Lu Jiang, Deyu Meng, Shoou-I Yu, Zhenzhong Lan, Shiguang Shan, Alexander Hauptmann
    NIPS 2014, [pdf], [supplementary materials], [project page]

  • Easy Samples First: Self-paced Reranking for Zero-Example Multimedia Search
    Lu Jiang, Deyu Meng, Teruko Mitamura, Alexander Hauptmann
    ACM MM 2014, [pdf], [slides]

  • Improvements to Speaker Adaptive Training of Deep Neural Networks
    Yajie Miao, Lu Jiang, Hao Zhang, Florian Metze
    SLT 2014, [pdf], [project page]

  • Towards Efficient Learning of Optimal Spatial Bag-of-Words Representations [Best paper candidate]
    Lu Jiang, Wei Tong, Deyu Meng, Alexander Hauptmann
    ICMR 2014, [pdf], [slides]

  • Zero-Example Event Search using MultiModal Pseudo Relevance Feedback
    Lu Jiang, Teruko Mitamura, Shoou-I Yu, Alexander Hauptmann
    ICMR 2014, [pdf], [slides]

  • Viral Video Style: A Closer Look at Viral Videos on YouTube [Best poster at CMU-LTI SRS14]
    Lu Jiang, Yajie Miao, Yi Yang, Zhen-Zhong Lan, Alexander Hauptmann
    ICMR 2014, [pdf], [slides], [dataset]

  • Leveraging High-level and Low-level Features for Multimedia Event Detection
    Lu Jiang, Alexander Hauptmann, Guang Xiang
    ACM MM 2012, [pdf], [slides]

  • Mining Learning-Dependency between Knowledge Units from Text
    Jun Liu, Lu Jiang, Zhaohui Wu, Qinghua Zheng, Yanan Qian
    The VLDB Journal, 20(3): 335-345, 2011, [pdf]

  • A Peep at Pornography Web in China
    Zhaohui Wu, Lu Jiang, Qinghua Zheng, Zhenhua Tian, Jun Liu, Junzhou Zhao
    Web of Science, 2010 [pdf]

  • Efficient Deep Crawling using Reinforcement Learning
    Lu Jiang, Zhaohui Wu, Qian Feng, Jun Liu, Qinghua Zheng
    PAKDD, 2010 [pdf]

    • Workshop Papers

    • Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
      Po-Yao Huang, Ye Yuan, Zhenzhong Lan, Lu Jiang, Alexander G. Hauptmann
      arxiv, 2017 [pdf]

    • Early Implementation Experience with Wearable Cognitive Assistance Applications
      Zhuo Chen, Lu Jiang, Wenlu Hu, Kiryong Ha, Brandon Amos, Padmanabhan Pillai, Alex Hauptmann and Mahadev Satyanarayanan
      WearSys at MobiSys, 2015 [pdf]

    • CMU-Informedia@ TRECVID 2014 [Top performer in the TRECVID multimedia event detection task 2014]
      Shoou-I Yu, Lu Jiang, Zhongwen Xu, Zhenzhong Lan, Shicheng Xu, Xiaojun Chang, Xuanchong Li, Zexi Mao, Chuang Gan, Yajie Miao, Xingzhong Du,Yang Cai, Lara Martin, Nikolas Wolfe, Anurag Kumar, Huan Li, Ming Lin, Zhigang Ma, Yi Yang, Deyu Meng, Shiguang Shan, Pinar Duygulu Sahin, Susanne Burger, Florian Metze, Rita Singh, Bhiksha Raj, Teruko Mitamura, Richard Stern and Alexander Hauptmann.
      NIST TRECVID 2014 [pdf]

    • CMU-Informedia@ TRECVID 2013 [Top performer in the TRECVID multimedia event search task 2013]
      Zhen-Zhong Lan, Lu Jiang, Shoou-I Yu, Shourabh Rawat, Yang Cai, Chenqiang Gao, Shicheng Xu, Haoquan Shen, Xuanchong Li, Yipei Wang, Waito Sze, Yan Yan, Zhigang Ma, Nicolas Ballas, Deyu Meng, Wei Tong, Yi Yang, Susanne Burger, Florian Metze, Rita Singh, Bhiksha Raj, Richard Stern, Teruko Mitamura, Eric Nyberg, and Alexander Hauptmann
      NIST TRECVID 2013 [pdf]

    • Informedia@ TRECVID 2011: Surveillance Event Detection [Best performer in the TRECVID surveillance event detection task 2011]
      Longfei Zhang, Lu Jiang, Lei Bao, Shohei Takahashi, Yuanpeng Li, Alexander Hauptmann]
      NIST TRECVID 2011 [pdf]