Zhen Liu

Google Scholar | Twitter

 

PhD candidate @ Mila and Université de Montréal

 

Incoming assistant professor @ CUHK-Shenzhen

 

Email : [firstname].[lastname].2[at]umontreal.ca


Bio

I am a PhD candidate in computer science at Mila and Université de Montréal, advised by Liam Paull and Yoshua Bengio. I received my M.S. and B.S. degrees in computer science with a minor in economics from Georgia Institute of Technology.

 

I am currently visiting at Max Planck Institute for Intelligent Systems, working with Michael J. Black and Bernhard Schölkopf.

 

News: I will join School of Data Science at CUHK-Shenzhen and lead CUHK-SZ GenLab as an assistant professor in Spring 2025.

 

I am actively looking for self-motivated PhD / MPhil students (Spring / Fall '25) and research assistants (see below for research directions). Please fill out the Google Form and then drop me an email through zliuacademia@gmail.com if interested.

Resaerch

I study representations and machine learning methods to build generalizable models and agents with spatial, physical and semantic understanding and to re-create / simulate our world.

 

Below are some more concrete research questions I am recently interested in answering:

  • 3D Content Generation. How to efficiently and effectively generate 3D shapes, materials and motions in a physics-aware way? How can we customize multi-modal models for automated creation of games/movies?
  • Assembly. Can we enable models to understand and assemble complex 3D structures for us?
  • Semantics. Can we achieve synergy of spatial and semantic understanding in foundation models?

Publications

(All / Selected)

Fast Diversity-Preserving Reward Finetuning of Diffusion Models via Nabla-GFlowNets
Zhen Liu#, Tim Z. Xiao*, Weiyang Liu*, Yoshua Bengio, Dinghuai Zhang#

Preprint, 2024

arXiv (to appear soon)

Can Large Language Models Understand Symbolic Graphics Programs?
Zeju Qiu*, Weiyang Liu*, Haiwen Feng*, Zhen Liu**, Tim Z. Xiao**, Katherine M. Collins**, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, Bernhard Schölkopf

Preprint, 2024

arXiv | code | project | bib 

      @article{qiu2024sgpbench,
        title={Can Large Language Models Understand Symbolic Graphics Programs?},
        author={Qiu, Zeju and Liu, Weiyang and Feng, Haiwen and Liu, Zhen and Xiao, Tim Z and Collins, Katherine M 
          and Tenenbaum, Joshua B and Weller, Adrian and Black, Michael J and Sch{\"o}lkopf, Bernhard},
        journal={arXiv preprint arXiv:2408.08313},
        year={2024}
      }

PuzzleAvatar: Assembling 3D Avatars from Personal Albums
Yuliang Xiu, Yufei Ye, Zhen Liu, Dimitrios Tzionas, Michael J. Black

ACM Transactions on Graphics (SIGGRAPH Asia), 2024

arXiv | video | bib 

        @article{xiu2024puzzleavatar,
          title={PuzzleAvatar: Assembling 3D Avatars from Personal Albums},
          author={Xiu, Yuliang and Ye, Yufei and Liu, Zhen and Tzionas, Dimitrios and Black, Michael J},
          journal={ACM Transactions on Graphics (TOG)},
          year={2024},
          publisher={ACM New York, NY, USA}
        }

Ghost on the Shell: An Expressive Representation of General 3D Shapes
Zhen Liu, Yao Feng*, Yuliang Xiu*, Weiyang Liu*, Liam Paull, Michael J. Black†, Bernhard Schölkopf†

International Conference on Learning Representations (ICLR), 2024 (Oral)

website | code | arXiv | pdf | bib 

  @InProceedings{Liu2024gshell,
        title = {Ghost on the Shell: An Expressive Representation of General 3D Shapes},
        author = {Liu, Zhen and Feng, Yao and Xiu, Yuliang and Liu, Weiyang and Paull, Liam and Black, Michael J. and Schölkopf, Bernhard},
        booktitle = {International Conference on Learning Representations},
        year = {2024}
      }

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Weiyang Liu*, Zeju Qiu*, Yao Feng**, Yuliang Xiu**, Yuxuan Xue**, Longhui Yu**, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, Bernhard Schölkopf

International Conference on Learning Representations (ICLR), 2024

arXiv | code | project | pdf | bib

  @InProceedings{liu2023boft,
        author = {Liu, Weiyang and Qiu, Zeju and Feng, Yao and Xiu, Yuliang and Xue, Yuxuan and Yu, Longhui and Feng, Haiwen and Liu, Zhen 
          and Heo, Juyeon and Peng, Songyou and Wen, Yandong and Black, Michael J. and Weller, Adrian and Sch{\"o}lkopf, Bernhard},
        title = {Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization},
        booktitle = {International Conference on Learning Representations},
        year = {2024}
      }

Controlling Text-to-Image Diffusion by Orthogonal Finetuning
Zeju Qiu*, Weiyang Liu*, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, Bernhard Schölkopf

Neural Information Processing Systems (NeurIPS), 2023

arXiv | code | project | bib

  @InProceedings{Qiu2023OFT,
        title={Controlling Text-to-Image Diffusion by Orthogonal Finetuning},
        author={Qiu, Zeju and Liu, Weiyang and Feng, Haiwen and Xue, Yuxuan and Feng, Yao 
          and Liu, Zhen and Zhang, Dan and Weller, Adrian and Schölkopf, Bernhard},
          booktitle={NeurIPS},
          year={2023}
      }

Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods
Yuchen Lu, Zhen Liu, Aristide Baratin, Romain Laroche, Aaron Courville, Alessandro Sordoni

Transactions on Machine Learning Research (TMLR), 2023

openreview | bib

  @article{
        lu2023expresiveness,
        title={Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods},
        author={Yuchen Lu and Zhen Liu and Aristide Baratin and Romain Laroche and Aaron Courville and Alessandro Sordoni},
        journal={Transactions on Machine Learning Research},
        year={2023},
        url={https://openreview.net/forum?id=BxdrpnRHNh},
        }

MeshDiffusion: Score-based Generative 3D Mesh Modeling
Zhen Liu, Yao Feng, Michael J. Black, Derek Nowrouzezahrai, Liam Paull, Weiyang Liu

International Conference on Learning Representations (ICLR), 2023 (Notable-top-25%)

openreview | website | slides | code | arXiv | pdf | bib 

  
        @InProceedings{Liu2023MeshDiffusion,
          title={MeshDiffusion: Score-based Generative 3D Mesh Modeling},
          author={Zhen Liu and Yao Feng and Michael J. Black and Derek Nowrouzezahrai and Liam Paull and Weiyang Liu},
          booktitle={International Conference on Learning Representations},
          year={2023},
          url={https://openreview.net/forum?id=0cpM2ApF9p6}
          }

Iterative Teaching by Data Hallucination
Zeju Qiu*, Weiyang Liu*, Tim Z. Xiao, Zhen Liu, Umang Bhatt, Yucen Luo, Adrian Weller, Bernhard Schölkopf

International Conference on Artificial Intelligence and Statistics (AISTATS), 2023

arXiv

Continual Learning by Modeling Intra-Class Variation
Longhui Yu, Tianyang Hu, Lanqing Hong, Zhen Liu, Adrian Weller, Weiyang Liu

Transactions on Machine Learning Research (TMLR), 2023

arXiv | OpenReview

Structural Causal 3D Reconstruction
Weiyang Liu*, Zhen Liu*, Liam Paull, Adrian Weller, Bernhard Schölkopf

European Conference on Computer Vision (ECCV), 2022

arXiv

Generative Flow Networks for Discrete Probabilistic Modeling
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, Yoshua Bengio

International Conference on Machine Learning (ICML), 2022

arXiv

Iterative Teaching by Label Synthesis
Weiyang Liu*, Zhen Liu*, Hanchen Wang*, Liam Paull, Bernhard Schölkopf, Adrian Weller.

Neural Information Processing Systems (NeurIPS), 2021 (Spotlight)

arXiv

A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning
Mingde Zhao*, Zhen Liu*, Sitao Luan*, Shuyuan Zhang*, Doina Precup, Yoshua Bengio.

Neural Information Processing Systems (NeurIPS), 2021

arXiv | TL;DR | project | code 

 
        We show a consciousness-inspired bottleneck (i.e. agent attending to limited environment entities at a time) enables 
        model-based RL agents with out-of-distribution generalization capability.

Orthogonal Over-parametrized Training
Weiyang Liu*, Rongmei Lin*, Zhen Liu, James M Rehg, Liam Paull, Li Xiong, Adrian Weller, Le Song.

Conference on Computer Vision and Pattern Recognition (CVPR), 2021 (Oral)

arXiv | TL;DR | project | code | slides | talk 

 
        We parametrize convolution operator with a learnable rotation matrix multiplied by a fixed randomly-initialized matrix 
        so that a minimal hyperspherical energy of a network, a measure of neuron diversity and generalization, is guaranteed
        during the entire training process.
        - Generalization and optimization landscape are improved;
        - Also helpful for large category training.

Learning with Hyperspherical Uniformity
Weiyang Liu*, Rongmei Lin*, Zhen Liu*, Li Xiong, Bernhard Schölkopf, Adrian Weller.

International Conference on Artificial Intelligence and Statistics (AISTATS), 2021

arXiv

Neural similarity learning
Weiyang Liu*, Zhen Liu*, James M Rehg, Le Song.

Neural Information Processing Systems (NeurIPS), 2019

paper

Exponential Family Estimation via Adversarial Dynamics Embedding
Bo Dai*, Zhen Liu*, Hanjun Dai*, Niao He, Arthur Gretton, Le Song, Dale Schuurmans.

Neural Information Processing Systems (NeurIPS), 2019

arXiv | code

Coupled Variational Bayes via Optimization Embedding
Bo Dai, Hanjun Dai, Niao He, Weiyang Liu, Zhen Liu, Jianshu Chen, Lin Xiao, Le Song.

Neural Information Processing Systems (NeurIPS), 2018

paper

Learning towards Minimum Hyperspherical Energy
Weiyang Liu*, Rongmei Lin*, Zhen Liu*, Lixin Liu*, Zhiding Yu, Bo Dai, Le Song.

Neural Information Processing Systems (NeurIPS), 2018

arXiv | code

Decoupled Network
Weiyang Liu*, Zhen Liu*, Zhiding Yu, Bo Dai, Rongmei Lin, Yisen Wang, James Rehg, Le Song.

Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Spotlight)

pdf | code | slide

Towards Black-box Iterative Machine Teaching
Weiyang Liu*, Bo Dai*, Xingguo Li, Zhen Liu, James Rehg, Le Song.

International Conference on Machine Learning (ICML), 2018

arXiv | code

SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, Le Song.

International Conference on Machine Learning (ICML), 2018

arXiv

Deep Forward and Inverse Perceptual Models for Tracking and Prediction
Alexander Lambert, Amirreza Shaban, Amit Raj, Zhen Liu, Byron Boots.

International Conference on Robotics and Automation (ICRA), 2018

arXiv

One Shot Learning for Semantic Segmentation
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, Byron Boots.

The British Machine Vision Conference (BMVC), 2017

arXiv | code

Motion Planning with Graph-Based Trajectories and Gaussian Process Inference
Eric Huang, Mustafa Mukadam, Zhen Liu, Byron Boots.

International Conference on Robotics and Automation (ICRA), 2017

pdf | demo