Deep Generative Models with Human Preferences
- Publication Type:
- Thesis
- Issue Date:
- 2023
Open Access
Copyright Clearance Process
- Recently Added
- In Progress
- Open Access
This item is open access.
Powered by the learning capacity of deep neural networks, generative models have facilitated the scalable modeling of complex, high-dimensional data and are extensively used in various fields. In practical scenarios, deep generative models (DGMs) are often required not only to produce authentic samples but also to optimize synthetic samples for some desired properties. While existing DGMs are capable of generating data meeting users' expectations using desired class/attribute labels or an off-shelf evaluator, acquiring complete knowledge pertaining to the target property is an indispensable prerequisite for obtaining the labels or the evaluator, which is unmet in many real-world applications. In addition, discrete labels have limited description capacity, which cannot capture intra-category differences.
This thesis resorts to human preferences that are more readily accessible, which are typically represented by comparisons among a list of samples and can provide fine-grained information.
Motivated by real-world problems, preferences-guided desired data generation can be defined in terms of the dataset level or the instance level, which means generating the desired data based on a given dataset or a single sample, respectively. This thesis focuses on deep generative modeling from human preferences in different scenarios.
(1) First investigation on preference-guided desired data generation at the dataset level. We incorporate pairwise preferences into the existing framework of DGMs by introducing an additional pairwise ranking loss over the critic of Wasserstein Generative Adversarial Network, which slightly shifts the learned distribution of the generative model towards the desired data distribution. Our model converges to the desired data distributions by multi-step distribution shifts.
(2) A new and more efficient generative modeling paradigm for preference-guided desired data generation at the dataset level. We learn the desired data distribution from partial preferences via an adversarial ranking framework, which is proven to estimate a relativistic $f$-divergence between the desired data distribution and the generated data distribution. This approach shifts the generative model's distribution towards the desired data distribution in a single step, resulting in reduced training expenses.
(3) Preference-guided desired data generation at the instance level. We propose an adversarial ranking paradigm for generating desired data for single input samples (a.k.a., fine-grained image-to-image translation) based on comparisons in terms of specified attributes. The adversarial training between the ranker and the generator enhances the ability of the ranker and encourages a better generator. Meanwhile, our ranker enforces a linearizedly continuous change between the generated image and the input image.
Please use this identifier to cite or link to this item: