Demo Video

Recent advancements in text-to-3D generation improve the visual quality of Score Distillation Sampling (SDS) and its variants by directly connecting Consistency Distillation (CD) to score distillation. However, due to the imbalance between self-consistency and cross-consistency, these CD-based methods inherently suffer from improper conditional guidance, leading to sub-optimal generation results. To address this issue, we present SegmentDreamer, a novel framework designed to fully unleash the potential of consistency models for high-fidelity text-to-3D generation. Specifically, we reformulate SDS through the proposed Segmented Consistency Trajectory Distillation (SCTD), effectively mitigating the imbalance issues by explicitly defining the relationship between self- and cross-consistency. Moreover, SCTD partitions the Probability Flow Ordinary Differential Equation (PF-ODE) trajectory into multiple sub-trajectories and ensures consistency within each segment, which can theoretically provide a significantly tighter upper bound on distillation error. Additionally, we propose a distillation pipeline for a more swift and stable generation. Extensive experiments demonstrate that our SegmentDreamer outperforms state-of-the-art methods in visual quality, enabling high-fidelity 3D asset creation through 3D Gaussian Splatting (3DGS).

An overview of SegmentDreamer: We begin by initializing a 3D representation θusing a 3D generator, such as Point-E. In each iteration, we randomly render a batch of camera views \( \mathbf{z}_0 \) from \( \theta \) and diffuse them into \( \mathbf{z}_m \) with fixed noise \( \boldsymbol{\epsilon}^* \). Next, we transform zsm into \(\tilde{\mathbf{z}}^{\boldsymbol{\Phi}}_t\) using either one-step or two-step unconditional deterministic sampling. During the denoising process, we first estimate \(\hat{\mathbf{z}}^{\boldsymbol{\Phi}}_s\) through one-step conditional deterministic sampling from \(\tilde{\mathbf{z}}^{\boldsymbol{\Phi}}_t\). Subsequently, we compute two consistency functions and utilize them to derive the loss \(\mathcal{L}_{\text{SCTD}}\), which is ultimately employed to optimize \(\theta\)