ComfyUI中的一些代码实现
| 步骤 | 细节 | 种类 | |
|---|---|---|---|
| Euler 和SDE的Euler–Maruyama 的解法不同 |
noise injection: - increased noise $\hat \sigma$ : $\hat \sigma\leftarrow \sigma_i + \gamma\sigma_i$ - sample x with increased noise: $\hat x \leftarrow x_i + \sqrt{\hat \sigma^2-\sigma_i^2}\cdot\epsilon$ Take Euler Step: - $dt=\sigma_{i+1}-\hat \sigma$ - $denoised=model(\hat x,\hat \sigma)$ - gradient: $d=(\hat x-denoised)/{\hat \sigma}$ - Euler step: $x_{i+1}=\hat x+dt \cdot d$ |
采用SMLD, $\alpha_t=1$ DDIM和Euler-method表达式一致, 不用另外写 |
|
| heun Euler方法的改进 |
noise injection: 得到$\hat\sigma, \hat x, denoised$ Left tangent prediction: - $x_2=\hat x + dt\cdot d$ Right tangent prediction: - $d_2=(x_2-model(x_2,\sigma_{i+1}))/\sigma_{i+1}$ 结果: - $d^\prime=\frac{d+d_2}{2}$ - $x_{i+1}=\hat x+d^\prime\cdot dt$ |
||
| Euler Ancestral | Take Euler Step to $\sigma_{down}$: - $dt=\sigma_{down}-\sigma_i$ - $denoised=model(x,\sigma_i)$ - numerical derivative: $d=(x-denoised)/{\sigma_i}$ - Euler step: $x_{down}=x+dt \cdot d$ Add ancestral noise: - $x_{i+1}=x_{down}+noise*\sigma_{up}$ |
$\sigma_{up}=\min(\sigma_{i+1},\eta\cdot\sqrt{(\frac{\sigma_{i+1}^2}{\sigma_i^2}(\sigma_i^2-\sigma_{i+1}^2))})$ $\sigma_{down}=\sqrt{\sigma_{i+1}^2-\sigma_{up}^2}$ |
|
| lms(linear multistep method ) | 用 Lagrange basis functions 拟合$\epsilon$ 即, $f=\sum_k P_k\cdot \epsilon_{i+k}$ , 令$f(\sigma_{i+k})=\epsilon(x_{i+k},\sigma_{i+k})$ $x_{i-1}=x_i+\int_{\sigma_i}^{\sigma_{i-1}}f d\sigma$ |
$f$ 是对$\epsilon$ 的估计值, k=0时退化成Euler method |
|
| deis & ipndm & ipndm-v |
实现不一样但是思路和lms差不多? | ||
| dpm_fast | 根据number of function evaluations (nfe) 的输入 设置dpm的次数. 优先使用dpm-solver-3, nfe不被3整除的情况下用dpm-solver-2或dpm-solver-1补充 |
DPM-Solver 根据probability ODE |
|
| dpm_adaptive | 不输入nfe 同时计算dpm-solver-2和dpm-solver-3 比较结果, 两者的差值(L2-norm的scale)作为pid- controller的输入, 由pid-controlller判断是否结束 |
PID 中实际使用最近的3个error 的-log $k_p(e_i-e_{i-1})+k_ie_i+k_d(e_i-2e_{i-1}+e_{i-2})$ error $\searrow$ , factor $\nearrow$ , accept ✔️ 和标准的pid不一样: 1. 积分项对应error 2. 只根据factor输出是否结束, factor不输出, 对每个step本身的步骤也没有影响 |
DPM-Solver 根据probability ODE |
| dpm_2 & dpm_2_ancestral |
noise injection: 得到$\hat\sigma, \hat x, denoised$ DPM-Solver-2: - 在$\sigma_{i+1},\hat \sigma$ 之间取: $\sigma_{mid}=e^{\frac{\log \hat\sigma+\log \sigma_{i+1}}{2}}$ - $dt_1=\sigma_{mid}-\hat \sigma$ - $dt_2=\sigma_{i+1}-\hat \sigma$ - $x_{mid}=\hat x+dt_1\cdot(\hat x-model(\hat x,\hat \sigma))/\hat \sigma$ 用$\sigma_{mid}$ 处的numerial derivative - $x=\hat x+dt_2\cdot(x_{mid}-model(x_{mid},\sigma_{mid}))/\sigma_{mid}$ |
用DPM-Solver-2的简化板, 把二分点取在中间 |
|
| dpmpp_2m | - $t=-\log(\sigma_i)$ , $t_{next}=-\log(\sigma_{i+1})$ , $t_{last}=-\log(\sigma_{i-1})$ - $h=t_{next}-t$, $h_{last}=t-t_{last}$ - $r=h_{last}/h$ - $denoised_d=(1+\frac{1}{2r})\cdot denoised-\frac{1}{2r}\cdot denoised_{old}$ - $x_{i+1}=\frac{\sigma_{i+1}}{\sigma_i}\cdot x_i-(e^{-h}-1)\cdot denoised_d$ |
DPM-Solver++ |
reference
[1 ] Karras, Tero, Miika Aittala, Timo Aila, and Samuli Laine. “Elucidating the Design Space of Diffusion-Based Generative Models.” arXiv, October 11, 2022. https://doi.org/10.48550/arXiv.2206.00364. Euler, Heun method
[2] Lu, Cheng, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps.” arXiv, October 13, 2022. https://doi.org/10.48550/arXiv.2206.00927.
[3] Lu, Cheng, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. “DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models.” arXiv, May 6, 2023. https://doi.org/10.48550/arXiv.2211.01095.
[4] Zhao, Wenliang, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. “UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models,” 2023. https://openreview.net/forum?id=hrkmlPhp1u&referrer=%5Bthe%20profile%20of%20Jie%20Zhou%5D(%2Fprofile%3Fid%3D~Jie_Zhou3).
[4] Zhang, Qinsheng, and Yongxin Chen. “Fast Sampling of Diffusion Models with Exponential Integrator.” arXiv, February 25, 2023. https://doi.org/10.48550/arXiv.2204.13902. DEIS, ipndm
[5] Luo, Simian, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference.” arXiv, October 6, 2023. https://doi.org/10.48550/arXiv.2310.04378. LCM
[5] 代码: https://github.com/zju-pi/diff-sampler 一些sampler
[6] 代码: https://github.com/comfyanonymous/ComfyUI/blob/master/comfy/k_diffusion/sampling.py https://github.com/comfyanonymous/ComfyUI/blob/master/comfy/samplers.py
[7] https://stable-diffusion-art.com/samplers/#DDIM_and_PLMS 不同sampler的比较测评