πŸ“–
Wiki
CNCFSkywardAIHuggingFaceLinkedInKaggleMedium
  • Home
    • πŸš€About
  • πŸ‘©β€πŸ’»πŸ‘©Freesoftware
    • πŸ‰The GNU Hurd
      • πŸ˜„The files extension
      • πŸ“½οΈTutorial for starting
      • 🚚Continue Working for the Hurd
      • πŸš΄β€β™‚οΈcgo
        • πŸ‘―β€β™€οΈStatically VS Dynamically binding
        • 🧌Different ways in binding
        • πŸ‘¨β€πŸ’»Segfault
      • πŸ›ƒRust FFI
    • πŸ§šπŸ»β€β™‚οΈProgramming
      • πŸ“–Introduction to programming
      • πŸ“–Mutable Value Semantics
      • πŸ“–Linked List
      • πŸ“–Rust
        • πŸ“–Keyword dyn
        • πŸ“–Tonic framework
        • πŸ“–Tokio
        • πŸ“–Rust read files
  • πŸ›€οΈAI techniques
    • πŸ—„οΈframework
      • 🧷pytorch
      • πŸ““Time components
      • πŸ““burn
    • 🍑Adaptation
      • 🎁LoRA
        • ℹ️Matrix Factorization
        • πŸ“€SVD
          • ✝️Distillation of SVD
          • 🦎Eigenvalues of a covariance matrix
            • 🧧Eigenvalues
            • πŸͺCovariance Matrix
        • πŸ›«Checkpoint
      • 🎨PEFT
    • πŸ™‹β€β™‚οΈTraining
      • πŸ›»Training with QLoRA
      • 🦌Deep Speed
    • 🧠Stable Diffusion
      • πŸ€‘Stable Diffusion model
      • πŸ“ΌStable Diffusion v1 vs v2
      • πŸ€Όβ€β™€οΈThe important parameters for stunning AI image
      • ⚾Diffusion in image
      • 🚬Classifier Free Guidance
      • ⚜️Denoising strength
      • πŸ‘·Stable Diffusion workflow
      • πŸ“™LoRA(Stable Diffusion)
      • πŸ—ΊοΈDepth maps
      • πŸ“‹CLIP
      • βš•οΈEmbeddings
      • πŸ• VAE
      • πŸ’₯Conditioning
      • 🍁Diffusion sampling/samplers
      • πŸ₯ Prompt
      • πŸ˜„ControlNet
        • πŸͺ‘Settings Explained
        • 🐳ControlNet with models
    • πŸ¦™Large Language Model
      • ☺️SMID
      • πŸ‘¨β€πŸŒΎARM NEON
      • 🍊Metal
      • 🏁BLAS
      • πŸ‰ggml
      • πŸ’»llama.cpp
      • 🎞️Measuring model quality
      • πŸ₯žType for NNC
      • πŸ₯žToken
      • πŸ€Όβ€β™‚οΈDoc Retrieval && QA with LLMs
      • Hallucination(AI)
    • 🐹diffusers
      • πŸ’ͺDeconstruct the Stable Diffusion pipeline
  • 🎹Implementing
    • πŸ‘¨β€πŸ’»diffusers
      • πŸ“–The Annotated Diffusion Model
  • 🧩Trending
    • πŸ“–Trending
      • πŸ“–Vector database
      • 🍎Programming Languages
        • πŸ“–Go & Rust manage their memories
        • πŸ“–Performance of Rust and Python
        • πŸ“–Rust ownership and borrowing
      • πŸ“–Neural Network
        • 🎹Sliding window/convolutional filter
      • Quantum Machine Learning
  • 🎾Courses Collection
    • πŸ“–Courses Collection
      • πŸ“šAcademic In IT
        • πŸ“Reflective Writing
      • πŸ“–UCB
        • πŸ“–CS 61A
          • πŸ“–Computer Science
          • πŸ“–Scheme
          • πŸ“–Python
          • πŸ“–Data Abstraction
          • πŸ“–Object-Oriented Programming
          • πŸ“–Interpreters
          • πŸ“–Streams
      • 🍎MIT Algorithm Courses
        • 0️MIT 18.01
          • 0️Limits and continuity
          • 1️Derivatives
          • 3️Integrals
        • 1️MIT 6.042J
          • πŸ”’Number Theory
          • πŸ“ŠGraph Theory
            • 🌴Graph and Trees
            • 🌲Shortest Paths and Minimum Spanning Trees
        • 2️MIT 6.006
          • Intro and asymptotic notation
          • Sorting and Trees
            • Sorting
            • Trees
          • Hashing
          • Graphs
          • Shortest Paths
          • Dynamic Programming
          • Advanced
        • 3️MIT 6.046J
          • Divide and conquer
          • Dynamic programming
          • Greedy algorithms
          • Graph algorithms
Powered by GitBook
On this page
  • Overview
  • Why we need it?
  • How doe LoRA work on Stable Diffusion models?
  • The cross attention layers
  • Tips
  • LoRA model needs the following phrase in the prompt
  • How to make sure the filename is correct?
  • Adjust the multiplier
  • Adjust for Dreambooth
  • Using multiple models at same time
  • Add the LoRA to the prompt
  • Credit

Was this helpful?

Edit on GitHub
  1. AI techniques
  2. Stable Diffusion

LoRA(Stable Diffusion)

PreviousStable Diffusion workflowNextDepth maps

Last updated 1 year ago

Was this helpful?

Overview

models are small Stable Diffusion models that apply tiny changes to standard checkpoint models. They are usually 10 to 100 times smaller than checkpoint models. That makes them very attractive to people having an extensive collection of models.

Why we need it?

It offers a good trade-off between file size and training power. For example, Dreambooth is powerful but results in large model files (2-7 GBs). are tiny (about 100 KBs), but you can't do as much.

LoRA sits in between: Their files sizes are manageable (2-200 MBs), and the training power is decent.

LoRA is an excellent solution to the storage problem. Like textual inversion, you cannot use a LoRA model alone. It must be used with a model checkpoint file. LoRA modifiles styles by applying small changes to the accompanying model file.

How doe LoRA work on Stable Diffusion models?

LoRA applies small changes to the most critical part of Stable Diffusion models:

The cross attention layers

It is the part of the model where the . According to the pager, it sufficient to fine-tune this part of the model to achieve good training. The are the yellow parts in the Stable Diffusion architecture below.

The weights of a cross-attention layer are arranged in matrices. A LoRA model fine-tunes a model by adding its weights to these matrices.

Tips

LoRA model needs the following phrase in the prompt

<lora:filename:multiplier>

filename

It is the file name of the LoRA model, excluding the extension (.pt, .bin.etc)

multiplier

It is the weight applied to the LoRA model. The defaults is 1. Setting it to 0 disables the model.

How to make sure the filename is correct?

Here we can use the model button

Click on the Lora tab. You should see a list of LoRA models installed. Click on the one you want to use. And the LoRA phrase will be inserted in the prompt.

Adjust the multiplier

You may adjust the multiplier to crank up or tune down the effect. Setting the multiplier 0 disables the LoRA model. You can adjust the style effect between 0 and 1.

Adjust for Dreambooth

Some LoRA models are trained with Dreambooth. You will need to include a trigger keyword to use the LoRA model. You can find the trigger keyword on the model’s page.

Using multiple models at same time

Add the LoRA to the prompt

Credit

Similar to , you can use multiple LoRA models at the same time. You can also use them with embeddings.

In AUTOMATIC1111, the LoRA phrase is not part of the prompt. It will be removed after the LoRA model is applied. That means you cannot use like [keyword1:keyword2: 0.8] with them.

πŸ›€οΈ
🧠
πŸ“™
embeddings
prompt syntax
LoRA(Low-Rank Adaptation)
Textual inversions
cross-attention layers
LogoWhat are LoRA models and how to use them in AUTOMATIC1111 - Stable Diffusion ArtStable Diffusion Art
image and the prompt meet
Source: Stable Diffusion Art
Source: Stable Diffusion Art
LORA fine-tunes the cross-attention layers (the QKV parts of the U-Net noise predictor). (Figure from .)
Stable Diffusion paper