🐳ControlNet with models

Using Preview with the ControlNet plugin

The first step is to choose a preprocessor and it is helpful to turn on the preview so we know what the preprocessor is doing. Once the preprocessing is done, the original image is discarded, and only the preprocessed image will be used for ControlNet.

Reduce the Control Weight will help with color issues or other artifacts

Preprocessor and models

Once we choose a preprocessor, we must pick the correct model. We need to do is select the model with the same starting keyword as the preprocessor.

OpenPose preprocessors

• OpenPose: eyes, noses, neck, shoulder, elbow, wrist, knees, and ankles

• OpenPose_face: Openpose +facial detail

• OpenPose_hand:OpenPose+hands and fingers

• OpenPose_faceonly: facial details only

• OpenPose_full: All of the above

Tile resample model

The Tile resample model is used for adding details to an image. It is often used with an upscaler to enlarge an image simultaneously.

Reference preprocessors

Reference is a new set of preprocessors that let us generate images similar to the reference image. The Stable Diffusion model and the prompt will still influence the images.

Reference preprocessors do not use a control model. We only need to select the preprocessor but not the model. And there are 3 reference preprocessors:

• Reference adain

  • Style transfer via Adaptive Instance Normalization
• Reference only

  • Link the reference image directly to the attention layers

• Reference adain+attn

  • Combination of above

Canny edge detector(preprocessor+model)

Canny edge detector is a general-purpose, old-school edge detector.

It extracts the outlines of an image. It is useful for retaining the composition of the original image.

Depth preprocessor

The depth preprocessor guesses the depth information from the reference image.

• Depth Midas

  • A classic depth estimator

• Depth Leres

  • More details but also tend to render background

• Depth Leres++

  • Even more details

• Zoe:

  • The level of detail sits between Midas and Leres.

Line Art preprocessors

Line Art renders the outline of an image. It attempts to convert it to a sample drawing. The are a few line art preprocessors

• Line art anime: Anime-style lines

• Line art anime denoise: Anime-style lines with fewer details

• Line art realistic: Realistic-style lines

• Line art coarse: Realistic-style lines with heavier weight

ControlNet Inpainting

ControlNet inpainting lets you use high denoising strength in inpainting to generate large variations without sacrificing consistency with the picture.

Credit

ControlNet v1.1: A complete guide - Stable Diffusion ArtStable Diffusion Art