# ControlNet with models

## Using Preview with the ControlNet plugin

The first step is to choose a preprocessor and it is helpful to turn on the <mark style="color:red;">**preview**</mark> so we know what the preprocessor is doing. Once the preprocessing is done, the original image is discarded, and only the preprocessed image will be used for ControlNet.

{% hint style="info" %}
Reduce the Control Weight will help with color issues or other artifacts
{% endhint %}

### Preprocessor and models

Once we choose a preprocessor, we must pick the correct model. <mark style="color:red;">**We need to do is select the model with the same starting keyword as the preprocessor.**</mark>

### OpenPose preprocessors

* **OpenPose**: eyes, noses, neck, shoulder, elbow, wrist, knees, and ankles
* **OpenPose\_face**: Openpose +facial detail
* **OpenPose\_hand**:OpenPose+hands and fingers
* **OpenPose\_faceonly**: facial details only
* **OpenPose\_full**: All of the above

### Tile resample model

The *<mark style="color:red;">**Tile**</mark>* resample model is used for adding details to an image. It is often used [<mark style="color:red;">**with an upscaler**</mark>](https://github.com/Coyote-A/ultimate-upscale-for-automatic1111) to enlarge an image simultaneously.

### Reference preprocessors

*<mark style="color:blue;">**Reference is a new set of preprocessors**</mark>* that let us *<mark style="color:purple;">**generate images similar to the reference image**</mark>*. The Stable Diffusion model and the prompt will still influence the images.

<mark style="color:red;">**Reference preprocessors do not use a control model. We only need to select the preprocessor but not the model. And there are 3 reference preprocessors:**</mark>

* Reference adain
  * Style transfer via [Adaptive Instance Normalization](https://arxiv.org/abs/1703.06868)
* Reference only
  * Link the reference image directly to the attention layers
* Reference adain+attn
  * Combination of above

### Canny edge detector(preprocessor+model)

[Canny edge detector](https://en.wikipedia.org/wiki/Canny_edge_detector) is a general-purpose, old-school edge detector.

It extracts the outlines of an image. It is useful for retaining the composition of the original image.

### Depth preprocessor

The depth preprocessor guesses the depth information from the reference image.

* **Depth Midas**
  * A classic depth estimator
* **Depth Leres**
  * More details but also tend to render background
* **Depth Leres++**
  * Even more details
* **Zoe**:
  * The level of detail sits between Midas and Leres.

### Line Art preprocessors

Line Art renders the outline of an image. It attempts to convert it to a sample drawing. The are a few line art preprocessors

* **Line art anime**: Anime-style lines
* **Line art anime denoise**: Anime-style lines with fewer details
* **Line art realistic**: Realistic-style lines
* **Line art coarse**: Realistic-style lines with heavier weight

### ControlNet Inpainting

ControlNet inpainting lets you use high denoising strength in inpainting to generate large variations without sacrificing consistency with the picture.

## Credit

{% embed url="<https://stable-diffusion-art.com/controlnet/#Canny>" %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://aisuko.gitbook.io/wiki/ai-techniques/stable-diffusion/controlnet/controlnet-with-models.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.