Answered step by step
Verified Expert Solution
Link Copied!

Question

1 Approved Answer

1 . Finetuning with LoRA adapters on custom dataset . LoRA adapters can be applied for finetuning large diffusion models, trained on diverse mix of

1. Finetuning with LoRA adapters on custom dataset .
LoRA adapters can be applied for finetuning large diffusion models, trained on diverse mix of data to improve image generation quality or text-image alignment on a specific domain.
Your task is to take pretrained Stable Diffusion model (or any other model at your preference) and finetune in on one of the two tasks listed below:
lambdalabs/naruto-blip-captions
Your notebook should contain the entire dataset preprocessing and training pipeline.
You may use any functionality from the diffusers, datasets, transformers, peft and accelerate libraries. You may adopt pieces of code, but not the entire training_script.
Use of the models hosted on the hub and trained on this dataset is strictly prohibited!
Data loading and visualization
Load the dataset and visualize some samples from it. How many samples are there? Split the dataset into train/test set in proportion 75%/25% at random. Fix all seeds to 42(torch, numpy, random).
LoRA preparation
Define LoRA config using the peft library. Which layers can you apply LoRA to?
Training
Train the model and save the adapter of the final model. Since there is no reasonable quantative metric for the quality of images, visualize 20 samples generated on the prompts from the hold-out set.
LoRA ablation
Experiment with different ranks of LoRA adapters. How does the quality / aligment to the target domain depend on the rank of adapter? Try 3-5 options for LoRA rank that yield noticeable difference in performance. Apply LoRA to different layers. What layers are better for LoRA application?
Impact of CFG scale
For a single model and a 2-3 prompts generate images with different classifier-free guidance scale (from 1 to some large value, like 20). What do you observe?

Step by Step Solution

There are 3 Steps involved in it

Step: 1

blur-text-image

Get Instant Access to Expert-Tailored Solutions

See step-by-step solutions with expert insights and AI powered tools for academic success

Step: 2

blur-text-image

Step: 3

blur-text-image

Ace Your Homework with AI

Get the answers you need in no time with our AI-driven, step-by-step assistance

Get Started

Recommended Textbook for

Microsoft Visual Basic 2008 Comprehensive Concepts And Techniques

Authors: Gary B. Shelly, Corinne Hoisington

1st Edition

1423927168, 978-1423927167

More Books

Students also viewed these Databases questions