New feature : Low-Rank Pivotal Tuning

[cloneofsimo]

New feature! Pivotal Tuning with LoRA

I have successfully adopted low-rank pivotal tuning, and it seems to perform very well compared to other settings.

Paper of Pivotal Tuning : https://arxiv.org/abs/2106.05744

With learning rate of 1e-4, 2500 steps, here are the results:

"female 3d game character , Skill magic geek inside matrix deepdream radiating a glowing aura stuff loot legends stylized digital illustration video game icon artstation lois van baarle, ilya kuvshinov, rossdraws"

"portrait of female 3d game character , impressionist style from the 19th century, claude monet, oil painting"

Basically, we first perform textual inversion for 1500 steps, then fine-tune LoRA for another 1000 steps.

You can checkout run_lorpt.sh for training script, and scripts/run_lorpt.ipynb in the develop branch.

[cloneofsimo]

By the way, compare the results with non-pivotal tuning 3000 steps-trained LoRA with 1e-4 with same prompt : Notice how reconstructiveness has decreased as it is simply overwhelmed by other prompts.

[cloneofsimo]

Thing to note : I've simply tried to jointly training them (text embedding + LoRAs), but that makes it overfit very easily with entangled features. Extensive comparisons on the tradeoff between training inversion + training adaptation is required here. I think this is the kinda the ultimate form of space of parameters we are going to get : although maybe reasonable next step, we can try multi-token inversion...

[cian0]

hello, sorry maybe a dumb question, so this helps in the flexibility of the concept in regards to promps? I'm trying it but not sure if batch_size affects embedding or lora training, or does it apply to both?

[cloneofsimo]

Batch size will effect the training for sure. Might need shorter/longer training steps than above one.

[cian0]

Thanks! I will try it out

[brian6091]

Oh, there goes my weekend ...

[cloneofsimo]

Going to share this here as well : Wednesday with 7 images, trained for ~3000 steps, (2000 steps were inversion)

Prompts to made them :

img_prompts = [
    "painting of <wday>, a starry night, style of vincent van gogh",
    "portrait of <wday> by mario testino 1950, 1950s style, hair tied in a bun, taken in 1950, detailed face of <wday>, sony a7r",
    "photof of <wday>, 50mm, sharp, muscular, detailed realistic face, hyper realistic, perfect face, intricate, natural light, <wday> underwater photoshoot,collarbones, skin indentation, Alphonse Mucha, Greg Rutkowski",
    "a photo of <wday> in advanced organic armor, biological filigree, detailed symmetric face, flowing hair, neon details, intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, octane, art by Krenz Cushart , Artem Demura, Alphonse Mucha, digital cgi art 8K HDR by Yuanyuan Wang photorealistic",
    "a photo of <wday> on the beach, small waves, detailed symmetric face, beautiful composition",
    "a photo of <wday> rainbow background, wlop, dan mumford, artgerm, liam brazier, peter mohrbacher, jia zhangke, 8 k, raw, featured in artstation, octane render, cinematic, elegant, intricate, 8 k",
    "photo of Summoner <wday> with a cute water elemental, fantasy illustration, detailed face, intricate, elegant, highly detailed, digital painting, artstation, concept art, wallpaper, smooth, sharp focus, illustration, art by artgerm and greg rutkowski",
    "<wday>, cyberpunk 2077, 4K, 3d render in unreal engine",
    "a pencil sketch of <wday>",
    "a minecraft render of <wday>",
    "young woman <wday>, eden, intense eyes, tears running down, crying, vaporwave aesthetic, synthwave, colorful, psychedelic, crown, long gown, flowers, bees, butterflies, ribbons, ornate, intricate, digital painting, artstation, concept art, smooth, sharp focus, illustration of <wday>, art by artgerm and greg rutkowski and alphonse mucha",
]

[cian0]

Hope you can paste the whole command including batch size, steps etc. I'm having trouble getting to mix the right config with 3090, i wanted to utilize the whole 24gb vram so i increase the batch size and decrease the steps but it seems it's better to leave it at batch size 1 with higher steps and not utilize max vram?

[cloneofsimo]

Increase batch, but you'll need to increase the learning rate as well.

[cloneofsimo]

Also know as Linear scaling rule : http://proceedings.mlr.press/v119/smith20a/smith20a-supp.pdf

[Boldor83]

can i use this in automatic1111?