▲ 340 points
back
171 comments
It's often said porn drives technology.
I clicked through the links in the article, since they sounded technically interesting. They led to AI-generated porn. Those, in turn, led to pages about training SD to generate porn. Now, two disclaimers:
1) I am not interested in AI-generating porn
2) I haven't followed SD in maybe 6-9 months
With those out-of-the-way, the out-of-the-box tools for fine-tuning SD are impressive, well beyond anything I've seen in the non-porn space, and the progress seems to be entirely driven by the anime porn community:
https://aituts.com/stable-diffusion-lora
10 images is enough to fine-tune. 30-150 is preferred. This takes 15-240 minutes, depending on GPU. I do occasionally use SD for work. If this works for images other than naked and cartoon women, and for normal business graphics, this may dramatically increase the utility of SD in my workflows (at least if I get around to setting it up).
I want my images to have a consistent style. If I'm making icons, I'd like to fine-tune on my baseline icon set. If I'm making slides for a deck, I'd like those to have a consistent color scheme and visual language. Now I can.
Thanks creepy porn dudes!
The other piece: Anyone trying to keep the cat in the bag? It's too late.
> the progress seems to be entirely driven by the anime porn community:
Its not entirely driven by porn communities, and the porn communities driving it aren’t entirely anime porn communities (and the anime communities driving it aren’t entirely porn communities.)
But, yeah, the anime + porn/fetish art + furry + rpg art + scifi/fantasy art communities, and particularly the niches in the overlap of two or more of those are, pretty significant.
> If this works for images other than naked and cartoon women
It does, and while it may not be large proportionally compared to the anime-porn stuff, there’s a lot of publicly distributed fine tuned checkpoints, LoRas, etc., demonstrating that it does.
It absolutely works for things other than naked and cartoon women. Here are some generations of my daughter and dog (together!). I believe most of these are from a fine tuned model of them and not an extracted LoRA, though I use that sometimes too: https://imgur.com/a/naHgnel
The official blog post from Stability is finally up and would probably be a better URL to link to than the TechCrunch coverage: https://stability.ai/blog/stable-diffusion-sdxl-1-announceme...
It’ll be "released" once the model weights show up on the repo or in HuggingFace… for now it’s "announced"
It should appear here at some point, currently only the VAE was added:
The release event is in like ~30 minutes on their discord, probably the announcement went out a bit early.
It does appear to be live on Clipdrop.
You get access to the weights instantly if you apply for them. It's basically not a hurdle.
(I've been having fun with this for a few days. https://huggingface.co/stabilityai/stable-diffusion-xl-base-... Not sure there's much of a difference with the 1.0 version.)
It sounds like after the previous 0.9 version there was some refining done:
> The refining process has produced a model that generates more vibrant and accurate colors, with better contrast, lighting, and shadows than its predecessor. The imaging process is also streamlined to deliver quicker results, yielding full 1-megapixel (1024x1024) resolution images in seconds in multiple aspect ratios.
Sounds pretty impressive, and the sample results at the bottom of the page are visually excellent.
They have bots in their discord for generating images bases on user prompts. Those randomize some settings, compare candidate models and are used for rlhf fine-tuning and that's the main source of refining which will continue even after release.
There were, IIRC, three different post-0.9 candidate models in parallel testing to become 1.0 recently.
I always wondered why the vision models don't seem to be following the whole "scale up as much as possible" mantra that has defined the language models of the past few years (to the same extent). Even 3.5 billion parameters is absolutely nothing compared to the likes of GPT-3, 3.5, 4, or even the larger open-source language models (e.g. LLaMA-65B). Is it just an engineering challenge that no one has stepped up for yet? Is it a matter of finding enough training data for the scaling up to make sense?
Diffusion is more parameter-efficient and you quickly saturate the target fidelity, especially with some refiner cascade. It's a solved problem. You do not need more than maybe 4B total. Images are far more redundant than text.
In fact, most interesting papers since Imagen show that you get more mileage out of scaling the text encoder part, which is, of course, a Transformer. This is what drives accuracy, text rendering, compositionality, parsing edge cases. In SD 1.5 the text encoder part (CLIP ViT-L/14) takes a measly 123M parameters.[1] In Imagen, it was T5-XXL with 4.6B [2]. I am interested in someone trying to use a really strong encoder baseline – maybe from a UL2-20B – to push this tactic further.
Seeing as you can throw out diffusion altogether and synthesize images with transformers [3], there is no reason to prioritize the diffusion part as such.
1. https://forums.fast.ai/t/stable-diffusion-parameter-budget-a...
They often reference this paper as the motivation for that https://arxiv.org/pdf/2203.15556.pdf I.e. training with 10x data and 10x longer can yield as good models as a gpt-3 model but with fewer weights (according to the paper) and the same principle applies in vision.
Diffusion is relatively compute intensive compared to transformers llms, and (in current implementation) doesn't quantize as well.
A 70B parameter model would be very slow and vram hungry, hence very expensive to run.
Also, image generation is more reliant on tooling surrounding the models than pure text prompting. I dont think even a 300B model would get things quite right through text prompting alone.
Do we know the amount of parameters Dall-e have these days, Firefly or Midjourney, etc?
If we are talking about Stable Diffusion, the reality is that... more parameters mean it will be hard to run locally. And let me tell you something, the community around Stable Diffusion only cares with NSFW... And want local for that...
Stable Diffusion 2 was totally boycotted by the community because they... banned NSFW from there. They had now to allow it again on SDXL.
Also, more parameters mean it will be more expensive to community finetunners to train as well.
I'm out of date on the image-generating side of AI, but I'd like to check things out. What's the best tool for image generation that's available on a website right now? Ie, not a model that I have to run locally.
If you want to play around with Stable Diffusion XL: https://clipdrop.co
https://playgroundai.com/create
Not affiliated in anyway and not very involved in the space. I just wanted to generate some images a few weeks ago and was looking for somewhere I could do that for free. The link above lets you do that but I suggest you look up prompts because its a lot more involved than I expected.
Midjourney right? Although, discord isn't a website I guess.
I've found https://firefly.adobe.com/ pretty good at composing images with multiple subjects. [disclaimer - I work at Adobe, but not in the Creative Cloud]
But I wouldn't say it's the "best." Just trained on images that weren't taken from unconsenting artists.
There are toy AI things, but there is nothing quite like Stable Diffusion running on Colab. Lots of people recommended Midjourney but that is like playing with MSpaint. If you can get Stable Diffusion going with Automatic1111, its AAA tier. Especially with Control-net, and dreambooth, but that is part 2.
Google: The Last Ben Stable Diffusion Colab
for a way to not run it locally, but get all the features.
Probably Midjourney, but I like Dreamstudio better.
Is there anything like this for the vector landscape?
This may just be due to the iterative denoising approach a lot of these models take but they only seem to work well when creating raster style images.
In my experience when you ask them to create logos, shirt designs, illustrations, they tend to not work as well and introduce a lot of artifacts, distortions, incorrect spellings etc.
If you mean raster images that look like vector and contain arbitrary text and shapes, controlnets/T2I adapters do work for this. You could train your custom controlnet for this, too. (it requires understanding)
As for directly generating vector images, there's nothing yet. Your best bet is generating vector-looking raster and tracing it.
Midjourney is still going to be hard to beat imo. Comparing SD to MJ is a little unfair considering their applications and flexibility, but I do really enjoy the "out of the box" experience that comes with MJ.
I tried it in dreamstudio. Like all the other image generators I've tried, it's rubbish at drawing a piano keyboard or an accordion. (Those are my tests to see if it understands the geometry of machines.)
A couple of accordion pictures do look passable at a distance.
Another test: how well does it do at drawing a woman waving a flag?
One thing that strikes me is that it generates four images at a time, but there is little variety. It's a similar looking woman wearing a similar color and style of clothing, a similar street, and a large American flag. (In one case drawn wrong.) I guess if you want variety you have to specify it yourself?
AI models seem to be getting ever better in resolution and at portraits.
I hope someday there’s a version of this or something comparable to it that can run on <8gb consumer hardware. The main selling point of Stable Diffusion was its ability to run in that environment.
Let's see wether derived models will suffer less from the 'same face actor'-model response to every portrait prompt. It's not trivial to get photoreal models not lookalike without resorting to specific, typically celeb based, finetunes.
I am completely uninformed in this space.
Would someone be kind to explain what the current state of the art in image generation is (how does this compare to Midjourney and others)?
How do open source models stack up?
Also what are the most common use cases for image generation?
I will wait for the automatic1111 web ui version
Is this pre-censored like their other later models?
I thought this release had been announced already? Or was that not 1.0? Could have sworn they released an "XL" variant a little while ago?
In the meantime I've been getting good mileage out of Kandinsky - anyone got a good sense of how they compare?
Amazing that their examples at the bottom of the page still show really messed up human hands.
Stability AI is awesome I love them
Can SD draw hands finally?
Still can't draw hands correctly it looks like.
Not actually released in the API unlike they said.
This explosion of AI-generated imagery will result in an explosion of millions of fake images, obivously. Perhaps in the short-term this is fun, but in the long-term, we will lose a bit more scarcity, which is not that great in my opinion.
Isn't the best part of a meal eating after you've not had anything to eat for a while? The best part about a kiss that you've quenched the pain of missing your partner?
The best part of art is that you haven't seen anything good in a while?
Scarcity is an underappreciated gift to us, and the relative scarcity per capita is in a sense what drives us to connect with other people, so that we may be priveleged to witness the occasional spark of creativity from a person, which in turn tells us about that person.
Although that sort of viewpoint has been declining for some time due to the intensely capitalistic squeezing of every sort of human endeavor, AI brings this to a whole new level.
I think if those making this software thought a bit about this, they might second-guess whether it is truly right to release it. Just a thought.