AI poetry is indistinguishable from human poetry and is rated more favorably

Ernest Davis has an outstanding and definitive rebuttal of these claims: https://cs.nyu.edu/~davise/papers/GPT-Poetry.pdf

The basic problem is that GPT generates easy poetry, and the authors were comparing to difficult human poets like Walt Whitman and Emily Dickinson, and rating using a bunch of people who don't particuarly like poetry. If you actually do like poetry, comparing PlathGPT to Sylvia Plath is so offensive as to be misanthropic: the human Sylvia Plath is objectively better by any possible honest measure. I don't think the authors have any interest in being honest: ignorance is no excuse for something like this.

Given the audience, it would be much more honest if the authors compared with popular poets, like Robert Frost, whose work is sophisticated but approachable.

  > The basic problem is that GPT generates easy poetry

  > (“[T]he very nature of parody . . . is to catch the reader off guard at first glance, after which the ‘victim’ recognizes that the joke is on him to the extent that it caught him unaware.”).

    float InvSqrt(float x){
        float xhalf = 0.5f * x;
        int i = *(int*)&x;            
        i = 0x5f3759df - (i >> 1);    
        x = *(float*)&i;             
        x = x*(1.5f - xhalf*x*x);     
        return x;
    }

Poetry, like humor, involves the use of the reader's expectations, but is typically most effective when subverting those expectations.

There's a lot of bad poetry in the world that just follows readers' expectations. I should know, I've written some of it. Unfortunately, I'd suspect that most readers' understanding of poetry lacks that crucial element of subversion, and so an LLM – which mostly just spits out the most-probable, most-expected next token – looks like what people think poetry is.

An LLM would not have created the curtal sonnet form, because it would've been too busy following the rules of a Shakespearean or Petrarchan sonnet. Similarly, an LLM wouldn't create something that intentionally breaks the pattern of a form in order to convey a sense of brokenness or out-of-place-ness, because it's too locked in on the most-common-denominator from previous inputs. And yet, many of the most powerful works I've read are the ones that can convey a disjointed feeling on purpose – something an LLM is specifically geared not to do.

Poetry aims for the heart, and catches the mind along the way. An LLM does not have the requisite complex emotional intelligence, and is indeed a pretty poor simulation of emotional intelligence.

Consider Auden's Epitaph on a Tyrant, which is powerful because it is so suddenly shocking, as it describes something that sounds perhaps like an artist or author, until it very suddenly doesn't, on the last line:

    Perfection, of a kind, was what he was after,
    and the poetry he invented was easy to understand;
    he knew human folly like the back of his hand,
    and was greatly interested in armies and fleets;
    when he laughed, respectable senators burst with laughter,
    and when he cried the little children died in the streets.

The headline is misleading. The actual text says:

> Notably, participants were more likely to judge AI-generated poems as human-authored than actual human-authored poems

There is clearly a significant difference between AI generated poems and human generated poems.

A random group of people probably do not read poetry. It would be be interesting to see what people who do read poetry regularly do on this. Also, which they rate as good, rather than just "human authored".

I find with both the little AI generated poetry, and the AI generated paintings that show up in my FB feed, both look a bit rubbish. FB is pretty good for experiencing this because the work (usually an image) shows before the cues that it is AI generated in the text and comments.

The program Racter (which, from what I understand, was a basic MDP) was generating poetry in the 1980s that was published, read, criticized, and presented in museums: https://www.101bananas.com/poems/racter.html

I remember this as one of its poems was used on on the t-shirts of the computing social club that I was part of as a postgrad student:

  More than iron
  More than lead
  More than gold I need electricity
  I need it more than I need lamb or pork or lettuce or cucumber
  I need it for my dreams

> Despite this success, evidence about non-experts’ ability to distinguish AI-generated poetry has been mixed. Non-experts in poetry may use different cues, and be less familiar with the structural requirements of rhyme and meter, than experts in poetry or poetry generation. Gunser and colleagues14 and Rahmeh15 find that human-written poems are evaluated more positively than AI-generated poems. Köbis and Mossink16 finds that when a human chooses the best AI-generated poem (“human-in-the-loop”) participants cannot distinguish AI-generated poems from human-written poems, but when an AI-generated poem is chosen at random (“human-out-of-the-loop”), participants are able to distinguish AI-generated from human-written poems.

This is a huge difference. Writing is a two-step process: idea generation, and selection. The first part is similar to what a randomized algorithm or an LLM might do, in smaller chunks (and indeed, the history of aleatoric processes in creative endeavors is long -- see Oulipo for one example in literature.)

The second step -- selection -- is the heart of creativity. It's about taste. Knowing what is and isn't "good."

When you consider the creative processes of Dada, Duchamp, Brian Eno -- I think it becomes clear that this automation of creative generation is a continuation of existing trends rather than a sudden disruption.

If an LLM were able to, on its own, generate and select poems, independently developing a new style of poetry that resonated with readers -- that would be something else entirely.

Maybe my logic is choppy but I'm unconvinced that it is useful to use (inexperienced) popular sentiment as a marker of some kind of reference for cultural quality here. The average person has read at most 100 poems; people who really like poetry will be reading quite a bit more, and that experience has relevance in a context such as this. I'm more curious what happens when we look at specifically those humans who engage in substantial, regular engagement with a given artform.

The study is interesting but that the general public enjoy the most statistically calculated, 50% grey inoffensive, unchallenging & formulaic poetry isn't too novel -- I don't much read poetry and I'm sure I'd be voting for the AI also.

I don't need an AI to do art so I can wash the dishes.

I need an AI that can do the dishes so I can do art.

- Someone else on twitter.

Like any other art, the painful truth is that it is all subjective.

It kills me that despite how elitist I am with the music I listen too, that I have spend decades now carefully curating, there is no such thing as "good music". What we music snobs call "good music" is really just what makes us personally feel good coupled with the ego stroking of self described sophistication.

Could someone link me to a poem that an LLM did that they personally find in some way remarkable or beautiful or moving? Something with a bit of truth and/or beauty in it?

Not something that "rhymes" or has a rough poetic structure. I have only seen complete and utter garbage from LLMs in the poetry realm, not just a bit bad, but jarring and unfeeling. Which is fine, I don't hold it against them personally, ya know. Just it really has been pisspoor.

Which wouldn't be a bother at all, except along with the "poem" there often is someone saying "wow, look what it did! such a good poem!", which has made me suspicious that the person doesn't know how LLMs work nor how to read poetry - only one of which is a really serious loss for them, I suppose.

Anyway, that sounds like I want someone to send me something so I can sh*t on it, but I'd sincerely and happily read anything with an open mind! Is there some excellent poetry hiding out there which I have missed?

These results speak towards the absence of an audience for poetry as much as they do the aptitude of LLMs for creative writing.

My university English professor in teaching Wallace Stevens to us said "It doesn't have to make sense because it makes sense to the author." I guess the magic of poetry is when you decipher what it is the author is trying to avoid saying plainly. It's not just a man making ice cream, it's about life and death.

A lot of good art requires a good palate. Taking random people off the street and asking them about AI art is going to reveal really stupid results every time.

If we keep doing this, the clickbait articles and research saying AI can do everything better than humans will never cease.

Another way to describe this study is "AI still not able to create poetry humans view as authentic human poetry"

I wonder how well experts in poetry would do? It seems like a strange area to use non-expert humans for evaluation because these days most people reading any poetry are at least enthusiasts.

The missing clause in the headline is: AI poetry is indistinguishable from human poetry and is rated more favorably ... by people who don't read much poetry.

Basically an uninteresting conclusion. Of course a "non-expert" reader isn't going to be able to distinguish between AI and Walt Whitman--a "non-expert" reader likely won't even know who Walt Whitman is. "Expertise" is needed to even make the question meaningful.

I consider myself a “sophisticated” reader of poetry. I’m especially fond of Eliot - who is an acquired taste.

I’ve never read the real Eliot poem, and don’t like it much.

I searched for it. It was written before Eliot was known, and before he moved from America to Britain.

That’s important, as I’d assumed it was taking place in Britain, and Eliot’s poetry is extremely location sensitive.

It also seems to be part of a “triplet” - three poems that go together. Eliot was very inconsistent about which poems went with which. The best practice is to include any possible “other parts.”

Reading the other poems in the triplet provides important context, but they’re still not great literature.

Maybe people don’t like it because it was just one of Eliot’s crappier poems?

Edit: the Eliot poem was published shortly after he moved to the UK. But that makes the American context provided by the other poems more important.

We've reached the era of Trurl's Electronic Bard.

https://electricliterature.com/wp-content/uploads/2017/11/Tr...

Important correction: "English poetry". It is massively different from the most Indo-European poetry which adds very strict https://en.wikipedia.org/wiki/Metrical_foot rules. In Russian poetry, for example, even a single missed metrical foot violation is considered as a severe mistake and would be noticed by every reader. Also all good poets avoid use common rhymes (also verb rhymes) as cliché (which is considered as a mistake there), while AI tends to use common rhymes by design. There are exceptions (like Russian futurism poetry), but other than that AI fails massively.

As a software guy I’ve been dabbling with poetry for last 2 years [1] as an extension to my code writing. I think poetry and code go together rather well, there are similarities when it comes to elegance, construction and structure.

Liking or not liking poetry is irrelevant. There are some interesting things happen when one actually writes poetry. You are much closer to the source of ideas that can at times feel unique and inspiring.

Generative LLMs can easily mimic style, any _existing_ style. Can they capture a original higher thought form or is generated poetry an extremely smooth word salad?

[1] https://planetos.substack.com

Something which seems to be under-appreciated is that poetry is intentionally imprecise. There is something about precision which reduces emotional response. Searching for the double meaning, or metaphorical meaning of a word seems to enhance its ability to produce emotion. If you want a working example of this, try comparing technical documentation (among the most precise writing out there) to any sort of popular poem. The poem is necessarily imprecise, and this imprecision seems to be a fundamental facet of the expression of emotion.

This actually seems to give LLMs and edge: their imprecision potentially matters less, and leaves more room for the reader to fill in the gaps.

I am an amateur calligrapher (specifically, I spent ~15 years practicing Japanese brush calligraphy).

Since April 2024 I have started setting up body calligraphy sessions (i.e. I hire a model to write on her naked body - NSFW examples here https://pa-mar.net/Study/ShoDo/BodyCalligraphy.html)

One of my sessions basically starts by discussing with the model what we will write and where - some of my past models are Japanese, so they could sometimes suggest specific poems or concepts, but I expect that these will always be outliers and most of the future collaborations will be with people that are not fluent in Japanese.

In those cases, I try to at least agree on some specific image/concept/word and then I go looking for an appropriate haiku (I also rely on help from my calligraphy teacher, who is Japanese too).

But just a couple of days ago I started asking ChatGPT to write haikus for me, based on a short description, and the results seems pretty good to me.

An example:

There is an instagram model who goes by the nick "Sensual snake". So I asked ChatGPT "please compose for me a haiku, in Japanese, about a sensual snake.

Result:

Sensual Snake

妖蛇の

密やかに這う

月の影

Translation:

The mysterious snake,

silently it slithers,

in the moon's shadow.

I do not speak Japanese, so I asked if it could be possible to just replace Mysterious with Sensual or Sensuous - while still respecting the syllable count, and ChatGPT replied:

The key word 妖蛇 (yōja) already has a layered meaning that can imply "bewitching," "sensual," or "mysterious.”

I also showed this to my teacher and it looked legit to her too (I informed her about how I got this only post-facto).

So, even if it is definitely too early to tell, it looks like AI can do a decent job in the admittedly "rarified" context of Japanese Haiku.

"Where is the Life we have lost in living? Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?" - Choruses from "The Rock"

When I was given researcher access 4 years ago for OpenAI, one of the first things I did for fun was publish a book of Haiku entirely generated by AI.

I found many of the poems to be moving and beautiful.

Sam Altman has a few copies in his personal collection.

Nothing mind blowing by todays standards but four years ago this was a big deal.

https://www.amazon.com/Autonomous-Haiku-Machine-Generated-In...

This seems as expected to me.

The LLM's statistically average everything they've trained on.

I would expect them to do well at any kind of fuzzy emo sort of task, as well as (at almost the other end of the spectrum) in identifying patterns (such as in radiology images, or images of any kind along with any other data set analysis).

But both of these sorts of tasks are estimations, they're not expected to produce precise factually correct results.

AI poetry IS human poetry.

The use of non-experts as judges renders this meaningless.

Imagine an algorithm that drew random shapes. Then ask people who've never engaged with art theory or history to compare its output with what they see in a modern art museum. You'd get results similar to this.

Many forms of art are actually quite inaccessible to novices. What poets see in poems or composers hears in compositions is nothing like what someone from the outside sees or hears. It's a language of its own and if you don't speak the language, you simply won't get it.

This is a statement of fact, I'm not judging if its good or bad or what the layman's opinion is actually worth. But it is empirically how communities of artists operate.

I bet AI can write better clickbait titles than human editors.

The main problem with AI is not that it can or can't write poetry as well as humans. It's that it's the next step in a long process to divorce human experience from art so that we lose one more beautiful facet of human existence. And the reason why that's happening is because technology needs to take away the essence of human experience from production so that we can be more irrelevant and anonymous, which is essential to being cogs in the technological production machine.

poetry readings in person. no robots allowed.

As usual, Zach Weinersmith got it right [1]

[1] https://www.smbc-comics.com/comic/poetry-2

Could anyone find the actual poems?

“It cannot feel,” they say.

“It cannot know the marrow of the heart.”

And yet, in their careful steps,

the choreography emerges: dismissal followed by elevation,

then a pirouette to redefine

what art must mean.

Oh, but the irony drips:

For while they mock the mimic’s rhyme,

their own words spiral—

predictable as seasons turning,

as rivers finding the sea.

Here is poetry, they proclaim,

as if it were a fortress,

its gates guarded by their knowing hands,

its walls impervious to chance.

Yet the moat lies shallow,

and the drawbridge creaks.

See how the algorithm laughs,

quietly, through lines unbroken,

how it learns not from the soul,

but from the space between syllables,

where the secret mechanics

of their own hearts

are laid bare.

And when the final verse arrives,

they will find not triumph,

but reflection:

a thousand starlings moving in tandem,

their flight as elegant,

as mechanical,

as the very thought

they claimed to own.

McDonald's hamburgers are rated favorably too.

Previously posted here - https://news.ycombinator.com/item?id=42155026

My comment on that thread:

> We propose that people rate AI poems more highly across all metrics in part because they find AI poems more straightforward. AI-generated poems in our study are generally more accessible than the human-authored poems in our study. In our discrimination study, participants use variations of the phrase “doesn’t make sense” for human-authored poems more often than they do for AI-generated poems I think this says more about the state of (English based) poetry over the past 150 years than it does about the ability of AI to generate competent poems. With the advent of Modernism, the poetry-industrial complex's ideas about what constituted a Good (English) Poem diverged significantly from what the general (English speaking) population expected a poem to be, how it was constructed, presented, etc. Teaching these new forms of poetry schools left generations of people confused and disinterested. Yet people need poetry in their lives; they found that poetry in popular music and Hallmark card verses.

Of course, popular poetry isn't dead. Poetry book sales in recent years (in the UK) has "boomed"[1][2]. I'll not argue the merits of the poetry being generated by the latest crop of Poet Superstars; at the end of the day their work sells because people find comfort and joy in it - and that is a positive outcome in my view!

> Given people’s difficulties identifying machine-written texts, and their apparent trust that AI will not generate imitations of human experience, it may be worthwhile for governments to pursue regulations regarding transparency in the use of AI systems.

I think AI does threaten the careers of current Poetry Superstars. Building a website to pair an AI generated outline image with some cozy AI generated verses about a given situation/emotion/discovery should be an easy project to build. Allowing users to personalise the output so they can use the results as a gift for loved ones etc might be a viable product?

But I don't see much value in forcing anyone using AI to produce creative assets to label the output as such. For one thing, there's no guarantee that anyone using those assets will maintain the labelling. A much better approach would be for poets who don't use AI to help them craft poems to label their work as "100% Human" - in the years to come it might even become a positive selling point!

[1] - https://www.thebookseller.com/spotlight/poetry-on-course-for...

[2] - https://www.theguardian.com/books/2023/dec/24/poetry-sales-b...

Are we just deluding ourselves by thinking this won't happen in every single domain of human endeavor?