Famous first lines with VQGan + CLIP

Can we learn something about a novel by generating images?

Text-to-Image generation is a curious field that, aside from producing a vast array of monstrosities, can also provide some insight into the visual conceptual embeddings inherent in social language. We can even exploit that social linguistic programming to reproduce some conceptual information that is associated with entities even when they aren’t fully contained in the input prompt.

To explore this idea, I’ve generated a list of famous opening lines from novels, ideas about which might be embedded in in the language corpus used to trained text-to-visual representation learning systems—after all, how many essays have posted online about 1984 that analyse the sentence “It was a bright cold day in April and the clocks were striking thirteen”. How many of those have some relationship to visual concepts that we would link to the book’s themes of state control, linguistic determinism, and the abscence of free will?

My intuition was that for the most famous of these cultural entities, a naive text-to-image generation process based on first lines alone should demonstrate at least some qualities we associate with the novel. Thankfully, adapting existing code for such a system is pretty straightforward since there are plenty of implementations online.

Check out:

The solution I adapted for local generation was initially created by Katherine Crowson. You can check out the original notebook in Colab:

Open In Colab

My solution, which you can easily adapt to any set of prompts by switching out the input CSV, can be found at ryangoslingsbugle/VQGan_CLIP.


Herzog Voyage of the Dawn Treader The Crow Road Neuromancer



CLIP is a supervised natural language learning model trained to classify images and optimised for maximum transfer potential across benchmark datasets. The idea is that not training for best performance on a single dataset will produce more robust outcomes, better suited to out-of-training-domain flexibility.

OpenAI have adopted a simple approach to producing transferable visual models with their CLIP network. This system follows the general process of masked language model learning, learning to predict which caption applies to which image, essentially producing a kind of visual concept embedding.

Read more in Radford, et al. (2021).

      title={Learning Transferable Visual Models From Natural Language Supervision}, 
      author={Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever},


VQGan is a combination of a convolutional representation generator with an auto-regressive transformer for reconstruction. This setup achieves extremely good results in image reconstruction tasks, and loading pre-trained models makes working with with it quite straightforward.

Read more in Esser, et al. (2020).

      title={Taming Transformers for High-Resolution Image Synthesis}, 
      author={Patrick Esser and Robin Rombach and Björn Ommer},


Below you can see the full set of novels chosen, the input text used, and the image generated after the 1000th training epoch.

Title Text Image
A Tale of Two Cities “It was the best of times, it was the worst of times”
1984 It was a bright cold day in April and the clocks were striking thirteen.
Fahrenheit 451 It was a pleasure to burn.
Anna Karenina Happy families are all alike; every unhappy family is unhappy in its own way.
Pride and Prejudice It is a truth universally acknowledged that a single man in possession of a good fortune must be in want of a wife.
Moby Dick Call me Ishmael.
The Hobbit In a hole in the ground there lived a hobbit.
Slaughterhouse Five “All this happened, more or less.”
The Metamorphosis As Gregor Samsa awoke one morning from uneasy dreams he found himself transformed in his bed into a gigantic insect.
Voyage of the Dawn Treader “There was a boy called Eustace Clarence Scrubb, and he almost deserved it.”
The Great Gatsby In my younger and more vulnerable years my father gave me some advice
The Go-Between The past is a foreign country; they do things differently there.
The Crow Road It was the day my grandmother exploded.
The Bell Jar “It was a queer, sultry summer, the summer they electrocuted the Rosenbergs, and I didn’t know what I was doing in New York.”
To Kill a Mockingbird “When he was nearly thirteen, my brother Jem got his arm badly broken at the elbow.”
Invisible Man I am an invisible man.
One Hundred Years of Solitude Colonel Aureliano Buendía was to remember that distant afternoon when his father took him to discover ice.
Don Quixote Somewhere in la Mancha a gentleman lived not long ago one of those who has a lance and ancient shield on a shelf and keeps a skinny nag and a greyhound for racing.
Scaramouche He was born with a gift of laughter and a sense that the world was mad.
The Trial “Someone must have slandered Josef K., for one morning, without having done anything truly wrong, he was arrested.”
The Stranger Mother died today.
The Old Man and the Sea He was an old man who fished alone in a skiff in the Gulf Stream and he had gone eighty-four days now without taking a fish.
The Middle Passage “Of all the things that drive men to sea, the most common disaster, I’ve come to learn, is women.”
Neuromancer “The sky above the port was the color of television, tuned to a dead channel.”
A Frolic of His Own “Justice? You get justice in the next world, in this world you have the law.”
Paradise They shoot the white girl first.
Notes From the Underground I am a sick man… I am a spiteful man.
A Clockwork Orange “There was me, that is Alex, and my three droogs and we sat in the Korova milkbar trying to make up our rassoodocks what to do with the evening.”
Chromos “The moment one learns English, complications set in.”
Back When We Were Grownups “Once upon a time, there was a woman who discovered she had turned into the wrong person.”
The Dark Tower The man in black fled across the desert and the gunslinger followed.
The End of the Affair A story has no beginning or end; arbitrarily one chooses that moment of experience from which to look back or from which to look ahead.
Their Eyes Were Watching God Ships at a distance have every man’s wish on board.
Catch-22 It was love at first sight.
I Capture the Castle I write this sitting in the kitchen sink.
Ethan Frome “I had the story, bit by bit, from various people, and, as generally happens in such cases, each time it was a different story.”
City of Glass “It was a wrong number that started it, the telephone ringing three times in the dead of night, and the voice on the other end asking for someone he was not.”
The Red Badge of Courage “The cold passed reluctantly from the earth, and the retiring fogs revealed an army stretched out on the hills, resting.”
Tracks “We started dying before the snow, and like the snow, we continued to fall.”
Tristram Shandy I begin with writing the first sentence - and trusting to Almighty God for the second.
The Napoleon of Notting Hill The human race has been playing at children’s games from the beginning and will probably do it till the end
Murphy “The sun shone, having no alternative, on the nothing new.”
Waiting “Every summer Lin Kong returned to Goose Village to divorce his wife, Shuyu.”
The Tin Drum “Granted, I am an inmate of a mental hospital; my keeper is watching me, he never lets me out of his sight”
Changing Places “High, high above the North Pole, on the first day of 1969, two professors of English Literature approached each other”
Herzog “If I am out of my mind, it’s all right with me, thought Moses Herzog.”
Gravity’s Rainbow A screaming comes across the sky.
The Third Man One never knows when the blow may fall.
The Satanic Verses “To be born again, sang Gibreel Farishta tumbling from the heavens, first you have to die.”
Middlemarch Miss Brooke had that kind of beauty which seems to be thrown into relief by poor dress.
Cat’s Eye “Time is not a line but a dimension, like the dimensions of space.”
The Silmarillion “There was Eru, the One, who in Arda is called Ilúvatar; and he made first the Ainur, the Holy Ones, that were the offspring of his thought”
Wide Sargasso Sea “They say when trouble comes close ranks, and so the white people did.”
Crash Vaughan died yesterday in his last car-crash.
The Sound and the Fury “Through the fence, between the curling flower spaces, I could see them hitting.”
A Portrait of the Artist as a Young Man Once upon a time and a very good time it was there was a moocow coming down along the road
Elmer Gantry Elmer Gantry was drunk.
Blown Away Psychics can see the color of time it’s blue.
Pale Fire I was the shadow of the waxwing slain By the false azure in the windowpane
Mrs Dalloway Mrs. Dalloway said she would buy the flowers herself.
A Farewell to Arms In the late summer of that year we lived in a house in a village that looked across the river and the plain to the mountains.
The Towers of Trebizond “Take my camel, dear, said my Aunt Dot, as she climbed down from this animal on her return from High Mass.”
Riddley Walker On my naming day when I come 12 I gone front spear and kilt a wyld boar he parbly ben the las wyld pig on the Bundel Downs
Richard Plant
Richard Plant
PhD Student

My research interests include natural language processing, privacy, and general machine learning.