Text-to-Image generation is a curious field that, aside from producing a vast array of monstrosities, can also provide some insight into the visual conceptual embeddings inherent in social language. We can even exploit that social linguistic programming to reproduce some conceptual information that is associated with entities even when they aren’t fully contained in the input prompt.

To explore this idea, I’ve generated a list of famous opening lines from novels, ideas about which might be embedded in in the language corpus used to trained text-to-visual representation learning systems—after all, how many essays have posted online about 1984 that analyse the sentence “It was a bright cold day in April and the clocks were striking thirteen”. How many of those have some relationship to visual concepts that we would link to the book’s themes of state control, linguistic determinism, and the abscence of free will?

My intuition was that for the most famous of these cultural entities, a naive text-to-image generation process based on first lines alone should demonstrate at least some qualities we associate with the novel. Thankfully, adapting existing code for such a system is pretty straightforward since there are plenty of implementations online.

Check out:

• olaviinha/NeuralTextToImage
• kcosta42/VQGAN-CLIP-Docker
• nerdyrodent/VQGAN-CLIP

The solution I adapted for local generation was initially created by Katherine Crowson. You can check out the original notebook in Colab:

My solution, which you can easily adapt to any set of prompts by switching out the input CSV, can be found at ryangoslingsbugle/VQGan_CLIP.

Samples

Method

CLIP

CLIP is a supervised natural language learning model trained to classify images and optimised for maximum transfer potential across benchmark datasets. The idea is that not training for best performance on a single dataset will produce more robust outcomes, better suited to out-of-training-domain flexibility.

OpenAI have adopted a simple approach to producing transferable visual models with their CLIP network. This system follows the general process of masked language model learning, learning to predict which caption applies to which image, essentially producing a kind of visual concept embedding.

Read more in Radford, et al. (2021).

@misc{radford2021learning,
      title={Learning Transferable Visual Models From Natural Language Supervision}, 
      author={Alec Radford and Jong Wook Kim and Chris Hallacy and Aditya Ramesh and Gabriel Goh and Sandhini Agarwal and Girish Sastry and Amanda Askell and Pamela Mishkin and Jack Clark and Gretchen Krueger and Ilya Sutskever},
      year={2021},
      eprint={2103.00020},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

VQGan

VQGan is a combination of a convolutional representation generator with an auto-regressive transformer for reconstruction. This setup achieves extremely good results in image reconstruction tasks, and loading pre-trained models makes working with with it quite straightforward.

Read more in Esser, et al. (2020).

@misc{esser2021taming,
      title={Taming Transformers for High-Resolution Image Synthesis}, 
      author={Patrick Esser and Robin Rombach and Björn Ommer},
      year={2021},
      eprint={2012.09841},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Outputs

Below you can see the full set of novels chosen, the input text used, and the image generated after the 1000th training epoch.

Title	Text	Image
A Tale of Two Cities	“It was the best of times, it was the worst of times”
1984	It was a bright cold day in April and the clocks were striking thirteen.
Fahrenheit 451	It was a pleasure to burn.
Anna Karenina	Happy families are all alike; every unhappy family is unhappy in its own way.
Pride and Prejudice	It is a truth universally acknowledged that a single man in possession of a good fortune must be in want of a wife.
Moby Dick	Call me Ishmael.
The Hobbit	In a hole in the ground there lived a hobbit.
Slaughterhouse Five	“All this happened, more or less.”
The Metamorphosis	As Gregor Samsa awoke one morning from uneasy dreams he found himself transformed in his bed into a gigantic insect.
Voyage of the Dawn Treader	“There was a boy called Eustace Clarence Scrubb, and he almost deserved it.”
The Great Gatsby	In my younger and more vulnerable years my father gave me some advice
The Go-Between	The past is a foreign country; they do things differently there.
The Crow Road	It was the day my grandmother exploded.
The Bell Jar	“It was a queer, sultry summer, the summer they electrocuted the Rosenbergs, and I didn’t know what I was doing in New York.”
To Kill a Mockingbird	“When he was nearly thirteen, my brother Jem got his arm badly broken at the elbow.”
Invisible Man	I am an invisible man.
One Hundred Years of Solitude	Colonel Aureliano Buendía was to remember that distant afternoon when his father took him to discover ice.
Don Quixote	Somewhere in la Mancha a gentleman lived not long ago one of those who has a lance and ancient shield on a shelf and keeps a skinny nag and a greyhound for racing.
Scaramouche	He was born with a gift of laughter and a sense that the world was mad.
The Trial	“Someone must have slandered Josef K., for one morning, without having done anything truly wrong, he was arrested.”
The Stranger	Mother died today.
The Old Man and the Sea	He was an old man who fished alone in a skiff in the Gulf Stream and he had gone eighty-four days now without taking a fish.
The Middle Passage	“Of all the things that drive men to sea, the most common disaster, I’ve come to learn, is women.”
Neuromancer	“The sky above the port was the color of television, tuned to a dead channel.”
A Frolic of His Own	“Justice? You get justice in the next world, in this world you have the law.”
Paradise	They shoot the white girl first.
Notes From the Underground	I am a sick man… I am a spiteful man.
A Clockwork Orange	“There was me, that is Alex, and my three droogs and we sat in the Korova milkbar trying to make up our rassoodocks what to do with the evening.”
Chromos	“The moment one learns English, complications set in.”
Back When We Were Grownups	“Once upon a time, there was a woman who discovered she had turned into the wrong person.”
The Dark Tower	The man in black fled across the desert and the gunslinger followed.
The End of the Affair	A story has no beginning or end; arbitrarily one chooses that moment of experience from which to look back or from which to look ahead.
Their Eyes Were Watching God	Ships at a distance have every man’s wish on board.
Catch-22	It was love at first sight.
I Capture the Castle	I write this sitting in the kitchen sink.
Ethan Frome	“I had the story, bit by bit, from various people, and, as generally happens in such cases, each time it was a different story.”
City of Glass	“It was a wrong number that started it, the telephone ringing three times in the dead of night, and the voice on the other end asking for someone he was not.”
The Red Badge of Courage	“The cold passed reluctantly from the earth, and the retiring fogs revealed an army stretched out on the hills, resting.”
Tracks	“We started dying before the snow, and like the snow, we continued to fall.”
Tristram Shandy	I begin with writing the first sentence - and trusting to Almighty God for the second.
The Napoleon of Notting Hill	The human race has been playing at children’s games from the beginning and will probably do it till the end
Murphy	“The sun shone, having no alternative, on the nothing new.”
Waiting	“Every summer Lin Kong returned to Goose Village to divorce his wife, Shuyu.”
The Tin Drum	“Granted, I am an inmate of a mental hospital; my keeper is watching me, he never lets me out of his sight”
Changing Places	“High, high above the North Pole, on the first day of 1969, two professors of English Literature approached each other”
Herzog	“If I am out of my mind, it’s all right with me, thought Moses Herzog.”
Gravity’s Rainbow	A screaming comes across the sky.
The Third Man	One never knows when the blow may fall.
The Satanic Verses	“To be born again, sang Gibreel Farishta tumbling from the heavens, first you have to die.”
Middlemarch	Miss Brooke had that kind of beauty which seems to be thrown into relief by poor dress.
Cat’s Eye	“Time is not a line but a dimension, like the dimensions of space.”
The Silmarillion	“There was Eru, the One, who in Arda is called Ilúvatar; and he made first the Ainur, the Holy Ones, that were the offspring of his thought”
Wide Sargasso Sea	“They say when trouble comes close ranks, and so the white people did.”
Crash	Vaughan died yesterday in his last car-crash.
The Sound and the Fury	“Through the fence, between the curling flower spaces, I could see them hitting.”
A Portrait of the Artist as a Young Man	Once upon a time and a very good time it was there was a moocow coming down along the road
Elmer Gantry	Elmer Gantry was drunk.
Blown Away	Psychics can see the color of time it’s blue.
Pale Fire	I was the shadow of the waxwing slain By the false azure in the windowpane
Mrs Dalloway	Mrs. Dalloway said she would buy the flowers herself.
A Farewell to Arms	In the late summer of that year we lived in a house in a village that looked across the river and the plain to the mountains.
The Towers of Trebizond	“Take my camel, dear, said my Aunt Dot, as she climbed down from this animal on her return from High Mass.”
Riddley Walker	On my naming day when I come 12 I gone front spear and kilt a wyld boar he parbly ben the las wyld pig on the Bundel Downs