Culture Crossover: AI becomes art historian at Rutgers Art and AI Lab


In our Culture Crossover series we pick up examples of projects that delightfully bridge the worlds of technology and culture. We'll be reviewing exhibitions, giving you a heads up on cultural events or talks coming up in the UK and highlighting the best techy art.

To read more instalments of Culture Crossover click here.

Dr Ahmed Elgammal had spent years trying to increase the intelligence of machines and he knew something was missing.

"It was very hard to advance AI beyond what we have right now without looking at this cultural human product, because in the end, artificial intelligence is about making a machine that has perceptual and cognitive abilities,"  Elgammal told Techworld at the World Congress on IT (WCIT) last week in the Yerevan, Armenia. "And when you look at art, that's what's happening."

He set up the Art and Artificial Laboratory at Rutgers University to explore the connection, by developing algorithms that could study and create art.

Around 700 miles south from the lab at the College of Charleston in South Carolina in New York state, Dr Marian Mazzone was working as history of art professor when she discovered Elgammal’s research. She contacted him to add the input of an art historian to the computer scientist's research and soon became a member of his lab.

The two then teamed up to investigate how machines classify syles of art and how that relates to the analysis of art historians. They decided to create a system based on the theories of Heinrich Wölfflin (1846–1945), a Swiss professor whose principles of classification were highly influential in the development of the discipline of art history.

Wölfflin's system divides paintings into five different pairs of concepts: linear and painterly; plane and recession; closed and open; multiplicity and unity; and absolute and relative. The approach excludes the subject matter from the analysis and instead focuses entirely on the "visual schema" of the work and thereby identify style patterns that evolve over time. Its emphasis on distinctive features and binary logic matched well with machine learning.

Deep convolutional neural networks were trained to classify these styles along a number of variables. They were fed almost 80,000 digitised paintings and trained to find the patterns. The system had been given no understanding of time or who created each artwork, but it nonetheless placed the paintings along a smooth chronology that was closely correlated with the times in which they were painted.

It placed them along a timeline starting at the renaissance and then progressing through baroque, neo-classicism, romanticism, impressionism, post-impressionism, expressionism, and cubism, before ending with abstract art.

They also trained the machine to measure creativity by spotting unusual data points and comparing them to what appeared in other artworks.

Certain artists were identified as particularly representative of their style, from Van Eyck's association with the northern renaissance to Picasso's connection with cubism. Others were pinpointed for having a special influence, such as French painter Paul Cézanne.

"Cezanne was a bridge between impressionism and cubism," said Elgammal. "It's not just conjecture or theories. You can see it in the data."

In the 19th century, it was particularly impressed by the creativity of Monet's Haystacks at Chailly, an astute observation given that the painting was unveiled five years before impressionism became a recognised movement, and 25 before Monet's more celebrated series of haystack paintings.

It also gave a very high score to Munch’s 1893 the Scream, noticing that it was one of the most copied artworks of the following century. But the highest score of the century was a Mondrian dated 1910. Elgammal’s investigations revealed why: the painting was not done in 1910, but the 1930s. The algorithm had been given the wrong data, but its observation was logical.

The machine confirmed much of what art historians believe, adding computational evidence of what had previously been based on subjective analysis.

Mazzone believes that its capacity to analyse thousands of artworks could identify fundamental changes in styles that a human eye could never see. It could even predict the artistic forms of the future.

“It makes very few errors," she said. "And even when it makes an error in some ways, it's just the machine seeing something different than what the human is seeing. And that is interesting, too. What is it seeing that is unlike what human beings perceive?"

To read more instalments of Culture Crossover click here.

This story, "Culture Crossover: AI becomes art historian at Rutgers Art and AI Lab" was originally published by

Copyright © 2019 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon