Compression ensembles quantify aesthetic complexity and the evolution of visual art

Compression ensembles quantify aesthetic complexity and the evolution of visual art

Abstract

object_of_studyconcepts↗unhoused_relationssensescontent_tagsTo the human eye, different images appear more or less complex, but capturing this intuition in a single aesthetic measure is considered hard. Here, we propose a computationally simple, transparent method for modeling aesthetic complexity as a multidimensional algorithmic phenomenon, which enables the systematic analysis of large image datasets. The approach captures visual family resemblance via a multitude of image transformations and subsequent compressions, yielding explainable embeddings. It aligns well with human judgments of visual complexity, and performs well in authorship and style recognition tasks. Showcasing the functionality, we apply the method to 125,000 artworks, recovering trends and revealing new insights regarding historical art, artistic careers over centuries, and emerging aesthetics in a contemporary NFT art market. Our approach, here applied to images but applicable more broadly, provides a new perspective to quantitative aesthetics, connoisseurship, multidimensional meaning spaces, and the study of cultural complexity.

1 Introduction

concepts↗claim_relationsclaim_relationsThe quantification of visual aesthetics and artistic expression goes back to Birkhoff [1] and Bense [2], inspiring several computational approaches [3–16]. Research drawing on information theory has shown repeatedly and in parallel that visual complexity can be estimated with some accuracy using compression algorithms such as zip or gif [17–29]. While some previous proposals were tested against perceptual human judgments, results diverge as to which compression algorithm or approach would be optimal. Elsewhere in cultural research, measures of compression length have been used at face value to compare the complexity of visual inputs [30–32]. Related information-theoretic approaches have also used entropy to quantify artistic styles and conceptual groupings [8, 12, 33].

mechanisms↗concepts↗concepts↗concepts↗unhoused_relationssensessensesclaim_relationscontent_tags↗content_tags↗Considering quantitative analysis of visual art, it makes sense to adopt algorithmic approaches, as creative processes themselves also follow a set of procedures — or algorithms, in the broadest sense — in which artists may be similar or differ [2, 3, 34]. Algorithmic complexity is best understood via information theory [35, 36], defining the Kolmogorov complexity of a string or dataset as the length of the shortest algorithm reproducing the data. Kolmogorov complexity is uncomputable, but can be approximated using the compression size of a dataset as its upper bound. In the case of images, the more compressible one is — i.e. the larger the difference between a bitmap and a compressed version e.g. gif — the visually simpler it is. This notion in turn can be extended to measures of algorithmic distance, one example being normalized compression distance (NCD) [37]. This however requires a separate compression for each comparison event, and has been rarely applied to visual materials [38]. Pairwise image comparison frameworks [19, 22], suffer from the same computational bottleneck. Furthermore, visual media such as art or photography can, intuitively, differ in various dimensions of complexity, such as color, composition and detail. A single value of complexity would likely fail to capture this multiplicity.

1.1 A method proposal

contributionclaim_typemechanisms↗concepts↗concepts↗content_tags↗content_tags↗We introduce “compression ensembles”, an efficient and explainable algorithmic comparison framework for images. Instead of attempting to find a single algorithm or metric to best match human judgments or perform on downstream tasks, we argue for an ensemble approach of concurrently using multiple measures. This consists of two steps: multiple transformations and subsequent compression. For a given input image, a set of alternative images is produced by applying a number of different image processing transformations, including various filters, blurs, distortions and color manipulations (see Fig. 1.A, Methods section, and Additional file 1, for details).

method↗mechanisms↗limitationsconcepts↗concepts↗unhoused_relationsunhoused_relationsconcept_edgescontent_tags↗content_tags↗These new images are compressed (possibly with multiple compression algorithms), and the resulting file sizes are divided by the compression size of the original bitmap; the unfiltered original input is also compressed but divided by the size of the bitmap input. This yields a vector of compression ratios. Further statistical transformations such as colorfulness metrics and fractal dimension [5, 16, 39, 40] (compressions in a broader sense) can also be added to these vectors, and rescaled if the magnitudes differ, which we do (see Methods). Given the perspective of artists as (or as executing) algorithms — in the very broadest sense — our approach aims to capture the residual signal of the generating process, the “algorithmic fingerprints” of an artwork, through the operationalization of various aspects of visual aesthetic complexity, as estimated by compression ratios of the visual transformations. This approach can, however, currently only assess the visual complexity apparent in the image. This may or may not correlate with production complexity, i.e. the time, effort, expertise, etc. that goes into producing an artwork.

phenomenonAs demonstrated below, compression ensembles can be used for estimating relative visual complexity as a multidimensional phenomenon; inferring specific aesthetic aspects of complexity where any two images or artist oeuvres differ; and as input to any predictive downstream tasks where image complexity (and by proxy aesthetic or stylistic) profiles matter. We also show its use as a “computational art historian” or “curator” algorithm to quantify and systematically explore the dynamics of art artistic careers in very large datasets. While we focus our showcase here on art, it is suitable for comparing any sort of images, and as described in the Discussion, straightforwardly extendable to other types of media to estimate their inherent aesthetic complexity.

1.2 Relation to other image embedding approaches

concepts↗claim_relationsCompression vectors can be rapidly compared, clustered, or used in image recognition and comparison tasks, as demonstrated below. Our method is comparable, but largely orthogonal to deep learning models of computer vision that also embed images in numeric vector spaces. Typically pre-trained on very large image databases, the latter excel at inferring image similarity (in the feature space), and can be tuned to predict discrete classes such as objects on the image, historical style, or authorship [9, 41, 42]. Compression ensembles can also be used to estimate similarity (in the complexity space), and nearby images often depict similar subjects — albeit incidentally, due to shared stylistic commonalities in how certain subjects are depicted (Fig. 1.B). However, the main function of the ensembles is to operationalize visual complexity as such. Our approach infers meaningful values for an input without requiring any costly pre-training, and therefore also entirely bypasses the “training set bias” problem inherent to pre-trained machine learning models. Fitting new images to an already generated vector space here does not require retraining or realignment of the space either, just a matching set of transformations.

concepts↗sensesWe are less interested in discrete classification (although we do run tests on these tasks, see Methods), and more in efficient exploration and curation of large image sets in the continuous complexity space. We show that this is also a good proxy for aesthetic complexity, and by extension, indeed notions of artistic style (see Figs. 1.B, 3). The issue with discrete style classification [5, 42] is that it inevitably requires training on some “ground truth”, which in the case of historical style periods is highly debatable. Our approach goes beyond limited discrete categories to operate in a continuous aesthetics space.

limitationssensescontent_tags↗To contrast another related body of literature, some previous research has tried to build statistical and machine learning models to explicitly predict what humans participants in psychology experiments perceive as intuitively complex, “aesthetic”, or “beautiful” [14, 20, 25, 43–45]. We did test our approach on a set of human complexity judgments and show that it performs very well (see Methods), but our goal, to provide a general framework, is broader. We use complexity as a practical comparative algorithmic measure, and, to be very clear, “aesthetic” not as a value judgment. Besides “beauty”, here we also do not attempt to quantify complexity in terms of iconography, semantics nor number of depicted subjects (but see Discussion for possible extensions).

concepts↗sensesOur approach is also conceptually related to “ensemble methods” in statistics and machine learning [46], but instead of a single aggregated prediction, the useful output here is the full vector of complexity estimates. Approaches using ensembles of various metrics have also been proposed in linguistics for language complexity [47] and economics for tax system complexity [48].

1.3 Explainable vector spaces

concepts↗concepts↗concepts↗concept_edgesconcept_edgesImportantly, unlike all of the aforementioned machine learning based image embeddings, the dimensions or variables in a compression ensemble remain interpretable: a compression ratio difference between two images for a given transformation indicates that they differ in this aspect. Applying a black-and-white transformation to a colorful image increases compressibility relative to the original (compression ratio ≪1), but has no effect on an already black-and-white image (ratio ≈1). Applying coarse pixelation to Piet Mondrian’s abstract paintings (Fig. 1.B.1) barely changes their compressibility, while it greatly increases compressibility for the highly detailed works of Hieronymus Bosch (Fig. 1.B.6). Therefore, images similar in multiple aspects end up close together in multidimensional compression space, while dissimilar ones stay far apart. Though our approach does not include any visual similarity or object recognition features in the machine learning sense, certain genres do appear more popular within certain regions of the space, depicting features yielding a similar complexity profile (e.g. human half-figures on dark background; Fig. 1.B.2).

mechanisms↗concepts↗concepts↗content_tagsThe model used in this contribution consists of 112 transformations (see Fig. 1.A for examples; Table S1 and Figure S1 in the Additional file 1 for the full list; we use lossless gif, and also png and lossy jpeg on a smaller subset). The exact nature and number of transformations is unimportant yet subject to optimization. More features increase computation time, but provide more information, i.e. the approximation honing in on the true uncomputable Kolmogorov complexity (see Methods). As demonstrated in the classification experiments in the Methods section, different transforms are informative for different tasks, and, given a specific task, a handful of well-chosen features can yield accuracy close to using a large ensemble. Some transformations, like blurs of various magnitudes, may well correlate. In applications where multicollinearity must be avoided or a lower number of dimensions is desired, methods such as Principal Component Analysis (PCA) or UMAP [49] can easily be applied. In this work we use both, as PCA is directly interpretable due to linear relationship with original variables, while UMAP arguably offers better low-dimensional representations [49].

Dimension reduction like UMAP can be used to produce a complementary “field of similarity” [50] where similar images cluster together intuitively, while remaining subject to the “curse of dimensionality” (Fig. 1.B). We can break the curse by repeatedly mapping individual transformations onto the common UMAP, effectively using it as a reference topography (Figs. 1.C and S2). The resulting “small multiple” of visualizations provides intuition why an ensemble of multiple transformations is necessary towards a fuller understanding of visual aesthetic complexity. It also relates to the explainability aspect: images being in different ends of a given transformation variable (colored brown to blue in 1.C) indicates they differ in the aspect of complexity represented by the given distortion.

In this contribution, we define and test the compression ensemble approach in a number of experiments from human complexity judgments to example downstream classification tasks (see Methods section). In the Results section, we showcase the utility of the approach for the exploration of art collections, quantifying global trends in historical art over the past six centuries in a large dataset, and the first half a year of a non-fungible token (NFT) art marketplace. On historical timescales, we introduce a temporal resemblance model to quantify artistic career trajectories, grouping them into qualitatively distinct types. We reveal artists that were well embedded in the historical tradition of their time, those who simultaneously experimented with different areas of the aesthetic complexity space, artists with transitory success, and those who were later seen as ahead of their time. Finally, we discuss the broader relevance of the approach to digital art history, cultural evolution, and extensions to other media and modalities.

2 Results

content_tagsWe make use of two large art corpora to proof the application of the compression ensemble approach for visual data, while exemplifying the exploration of historical and contemporary dynamics of visual art. The first dataset which we denote as “Historical” (henceforth capitalized when being referred to) is illustrated in Fig. 1.B. It is sourced from the art500k project [42], filtered to only include two-dimensional art with a retrievable year of creation. Our subset contains 74028 (primarily Western) artworks representing 6555 artists from the years 1400-2018 (older art exists in the dataset but is sparsely distributed). This filtered dataset ends up consisting mostly of items art500k had in turn sourced from Wikiart.org. The latter is an online, user-editable, encyclopedic collection of mostly Western art images, also frequently used in computer vision research.

limitationsclaim_relationsIn the case of art collections or databases like Wikiart and art500k, it is important to be clear that these consist of small, curated, often biased samples of art of some place and period. As such, they represent the historiography of art first and the actual history of art second [12] (see Data Limitations in the Methods section for further discussion). When we make claims here about the history or dynamics of visual art, we are only referring to information derived from the sample — but we make the assumption that the sample is reasonably representative and as such informative of the population of Western art in the time periods we cover. This means that the figures depicting historical changes may look different if more data would be available. However, our quantitative method could also be used for systematic study of data set bias.

concepts↗content_tagsThe second dataset, denoted “Contemporary”, is mined from Hic et Nunc, a Tezos blockchain-based NFT art marketplace, representing the first 175 days of its existence (March to August 2021; 51640 artworks, 7284 artists). It contains 31% of all the objects added to the marketplace during our observation period. We only include static images (jpeg, png), as the currently presented approach does not yet extend to multi-frame objects such as animated gifs and videos. We also exclude low resolution images (such as icons), and a subset for which the data collection process failed to retrieve the image. Unlike the Historical dataset which consists of digitized art, the vast majority in the Contemporary set are born-digital images. For an overview of the NFT-driven “crypto art” market, see references [51, 52].

research_questionThe obvious question is whether simple vectors of file size ratios are provably informative about visual complexity or artistic aesthetics. We carried out two sets of experiments to verify that the compression ensembles are fit for this purpose, and show that they allow us to meaningfully track the evolution of complexity in historical art. As detailed in the Evaluation subsection under Methods, we first tested the approach against two sets of human visual complexity judgments, where it performed very well, aligning with what people would judge to be visually simple or complex. This indicates the method is cognitively plausible as a visual complexity estimator. Secondly, we devised a set of classification experiments to determine if there is enough information in a compression ensemble to meaningfully delineate aspects of interest in visual art such as style, genre, authorship and medium. The model yields reasonable accuracy in all cases (and its mis-classifications make sense from an art historical point of view). This indicates the method is fit for purpose for the exploratory tasks showcased in the next section.

2.1 Tracking historical and contemporary art dynamics

limitationsGiven the explainable nature of compression ensembles, and its demonstrable cognitive and technical plausibility, we proceed to use the method to investigate and interpret aesthetic trends over time. We do this for both the Historical and the Contemporary NFT datasets. To simplify this task, we apply Principal Component Analysis: compression vectors for both the Historical and the Contemporary datasets are fitted in the same PCA space for comparability. An alternative would be to map change over each individual transformation variable, but PCA conveniently allows for focusing on decorrelated latent aspects with most variance, while remaining interpretable through the transformations that load onto each component. Figure 2 depicts change over time in the two first most informative components. The Historical dataset is limited to 1500-2000 on the graph, as both ends outside of that range are quite sparse. The “trend lines” are estimates from a rolling window of ±10 (years of Historical data, days of Contemporary Hic et Nunc). Where there is insufficient data, the window is stretched up to size 50 to include at least 1000 artworks where possible; these broader estimates are reflected by decreased line opacity. We do not engage in statistical testing here, as this exercise is explorative (for dating and style classification testing, see Evaluation in the Methods section).

content_tagscontent_tags↗Changes in the trends of the half-millennium Historical dataset correspond broadly to art historical style classifications. PC1 in this model corresponds to texture and detail complexity (loading onto blurs, despeckle filters, and the Canny edge transform). A set of the more frequent style period labels are shown in Fig. 2.A, arranged by the median year of the respective artworks. Visible in the right half of (A), there is a visible median complexity decrease between the period of detailed paintings of Realism and Impressionism and the second half of the 20th century where (in this dimension less complex) styles such as Abstract Expressionism and Pop Art become more prevalent.

content_tags↗PC2 corresponds to overall compressibility (loading onto compression of the original unfiltered image but with different compression algorithms). The median in the Historical dataset is somewhat lower where the dataset contains many Rococo style portraits (in the middle of Fig. 2.B), which typically contain plain (easily compressible) backgrounds — not unlike the pixel-art portraits of Hic et Nunc (cf. days 100-150 in Fig. 2.C-D). PC2 values in Historical (Fig. 2.B) go up around the onset of Impressionism, and the bounds are pushed once more with Cubism, Expressionism, Surrealism, and the general diversification of classic modern “-isms”. As demonstrated in the Evaluation section in Methods, given a sufficient number of transformations, such differences are consistent and diverse enough to predict style periods with reasonable accuracy.

The Historical and Contemporary sets are combined in the same space, but the ranges of the vertical axes representing the components in Figs. 2.A-B versus C-D are intentionally different, as the two datasets occupy markedly different ranges in the complexity space, with much higher variance in the Contemporary Hic et Nunc dataset compared to the more conventional Historical dataset. This does not necessarily mean that art in the last 500 years has been less creative or explorative. The relative boundedness instead is more plausibly rooted in a combination of material affordances and limits of curation and scholarship. The latter is a function of cultural selection, as collectors, audiences, and art historians put a bound on what has been and is considered worthwhile of adding to collections from the time of creation to current retrospectives.

In contrast, anybody who is able to pay the fairly low “minting” fee can upload an artwork to blockchain art market places such as Hic et Nunc, making their creations public in an attempt to get attention and sell. The Historical broadening of the parameter space goes in lockstep with the fraction of noted creatives growing faster than world population in the last five centuries [53]. It is broadly established knowledge in art history that new technologies and concepts, from pigments to theories of perception [54], were harnessed by said creatives. Examples include the emergence of more affordable blue pigment alternatives to the rare and expensive azurite and lapis lazuli, or (color) photography, which put traditional pictorial conventions of depiction into question. Another striking difference between the Historical and the Contemporary NFT dataset becomes visible in Figs. 2.A-B versus C-D when we focus on the range of colors in the single-pixel reductions of the artworks. The digital NFT images appear darker and more saturated, as they are using the full RGB color space, while the dominant color of Historical artworks tends to remain in the range of “natural” pigments, which one could buy in a physical art supply store.

Since we have information on transactions in the Hic et Nunc dataset (as of the data collection time, 22 August 2021), successful sales are shown as inset heatmaps (E, F) in Fig. 2.D. The heatmaps show the fraction of sales across the first and second principal components respectively. About half the objects in the Hic et Nunc sample in total were sold off by their authors during our observation period, with some areas — dark red in the insets — being clearly more conductive to sales, while others do not sell at all.

content_tagsEven qualitatively, one can see revealing patterns, such as the mass-minting of initially non-selling NFT portrait images starting around day 110 in mid 2021. These can be described as simple, typically procedurally generated, mugshot or portrait-style images depicting various human, humanoid or cartoon characters. At the height of the NFT boom, the perhaps more widely known examples of this trend on other platforms included the “CryptoPunks” and “Bored Ape Yacht Club” series [51]. On Hic et Nunc, such series are titled for example “AI Pokemon”, “Dino Dudes”, and “NFT-People”. An emergent quality of these mass-produced images is that their texture and detail complexity (PC1, Fig. 2C/E, day 100-150) is substantially lower than the all preceding art, putting them more in the realm of icons or brand logos. At the same time their overall compressibility (PC2, Fig. 2D/F) is not only systematically lower, but also subject to less variance. This could potentially indicate low effort attempts at production of these simple images with hopes of fast monetary gain in the marketplace. The narrowness of the mass-produced NFT series also expresses itself in their skew towards highly saturated primary dominant colors. In at least one case, this indeed seemed to work, where sales follow in the wake of a strong minting burst, mostly consisting of the “NFT-People” and “NFT Kids” series (cf. the rightmost vertical blue line in Fig. 2.E, followed by a light red wake).

These initial observations could of course be augmented with more systematic statistical or predictive modeling in future research. As an example, we trained a simple classifier, Linear Discriminant Analysis (see Evaluation section under Methods) on the sales data, predicting whether an NFT art piece was sold or not, based on the values in the compression vectors. Using training sets of size 20k per class and separate test sets of 5k (and replicating the model 500 times), the model predicts sales at an average accuracy of 58% (or a 17% kappa, given the 50-50 baseline). This is despite containing no information on the prestige or reach of the artists, past sales, the depicted content, nor trends of the market of the respective time. A linear regression model fitted to 23370 sold items predicting log price (excluding zero-price giveaways) by all the compression variables describes about 6% of variance (adjusted \(R^{2}\)); allowing for interaction with the time variable improves this to 8%. While these are all fairly low scores in absolute terms, we consider this a promising result for future research, likely improved by combining our aesthetics model with the aforementioned variables of author properties, sales history and past trends [44, 52], to predict future trends in evolving art markets.

2.2 Quantifying temporal resemblance in artistic careers

In the previous section, we took a look into art history as a whole. We can also use compression ensembles to investigate how the oeuvres of individual artists progress and are situated in their eras. Tracing “the lives of the artists” has been a foundational and central direction in the historiography of art since the 1550 book by Giorgio Vasari which initiated the genre [55], followed by a great number of artist monographs and critical catalogs. More recently, multidisciplinary science has tackled the issue using methods of network science and quantitative measures of success, making use of data such as demographic and migration records [53, 56, 57], museum, exhibition and art market price information [58, 59], but also visual aesthetic aspects using information theory or machine learning [7, 12, 60].

concepts↗content_tags↗We introduce “temporal resemblance”, a simple metric to summarize and compare artistic careers (Fig. 3). This could be applied to any numeric space (including deep learning embeddings) that includes temporal metadata, but its interpretation of course depends on the space. Here, it represents resemblance in the compression ensemble approximated aesthetic complexity space.

Figure 3.A provides an illustration, using the oeuvre of Piet Mondrian: the rows are transformations; the columns his works, arranged diachronically. For this example, the compression ratio values are z-scored using the mean and standard deviation of his era. This means that the blue to red scale is interpretable as “higher or lower than contemporaries”, and white as being close to the mean. Mondrian’s departure from the era’s mainstream is quite clear from the increase of darker reds and blues. The transformations provide insight as to which areas are most contrasting (labels on the dendrogram in 3.A).

mechanisms↗content_tags↗The temporal resemblance method generalizes this comparison. Given the vector space of the Historical set (decorrelated using PCA), we can calculate the nearest neighbors for each artwork vector (like the columns in 3.A). We use cosine similarity and the top closest 100 neighbors (excluding works by the same artist). The median temporal distance between these neighbors and the target work indicates if it resembles the past or anticipates some yet unseen future. This allows us to group artists who are traditionalist or historicist, those who stay current, and those ahead of their time. It is necessary to adjust the median time distances to account for the boundedness and density bias of the dataset: the metric reported here is derived from the residuals of a generalized additive regression model (GAM), still on the same yearly timescale (see Additional file 1 for technical details of the adjustment). Figure 3.B again depicts Mondrian’s works, with temporal resemblance now on the vertical axis.

This metric is relative to the point in time of each work, and all measures are relative to all other works. Therefore, curves that stay close to the zero line in Figs. 3.B/C should be interpreted as artists who produce works that are similar to other artworks made in the same years, in terms of aesthetic complexity (and thus aspects of their style — but we are not quantifying the subjects they depict). That does not preclude changes in their style, if the changes in the artist and their era correlate. Staying around the zero may also indicate that a given artist is surrounded by a handful of prolific contemporaries with very similar output, who as a group may not be representative of the mainstream. Descending curves can indicate an artist who becomes more traditional, the world catching up to an artist’s style, or the world adopting other new styles.

Figure 3.C depicts the careers of 20 artists, grouped by career trend similarity, revealing different modes of artistic existence, similar yet not identical to the narrative types of Vonnegut [61]. To trained art historians (as represented among the authors of this paper) these results are intuitively correct — but of course the career comparisons discussed here only refer to artworks that are present and dated in our dataset. Some such as Piet Mondrian or Mark Rothko, “rise above the flock”, starting out in the mainstream, growing into their own distinct style, with works that could be considered ahead of their time. Paul Cezanne and Mary Cassatt instead become “constant innovators”, starting out by producing conventional, retrospective works, but growing and remaining innovative throughout the rest of his careers. Albert Bierstadt and Camille Corot represent “mainstream artists”, appearing more narrow in their practice, and remaining consistent with the current of their peers. There are also those who “rise and fall” (see James Mcneill Whistler or William Merritt Chase), growing to their moment in history, then becoming more conventional again over the course of their careers. The metric highlights Eastman Johnson, who was predominantly drawing inspiration from the past, even at the height of his career. But even innovative careers may include revivals on occasion; e.g. Paul Cezanne also has works resembling the complexity profiles of art preceding his own by 100-200 years.

As a static graph, Fig. 3 just exemplifies how compression ensembles (or other embeddings) can be used to filter and cluster artistic trajectories. Figure S5 provides an alternative version of Fig. 3.B/C with larger thumbnails. Interactive versions of such plots could function as a research instrument for qualitative experts, to investigate the quantitative model and dataset biases, and compare artists between different datasets. In a related project [62], we developed an interactive web interface titled Collection Space Navigator, which allows for on-the-fly visualization of large collections, and operations such as zooming, filtering, and hovering for more information. Its online demo also features a large subset of the Historical dataset used here, including functions to operate with compression ensembles (link in the Availability of data and materials section below).

3 Discussion

Products of human culture, such as art, language and music are all subject to ongoing change, complex dynamics, and cumulative evolution [63–66]. And even though complexity could emerge from a simple generating mechanism in principle, a single unidimensional measure would likely prove insufficient to capture the polymorphic complexity of human cultural interaction and cultural products [67]. Here, we have demonstrated the utility of explainable compression ensembles to quantify polymorphic visual aesthetic complexity. We showed how the approach can recover and reveal meaningful patterns in datasets of historical and contemporary art, and, in the methods section, we evaluated the cognitive plausibility of our approach, tested its viability at author, date, style, genre, and medium detection tasks. Given the increasing availability of cultural datasets in machine-readable form, this operationalization opens up new avenues to study the dynamics of visual art aesthetics at scale, over long time spans and almost in real time. As such, the approach may help to transcend the still considerable specialization and bifurcation (by artist, period, style, etc.) of qualitative art historical scholarship. Our approach could be used to fill a similar niche in art history as computational corpus linguistics does in relation to the qualitative study of literature.

3.1 Cultural evolution and aesthetic value

concepts↗concepts↗sensesWhile we focus on complexity, the same approach could be used on other related phenomena. For example, Sinclair et al. [68] raise the concept of “aesthetic value” or “attractiveness” of a given cultural product, to discuss whether art and music indeed could be considered as products of cumulative cultural evolution [66] or not, as “cumulative” would allude to objective improvement over time [69]. From the art historical perspective, a style that builds on or grows out of another style may not be necessarily objectively better, but may better meet the preferences of its consumers in a given time, place, or ecological niche. This is not unlike the concept of communicative need in language: a structure or lexical configuration might not be better in some absolute evolutionary terms, but can be more optimal or efficient given the usage tendencies or needs of a given language community [70, 71].

Our method goes beyond discrete categories and facilitates studying visual culture using a continuous form of aesthetic representation to examine such questions, as exemplified in the temporal resemblance section above. Naturally, the extent that need or preference can be studied depends on available data. E.g. the Historical set represents only a rough estimate of a (primarily Western, biased, somewhat dated) preference consensus, while the Hic et Nunc data includes artist, collector, trade and price info, which could be (carefully) interpreted as preference, as well as linked with the (social) media activity of the sellers and buyers.

3.2 Family resemblance and connoisseurship

concepts↗concepts↗unhoused_relationsThe vector space of the ensemble allows for flexible operationalization and visualization, e.g. a single figure (e.g. 3) can summarize careers of several artists. This is not entirely dissimilar to the intuition of an human connoisseur trained on a given corpus of art. As each transformation in an ensemble represents a tangible visual aspect e.g. abundance of detail or colorfulness, as a whole, it constitutes an estimate of the philosophical and cognitive concept of polymorphic family resemblance, originally used to characterize similarity of games such as chess and soccer, later extended to polymorphic visual perception [72–74]. The recognition of visual family resemblance is arguably foundational and intrinsically mastered by trained human art connoisseurs, yet “exists at an unarticulated level, easy to invoke but difficult to explicate” [3]. Such recognition skills are also required of other visual experts e.g. radiologists, detecting health issues in medical imaging. Deep learning models have made headway in solving the latter among other object recognition tasks, but are not very good at explaining why or how they recognize something either (though cf. [29]).

Our explorations of the Historical dataset yield results that meaningfully reflect the art historical scholarship underlying the dataset (Figs. 2, 3). The model captures enough family resemblance to cluster together similar styles or works by the same artist (as verified in Methods), demonstrating the explanatory power of using an ensemble of multiple transparent transformations, effectively addressing what Friedländer in his foundational art connoisseurship book [75] called “the visible in its manifoldness and unity, bristling against conceptual segmentation, so that the boundaries between the species of images get into flow” or get “blurred” (p. 60; our translation).

content_tagsAs an interesting byproduct, the continuous multidimensional ensemble space of compression ratios also allows for mathematical vector operations (not unlike in word embeddings [76]) and explainable latent space exploration. Adding the vectors of Example 4 and 6 of Fig. 1.B, an etching or woodcut print plus the “Garden of earthly delights”, yields a vector where the closest neighbors are fittingly prints of trees and nature (cf. Figure S3). Multiplying a Mondrian vector of Example 1 with an averaged vector of all landscape paintings nets an abstract landscape. Exploring these operations could be an interesting venue of future research.

3.3 A window to cultural meaning space

We used three kinds of vector spaces: the full multidimensional ensemble space of compression ratios, the decorrelated multidimensional space of associated PCA components, and a dimension-reduced UMAP space providing a proxy topography. These can also be understood as subspaces of more general cultural meaning spaces, and interpreted though or used in various other approaches to culture, briefly exemplified below, with potential to be developed further in future work on quantitative aesthetics. In Cassirer’s most general reference framework they would be “spaces of geometric intuition” [77–79], also later referred to in art history as “iconologic” aspects, complementing the associated contextual information including “iconographic” aspects [80]. They resonate with the conceptual spaces theory of Gärdenfors [81, 82], as well as with the notion of information space [83]. Future work could also look into resonance between our and the recent deep learning driven “distributed information bottleneck” proposal [29] which also involves visual transformations in the case of images.

3.4 Extensions to other visual media, audio, video and text

Here we focused on static, 2D art such as paintings and digital drawings. However, there is no reason the same methodology could not be applied to quantify other visual media such as photographs, maps [11, 17], websites [84] or natural patterns [27] to assess their aesthetic complexity (and by proxy, style) in a transparent, explainable framework. Multi-frame visual media such as film and animation could be split up by frame or shot, and represented in a compression ensemble as ordered sets of vectors, or alternatively, transformed using video filters and compressed directly using video compression algorithms. Similarly, the aesthetic complexity of sound and music [85, 86] could be inferred by either applying the same visual transformations to spectrograms, or by using audio filters as the transformations followed by audio compression. 3D objects such as sculptures, architecture, or clothing can be operationalized by systematically scanning them from multiple angles, or using 3D versions of transformations and compressions (voxels instead of pixels). For written text, an ensemble of byte pair encoding [87] models (with variable parameters or trained on different genres) could be used as the transformations.

3.5 Combining compression ensembles with other vector spaces to study culture at scale

For multi-modal media, multiple ensembles or embeddings can be concatenated, provided a principled way to weigh or normalize their contribution [88]. If both feature similarity and aesthetic complexity matter in a given application, a compression ensemble could be horizontally aligned and concatenated with a suitable deep learning embedding [42], an approach shown to be fruitful in NLP [89]. A scene in a film or a recorded theater play could be represented by the concatenation of a visual compression ensemble, an audio compression ensemble, an image embedding, and a language model embedding [90] of the spoken dialogue. A compression ensemble could also be combined with the larger apparatus of art history, in the form of socio-cultural context information from relevant databases and knowledge graphs [91] or any other visual feature vectors [13, 16, 92]. Furthermore, features or objects (e.g. humans) could be extracted using a deep learning classifier, followed by a compression ensemble of these sub-images, producing e.g. an aesthetic vector space of human pose, effectively a further operationalization of Aby Warburg’s Mnemosyne [93, 94].

Understanding cultural products at scale is not only relevant just because of the growing body of born-digital culture and digitization efforts of non-digitized culture. We stand at the threshold of an AI revolution, where the fully automated generation of photo-realistic, artistic and otherwise previously primarily human-produced visual content (and soon likely multimedia too) has suddenly become feasible, accessible and affordable, using e.g. models like Dall-E or Stable Diffusion [95]. This is likely to transform multiple entire industries, but the functionally near-infinite content quantity will require curation and understanding to be made efficient use of. Pretrained deep learning embeddings can be used to calculate and cluster similar items in a feature space, or be trained to predict preset objects, styles or human preferences. Our explainable compression ensembles are however well-positioned to make sense of such spaces and navigate aesthetics, without requiring any “training” — indeed, consisting of already meaningful values, they are not constrained by the need to predict anything specific to be functional.

4 Methods

4.1 Constructing a vector space of algorithmic distance

As discussed in the Introduction, compression as such has been used to estimate visual and aesthetic complexity before. In some applications, it has also consisted of or included combination with limited visual transformations [5, 24, 25, 27–29, 40, 44]. However, the fairly large number of transformations is key to our approach, with the following rationale.

Consider two algorithmically similar uncompressed images A and B, for example two versions of the same famous view of Rouen cathedral by Claude Monet (of which the artist painted more than 30 in 1892/93). These two images will yield similar compressed sizes for the same compression algorithm because the “algorithm” that generated them (being a function of Monet’s perspective, style, and execution) is similar. Another artwork C, e.g. a late, abstract work by Piet Mondrian will, due its lack of detail, likely have a much smaller compression size. However it is entirely conceivable that a work D that is stylistically very different to Monet’s Rouen cathedral, e.g. a surrealist painting by Salvador Dali, might by chance have a very similar compression size. The “algorithms” used by Monet and Dali differ greatly, and an equal compression size does not imply that they are of equal algorithmic complexity either, as the efficiency of the compression algorithm itself will differ depending on the detailed characteristics of the images.

mechanisms↗concepts↗However, now consider an image transformation T (e.g. Gaussian blur), which we apply to the uncompressed versions of our four images A, B, C, and D before compressing them. The compressed sizes of \(T(A)\) and \(T(B)\) are still likely to be very similar, as the algorithms that generated the original images are very similar, and the transformation and compression algorithms are identical. \(T(C)\) is very likely to still be very different to \(T(A)\) and \(T(B)\). While the compressed size of D was similar to A and B by chance, it is much more unlikely that \(T(D)\) is also similar to \(T(A)\) and \(T(B)\), as the interaction between the transformation T and the generative algorithm of D would have to change the compressibility in the same way as the interaction of T and A/B. Put more intuitively, a Gaussian blur is very likely to affect the compressibility of a Monet very differently from the compressibility of a Dali. Thus, more generally speaking, two images with similar compressed sizes are much less likely to still yield similar compressed sizes by chance after a transformation unless they are algorithmically similar to start with in which case the combined algorithms of generation and transformation (and their interaction with the compression algorithm) remain similar. If we now consider the application of N different transformations of an uncompressed image A, each applied before a subsequent compression, the compressed sizes (including of the untransformed image) \(c(A)\), \(c(T_{1}(A)),c(T_{2}(A)), \dots, c(T_{N}(A))\) form a vector \({\mathbf{v}}(A)\) of length \(N+1\). It follows from the above argument about coincidental proximity that it becomes increasingly unlikely for two algorithmically dissimilar images to remain close together as N increases. Thus the resulting vector space of compressed sizes provides an indication of algorithmic distance between images.

4.2 Data processing

In practice, we use normalized compression lengths. The compression size of the original image without transformations is divided by the size of the original bitmap image. Compressions of transformations are divided by the size of the original compression. In most applications discussed in this paper, it also makes sense to rescale the vector space components (we use z-scoring), to put the compression ratios and the additional statistical transformations (fractal dimension, colorfulness metrics, etc) on a comparable scale. For more technical details on the processing pipeline, list of the transformations and implementation, see Additional file 1.

Both the Historical and Contemporary Hic et Nunc dataset are preprocessed the same way, downscaling images to 160,000 pixel bitmaps (400 × 400 in the case of a perfect square) while retaining aspect ratio. Smaller images up to 50% of that size are allowed (but not upscaled), smaller images are discarded. Another option would be to resize all images to identical squares, but that would distort the composition of wide or tall artworks. The aspect differences, size differences resulting from integer division of the 160,000 and the inclusion of smaller images, are all controlled for in the next step. The assigned file size of a compressed image (or its transformation) is actually the mean of two compressions, of the original and its 90 degree rotation. The compression ratios are calculated in terms of the respective downscaled bitmaps. Furthermore, one of our visual transformations is the Fast Fourier Transform; given its square-shaped output components, the transform is applied twice, on the original and its rotation, and the resulting components are also additionally rotated for compression.

4.3 Data limitations and biases

limitationsThis approach to homogenizing the images is far from perfect, as the size of the originals that these photographs and scans represent may well range from the size of a postcard to that of an altar piece. Not only that, but the latter may well be represented by a lower resolution image than the former, with better or worse color grading, etc. The dataset contains sparse metadata on original size and we have no way to systematically quantify this issue at scale, which remains a limitation of the current study. However, in a sense, our approach is not very different from making art historical inferences by going through and looking at large visual resource collections, much like students of art history examining art historical survey literature, or an art connoisseur training their eye using a large comprehensive 35 mm slide collection of a library of photos, which historically served exactly this very purpose.

Since we are interested in making comparisons over time, the Historical dataset was also filtered for items with an identifiable creation date. We carried out some preprocessing of the date metadata, retrieving four-digit years from descriptions that included them. However, much of earlier art is tagged with heterogeneous and approximate descriptions such as “early XVI century”. Discarding these made the earlier end of this dataset even smaller, which is why we limit some analyses to the 19-20th century.

limitationsOur Historical dataset is also likely biased in a number of ways. It features primarily Western art, most of the data is concentrated in the 20th century, the metadata quality varies and is of unidentifiable origin, the sampling mechanisms are unknown but likely biased by archival and selection practices of the various museums and collections in which these reproductions originate and the websites that house them. Still, from an art historical standpoint the dataset provides a reasonable and sufficient proxy benchmark to show the feasibility of our approach. Known biases of the Historical dataset include reliance on partially dated literature, including a corresponding gap of 18th century art, and very likely some variation in terms of reproduction quality due to the broad variety of the crowdsourced images, either found in the public domain or taken from a great variety of literature and online sources on the basis of fair use. Digitizing larger amounts of visual cultural heritage in high resolution, consistent quality, and minimal bias is a generational challenge. While the Historical dataset is sufficient for our proof of concept, as more data in better quality becomes available, descriptions based on our method are expected to also become more precise and representative.

4.4 Evaluation

We evaluate the compression ensemble approach extensively using three datasets and two methodologies, (1) examining correlations of our model predictions with human judgments of visual complexity, (2) using the model to perform authorship and style attribution. We show that our model performs very well on the first task and with fair accuracy on the second task (despite not being trained for the specific purpose). The second experiment also demonstrates explicit connections between specific dimensions in the vector space of compression ratios and particular aspects of the corresponding artworks. For example, the compression ratio of edge-filter transformations are informative regarding the genre of the work (portraiture versus landscape), while color-affecting transforms can help predict the medium (drawing vs oil painting).

4.4.1 Human complexity norms

We assess the cognitive plausibility of the compression ensemble approach by comparing its predictions of visual complexity with human judgment norms from two datasets. The first dataset, MultiPic [96] consists of 750 colored pictures of concrete concepts, and human judgments on various aspects of visual perception, including complexity, based on experiments with 620 participants from six language communities (British English, Spanish, French, Dutch, Italian, German; see Fig. 4.A). The dataset includes means for each image for a given language sample. The second dataset, Fractals [26] consists of 400 abstract fractals and related norms, again as mean judgments of visual complexity, by 512 German-speaking participants (Fig. 4.B). Previous research has engaged in analogous exercises of evaluations against human complexity judgements [24, 28]. We use the datasets described here as they are both publicly available while representing fairly large pools of participants.

content_tagscontent_tagsWe generate the compression ensemble vectors separately for each of the two datasets, then carry out repeated out-of-sample evaluation where we train a linear regression model on a set of vectors to predict human scores, then test its accuracy on a separate test set. The results are very good, with median absolute error ranging from 0.19 (Multipic English) to 0.23 (Multipic Flemish) on a scale of 0 to 5. To put this in perspective, this is smaller than the differences between languages in this dataset (the median standard deviation of complexity scores per image across languages is 0.24). In Fractals, median absolute error is 0.46 on the same scale of 0 to 5. The linear regression model with compression ratios as predictors describes most of the variance (measured as adjusted \(R^{2}\)) in human visual complexity ratings: 73% (Multipic Italian) to 83% (Multipic Flemish), and 32% in Fractals. By comparison, using gif compression alone describes just \(37-44\%\) (Multipic) and 10% (Fractals). These results provide us with confidence that the approach is cognitively valid, correlating with what the human eye would consider visually complex.

4.4.2 Artist, date, style, genre, and medium classification

The second evaluation involves the Historical dataset, in the form of a number of retrieval or classification experiments. We generate the compression ensemble vectors for the entire dataset, and extract the following subsets, where each included class has at least 1100 unique examples: 13 style periods as per metadata (5 of which are exemplified in see Fig. 4.C), 7 centuries, drawings vs oil paintings, landscape paintings vs human portraits, and 91 artists with at least 110 artworks each (3 of which are exemplified in see Fig. 4.D).

We perform out-of-sample evaluation where we repeatedly train a classifier for each subset, on a randomly sampled set of vectors from each class in the subset to predict the relevant class labels such as style period (\(n= 1000\) per class, except 100 for authors due to limited data), then test its accuracy on a separate test set (\(n= 100\) per class, except \(n= 10\) per author). We use Linear Discriminant Analysis — a simple, computationally lightweight supervised machine learning model that straightforwardly generalizes to multi-label classification. To probe how well the ensembles work on this task given different amounts of data and number of transforms, we carry this out in a step-wise manner, as depicted in Fig. 4.E-I. Each classifier is trained on 10, 100 and 1000 examples of each class, and employing an increasing number of transforms, starting from the baseline of gif compression (ratio to raw bitmap file size). The rest of the features are ordered by a rough estimate of variable importance (derived from repeatedly training binomial logistic regression classifiers on all possible pairs of classes and averaging the t-statistics of the variables).

content_tagscontent_tagsEven with a handful of examples and a couple of the most informative transforms, the simple classifier is able to detect above chance the creator, the date, style, genre, and medium of a given artwork. With a 100 examples and the full ensemble of transforms, author (\(n= 91\)) detection accuracy is 38%, which is much higher than the accuracy of 1.1% that random attribution would achieve by chance. Provided 1000 examples per class, oil paintings are distinguished from drawings about 86% of the time, same for landscapes vs human portraits (both have 50% random chance baseline), style period 34% (baseline \(\sim 8\%\)) and century 44% (baseline \(\sim 14\%\)).

The ranking of the transforms beyond the compression baseline (as depicted in Fig. 4.E-I) is also informative. Some of the aspects represented by the transformations are more useful than others for the prediction task; for example, gray-scaling distinguishes pencil drawings from colorful oil paintings, because this is one of the primary aspects they differ in. Turning this around, the explainable features of the compression ensemble can be used to describe how any two images (or sets of images) differ, by looking into which transformation dimensions describe the most variance.

content_tagsInspecting the relevant confusion matrices reveals the errors are fairly systematic and intuitive, as classification errors are more likely between adjacent style periods and artists. In the set of 91 artists, Thomas Lawrence and George Romney are most often confused with each other by the model — and indeed, both are portrait artists from roughly the same period (see Fig. 4.D). Conversely, artists in a distinguishable style or genre are easy to identify, for example the 19th century engraver Charles Turner is detected at 97%. Rococo, also known as Late Baroque, is correctly labeled in 47% of the tests, while 16% of it is misclassified as Baroque. Impressionism is easiest to identify (53% correct) — but confused with Post-Impressionism (14%). Expressionism is by far the hardest to put a finger on at 12% (cf. Figure S4 for the confusion matrix of style periods). These error structures are interesting in themselves, and could be investigated in future research.

It is important to be clear about the purpose of the experiments here: these are to verify that the compression ensembles approach, given a sufficient number of transformations, is capable of delineating art historically interesting aspects that may differ in aesthetic complexity. The results indicate this to be the case, and lends confidence to explorative findings in the Results section.

concept_edgesDeep learning models, which typically involve lengthy pre-training on large image databases, can also be tuned to perform similar tasks [9, 42, 97]. Our method is primarily a general zero-shot complexity estimation algorithm and does not involve pre-training; but indeed its output is informative enough to perform some such downstream tasks with reasonable accuracy. The purpose of this exercise here however is not to compete with these approaches, but to show that a compression ensemble — despite consisting of no features other than file size ratios and statistical transformations, and containing no pre-trained baseline — still captures and disambiguates enough family resemblance to place stylistically similar artworks close together and dissimilar ones apart, with a non-random error structure.

claim_relationsNevertheless, below are some analogous results, to give a sense of machine learning accuracy in similar classification tasks using similar art corpora. These are however not directly comparable to ours due to training and test set differences. Mao et al. [42] report a 39% accuracy for style period and 30% for author retrieval; Tan et al. [41] report 55% for style and 76% for artist (but that is between just 23 artists with the most training data). If for example authorship attribution was the goal of a given application, we envision that its accuracy could likely be improved by concatenating image embeddings with compression ensembles, as mentioned in the Discussion.

Availability of data and materials

The datasets supporting the conclusions of this article are available through publicly available databases, as detailed in Results and Methods sections, in particular these sources: https://deepart.hkust.edu.hk/ART500K/art500k.html (art500k), https://www.bcbl.eu/databases/multipic (Multipic stimuli) and https://osf.io/ckfmv (fractals stimuli). The R code to run the compression ensemble models as presented here is available at https://github.com/andreskarjus/compression_ensembles. See the code repository for details on running the code. The code is based on R and ImageMagick. An optimized, more efficient and modular implementation of the compression ensemble algorithm is available here: https://github.com/Collection-Space-Navigator/compression_ensembles. It is based on Python, and uses OpenCV and PIL instead of ImageMagick. An interactive web demo that includes a compression ensemble embedding of a subset of the Historical art dataset is available here: https://collection-space-navigator.github.io.

Abbreviations

non-fungible token

normalized compression distance

Principal Component Analysis

Uniform Manifold Approximation and Projection

generalized additive regression model