It is well known that a cross-modal perception of auditory and visual stimuli does not equal a mere summation of individual auditory and visual perceptions. As with the Gesamtgestalt of the perceptual mechanism, sounds are not apprehended individually, but shaped as a Tongestalt, in which auditory and visual stimuli are combined audio-visually, similarly to the perception of auditory apparent motion.

In recent scientific research, audio-visual perception has been usually conceptualised in relation to synaesthesia. However, the notion of synaesthesia refers to individual nerve structures, which do not encompass the implications of Gesamtgestalt. In fact, the latter describes a selective sensory property within the perceptual process, which can be observed in every human being. Recent neurobiological studies seem to underline the universality or non-individuality of cross modal perception. Therefore, we are looking for an alternative hypothesis that explains what happens, when auditory and visual stimuli are perceived at once.

Naturally, it is easy to conduct a perceptual study that records what is shown [input] on the one hand and what is perceived

In the case of motion, perception follows a horizontal timeline according to the physical continuity of motion. However, sound features a physical continuity as well: It moves both on the horizontal and the vertical timeline. While listening to music, the audience follows automatically, but the brain perceives the structure of music congruently in temporal and constitutive in non-temporal. Also, each sounds, articulation, melody and rhythm are is perceived in relation to a prior and consequent element of the same structure. This, in turn, indicates that during the experience of music our senses enter another world, where present, past and future coexist simultaneously, which further complicates our understanding of audio-visual perception and the ruling mechanism behind it.

All of this leads to the question of how to figure the Gesamtgestalt of perception. In our approach we focus on the cutting point of audio-visual perception and try to embody a cross-modal Gesamtgestalt that links sound (=tone) and vision (=motion) in the perceptual process. In this context, the art-machine “Visible Canon” is conceived of as an experimental device for the exploration of audio-visual perception.

The machine looks like a rolling ball structure that plays music and works like a three-dimensional chart. It plays three tracks of various Canons, including Johan Pachelbel’s Canon, and displays the kinetic movement of an iron ball at the same time. Through the combination of the auditory and the visual structure of the canon the uncertainty and complementary relation of the stimuli calls up the Gesamtgestalt of audio-visual perception.

In the case of Pachelbel’s Canon, it might be less important to envision its structure visually and kinetically at the same time, since it will be perceived automatically as rhythm and melody. However, if a Canon has a more complicated composition, rhythm and melody cannot be followed as easily; in the process, the visual information provides the viewing audience with a powerful key to the understanding the underlying structure of the Canon on the basis of the cross-modal Gesamtgestalt.

With the above method we endeavour to embody the Gesamtgestalt of audio-visual perception. At the same time we have explored the new role of art in modern sciences that provides an experimental device able to overcome the shortcomings of the classical methods.

Researchers working with digital data sets fall into a rabbit hole of visual image immersion. As a bygone world takes shape before their eyes, they loose their bearings, let go of “now” and respond to “then.” When these “time travelers” reemerge how do they tell their tale? In the digital age, readers can experience the primary sources that previously were only available second-hand, filtered through a historian’s prose. Visual artifacts, when arranged by knowing hands, communicate historical narrative directly with a sensory form of knowledge.

Visual narrative is atmospheric rather than precise, characterized by ambience, aura, character, feel, mood, tone, and undercurrent. The grammar of visual narrative derives from the source materials made by vanished citizens of the time period in question and readers are literally able to see for themselves. Like the researchers, readers are also asked to let go of their moorings, become comfortable with a state of not knowing in order to discover the unknown. Authors are charged with creating this exploratory state in readers using emerging models of visual narrative that are not only technically innovative, but also informed by the ways knowledge is evolving toward visual thinking in response to today’s “visual tsunami.”

China’s Boxer Uprising of 1900 is the case study for this research. When an unprecedented military alliance of eight international powers overran Beijing, the global response generated a vast visual record that is now being collected in the digital medium. Immersion in these visual data sets mirrors the pursuit of knowledge in science as described by neuroscientist Stuart Firestein in his 2012 book, Ignorance: How it Drives Science. Firestein writes: “The facts serve mainly to access the ignorance….science traffics in ignorance, cultivates it, and is driven by it. Mucking about in the unknown is an adventure…” and asks “Can we construct an epistemology of ignorance like we have one for knowledge?”

Digital visual narrative is in its infancy, but the awakening of the visual sense in social/cultural historical research is growing within what is traditionally a vehemently text-based field. My own project at MIT, Visualizing Cultures, is in its tenth year of collaboration with historians to explore the visual record through new media. Earlier at MIT, Gyorgy Kepes, founder of the Center for Advanced Visual Studies, included in his series on art and science, a volume called Education of Vision (1965). Essays by such scholars as Rudolf Arnheim shine light on the secondary, hidden, and often denied role of visual thinking. Now, like driving a car in a new place, states of ignorance and disorientation indicate a move toward a sense-based awareness with which the mental mastery of facts must collaborate. This bilateral imaginary is at the root of visual narrative.

This practice has also extended to collecting every text message the couple have sent each other since 2006, a collection that has formed the material for the performance ‘A Message To You’ in which a year’s worth of text messages are shouted out in a public space, as well as ‘The printing of every text message we sent each other since 2007′, a work for dot matrix printer that was seen at the exhibition ‘Tracing Mobility’ at the Haus der Kulturen der Welt in December 2011.

Multi-disciplinary artist Gretta Louw devised her durational, online performance, Controlling_Connectivity, as a ‘self-experiment’ testing the psychological burden associated with our increasingly ‘online’ lives. The rapid penetration of these technologies into all facets of life was taken to the extreme as the artist completely surrendered her time to unknown internet users. From 2-12 November 2011, she lived in total isolation within the gallery space, except interaction via the internet. Supplies were stored within the gallery with no one entering or leaving during the performance, and the windows were blacked-out to preclude and normalising effects of natural light or outside rhythms.

For 240 hours Louw lived in a state of constant online readiness, responding immediately to Facebook, Skype, Google +, or Twitter contact. She conducted interviews with extreme Second Life users, whose concepts of ‘family’ challenge the idea of mutually exclusive ‘real’ and ‘virtual’ worlds and lend weight to DeLanda’s term “real virtuality”; she investigated the rapidly forming world of online video ‘hangouts’ on the Google + platform, exploring the fanatical brand loyalty and undercurrent of ‘internet stardom’; she celebrated her 30th birthday digitally from within the gallery, testing the limits of online mediums in conveying emotion and replacing physical closeness.

Subsequently an exhibition at Art Laboratory Berlin sought to interpret the performance and present the viewer with a glimpse into the psychological and physical experience of 240 hours online. In early 2012 the artist released a book, which further analysed the results of the self-experiment, incorporating essays from curators Regine Rapp and Christian de Lutz as well as from media theorists and both online and traditional psychology professionals. The book also presented results from pre- and post-performance psychological testing that the artist underwent to measure psychological factors such as depression, general well-being, and quality of life – the results were quite startling, showing a significant reduction in well-being and an increase in stress and anxiety post-performance.

The pressure to be connected/online/constantly available has never been so great, and is only growing with the advent of smartphones and tablets. It is argued that participation in the elaborate communication networks that now underlie social interaction is no longer voluntary, since failure to participate is akin to social withdrawal. The consequent erosion of true leisure time, the blurring of the traditional professional/personal, public/private dichotomies and an information overload are creating hitherto unknown levels of psychological pressure as the debate about a so-called ‘internet addiction’ disorder rages in the academic community .

As psychoanalyst and media curator/theorist Leon Tan writes “The chief merit of the [Controlling_Connectivity] project is perhaps in pushing us to take our networked experiences seriously, to reflect on the consequences of reconfiguring co-presence.” Curator Christian de Lutz comments:

“What would be the cost, professionally and personally, if we ignored various social networks? We almost seem to have forgotten that hardly any of this existed 20 years ago. In all the speed of acclimatising to new technologies, we have had little, if any time to reflect on how it has changed us; changed our minds, souls and our society. Gretta Louw’s project Controlling_Connectivity provides a unique and long overdue starting point.”

What are the revolutionary, democratising aspects of online social media and is it all as utopian as some commentators would have us believe? Is there such a thing as ‘internet addiction’; are we creating new psychological disorders through our fanatical adherence to a path of technological advancement? If so, is it worth it? At MutaMorphosis the results of Controlling_Connectivity will be discussed with a view to analysing the role of social media and the internet in, as the MutaMorphosis website calls it, “a world undergoing a complete mutation”.