Semantic Meaning & Analysis

See Related Papers and Related Projects:
## AI-based Exploration of Art and Design ## AI Anonymization ## AI Cognitive Creativity
## Sensing Humans (Bio/Brain/Face/Movement/VR) ## AI Modeling Thought & Language

Reseachers: Steve DiPaola, Suk Choi, Meehae Song, Nouf Abukhodair, Vanessa Utz

The Research: Early Deep Learning systems are trained on huge datasets including text/image pairs that while can embed simple meaning, are still mainly trained on nouns like people and objects. Our main research area is to go beyond that to deeper Semantic Meaning & Analysis work. Which parses and models more multimodal and deeper semantic meaning to AI work. We do this by both analyzing meaning space (cognitive science of the arts, gesture, emotion, empathy, creativity, behavior etc.) but also then build models from cognitively and rigorously mapping these spaces that we can then use in our new emerging systems (with application in health, the arts, and education).

See samples of our work in the figures below:


Our work into a new method to understand, measure and model “aesthic emotion” (see studies and papers): Figure 1: Overview of the Emotion Wheel created by Song & Abukhodair

Figure 2: Explanation of the Emotion Wheel

Figure 3: Explanation of the Emotion Wheel - Secondary Emotion Selection


Figure 4: Reverse engineering generative visual AI systems to better understand current meaning making to improve them. Here an emotionally complex prompt “an angry man punching a bag in a crowded joyful gym”.

Figure 5: The diffusion based output where we then used an advanced system to reverse detect and heatmap the meaning of every word back through the system.




Deep studies with eye tracking, on how an art viewer perceives (cognitively) artwork where we AI redraw masterwork (switch left/right eye detail & lost and found edges near chin). See several papers and over 200 press articles on our findings - like this one Magic of Rembrandt’s Painting Technique Revealed: Figure 6: Diagram from eye-tracking study showing that greater detail on a painted eye leads to quicker, longer and more frequent fixations.


Additional studies on deep meaning in aesthics, emotions, creativity, empathy etc ( see papers): Figure 7: Desk setup for study on an artist’s creative process.

Figure 8: Diagram of eye-tracking data during an artist’s creative process during a sketching task.

Figure 9: Outling the Source-Path-Goal Schema as applied to creative interactivity.

Video 1: Video of the study shown in Figures 7, 8 and 9.

Video 2: Video of demo - system to track facial emotion expression to adapt visual appearance of video (to reflect subject’s emotional state).




—— PAPERS: Semantic Meaning & Analysis ——

Designing a Wheel-based Assessment Tool to Measure Visual Aesthetic Emotions

by Abukhodair N; Song M; DiPaola S – Journal – 23 pages
Cognitive Systems Research (2023)

Lucid Loop; A Virtual Deep Learning Biofeedback System for Lucid Dreaming Practice

by Kitson A; DiPaola S; Riecke B – Conference – Paper LBW1322. 6 pages
ACM CHI Conference on Human Factors in Computing Systems (CHI) (2019)

Deep Learning for Classification of Peak Emotions within Virtual Reality Systems

by Quesnel D; DiPaola S; Rieke B – Conference – pp. 6-11
Semantic Ambient Media Experiences (SAME); Artificial Intelligence MEETS Virtual and Augmented Worlds (AIVR) with SIGGRAPH Asia (2018)

Engagement with artificial intelligence through natural interaction models

by Feldman S; Yalcin ON; DiPaola S – Conference – British Computer Society. pp. 296-303
Electronic Visualisation and the Arts (EVA) (2017)

Movement Awareness through Emotion Based Aesthetic Visualization

by Salevati S; DiPaola S; Carlson K – Conference – British Computer Society. 8 pages. London
Electronic Visualisation and the Arts (EVA) (2016)

Touch of the Eye; Does Observation Reflect Haptic Metaphors in Art Drawing?

by Choi S K; DiPaola S – Conference – pp. 579-588
ACM Conf on Human Factors in Computing Systems (CHI) (2015)

A Creative Artificial Intelligence System to Investigate User Experience; Affect; Emotion and Creativity

by Salevati S; DiPaola S – Conference – British Computer Society. 8 pages. London
Electronic Visualisation and the Arts (EVA) (2015)

Enhancing Viewer's Emotional Connections to The Traditional Art Creative Process Via an AI Interactive System

by Salevati M; DiPaola S – Conference – British Computer Society. 8 pages. London
Electronic Visualisation and the Arts (EVA) (2015)

Using a Contextual Focus Model for an Automatic Creativity Algorithm to Generate Art Work

by DiPaola S – Journal – Special Issue; Bio Inspired Cognitive Architectures. Vol 41. pp. 212-219
Procedia Computer Science (2014)

Face; Portrait; Mask - Using a Parameterized System to Explore Synthetic Face Space

by DiPaola S – Book – Bowen. Keene. Ng. (Eds). pp. 213-227. Springer
Book Chapter. Electronic Visualisation in Arts and Culture (2013)

The Role of Micronarrative in the Design and Experience of Digital Games

by Bizzocchi J; Nixon M; DiPaola S; Funk N – Conference – Atlanta. Georgia. pp. 161-197
Digital Games Research Association Conference (DIGRA) (2013)

How a Painter Paints; An Interdisciplinary Understanding of Embodied Creativity

by Choi S K; DiPaola S – Conference – British Computer Society. pp. 127-134. London
Electronic Visualisation and the Arts (EVA) (2013)

How Did Humans Become So Creative? A Computational Approach

by Gabora L; DiPaola S – Conference – pp. 203-211
International Conference on Computational Creativity (ICCC) (2012)

Exploring the Effect of Color Palette in Painterly Rendered Character Sequences

by Seifi H; DiPaola S; Enns JT – Conference – pp. 89-97
International Symposium on Computational Aesthetics in Graphics; Visualization & Imaging (2012)

Formalizing an Interconnected Syntax For Picassos Creative Process In Producing Guernica

by DiPaola S; Smith A – Conference – 6 pages
Conceptual Structure; Discourse and Language (2012)

The Tacit and the Trace; Towards Syntax Of The Creative Act

by Choi S K; DiPaola S; Schiphorst T – Conference – 6 pages
Conceptual Structure; Discourse and Language (2012)

Rembrandts Textural Agency; A Shared Perspective in Visual Art and Science

by DiPaola S; Riebe C; Enns JT – Journal – Vol 43. No 3. pp. 145-151
Leonardo (2011)

Following the Masters; Viewer Gaze is Directed by Relative Detail in Painted Portraits

by Riebe C; DiPaola S; Enns JT – Journal – Vol 9. No 8. pp. 368-368 (abstract)
Journal of Vision (2009)

Incorporating Characteristics of Human Creativity into an Evolutionary Art Algorithm

by DiPaola S; Gabora L – Journal – Vol 10. No 2. pp. 97-110
Genetic Programming and Evolvable Machines Journal (2009)

Following the masters; Viewer gaze is directed by relative detail in painted portraits

by Riebe C; DiPaola S; Enns JT – Conference – 9th Annual Meeting.
Abstracts of the Vision Sciences Society (2009)

Quantifying artist's use of human vision constructs to influence viewer eye gaze

by DiPaola S – Conference – 6 pages
SPIE Human Vision and Imaging. Int. Society for Optical Engineering (2009)

Darwins Enduring Legacy

by DiPaola S – Journal – Images of my research in computer model of evolution ? selected by the Nature editors to accompany this essay (invited - not peer reviewed). Vol 451. pp. 632-633
The Journal Nature (2008)

The Trace and the Gaze; Textural Agency in Rembrandt?s Late Portraiture from a Vision Science Perspective

by DiPaola S – Conference – British Computer Society. 8 pages. London
Electronic Visualisation and the Arts (EVA) (2008)

Emotional Remapping of Music to Facial Animation

by DiPaola S; Arya A – Conference – pp. 143-149
ACM SIGGRAPH - Symposium on Videogames (2006)

Evolving Portrait Painter Programs using Genetic Programming to Explore Computer Creativity

by DiPaola S – Conference – 7 pages
International Digital Media and Arts Association (IDMA) (2006)

Simulating Face to Face Collaboration for Interactive Learning Systems

by DiPaola S; Arya A; Chan J – Conference – 6 pages. Vancouver
E-Learn 2005 (2005)

Towards an Interactive Visualization of Game Design Patterns

by Tolmie J; DiPaola S; Charles A – Conference – Vancouver
Digital Games Research Association (2005)

Affective Communication Remapping in MusicFace System

by DiPaola S; Arya A – Conference – British Computer Society. 8 pages. London
Electronic Visualisation and the Arts (EVA) (2004)

Investigating Face Space

by DiPaola S – Conference – pp. 207-207
ACM SIGGRAPH - Conference Abstracts and Applications (2002)