AI Affective Virtual Human

Researchers: Steve DiPaola, Mahdi Davoodikakhki, Andrey Goncharov, Nilay Ozge Yalcin

The Research Our open-source toolkit/cognitive research in AI-based 3D Virtual Human (embodied IVA: Intelligence Virtual Agents): a real-time system that can converse with a human by sensing their emotions and conversation (via facial emotion recognition, voice stress, semantics of the speech and words) and respond affectively, emotionally (voice, facial animation, gesture etc.) to a user in front of it via a host of gestural, motion and bio-sensor systems, with several in lab AI systems and give a coherent, personality-based conversational answers via speech, expression and gesture. The system uses Unity and SmartBody (USC) API who we have collaborated with for years. We use cognitive modeling, empathy modelling, NLP and a variety of AI-based modules in our system (see papers).

Please take a look at the videos below to see our affective real-time 3D AI virtual human setup with face emotion recognition, movement recognition and hand gesture recognition:

------ PAPERS: Virtual Humans ------

Expectations vs. Reality; The Impact of Adaptation Gap on Avatars in Social VR Platforms by Goncharov A; Yalcin ON; DiPaola S – Conference – workshop; ACE; how Artificial Character Embodiment shapes user behaviour in multi-modal interactions. 8 pages
ACM International Conference on Multimodal Interaction (ICMI) (2023)

Expressing Personality Through Non-verbal Behavior in Real-Time Interaction by Saberi M; DiPaola S; Bernardet U – Journal – Vol 12. 5474. 19 pages
Frontiers in Psychology (2021)

Speech Breathing in Virtual Humans; An Interactive Model and Empirical Study by Bernardet U; Kanq S; Feng A; DiPaola S; Shapiro A – Conference – pp. 1-9. Osaka
IEEE Virtual Humans and Crowds for Immersive Environments (VHCIE) (2019)

A computational model of empathy for interactive agents by Yalcin ON; DiPaola S – Journal – Vol 26. pp. 20-25. BICA Research Award Winning Paper
Biologically Inspired Cognitive Architectures Journal (BICA Journal) (2018)

An Eye Gaze Model for Controlling the Display of Social Status in Believable Virtual Humans by Nixon M; DiPaola S; Bernardet U – Conference – 8 pages. Maastricht. Netherlands
IEEE Conference on Computational Intelligence and Games (IEEE CIG) (2018)

A Dynamic Speech Breathing System for Virtual Characters by Bernardet U; Kang S; Feng A; DiPaola S; Shapiro A – Conference – pages 43-52
Intelligent Virtual Agents (IVA) (2017)

Simulink Toolbox for Real-time Virtual Character Control by Bernardet U; Saberi M; DiPaola S – Conference – Lecture Notes in Computer Science. Springer
Intelligent Virtual Agents (IVA) (2016)

m+m; A novel Middleware for Distributed; Movement based Interactive Multimedia Systems by Bernardet U; Adhia D; ? DiPaola S; ?; Schiphorst T – Conference – pp. 21-30
ACM Movement and Computing (MOCA) (2016)

A Framework for Exogenous and Endogenous Reflexive Behavior in Virtual Characters by Bernardet U; DiPaola S – Conference – pp. 132-136. Springer
International Conference on Intelligent Virtual Agents (IVA) (2015)

Exploring Facial Expressions for Human-Computer Interaction; Combining Visual Face Tracking and EMG Data to Control a Flight Simulation Game by Bayatpour S; Bernardet U; DiPaola S; Kitson A; Riecke B – Conference – Vancouver. 6 pages
International Symposium on Electronic Art (ISEA) (2015)

An Architecture for Personality-Based; Nonverbal Behavior in Affective Virtual Humanoid Character by Saberi M; Bernardet U; DiPaola S – Journal – Special Issue; Bio Inspired Cognitive Architectures. Vol 41. pp. 204-211
Procedia Computer Science (2014)

A Model for Social Spatial Behavior in Anthropomorphic Virtual Characters by Karimaghalou N; Bernardet U; DiPaola S – Journal – Vol 25. No 3-4. pp. 507-519
Computer Animation and Virtual Worlds Journal (2014)

Affective Response Patterns as Indicators of Personality in Virtual Characters by Bernardet U; DiPaola S – Conference – 2 pages. Qu‚bec City
Cognitive Science Society (2014)

A Generic Approach to Challenge Modeling for the Procedural Creation of Video Game Levels by Sorenson N; Pasquier P; DiPaola S – Journal – Vol 3. No 3. pp. 229-244
Transactions on Computational Intelligence and AI in Games (2011)

Face; Portrait; Mask Using a Parameterized System to Explore Synthetic Face Space by DiPaola S – Conference – British Computer Society. pp. 296-302. London
Electronic Visualisation and the Arts (EVA) (2010)

Perceptually Valid Facial Expressions for Character-based Applications by Arya A; DiPaola S; Parush A – Journal – Vol 2009. Article ID 462315. 13 pages
International Journal of Computer Games Technology (2009)

Intelligent Expression-based Character Agent Systems by DiPaola S – Conference – Keynote Short Paper. Springer. Amsterdam. pp. 3-7
ACM Intelligent Virtual Agents (IVA) (2009)

Face Modeling and Animation Language for MPEG-4 XMT Framework by Arya A; DiPaola S – Journal – Vol 9. No 6. pp. 1137-1146
IEEE Transactions on Multimedia (2007)

Facial Actions as Visual Cues for Personality by Arya A; Enns JT; Jefferies L; DiPaola S – Journal – Vol 17. No 3-4. pp. 371-382
Computer Animation and Virtual Worlds (CAVW) Journal (2006)

Socially Communicative Characters for Interactive Applications by DiPaola S; Arya A – Conference – 8 pages
Visualization and Computer Vision (2006)

Socially Expressive Communication Agents; A Face-centric Approach by DiPaola S; Arya A – Conference – British Computer Society. 8 pages. Florence
Electronic Visualisation and the Arts (EVA) (2005)

Face; Portrait; Mask - the Virtuality of the Synthetic Face by DiPaola S – Conference – Vancouver Art Gallery. 7 pages
Old and New Forms Conference (2004)

Face as a Multimedia Object by Arya A; DiPaola S – Conference – 5 pages. Lisbon
International Workshop on Image Analysis for Multimedia Interactive Services (2004)

FaceSpace; A Facial Spatial-Domain Toolkit by DiPaola S – Conference – pp. 105-109. London
IEEE Information Visualization (2002)

Book Excerpt; pp. 193-234. 446-448. Cover Art by DiPaola S – Book – Peach Pit Press
Book; Avatars by Damer B (1998)

Book Excerpt; pp. 101-104. 214-219. cover artwork by DiPaola S – Book – by Parke & Waters. AK Peters
Book; Computer Facial Animation (1996)

Extending the Range of Facial Types by DiPaola S – Journal – Vol 2. No 4. pp. 129-131
IEEE Journal of Visualization and Computer Animation (1991)

3D Parameterized Facial Modeling & Animation by DiPaola S – Conference – 8 pages
ACM SIGGRAPH - Facial Animation Tutorial (1989)