AI Affective Virtual Human

Researchers: Steve DiPaola, Mahdi Davoodikakhki, Andrey Goncharov, Nilay Ozge Yalcin

The Research Our open-source toolkit/cognitive research in AI-based 3D Virtual Human (embodied IVA: Intelligence Virtual Agents): a real-time system that can converse with a human by sensing their emotions and conversation (via facial emotion recognition, voice stress, semantics of the speech and words) and respond affectively, emotionally (voice, facial animation, gesture etc.) to a user in front of it via a host of gestural, motion and bio-sensor systems, with several in lab AI systems and give a coherent, personality-based conversational answers via speech, expression and gesture. The system uses Unity and SmartBody (USC) API who we have collaborated with for years. We use cognitive modeling, empathy modelling, NLP and a variety of AI-based modules in our system (see papers).

Please take a look at the videos below to see our affective real-time 3D AI virtual human setup with face emotion recognition, movement recognition and hand gesture recognition:

—— PAPERS: Virtual Humans ——

Expectations vs. Reality; The Impact of Adaptation Gap on Avatars in Social VR Platforms

by Goncharov A; Yalcin ON; DiPaola S – Conference – workshop; ACE; how Artificial Character Embodiment shapes user behaviour in multi-modal interactions. 8 pages
ACM International Conference on Multimodal Interaction (ICMI) (2023)

Expressing Personality Through Non-verbal Behavior in Real-Time Interaction

by Saberi M; DiPaola S; Bernardet U – Journal – Vol 12. 5474. 19 pages
Frontiers in Psychology (2021)

Speech Breathing in Virtual Humans; An Interactive Model and Empirical Study

by Bernardet U; Kanq S; Feng A; DiPaola S; Shapiro A – Conference – pp. 1-9. Osaka
IEEE Virtual Humans and Crowds for Immersive Environments (VHCIE) (2019)

A computational model of empathy for interactive agents

by Yalcin ON; DiPaola S – Journal – Vol 26. pp. 20-25. BICA Research Award Winning Paper
Biologically Inspired Cognitive Architectures Journal (BICA Journal) (2018)

An Eye Gaze Model for Controlling the Display of Social Status in Believable Virtual Humans

by Nixon M; DiPaola S; Bernardet U – Conference – 8 pages. Maastricht. Netherlands
IEEE Conference on Computational Intelligence and Games (IEEE CIG) (2018)

A Dynamic Speech Breathing System for Virtual Characters

by Bernardet U; Kang S; Feng A; DiPaola S; Shapiro A – Conference – pages 43-52
Intelligent Virtual Agents (IVA) (2017)

Simulink Toolbox for Real-time Virtual Character Control

by Bernardet U; Saberi M; DiPaola S – Conference – Lecture Notes in Computer Science. Springer
Intelligent Virtual Agents (IVA) (2016)

m+m; A novel Middleware for Distributed; Movement based Interactive Multimedia Systems

by Bernardet U; Adhia D; ? DiPaola S; ?; Schiphorst T – Conference – pp. 21-30
ACM Movement and Computing (MOCA) (2016)

A Framework for Exogenous and Endogenous Reflexive Behavior in Virtual Characters

by Bernardet U; DiPaola S – Conference – pp. 132-136. Springer
International Conference on Intelligent Virtual Agents (IVA) (2015)

Exploring Facial Expressions for Human-Computer Interaction; Combining Visual Face Tracking and EMG Data to Control a Flight Simulation Game

by Bayatpour S; Bernardet U; DiPaola S; Kitson A; Riecke B – Conference – Vancouver. 6 pages
International Symposium on Electronic Art (ISEA) (2015)

An Architecture for Personality-Based; Nonverbal Behavior in Affective Virtual Humanoid Character

by Saberi M; Bernardet U; DiPaola S – Journal – Special Issue; Bio Inspired Cognitive Architectures. Vol 41. pp. 204-211
Procedia Computer Science (2014)

A Model for Social Spatial Behavior in Anthropomorphic Virtual Characters

by Karimaghalou N; Bernardet U; DiPaola S – Journal – Vol 25. No 3-4. pp. 507-519
Computer Animation and Virtual Worlds Journal (2014)

Affective Response Patterns as Indicators of Personality in Virtual Characters

by Bernardet U; DiPaola S – Conference – 2 pages. Qu‚bec City
Cognitive Science Society (2014)

A Generic Approach to Challenge Modeling for the Procedural Creation of Video Game Levels

by Sorenson N; Pasquier P; DiPaola S – Journal – Vol 3. No 3. pp. 229-244
Transactions on Computational Intelligence and AI in Games (2011)

Face; Portrait; Mask Using a Parameterized System to Explore Synthetic Face Space

by DiPaola S – Conference – British Computer Society. pp. 296-302. London
Electronic Visualisation and the Arts (EVA) (2010)

Perceptually Valid Facial Expressions for Character-based Applications

by Arya A; DiPaola S; Parush A – Journal – Vol 2009. Article ID 462315. 13 pages
International Journal of Computer Games Technology (2009)

Intelligent Expression-based Character Agent Systems

by DiPaola S – Conference – Keynote Short Paper. Springer. Amsterdam. pp. 3-7
ACM Intelligent Virtual Agents (IVA) (2009)

Face Modeling and Animation Language for MPEG-4 XMT Framework

by Arya A; DiPaola S – Journal – Vol 9. No 6. pp. 1137-1146
IEEE Transactions on Multimedia (2007)

Facial Actions as Visual Cues for Personality

by Arya A; Enns JT; Jefferies L; DiPaola S – Journal – Vol 17. No 3-4. pp. 371-382
Computer Animation and Virtual Worlds (CAVW) Journal (2006)

Socially Communicative Characters for Interactive Applications

by DiPaola S; Arya A – Conference – 8 pages
Visualization and Computer Vision (2006)

Socially Expressive Communication Agents; A Face-centric Approach

by DiPaola S; Arya A – Conference – British Computer Society. 8 pages. Florence
Electronic Visualisation and the Arts (EVA) (2005)

Face; Portrait; Mask - the Virtuality of the Synthetic Face

by DiPaola S – Conference – Vancouver Art Gallery. 7 pages
Old and New Forms Conference (2004)

Face as a Multimedia Object

by Arya A; DiPaola S – Conference – 5 pages. Lisbon
International Workshop on Image Analysis for Multimedia Interactive Services (2004)

FaceSpace; A Facial Spatial-Domain Toolkit

by DiPaola S – Conference – pp. 105-109. London
IEEE Information Visualization (2002)

Book Excerpt; pp. 193-234. 446-448. Cover Art

by DiPaola S – Book – Peach Pit Press
Book; Avatars by Damer B (1998)

Book Excerpt; pp. 101-104. 214-219. cover artwork

by DiPaola S – Book – by Parke & Waters. AK Peters
Book; Computer Facial Animation (1996)

Extending the Range of Facial Types

by DiPaola S – Journal – Vol 2. No 4. pp. 129-131
IEEE Journal of Visualization and Computer Animation (1991)

3D Parameterized Facial Modeling & Animation

by DiPaola S – Conference – 8 pages
ACM SIGGRAPH - Facial Animation Tutorial (1989)