Chapter 5 explores the use of both symbolic/language and non-symbolic forms of communication which a virtual human might adopt. This page will no doubt break out into sub-pages as we develop more on each area. Note that the area of interest is in the communication of digital avatars within virtual worlds, or of digital avatars reaching out into the physical world (e.g. through a computer screen), not in physical robots or androids per se. Our working assumption is that if you can get something working in the virtual world then it's just an engineering challenge to get it working in the physical world!
The areas we look at in the book for non-symbolic communications are:
- Facial Expressions
- Gesture
- Stance and Body Language
Facial Expressions
There are two sides to this - being able to create facial expressions - covered in Chapter 3/Bodies & Avatars and how to read them.
A quick recap on quality of facial expression at https://www.cubicmotion.com/cm-live
A quick recap on quality of facial expression at https://www.cubicmotion.com/cm-live
Research on reading facial expressions is being driven a lot by being able to use them to drive avatars in games/virtual worlds in real-time, and just using a web cam. Examples are:
Relevant papers are:
- https://www.youtube.com/watch?v=eOjzC_NPCv8 (High Fidelity)
- https://www.auduno.com/clmtrackr/examples/clm_emotiondetection.html
- https://www.smithsonianmag.com/innovation/app-captures-emotions-real-time-180951878/ (app now dead)
- https://realpython.com/face-detection-in-python-using-a-webcam/
- https://nordicapis.com/20-emotion-recognition-apis-that-will-leave-you-impressed-and-concerned/
- https://core.ac.uk/download/pdf/160107352.pdf
- https://www.cmu.edu/news/stories/archives/2015/december/facial-image-analysis.html
- http://www.humansensing.cs.cmu.edu/node/310
- https://github.com/TadasBaltrusaitis/OpenFace
- https://www.noldus.com/facereader/facereader-online
- https://www.dotmug.net/en/2017/07/25/the-apps-that-guess-your-tastes-from-your-facial-expressions/
- https://sightcorp.com/crowdsight/
- https://link.springer.com/article/10.1007/s12652-012-0107-1
Relevant papers are:
- Benoit, A., Bonnaud, L., Caplier, A., Ngo, P., Lawson, L., Trevisan, D. G., ... & Chanel, G. (2009). Multimodal focus attention and stress detection and feedback in an augmented driver simulator.Personal and Ubiquitous Computing,13(1), 33-41.
- Ko, B. C. (2018). A Brief Review of Facial Emotion Recognition Based on Visual Information. sensors, 18(2), 401.
- Pfister, T., Li, X., Zhao, G., & Pietikäinen, M. (2011, November). Recognising spontaneous facial micro-expressions. InComputer Vision (ICCV), 2011 IEEE International Conference on Computer Vision(pp. 1449-1456). IEEE
- Poh, M. Z., McDuff, D. J., & Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. Biomedical Engineering, IEEE Transactions on58(1), 7-11
- Ramirez, G., Fuentes, O., Crites, S. L., Jimenez, M., & Ordonez, J. (2014, June). Color Analysis of Facial Skin: Detection of Emotional State. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on (pp. 474-479). IEEE.
Gesture
Again, two sides - being able to animate the limbs to create the gestures - again see Chapter 3/Bodies & Avatars, and how to read the gestures. Of course in a virtual world there may be short-circuit to all three of these tasks (expression, gesture, stance) by reading the commands to the other persons avatar or mesh if available, but it may still be simpler to handle at an image (even if not rendered) level rather than as a sequence of x/y/z transforms.
Interesting examples:
Interesting examples:
- https://medium.com/@muehler.v/simple-hand-gesture-recognition-using-opencv-and-javascript-eb3d6ced28a0
- http://www.arcsoft.com/technology/gesture.html
- https://www.youtube.com/watch?v=YLonOoXy8-Y
And papers:
Stance and Body Language
Same logic as the other two - but seems to be less studied - although there is some interesting work around detecting the intent of people to jump in front of trains/onto train tracks!
Interesting examples:
And papers:
Interesting examples:
- https://www.cmu.edu/news/stories/archives/2017/july/computer-reads-body-language.html
- https://www.techemergence.com/computer-vision-for-body-language/
And papers:
- https://www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/Body-Language-Around-the-World.aspx
- http://www.backwoodshome.com/body-language-and-threat-recognition/