These are the sub-headings we use in the book.
Sight covers work in computer vision - enabling the virtual human to make sense of webcam, CCTV, video, TV and other video feeds. Of particular interest is using video to recognise human emotions and gestures. Interesting articles and links are:
Hearing covers the ability for the virtual human to identify different types of sounds, identify music and potentially identify environments from their soundscapes. Note that speech recognition and voice (speaker) recognition are dealt with under communications.
Interesting articles and links are:
Whilst the main focus of the book is on virtual humans, and so any physical manifestation is not of primary interest, it's interesting to see what work is going on to give robotic hands fine touch sensors. Interesting articles and links are:
Again not a primary focus as taste pre-supposes a physical existence but it's interesting that there is even work going on around this for robots, or as least for robotic/electronic devices. Interesting articles and links are:
The idea of "electronic noses" is more prevalent than that of an electronic tongue - particularly in applications like explosive or gas detection. Interesting articles and links are:
The proprioceptive sense if that of our awareness of the position and movement of our own body. Even an avatar with a virtual body may have a use for this. Interesting articles and links are:
The interoceptive sense is our awareness of our internal body - our internal organs, muscles, heart-rate and so on. Possibly less relevant for a virtual human, but if the virtual human has a physiological model then the interoceptive sense is how the virtual human is made aware of what is going on in that model. Interesting articles and links are:
If the virtual human is embedded in a virtual world then it may be able to "read" the data stream that represents that world - the locations and descriptions of objects and avatars and so-on (in fact not dissimilar to the the falling green binary of the Matrix! Our Halo test-bed makes use of this to circumvent the need to do things like computer vision to understand the scene in the same way that a human would. There are however some situations where rendering a scene might be a better way to make sense of it (e.g. a complex mesh may be obviously a church), or where the descriptors may be unhelpful (e.g. object1234) or plain wrong.
The virtual human may read data in from the physical world. Forms such as video and audio (and other senses) have been described above.
There is no reason why the virtual human should not have a "native" connection to the Internet, so that it can consumer web pages, RSS feeds, JSON data and other net media. The virtual human again has no need to render this data (e.g. laying out a web page), but again in some cases with complex CSS it may be easier to render the web page and apply computer vision rather than to identify information clusters and importance from the native CSS markup. With cyber-data the virtual human can also engage in two-way interactions to search and query data.
There is of course no reason why the virtual human should be limited to human senses, in either the physical or virtual world. A virtual human could be given an electro-magnetic sense, endowed with a form of tele-empathy or mind-reading through merging net data with physical data, or using webcams to detect heart-rate and micro-gesture "tells" to better read emotions.
- Sight
- Hearing
- Touch
- Taste
- Smell
- Proprioceptive (sense of body in space)
- Interoceptive (sense of the inner body - organs, aches, pains etc)
- Data
- Virtual World
- Physical World
- Cyber World
- Non-Human
Sight
- Sover (2015) - COMPARATIVE STUDY OF DIFFERENT EMOTIONS DETECTION METHODS
- Poh, M. Z., McDuff, D. J., & Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE transactions on biomedical engineering, 58(1), 7-11.
- Lozano-Monasor, E., López, M. T., Fernández-Caballero, A., & Vigo-Bustos, F. (2014). Facial expression recognition from webcam based on active shape models and support vector machines. In Ambient Assisted Living and Daily Activities (pp. 147-154). Springer International Publishing. London, UK.
Hearing
Hearing covers the ability for the virtual human to identify different types of sounds, identify music and potentially identify environments from their soundscapes. Note that speech recognition and voice (speaker) recognition are dealt with under communications.
Interesting articles and links are:
- Shazam - music recognition. Imagine a VH hooking that up to Discogs (https://www.discogs.com/)
- Lin, Y. C., Yang, Y. H., & Chen, H. H. (2011). Exploiting online music tags for music emotion classification. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 7(1), 26. ACM. New York, USA.
- Lu, H., Pan, W., Lane, N. D., Choudhury, T., & Campbell, A. T. (2009, June). SoundSense: scalable sound sensing for people-centric applications on mobile phones. In Proceedings of the 7th international conference on Mobile systems, applications, and services (pp. 165-178). ACM. New York, USA.
Touch
Whilst the main focus of the book is on virtual humans, and so any physical manifestation is not of primary interest, it's interesting to see what work is going on to give robotic hands fine touch sensors. Interesting articles and links are:
- Bartolozzi (2016) - Robots with a sense of touch.
Taste
Again not a primary focus as taste pre-supposes a physical existence but it's interesting that there is even work going on around this for robots, or as least for robotic/electronic devices. Interesting articles and links are:
- Ceto (2017) - Electronic tongues to assess wine sensory descriptors
- Veloso (2018) - Honey Evaluation Using Electronic Tongues: An Overview
Smell
The idea of "electronic noses" is more prevalent than that of an electronic tongue - particularly in applications like explosive or gas detection. Interesting articles and links are:
- Electronic noses overview - https://www.sciencedirect.com/topics/medicine-and-dentistry/electronic-nose
- Loutfi (2015) - Electronic noses for food quality: A review
- Seesaard (2015) - Development of Fabric-Based Chemical Gas Sensors for Use as Wearable Electronic Noses
Proprioceptive
The proprioceptive sense if that of our awareness of the position and movement of our own body. Even an avatar with a virtual body may have a use for this. Interesting articles and links are:
- Proske (2015) - The role of muscle proprioceptors in human limb position sense: a hypothesis
- TechExplore (30 Jan 19) - A step closer to self-aware machines—engineers create a robot that can imagine itself
Interoceptive
The interoceptive sense is our awareness of our internal body - our internal organs, muscles, heart-rate and so on. Possibly less relevant for a virtual human, but if the virtual human has a physiological model then the interoceptive sense is how the virtual human is made aware of what is going on in that model. Interesting articles and links are:
- Forkmann (2016) - Making sense of what you sense: Disentangling interoceptive awareness, sensibility and accuracy
Data
For a virtual human their "natural" sense is that of data. At an absolute level all of the above senses are probably experienced by the virtual human as "data" - but perhaps this is no more so than the fact that all human senses are experienced by the brain as some form of electro-chemical signal. In the book we consider some of the main categories of data that a virtual human might experience, in addition to those of the more human senses.
Virtual World Data
If the virtual human is embedded in a virtual world then it may be able to "read" the data stream that represents that world - the locations and descriptions of objects and avatars and so-on (in fact not dissimilar to the the falling green binary of the Matrix! Our Halo test-bed makes use of this to circumvent the need to do things like computer vision to understand the scene in the same way that a human would. There are however some situations where rendering a scene might be a better way to make sense of it (e.g. a complex mesh may be obviously a church), or where the descriptors may be unhelpful (e.g. object1234) or plain wrong.
Physical World Data
The virtual human may read data in from the physical world. Forms such as video and audio (and other senses) have been described above.
Cyber World Data
There is no reason why the virtual human should not have a "native" connection to the Internet, so that it can consumer web pages, RSS feeds, JSON data and other net media. The virtual human again has no need to render this data (e.g. laying out a web page), but again in some cases with complex CSS it may be easier to render the web page and apply computer vision rather than to identify information clusters and importance from the native CSS markup. With cyber-data the virtual human can also engage in two-way interactions to search and query data.
Non-Human
There is of course no reason why the virtual human should be limited to human senses, in either the physical or virtual world. A virtual human could be given an electro-magnetic sense, endowed with a form of tele-empathy or mind-reading through merging net data with physical data, or using webcams to detect heart-rate and micro-gesture "tells" to better read emotions.