Senses

This page will no doubt break out into sub-pages as we develop more on each area. Note that the area of interest is in the senses of digital avatars, or digital avatars reaching out into the physical world, not in physical robots or androids per se - although we may add a page to deal with those later.

These are the sub-headings we use in the book.

  • Sight
  • Hearing
  • Touch
  • Taste
  • Smell
  • Proprioceptive (sense of body in space)
  • Interoceptive (sense of the inner body - organs, aches, pains etc)
  • Data
    • Virtual World
    • Physical World
    • Cyber World
  • Non-Human




Sight


Sight covers work in computer vision - enabling the virtual human to make sense of webcam, CCTV, video, TV and other video feeds. Of particular interest is using video to recognise human emotions and gestures. Interesting articles and links are:





Hearing


Hearing covers the ability for the virtual human to identify different types of sounds, identify music and potentially identify environments from their soundscapes. Note that speech recognition and voice (speaker) recognition are dealt with under communications.


Interesting articles and links are:





Touch


Whilst the main focus of the book is on virtual humans, and so any physical manifestation is not of primary interest, it's interesting to see what work is going on to give robotic hands fine touch sensors. Interesting articles and links are:





Taste


Again not a primary focus as taste pre-supposes a physical existence but it's interesting that there is even work going on around this for robots, or as least for robotic/electronic devices. Interesting articles and links are:





Smell


The idea of "electronic noses" is more prevalent than that of an electronic tongue - particularly in applications like explosive or gas detection. Interesting articles and links are:





Proprioceptive


The proprioceptive sense if that of our awareness of the position and movement of our own body. Even an avatar with a virtual body may have a use for this. Interesting articles and links are:





Interoceptive


The interoceptive sense is our awareness of our internal body - our internal organs, muscles, heart-rate and so on. Possibly less relevant for a virtual human, but if the virtual human has a physiological model then the interoceptive sense is how the virtual human is made aware of what is going on in that model. Interesting articles and links are:





Data


For a virtual human their "natural" sense is that of data. At an absolute level all of the above senses are probably experienced by the virtual human as "data" - but perhaps this is no more so than the fact that all human senses are experienced by the brain as some form of electro-chemical signal. In the book we consider some of the main categories of data that a virtual human might experience, in addition to those of the more human senses.


Virtual World Data


If the virtual human is embedded in a virtual world then it may be able to "read" the data stream that represents that world - the locations and descriptions of objects and avatars and so-on (in fact not dissimilar to the the falling green binary of the Matrix! Our Halo test-bed makes use of this to circumvent the need to do things like computer vision to understand the scene in the same way that a human would. There are however some situations where rendering a scene might be a better way to make sense of it (e.g. a complex mesh may be obviously a church), or where the descriptors may be unhelpful (e.g. object1234) or plain wrong.


Physical World Data


The virtual human may read data in from the physical world. Forms such as video and audio (and other senses) have been described above.


Cyber World Data


There is no reason why the virtual human should not have a "native" connection to the Internet, so that it can consumer web pages, RSS feeds, JSON data and other net media. The virtual human again has no need to render this data (e.g. laying out a web page), but again in some cases with complex CSS it may be easier to render the web page and apply computer vision rather than to identify information clusters and importance from the native CSS markup. With cyber-data the virtual human can also engage in two-way interactions to search and query data.




Non-Human


There is of course no reason why the virtual human should be limited to human senses, in either the physical or virtual world. A virtual human could be given an electro-magnetic sense, endowed with a form of tele-empathy or mind-reading through merging net data with physical data, or using webcams to detect heart-rate and micro-gesture "tells" to better read emotions.