| |
|
Introduction
3D computer graphics routines can be characterised on a spectrum from
physics to faking it. The more physics involved the more realistic, though
slow, while the more faking the faster. We also see that the more faking
required, the more the sensibilities of the artist are needed. Real-time
virtual reality, with its high demands on processing power, involves more
faking it than in other 3D computer graphics applications, but the fact
is that the artist's eye is needed across the spectrum, and this is only
for appearances: the artist's wider cultural antennae are also
vital for content.
This paper examines the range of the artistic skills necessary in VR,
and at the same time looks at some of the philosophical implications of
the technology.
What is VR?
There are different ways of defining virtual reality. For the purposes
of this paper I will consider it simply as an interactive technology that
recreates some aspect of human experience. Not all would agree that interactivity
is essential to virtual reality systems, but without it we leave out a
prime element in human experience: free will. Free will, from a philosophical
standpoint, is a difficult thing to pin down, but a virtual reality system
that leaves out any possibility for the user to make an input turns it
into a version of cinema. However, as this paper focuses on the visual
clues that make us accept a computer-generated version of reality as worth
engaging with, the interactivity question is not so central. How systems
can model aspects of human experience and what 'real' reality is becomes
relevant however.
Reality and the Role of the Brain
One of the reasons why the subject of virtual reality is so interesting
is that it can challenge our thinking about ordinary reality. In fact,
the question of what 'reality' is and to what extent the mind and brain
construct it for us is one that has engaged thinkers all through history,
and recent research has made the answers no easier to come by. Some groups,
often with a religious or mystical background, suggest that the world
we encounter through our sense perceptions is less real than their essences
(Plato's 'forms' [1]), while
others propose a spirit world which is more 'real' to them than the material
(Rudolf Steiner for example [2]). The Scottish philosopher David Hume
was much taken up with the problem of how we seem to construct an orderly
world from disparate and jumbled sense impressions; he found that the
more one concentrated on them the more the world seemed to fall apart
in fact. In the end he despaired of a solution and suggested (perhaps
ironically) that 'the only remedy was inattention'. The work of contemporary
philosopher Mary Warnock has explored this theme further, with reference
also to Kant [3]. Most empiricists however are quite certain that there is
a real object world 'out there'. Empiricists, unlike Plato and his followers,
believe that knowledge of the world can only come from careful observation
of it as presented through the five senses, and through instruments, such
as the telescope or microscope, which extend them. Interestingly not all
scientists hold this view, Erwin Schroedinger for example saying that
'the world is given to me only once, not one existing and one perceived'
[4]. To firmly hold either one or the other view seems unreasonable
to me however; it is more interesting to suspend final judgement for the
moment and look at what open-minded research is telling us.
One interesting area of research that poses problems for the question
of a 'reality-out-there' is dream studies. Brain researchers have found
that the chemical and electrical activities in the human brain are different
and distinct in the three main states of consciousness: waking, REM sleep,
and dreamless sleep. (REM sleep is named after the Rapid Eye Movements
which takes place in dreaming). Research has shown that the visual cortex
of the brain (the part involved in vision) is stimulated in dreaming as
if impulses were coming from the eyes: the brain is 'constructing' a visual
reality along with the tactile and emotional drama of the dream. Even
more surprising is the fact that the dreamer responds to the dream environment
by generating motor signals that would normally cause walking, running,
fighting and so on. A chemical that floods the brain during sleeping normally
inhibits the transmission of these motor impulses to the rest of the body,
though with one exception: the eyes. Technically the body is said to be
in a state of suppressed muscle tone. Sometimes, if the impulses are too
strong (as in a nightmare perhaps) the dreamer may move or speak, or even
wrestle with phantom opponents. Genuine sleepwalking is an extreme case
of this. Interestingly, the same is true of many higher mammals, cats
for instance. One can often observe a cat's tail twitch as it sleeps,
and science has shown that it must be dreaming because all the chemical
and electrical states in the cat's brain correspond closely to those in
humans.
A form of dreaming where the participant can control their dream environment
is called lucid dreaming, and has been the subject of recent research.
It has been discovered that most people make good subjects for lucid dreaming
and can be trained to gain degrees of control over their dreams. They
have even been able to receive messages from lucid dreamers via eye movements,
as these do not share the suppressed muscle tone of the rest of the body.
The messages have been used to confirm events in dreams, and these moments
can be correlated with measured brain activity and the dreamer's subsequent
account of events. A group in the US have been working with artists and
lucid dreaming with interesting results on their paintings.[5].
The conclusion of some who are involved in this research is that 'waking
is like dreaming with the imagination suppressed, while dreaming is like
waking with muscle tone suppressed.' The relevance of this to virtual
reality is partly philosophical, and partly practical, as there is no
doubt that the impetus of VR research will be to move inexorably from
modern total-immersion systems involving computer graphics visual systems
to what is loosely called 'brain-stem' virtual reality where the user
is 'plugged in' directly to the technology. Such possibilities remain
at present in the domain of science fiction, such as Neuromancer, [6] or in the wildly speculative physics of Frank Tipler in The
Physics of Immortality. [7]
A Taxonomy of VR
Before contemplating the future possibilities of brain-stem VR, let us
draw up a rough and ready taxonomy of virtual reality systems, so that
we can better see where computer imaging fits in at present.
·
text-based, such as Muds and Moos over the Internet: keyboard
and on-screen text - input via keyboard, feedback via text on screen
· graphics-based, such as games: input via mouse or
joystick, feedback via graphics on screen, sound cues
· professional simulators, such as aircraft, ship, tank
and road vehicle systems: input via mock-driving controls such as
steering wheel, rudder, flight console, feedback via graphics on wide
screen, with the addition of sound and possibly kinetics such as cockpit
judder, lean etc.
· low-level immersion, such as homebrew VR kits, arcade
VR games consoles: input via a range of devices including the dataglove,
bodysuit, spatial positioning sensors, feedback from head-up display,
audio cues.
· high-level or 'total' immersion: same as the
previous category but using state of the art devices and supercomputer
processing power.
· brain-stem VR: a direct connection between a computer
and our brain via synaptic connections (an idea popularised by William
Gibson in Neuromancer).
· up-loading the mind: leaving the body behind altogether
and digitally encoding (uploading) our personalities into the computer
(a highly speculative idea proposed by the physicist Frank Tipler).
Constructing a World
Whatever level of virtual reality we deal with the creators of the system
need to construct an artificial or imaginary world. With the exception
of text-based systems this world is presented via computer-generated imagery
of varying sophistication. At its simplest a world is 'modelled', that
is a database of elements and their relationships and behaviours are constructed,
and then 'rendered', that is made visible through the use of computer
imaging. As our normal world is three-dimensional most VR worlds are modelled
in 3D. In text-based systems the model is described textually, and the
'rendering' of it takes place in the user's imagination, ranging from
a typical dungeon in a castle to the hot-tub in LambdaMoo.
Total immersion: Brains in Vats?
Before looking more closely at computer graphics issues in VR, I want
to make another philosophical detour. Daniel Denett tells us in his Consciousness
Explained that philosophers in the last century were interested in
the 'brain in a vat' problem [8].
The idea here is to explore the philosophical implications of a scenario
where a mad (or evil) scientist has scooped out our brain, placed it in
a vat, and connected us up to electrical apparatus which receives and
returns synaptic impulses so as to fool us that we were living an ordinary
life. Denett briefly mentions VR in this context, but we are in a position
here to pursue this a little further. The main technical objection to
the scenario is in what is known as 'combinatorial explosion'. Because
the brain's owner may make over a period of time so many unpredictable
decisions (because it must still be allowed to exercise 'free will' for
the illusion to work), it would become impossible to calculate all the
possible activities of the person, and so be able to provide electrical
impulses that covered all the eventualities. For example, one might walk
the same route to work every day, which means that the mad scientist only
needs to construct part of a city (in computer terms: model and render).
But what if our person in the vat decides, on impulse, to turn down a
side-street one day? Or, far worse, suddenly decides to go on a holiday
to a foreign city and, again on impulse, turn into an obscure alleyway?
Our scientist would need to reconstruct the entire planet and all its
inhabitants in order to guarantee that the illusion never broke down.
The scenario was dropped by philosophers because of this basic objection
to it, but with VR we have to consider it again. Even the most basic of
computer games programmers is in the mad scientist's position: how much
of a 'world' is one going to construct for the interactive participant?
We find even at this crude level of VR an answer to combinatorial explosion:
procedural modelling. This is a technique whereby, for example,
cities can be constructed using a rule-based system: by abstracting out
the main principles whereby cities grow and their elements are constructed
and appear to us, we can generate cities (or whatever) 'on the fly'. In
addition we need (in visual terms) to be able to render any view of these
constructed environments on the fly, but this is a separate problem requiring
only that there is adequate processing power. (For the brain in the vat
an inadequate external processing system might result in 'picture loss'
if turning one's virtual head rapidly.) Recent advances in computer graphics
modelling and rendering techniques mean that the philosophers will need
to revisit the brain in the vat problem.
In The Physics of Immortality the physicist Frank Tipler makes
the assumption that computers in the far future will be powerful enough
to serve a vast number of 'brains in vats', in fact every one who ever
lived (or might have lived). Tipler does not in fact suggest preserving
peoples' brains: his theory proposes that our personalities (or souls
or whatever) are uploaded into the computer itself, and that the machine
will be powerful enough to simulate life for everyone who every lived
a digital immortality. This is the most extreme form of VR yet proposed,
and, although Tipler argues for it in great detail (with considerable
support from equations and physics) most scientists are highly sceptical
of his ideas. Probably the biggest stumbling block is the question of
the computability of mind, that is whether the mind/brain functioning
can be simulated, even in principle, by a computer (or technically a Turing
machine). This is a very interesting debate which I discuss further in
Artificial Consciousness Artificial Art [9]. The next section will throw up further
doubts about Tipler's proposal.
Physics vs. Faking it
Apart from text-based and 2D VR systems we are mainly dealing with 3D
environments, all of whose features can be expressed, understood, and
coded via the laws of physics. In practice however this is rarely done
because of the amount of computing power required, and the whole evolution
of computer graphics has in fact been a history of 'faking it.'
Faking starts at the modelling stage, where an internal representation
or 3D database is built of the artificial world. Various techniques exist,
but the commonest method is to represent objects using polygons. A great
deal of skill is involved in modelling objects to an appropriate level
of accuracy using the minimum number of polygons: in the games industry
for example typical 3D scenes may have a maximum polygon count in the
low hundreds to a few thousands. An artists' job working on such a game
may involve the modelling of a realistic looking car with just 30 polygons.
This is a skill that artists develop from their traditional training in
the arts, and involves an intuitive understanding of how visual clues
work, and how to abstract the key visual characteristics from complex
natural phenomena. What if we were to turn to physics to help us in the
modelling stage? If we really wanted to build a virtual reality that imitated
beyond any doubt the real world then we probably would have to use physics
right down to the molecular level. The appearance and behaviour of objects
depends on this: the exact distribution of mass in a car for example determines
the way it corners; the exact distribution of pigments and carriers in
the car's paintwork determines its finish (and whether the car looks new
and expensive or old and cheap). This, for me, is another of the objections
I have to Tipler's virtual universe: a convincing reality requires modelling
at the molecular (or even atomic) level, and you would need a processor
for every molecule or atom. 'Molecular computing' as it is called does
look in fact like a possibility, though a remote one, but even if we could
build an information processor at the molecular size, we would land up
needing one per molecule in our model: in other words you would need a
whole universe to model a universe!
A more realistic use of physics in modelling has been mentioned earlier:
procedural modelling. This requires again that we abstract out some high-level
behaviour from a natural phenomenon, for instance rules about the way
plants grow, clouds form, and so on. One common technique in this area
is called particle systems, where simple physical laws are used to control
large numbers of particles (represented usually only by a location and
a small amount of other data such as mass, direction, speed, and colour).
This is a solution somewhere between (in terms of the processing power
needed) molecular representation and polygon modelling.
However, modelling plays only a small part of the overall process of producing
'cheap' but convincing VR. Turning now to the rendering stage of computer
visualisation, we find that even more faking it goes on, though the physics
in fact is well-understood. There are two main areas for faking: firstly
in a range of texturing techniques that are used to make up deficiencies
in the modelling, and secondly in the rendering itself. The earliest
faking of a 'realistic' scene derived from the polygon representation
of objects: simply shade each polygon according to how much light falls
on it. This gives a crude faceted realism to a scene that a wire-frame
rendering can never have. Progress in rendering involved better faking:
if shading levels are averaged across the polygons (Gouraud shading) there
is a better illusion of smoothness, if polygon normals are interpolated
(Phong shading) specular highlights can be introduced to make things look
shiny. These are called shading models, and they work in conjunction
with lighting models. The simplest lighting model is a classical
bit of faking-it given the grand title of Lambert's lighting law. What
was needed was a simple mathematical relationship that took the angle
between the light direction falling at a point on the surface of an object
and the normal to the plane and calculated a light intensity. When this
angle is zero the intensity should be a maximum, and when the angle is
ninety degrees it should be zero; what, asked Mr Lambert, is a simple
mathematical function that does this? Answer: the cosine. The result works,
that is it gives reasonably realistic 3D shaded objects, but the maths
used bears no relation to the physics of light involved.
A more sophisticated strategy called ray-tracing allows more of the laws
of physics to be used, specifically optics. This is a shading technique
that can be used with different lighting models, as with the previous
shading techniques, but much more realistic results can be obtained. Ray-tracing
follows imaginary rays of light through a scene, but in reverse, starting
with the eye-point, moving through a pixel of the final image into the
3D space where it may encounter objects, bounce off them or pass through
them in differing proportions, and finally encounter a light source. A
development of this called radiosity allows for subtle lighting effects
to be computed based on the equilibrium of light energy in confined spaces.
Ray-tracing takes much more processing power than the rendering techniques
described above, and radiosity calculations that much more again. The
kind of imagery obtainable is described as photo-realistic, and, as this
name suggests, the artistic sensibilities required to work successfully
with it are closer to that of the photographer than painter.
Because ray-tracing and radiosity techniques are so processor-intensive
they tend to be used for still imaging and animations with high budgets.
VR, with its requirements for real-time rendering (absolute minimum 6
frames a second, but preferably 25 up to 60) rules out both of these techniques
for the time being. In fact, even more conventional rendering techniques
are often too slow, which is where texturing comes in. In any VR system
there will be a polygon 'ceiling', that is a maximum number of polygons
that the system can render at acceptable frame rates. To improve on the
crudity that this would otherwise impose on the look of the virtual world,
digital images known as maps are added to the polygons. The most important
are texture maps (images such as wood or brick), transparency maps (providing
holes in objects) and reflection maps (making objects look shiny). In
each case the application of the map reduces the polygon count, often
quite dramatically. In a typical top-end flight simulator one might need
trees at the end of a runway for realism, but modelling a realistic-looking
tree with thousands of polygons is out of the question. What is the minimum
number of polygons that we can use to simulate a tree and get away with
it? The surprising answer (in the specific context of the flight simulator)
is one! A single polygon with the appropriate texture map (showing trunk,
branches and leaves) and an appropriate transparency map (a silhouette
of the tree) makes for an acceptable tree. This works because the angle
of approach to the tree when landing will have only minute variations
otherwise it would soon be apparent that it was a cardboard cut-out. The
artistic skills here, as in a range of VR applications with limited view-points,
is more that of the stage-set designer.
If we can use ray-tracing and radiosity we can obtain very good shiny-looking
objects because of the accurate reflections in them. Interestingly, even
here, a degree of faking is often needed because of the cost of modelling
enough world to make realistic reflections. A spoon, held up to the face
in a busy street, reflects large chunks of city especially if moved a
modelling task that can be avoided if the reflections don't have to be
accurate. The fact is, and artists know this, that the reflection in the
spoon is read by the eye/brain merely as a visual clue: the eye is not
focused even on the reflected image but uses it in determining that it
is a shiny metal (and whether or not it is clean and suitable for eating
with, for example). The geniuses of faking-it quickly spotted that almost
any reflected image would do, and many systems merely provide a typical
indoor reflected image (usually containing a window) and a typical outdoor
reflected image (usually with a sky and ground). You've guessed it: another
map is used, the reflection map. (For a good discussion of these techniques
and how 'chrome' has come to symbolise modern computer graphics, and how
the artist's sensibilities are involved in this see 'The Chrome Age' by
Sonya Shannon. [10])
If we consider now what this spectrum of physics-to-faking-it implies
we realise that it is all about trade-offs. Faking it gives us speed;
physics needs a lot of processing power. Hence the demands of VR for real-time
graphics means, for the foreseeable future that more faking than physics
is needed. Does our spectrum imply as processing power increases (it will
need to by orders of magnitude) that the artistic input will be less?
I don't believe so, but the type of artistic skill will change:
real-time ray-tracing and radiosity will require the eye more of the photographer
than the painter.
Artists' Visual Clues
We have shown
that both the modelling and the rendering stage of VR graphics require
the artists' sensibilities, given that until Tipler's world becomes true
we cannot rely on physics alone to build our virtual reality systems.
Let us explore how pre-digital art has exploited a similar strategy of
visual clues, albeit rendered in paint or other artistic media. Artists
have always abstracted key visual features of the natural and man-made
objects that make up their compositions, though it is only in this century
that the degree of abstraction has made a quantum leap beyond its previous
parameters. If we stick to representational art however, it is clear that
the artist somehow 'models' the objects to be represented in their minds,
and then uses visual clues drawn from a stock of such clues that comprise
a language of the visual.
When an artist wishes to represent a scene, whether drawn from real life
or imaginary, they have to consider how it is lit: for the picture to
be convincing there will generally have to be a global light direction
which determines which side of objects will be brighter than others, and
how shadows fall. Many decisions regarding light have to be taken (perhaps
unconsciously), but they are all readily translatable into 3D computer
graphics. While a painter has a greater freedom and flexibility to use
light in a personal way than with a 3D computer system, he or she still
has to be consistent and follow certain rules otherwise the marks made
on the canvas will not be readable. Visual clues can cancel each other
out (though of course illusionists such as Escher can make use of this.)
Once global decisions are made about lighting, from which many of the
necessary visual clues and their execution follow in paint, or other media,
objects within the scene have to be differentiated using different surface
qualities. Thus apples are shiny (a dab of white is required for the highlight),
a carpet is dull but textured, a person's face rich in subtle colouring,
wet things reflect other objects.
Reflection in water and other reflective surfaces has long been a stock
trick for artists; it is instructive to look at master cartoonists like
Giles or Steve Bell to see how, with a few simple vertical strokes, they
transform a dry city pavement into a wet, sodden, slippery surface. Refraction,
the bending of light through transparent objects, is another easy trick:
just shift around the background a bit, or, if it is a severe case, like
a bottle or glass sphere, just use background elements in a murky swirl.
(Incidentally the long-running Smirnov vodka ads use no refraction for
the scene seen through the bottle probably for clarity , and it does not
detract from the message.)
Artists also use what is known as depth cueing to give the impression
of space: objects in the distance are reduced in clarity because of atmospheric
effects such as haze or fog. The simplest visual clue here is a desaturation
of colour (see Shannon's article again for a discussion of this).
One of the difficult problems in painting is to give the elements of the
composition sufficient separation, because, lacking stereo vision and
parallax, the viewers' normal strategies for disentangling objects are
restricted (this comes back to Hume's perceptive comments about how on
earth we extract discrete objects from an environment served up to us
as a 'porridge' of visual sense impressions). Some painters use thick
black lines around objects, while others use a variety of depth clues,
including of course the natural one of scale.
Finally, objects that move rapidly in a scene can be given the illusion
of motion by blurring them. All these techniques, including this motion
blurring, are translatable into software techniques for VR, and all of
them require the artists' eye to work successfully.
3D Studio
Given that 3D graphics systems whether for VR or anything else, have to
provide the user with a balance on the physics-to-faking-it spectrum combined
with a reasonable estimate of the artistic skill the user will have, how
do well-known systems compare? How do their designers pitch the functionality
of their systems? They also need to take into account how much physics
a user knows.
Materials: just provide a name, e.g. marble, brushed aluminium, and leave
it to the system? Or provide the tools to create these? The exact balance
is difficult, and is matched with the difficulty throughout the modelling
and rendering options that can be provided. At London Guildhall University
we use a 3D modelling and animation package called 3D Studio [11], which shows an interesting range
of choices in the modelling and rendering functions offered. The software
is designed to run on a fast PC, such as a Pentium, but the performance
of the average machine is way below that provided by workstations such
as the Silicon Graphics machines. Hence ray-tracing is only provided for
accurate shadows, and the rest is rendered by a scan-line algorithm.
Virtual Worlds and the Artist
So far we have discussed mainly the artisanship of the artist in
constructing acceptable virtual worlds, acceptable from a visual standpoint
that is. The real challenge of VR is to harness the available technology
at any stage in its development to provide an experience that really engages.
It is obvious from people like Tipler that there is a desire to play God,
to 'improve' on real reality. This impulse may be the same as the Utopian
one that appears through history from Plato to Aldous Huxley, but it may
well be that the more utopian the virtual world the more boring it will
be. In other words the ground-rules for life in the real world seem to
provide for a pretty engaging drama, and it is more likely that good VR
scenarios will not abandon the age-old bugbears of the utopian dreamers,
such as evil, profit and loss, and death. The artists' role in VR is much
greater than the merely visual, encompassing all the creative arts, many
of which are akin to those found in theatre or film production. The visual
artist has, however, a key role in the look and feel it of it.
Conclusions
We have seen that virtual reality systems comprise a spectrum of applications,
most of which rely heavily on the visual. Because of the processing power
necessary to provide sophisticated computer-generated imagery, much of
it is achieved through cheaper forms that can be called faking. Computer
imaging techniques are themselves drawn from a spectrum that can be described
as 'from physics to faking it', and the right choice is critical for a
workable VR scenario. The skills of the visual artist are essential in
working across this spectrum; at one end the visual clues used by painters
and illustrators have to be understood, while at the other end the sensibilities
of the photographer are essential.
The philosophical implications of VR are another challenging aspect to
the artist, in that the designer of a virtual world has to have a profound
grasp of the drama of the real world. I leave you with a philosophical
thought: would you really find a virtual world designed by utopian thinkers
like Plato or Tipler more interesting than one designed by a dramatist
like Shakespeare or Hitchcock?
References
[1]
See for example Plato, The Last Days of Socrates, Trans.: Hugh
Tredennick, Harmondsworth: Penguin, 1969, around p. 110 or Plato, The
Republic, Trans.: Desmond Lee, London: Penguin Books, 1987, around
p. 252
[2] See for example Steiner,
Rudolf. The Evolution of Consciousness, Rudolf Steiner Press, 1966
[3] Warnock, Mary, Imagination,
London: Faber, 1980 (This deals extensively with Hume as well)
[4] See Wilber, Ken, Quantum
Questions - Mystical Writings of the World's Great Physicists, Boston
and London: Shambhala, 1985, p. 79, or Schroedinger's own writings.
[5] Bogzaran, F. 'Images of the
Lucid Mind' in Consciousness Research Abstracts, proceedings of
the "Tucson II" conference (Journal of Consciousness Studies)
University of Arizona, 1996
[6] Gibson, William, Neuromancer,
London: Harper Collins, 1993
[7] Tipler, Frank J. The Physics
of Immortality - Modern Cosmology, God and the Resurrection of the Dead,
London: Macmillan, 1994
[8] Dennet, Daniel C., Consciousness
Explained, Allen Lane, The Penguin Press, 1991, p. 3-4
[9] King, Mike, 'Artificial Consciousness
- Artificial Art', chapter for a book Art and Computers, (Ed. Stuart
Mealing), to be published by Intellect. See also a shorter version in
Proceedings, International Symposium on Electronic Art 1995, Montreal,
ISEA'95 Montreal, 1995, p. 137 - 140
[10] Shannon, Sonya, 'The Chrome
Age: Dawn of Virtual Reality' in Leonardo (3rd Annual New York
Digital Salon), Vol 28, N. 5, pp. 369 - 380
[11] Produced by Autodesk. The
most recent version, 3DS Max, runs under Windows NT and is a competitor
to high-end systems like Softimage.
|
|
