Other web sites about Avatar Puppeteering:
1. Introduction
|
Virtual World Design has made great advances in the past few decade in many areas.
But the ability for users
to control their behaviors online remains a limiting factor, as compared to the current level of visual realism
in virtual worlds. Consider the deep and
vast expressivity of our body language in daily life: our facial expressions, gesticulations,
how we set our gaze, all these forms of physical expression that are
so ubiquitous in our normal lives.
|
In fact, it is so natural and ubiquitous (i.e., invisible) that developers of virtual worlds
often neglect to add the necessary technologies to facilitate this. And so, expressive technologies
end up being patched onto the code late in the game.
Add to this the fact that virtual world software is
usually created by young males from the Games industry, where self expression takes a back seat to the art of blowing things up
and making it look real. |
Making the Avatar More Expressive
An underlying motivation for Avatar Puppeteering is to enable a
more fluid, more direct way to manipulate the virtual object which represents the user's identity and which is the primary
vehicle of expression: the avatar.
As the original avatar developer at
There.com
,
I was able to get some avatar expressivity baked into the underlying
codebase, and many other folks contributed to the expressive avatars in There. During my two years as a Senior Developer at
Linden Lab, Avatar Puppeteering was my most ambitious attempt at adding a layer to the avatar technology that would allow more fluid and
immediate interaction with the world and with the user's touch. My other major accomplishments were
Flexi Prims
and
FollowCam.
|
In Second Life, residents have extensive tools to build objects, and script
their behaviors online, in a collaborative manner. But there is not as much
manipulation and creative freedom in terms of developing behaviors, expressions, and movements
in one's avatar - the fulcrum of social connectivity in Second Life. This technology
has the potential to provide a way for residents to manipulate their avatars
through
motion-capture, using input devices similar to the Wii.
As of now (March, 2008) the technology has not yet been fully deployed, but I hope it will soon be re-visited, as I
think it would add a level of expressivity and direct expression to the avatars in Second Life.
|
The Physical Avatar
Techniques for animating avatars have come a long way, although they are still
largely based on the linear narrative of film technology, whereby character
animations are crafted beforehand using keyframing, inbetweening, etc., stored
in a file, and then played using various triggers in the context of a dynamic
virtual world. The animations look exactly the same each time they are played -
which is of course desirable in the case of a feature film - but pretty useless
in terms of spontaneous expression in an online world.
To accommodate the novelty and freshness of online virtual worlds,
hybrid layers of procedural animation
have been introduced, which
allow the avatar to adapt to an unpredictable environment and to respond
to user intent (although still to a limited degree, in my opinion).
|
Avatar Puppeteering introduces a
completely physics-based means of naturalistically animating the avatar,
in which every joint can be pushed, pulled, or rotated in real time for
maximum expressivity and responsiveness.
The underlying technology for Puppeteering is called "Physical Avatar" - it
is called that because it uses
forward dynamics .
to affect the positions of
avatar body parts in the virtual world. This is explained in the sections below.
|
2. Avatar Representation
|
To explain how Puppeteering works, I will first describe the avatar
skeleton. The skeleton consists of a set of joints connected hierarchically.
|
|
The pelvis is the parent of the two hip joints and the torso joint.
We also refer to the hips and torso joints as "children" of the pelvis joint.
Notice the five
end-effectors
(head top, fingertips, and toes). These are not part of the standard skeleton,
but are added in the Physical Avatar representation, for reasons that are explained below.
|
The Root
The position and rotation of a character in a virtual world is typically stored both on a server and on various clients,
along with the positions and rotations of other avatars and objects. Most of these objects
are also represented by polygonal bodies, simulated using the
Havok
physics engine.
When
a user controls his/her avatar by walking around, the changes in position and rotation are sent up to the server, so
it can update the root position and rotation accordingly, and pass it down to all the clients, so it can be rendered
accordingly.
We often refer to this position and rotation pair as the "root". The root is the parent of the pelvis joint, which is itself the parent of all
the rest of the skeletal joints.
|
This picture shows the root position as a white dot. The
pelvis position can be offset from the root position while puppeteering, as shown.
|
Joint Representation
Every joint has a position and a rotation in 3D space.
A "bone" is the line segment that connects two neighboring joints. Joint rotation is
typically represented in the coordinate system of its parent
joint rotation, referred to as "local", or "parent-relative"
rotation. In Second Life, rotations are stored as
quaternions
.
Quaternions provide an efficient representation of rotation for
computer graphics, and they avoid certain problems associated with other representations,
such as
Euler angles
.
|
Physical Avatar requires several layers of joint representation, including a joint's world-coordinate
position, its world-coordinate rotation, its parent-relative position, and its parent-relative rotation.
It also uses other attributes which are specific to the algorithm, and not very interesting in this context.
The most important point is that joints use world-coordinate positions, and this is necessary for physical
simulation, as described below.
|
3. User Interaction
|
Hovering Over Joints
At any time while the user's avatar is within view, the user is able to press the CONTROL key on
the keyboard and pass the mouse cursor over his/her avatar. When the cursor passes-over one
of the avatar's 24 grabbable joints, that joint is highlighted with a translucent symbol,
and any bones existing on either side of that joint are also highlighted, as shown here.
Click-Dragging Joints
This provides visual affordance for the user, which has two purposes: (1)
discoverability, and (2) quick-targeting. Since the CONTROL key is also
used for manipulating other objects in the world, users are accustomed to
this bit of UI. If the user discovers that the avatar's joints light up
while holding the control key, he/she may conclude that the avatar is
manipulable in some way. Clicking and dragging will have the immediate
result of moving the joint, and the user will learn what the UI is for.
Then, once the user has become accustomed to this UI, he/she will be able
to use the appearances and disappearances of these hover dots as visual
feedback to rapidly manipulate joints.
|
4. Algorithm for Physically-based Manipulation
Now we get into the nuts and bolts of how the Physical Avatar works.
The process will be described in terms of the following three
primary steps:
5.1. Conversion from Rotational-space to Physical Representation
5.2. Applying Forces
5.3. Conversion Back into Rotational-Space
|
5.1 Conversion to
Physical Representation
As mentioned above, puppeteering starts when the user hovers the mouse cursor over a joint,
presses CONTROL, and clicks the mouse button. This initializes the
Physical Avatar engine. At initialization, an array of 24 world-coordinate positions
is generated. Starting with the root position and rotation, the hierarchy of
parent-relative joint rotations is recursively
traversed to calculate the
world-coordinate positions. These become the joints of the Physical Avatar skeleton.
The default pose array and the parent-relative rotations of each joint are also stored
when Physical Avatar is initialized.
The world-coordinate positions are used as the locations for a
collection of physically-simulated balls floating in space, which
correspond to the joints of the avatar at the time of initialization.
Each ball is constrained
to its neighbors by way of
spring forces
. "Neighbor" is defined
here as the joint's parent and its children, if any.
|
The springs
are highly-dampened with ambient friction, and they use a "relaxation"
technique - whereby the locations of the balls on either end of a spring
are adjusted in the same direction as the spring force.
These balls also have soft collision interactions: if any two of them
penetrate, they push apart.
They also respond to collisions with the
ground surface, and their collision response is affected by ground
surface
normal for more realism.
|
5.2 Applying Forces
With this ball-and-spring physical representation we can now
apply forces in the world-coordinate system on these joint
balls, to create naturalistic motions. The key forces are
described below.
Tetrahedral Body Forces
There are two regions in the skeleton in which a joint has more
than one child. These are the pelvis and chest regions. Consider
the pelvis: the ancestor of all the joints. In a typical avatar
animation, the torso and hip joints are rotationally "fused" to the
pelvis joint.
In order to calculate a coherent rotation in the
pelvis, we must secure its three children in relation to each other,
so that they act as one rotational body. This is done by adding
three special spring forces which constrain the two hip joints to
each other, and the torso to the hips, as shown with the thick blue lines.
Like the bone-springs (the white lines), these springs are highly dampened
and use relaxation, and so they are quite stable. These triangular structures
are needed in step 5.3, for generating coherent rotations.
|
|
Similar to the pelvis, the chest region has three springs that force the chest
joint, neck joint, and the two collars to become fused, and act as
one rotational unit.
The image at right provides a link to a Quicktime video showing how
these spring tetrahedra hold the pelvis and chest rotations stable.
You will notice that these tetrahedra are very flat. But they still
maintain their
tensegrity-like
stability.
|
|
Humanoid Constraints
If you don't think it's important to apply human-like constraints
on these joints, try
bending an elbow backwards, or pulling the head
back 120 degrees. The pain endured from just watching your avatar become twisted-sister is enough
to convince anyone of the importance of applying limits on
how these joints can move in relation to each other. But since the
simulation uses balls and springs, with no explicit rotations to the
bones, you cannot simply say, "don't allow the knee rotation to go less
than zero".
|
Instead you have to measure relative distances, calculate
cross-products
and
dot-products, and things like that, in order to
determine when an implicit rotation has gone past a threshold and also
in order to force it back to its preferred state. I will not go into the
details of this technique except to say that what is important here are the relative
positions of the balls to each other that determine these constraints, and they generally
require about three balls to do the calculations.
|
|
|
User-Manipulation Forces
Now I will explain how the user clicks and drags joints. While puppeteering,
the user's mouse cursor position is used to
project a ray
out
into the 3D scene (the normalized vector from the camera viewpoint to the
cursor location on the view plane. This vector is scaled in length to equal the
distance from the camera viewpoint to the ball which the user clicked on to start puppeteering.
Once clicked and dragging, this distance remains constant, and the ball sticks to the surface
of an implicit sphere surrounding the camera viewpoint.
Since the ball being dragged is connected to at least one other ball by a spring force, all neighboring balls
get dragged along with it. As explained above, there are various constraints that keep
the balls from moving unnaturally. There is also a constraint that keeps the ball being dragged from
moving too far away from its original position when clicked. This keeps things fairly well-behaved while
puppeteering (although it is turned off for full-blown
ragdoll mode, as the video below shows).
|
|
Gravity, Wind, and...Catapults?
Notice that when the mouse cursor is projected into the scene, it basically becomes a
force that constrains one of the balls in the 3D space. But
any other force can be applied to the balls as well. For instance, when ragdoll mode is turned on, gravity
adds a downward force to all the balls at a constant rate.
Other forces could just as easily be used, such as wind, slaps in the face, a
wedgie, or being thrown by a catapult.
|
|
5.3 Conversion Back
into Rotation-Space
OK, it's fun to talk about wedgies, catapults, and stuff.
But now it's time to explain the technique for taking the positions of
these physical balls and converting all that information into pure rotation.
Remember that avatar animation in Second Life (as well as in most systems) is
specified using "parent-relative rotations". In other words, each joint's rotation
represents a "rotation off it's parent's rotation". Consider the outward-aiming
arrows in the image below. Think of the base of each arrow as a joint which can pivot off
the tip of the arrow that it is attached to.
|
This is basic
hierarchical modeling.
You could say that the information
flows from the pelvis outward to all the joints. Now, I would like to propose that the Physical
Avatar system does just the opposite. The information flow goes inward
instead of outward. In this sense it is more like
inverse kinematics.
In hierarchical modeling, the rotation of the
elbow determines the position of the wrist. But in the Physical
Avatar, the position of the wrist determines the rotation
of the elbow (plus a few other things for reference, as explained below).
|
|
|
Default Pose as Reference
The default pose, or "T-Pose", is the avatar stance shown above - the avatar's
arms are extended out and the joints are mostly straight. In this pose, all the joint
rotations are set to zero, or
identity.
Since all rotations are parent-relative,
the default pose represents the joint rotations that are used to compare the rotational offsets.
In quaternion terms, the difference between the position of a joint and the position of where the
joint would be if it were in its default pose determines an arc sweep (a
spherical interpolation.),
expressed as a quaternion.
|
|
5. Networking
|
Well, all this time I've been explaining the algorithms and interaction associated with manipulating
the avatar in real-time. But I haven't said anything about how the user's actions get transmitted
over the internet so that it is distributed within the online virtual world.
|
In some ways, this is the most important part of expressing yourself - after all, if these avatar motions cannot
get transmitted across the internet, then it doesn't count as expression (OK, maybe it counts as self-indulgent
interpretive dance, but that's all).
|
The question is: what exactly do you have to transmit over the internet? Do you send the entire
array of avatar polygons? Of course not - because every client already has the geometry of each
avatar within view, and only needs to get updates from the skeleton - the rendering is taken care
of locally - on the client. But then there is still the question of what aspects of the skeleton to send
over the internet. One solution is to send the joint representation that the avatar animation system normally
expects: parent-relative rotations.
The image at right illustrates the idea of sending the entire array of joint rotations over the internet
whenever I puppeteer my avatar. Any client (viewer) in which my avatar is within view
then gets the stream of changing joint rotations downloaded so that my avatar can be seen moving.
All very cool. But there is actually a more efficient and clever (thought tricky) way of doing this:
|
|
Only Transmit the
Joint Being Dragged
Recall that there is a physics algorithm being employed when a user is puppeteering his/her avatar.
This physics code exists on every user's client, and is put into action whenever that user wants to do a little puppeteering.
But consider this physics code as being put into action when a REMOTE avatar is being pupeteered. In other words, all we have
to do is transmit the ONE SINGLE joint being dragged, and as long as the physics code is running on every client, then
the remotely-puppeteered avatar will animate correctly.
|
|
|
This is an example of the kind of technique which makes the client do most of the
work so that the server doesn't have to do so much communication (which can clog the internet pipes with redundant data).
By transmitting only the puppeteer's joint manipulations, the server's job is simplified - it is only to communicate the
essence of the user's expression - nothing more.
All the rest is taken care of on the client.
|
There is a lot more involved on the networking side of things.
I was not as involved in this aspect of the code as others - most notably - Cube Linden (Qdot).
So I cannot say much more about this. Most of the details are
mired in the idiosyncrasies of low-level Linden Lab code. It's not pretty.
|
Acknowledgements
There were several talented developers at Linden Lab who
worked on this project
with me.
Most importantly were the stellar devs Samantha Patterson and Kyle Machulis (Cube). Andrew Meadows helped in the early
stages with math, physics, and architecture.
|
Mike Schlacter did some work on the networking code. Thanks to
Mark Lentczner, Periapse, Cocoa, Qarl, Vector, Kona, and many others. Finally,
thanks to Cory and Philip for seeing the promise of this technology.
|
The Avatar Puppeteering code is owned by
Linden Lab. It is not
yet available in Second Life or as part of the
Open Source Viewer.
| 
