SEMESTER PROJECT


Developing Real-time Interactive Computer Graphics
for the Theatrical Stage

Documenting the development of interactive sound and graphics for a live dance production (with optical motion capture)proved to be a more valuable and practical VE project than developing a simple mixed reality boundary.

PROJECT GOALS

      • Create an environment where a user (in this case a live dance performer) in motion capture (or with any two points tracked in space with the hands) can put their hands together, create a bubble, scale it accordingly and freely, and release the bubble when the distance between the hands are wide enough apart
      • Layout the basic concept of my dynamic forced perspective in the theatre using real-time computer graphics.

     

 

Project Origins

A collaborative effort between the Dance Division, Theatre Division, and Computer Graphics Technology began in 2003 to experiment with mixing real time computer graphics (using motion capture) and live performance. The third performance of this group was premiered on April 30th and May 1st of 2004.

Part of my contribution to this third performance centered on the interactive elements between dancer and graphics. The most complex interactive component of the piece was the "bubble creation" scene. The goal was to have a dancer in motion caputure that could create, play with, and release, bubbles by putting her hands together and moving them apart.

Keeping in line with the definition of Virtual Environements, the performers in the dance project were not visually viewer-centered, but they were, however, spatially "viewer-centered"-- i.e. the virtual world was mapped the to motion capture performance space and movements/actions between the real/virtual space were directly related. The dancers navigated a virtual space that had a number of triggerable events and releationships with virtual objects and sounds.

In a typical performance environment, the point of view is usually given to the audience members.

A view from the audience: The stage consisted of a full, front-projected, downstage scrim (semi-transparent projection material). Behind the scrim a motion capture area was set up using a rectangular grey carpet that served a dual purpose--first and foremost, the carpet absorbed the infared glare that was generated on the stage marley. The carpet also served as a useful "grid" to mark off the extent of the motion capture area including other useful marks for the dancer. Above the motion capture area there were six infared cameras laid out in a hexagonal pattern. These cameras were hung and flown in and out from 3 battons. Behind the motion capture area was a large, 10K, rear-projection screen.

  • Foot Markers were tracked to trigger sound and video events based on the rise and step of each foot. This allowed for a very fun and playful opening of the dance piece
  • The second scene consisted of tracking each hand marker so that when the hands were put together a bubble would be created between the hands (centered on the midpoint between the hands. This bubble was made scalable so that increasing or decreasing the distance between the hands made the bubble expand and contract. When the hands were at a very large distance from each other, the bubble was released and floated off stage towards a pre-defined target. Five bubbles were created and released, all with different trajectories and designs.
    • The first bubble floated up above the stage, "popped", and a particle system of bubbles was released on the upstage and downstage screens.
    • The second bubble floated in a circular pattern around the motion capture data while staying in view of the audience for the entire scene
    • The third bubble, when released, floated towards the audience nearly running into the virtual camera that was "positioned" in an optimal viewing spot in the theatre, or "the king's seat".
    • The fourth bubble floated off in a general direction
    • The fifth bubble, when released, expanded quickly in scale from it's pivot point between the hands--giving an effect of an "atomic bubble"
  • The third scene consisted of a bridge with guy wires divided across the stage. One side of the bridge is on the downstage scrim and the other side is on the upstage projection screen. When the dancer reaches downstage towards the guy wires, she can trigger spatial cues that "play" the wire like a harp.
  • The fourth scene consisted of leaves and a leaf character that emitted particles (leaves) from the feet as it "danced" with the dancer..
  • The final scene did not use any motion capture data.
 

 

Interaction Algorithm Development

All of the interaction for the bubbles and bridge algorithms were developed in Kaydara's Mocap/Motion Builder. Below (Figure 1)gives an early "bird's eye view" of the relationships panel designed to create the bubble functionality. Each color coded area is shown in detail later in the paper.


FIGURE 1

The Bubble Algorithm consisted of several key components

  1. Importing Geometry into Kaydara Software
  2. Texturing Geometry
  3. Positioning Bubbles
  4. Animating Visibility of Bubbles
  5. Scaling Bubbles
  6. Releasing Bubbles
  7. Controlling Bubbles After Release
    _________________________

 

Importing Geometry into Kaydara Software:

A geosphere was created and imported from 3DSMax because Kaydara Software does not support modelling (except for planes and cubes).

 

Texturing Geometry:

Once geosphere was imported it needed to be textured and shaded. Textures were designed in Adobe Photoshop with alpha chanels and exported as tifs.

A shader was added to the bubble to give realistic lighting effects as well as some aditional control over the transparency. (See Figure 2 below)


FIGURE 2

The texture was animated in Motion Builder to give a more "floaty" and light movement to the bubble. (See Figure 3 below)


FIGURE 3

Positioning Bubbles:

After many design iterations it was decided that the best approach to this problem would be to keep all bubbles positioned at a "MagicSpot" between the hands throughout the piece until they were released. The position markers for the hands of the motion capture data were used as source points and a relationship between these two markers were made by attaching a null object at the midpoint between the hands. This way the dancer had complete control over the bubbles and always knew where to find them--a virtual point between the hands no matter where the hands were in space. This null object was named "MagicSpot" because it became the central node for most activities associated with the bubbles. All of the bubbles were parented to this spot, as well as all of the bubble targets off in space to which the bubbles would travel towards as they were released. The MagicSpot's rotational data was also manipulated by some subtle sine waves to make the movements more natural and dynamic. The MagicSpot became like a sun in a solar system where targets orbited/floated around it and bubbles were centered upon it.

Animating Visibility of Bubbles:

Of course there is the problem of having all five bubbles on at all times, so the visibility was animated for all bubbles and an algorithm was developed to fade in the visibility of all bubbles when needed. (See Figure 4 below)


FIGURE 4

Scaling Bubbles:

A major interactive component of the bubble scene was the ability of the dancer to scale the size of the bubble by opening and closing her hands. This was done based on the distance between the hands, but became complicated once the dancer released the bubble. This problem was solved by an "IF Cond Then A" object (see Figure 6 below)

 

Releasing Bubbles:

Once bubbles were release a complicated set of operations had to be made. The scaling had to be disabled, the size of the bubble upon release had to be maintained throughout the rest of the scene, the parenting relationship between the bubble and the MagicSpot has to be decreased to zero, and the positional relationship between the bubble and the bubble target had to be incresed to 100percent. This was done using the relationships algorithms below (Figure 5, 6).


FIGURE 5


FIGURE 6

 

Controlling Bubbles After Release:

Once bubbles were released, and all of the requirements of the algorithm were met (see above) the bubbles had to "look like bubbles" which was quite a challenge. Getting a proper weighting between the simultaneous decrease of the parenting constraint and the increase of the bubble target constraint was difficult in general and to get the passoff of weighting to be elegant and "bubble like" was no small task.

 

 

A Renaissance in Forced Perspective Theatre Design
Using Real-Time Computer Graphics

The most important development to come out of this project was my realization that early Renassance designs that used perspective scenery in theaatre productions could be revisited in ways unheard of before. See "Italian Renaissance" section of the following link: http://www.northern.edu/wild/th241/sdhist.htm

With real-time computer graphics elaborate sets can be designed and interacted with. Whereas the center of projection for the perspective scenery was positioned in the King's Seat (and the further away one sat from the king, the more distorted the perspective became), with computer graphics, the center of projection can be moved dynamically which opens up many new possibilities for interactive theatre. Especially if one was able to bring laptop computers into the theatre.