The Theme

The Ouroboros – but no explanation was given.  We just showed the jammers the image and it was up to them to interprete it.

LOTR Body Snatching

Rock, Paper, Scissors with body snatching!
Avoid or possess enemies to stay alive and win.

Low Power

In order to survive Low Power you must consume yourself (like the Ouroboros) to survive.  You control a robot whose battery is constantly draining but there are valuable energy cores scattered throughout the level that will sustain your life.  There are all kinds of dangerous environmental hazards that unless you enable your sensors/abilities (lights, shield, ground sensor, microphone) you’ll never survive.  However enabling the different systems will drain your battery even faster!

Parasite

You are a Parasite that leeches off battle ships. Enemies will constantly come in formations to destroy the ship you are occupying. Using the possessed ship, you must defeat your enemies! However, as you leech your ship it slowly dies, you will constantly need to take control of and leech of another ship. With out a ship for protection you are helpless!

Planes on a Snake

A rift in spacetime has resulted in a large number of World War II era planes getting stuck on the world snake.  Join the frequent fliers club of Ouroboros Airlines, racking up points while taking advantage of the torus nature of your new environment.

Roller Snake: The Quest for Chili Dogs

Control “Hardy the Hoop Snake, Jr., III, IV, and V” in his quest to eat as many chili dogs as possible while winding through ‘Catastrophe Canyon’

Snake Run

You are the snake’s guardian, you must ensure that balance is maintained.  Let enough creatures be eaten by the snake to ensure he doesn’t starve, but not enough that he frenzies and destroys everything.  All the while outrunning him!

In other news – The global game jam starts this Friday (January 27th).  You should signup and find a local jam site you can attend.  It’s totally worth a weekend of your time.

I’ll start with the worst input paradigm we tried go from there.

#5 The Faux Mouse

The Idea
First we just tried mapping the hands to the screen as if they were each a mouse or finger.  Each would control a hand like cursor and move it around the screen.  Clicks could either be performed by grasping the hands or pushing towards the screen.  The game would be played normally – except you’d click on everything using your hand.

The Reality
This input method was by far the worst.  Your hands are simply not mice, they get tired much faster and are not as dexterous.  Even with heavy filtering and snap-able buttons, the interaction is just too nuanced and motor skill intensive an operation to click on a button or object on the screen using your hands as mice.  The time it took us to play a level vs. someone playing it on the iPhone was an order of magnitude or longer a task.

When pushing towards the screen a lot of care had to be taken to deal with user drift.  When a user pushes towards the screen, they may in fact be pushing towards many possible locations.  They may drift towards the TV, or the camera if that’s their focus.  They may also consciously attempt to maintain a straight hand as they push forward.  No matter the case – a user will drift even off their intended target.

When we tried grasp detection using computer vision – again we saw drift.  When a user opens and closes their hand the volume of the hand is changed from the point of view of the camera.  The result ends up displacing the center of mass of the hand causing it to shift downward when the hand is closed.  There are solutions to this problem such as eliminating the fingers from the hand volume calculation but this is a difficult problem given the quality of the data.

In the end the drift issues prevented us from playing as well as we could with the real mouse or on the iPhone and since Angry Birds is both a game of quick victory and defeat as well as a game of precision.  Having both large problems with precision and game play progression speed – we ditched this idea.

#4 Voice

The Idea
This one is exactly what you might expect – firing the bird using your voice.  Either by saying “Fire!” or maybe… “Ca Cah!”.

The Reality
We really were not sure how this method would perform.  The voice portion was layered onto the mouse control system.  Essentially you would click on the slingshot – move your hand to aim and exclaim viciously to let loose the birds of war.  The hope was that it would alleviate the drift problems, which it did.  Without having to push your hand forward to fire the drift was eliminated from firing.

However there were other problems, there was delay in the speech recognition and often outright failure if you didn’t say the words just right.  We tried several tricks, like using multiple words to identify “Fire”.  One good way to generate that list is to take the top 10 words it mistakened someone saying fire for and add those to the dictionary as triggers for firing the bird.

Moving Away From Faux Mouse

After the failures with both #4 and #5 we went back to the drawing board.  We needed to get away from the mouse or touch device centered thinking.  The worlds are simply too different to treat it the same.  So we prototyped 3 other solutions that ended up all being better than the mouse style interface.

The ideas all stem from an understanding that when you go to port an application from a touch enabled world you need to think about 2 things primarily,

• Context
• Automation

Context – How can you reduce and scope the options to the user so that a broad array of options can be presented to the user – but with only a few usable at any given time with a small vocabulary of motions.

An example from Angry Birds is all the options the user can perform in the game:

• Fire Bird
• Activate Bird Special Attack
• Restart
• Pan
• Zoom
• Return to the Menu

We needed to find a way to contextualize these options when moving away from a mouse driven style interface.

Automation – Find the items in the application that everyone does without thinking about it and automate them.  If they aren’t relevant to game play find a way to make them irrelevant in a NUI application.

An example from Angry Birds is activating the slingshot.  You probably don’t think about it when playing the game but to fire a bird you must first place your finger on the slingshot before drawing back the bird to fire it.  While this is unbelievably trivial to the point of not thinking about it on an iPhone, it’s a huge pain in an environment where you have to get a virtual hand cursor over it, even more so if you then need to push forward to activate it.

So we needed to find a way to automate clicking the slingshot.  That way instead of clicking the slingshot explicitly it would be implicitly activated by performing some gesture to begin the act of firing a bird, that would be disconnected from the onscreen location of the hand relative to the slingshot on the screen.

#3 Arclight

The Idea
You would draw back the slingshot by bringing your hands together.  Then bring your hands apart and rotate them around your center of mass to change the firing angle on the slingshot.  Then once you’ve settled on the firing angle, bring your hands together fast to trigger the fire.

The Reality
The problem with this kind of activation of the slingshot was the drift when the hands come together.  This can be partially accounted for but it’s heuristically based and can be erroneous.  Additionally activating the bird’s special power was difficult.  You would have to choose a different kind of interaction to activate the special power which would complicate the process.

#2 Stretch and Snap

The Idea
This idea grew out of the Arclight firing system.  To attempt to solve the problem of drift, have the slingshot fire as soon as the arms reach a certain distance apart.

The Reality
With this firing system you still have the problem of determining how to activate the bird special power.  You also introduce a new problem – all birds are fired at maximum drawback.  You also need to make sure to provide the user with feedback so that they know how close the user is to the *snap*, some kind of progress firing bar.

#1 Axis Separated

The Idea
For this idea I separated the functions of the hands into distinct responsibilities.  Your left hand activates the slingshot by pushing forward (doesn’t matter where).  After a threshold is crossed the slingshot is activated, from then on an angle is calculated between the shoulder location and the left hand’s location relative to it, to produce the slingshot firing angle.  To fire the bird the right hand is pushed forward and pulled back, this sends the bird flying.  To activate the bird’s special power you again push the right hand forward.

The Reality
This method ended up working perfectly.  It doesn’t result in any drift when firing is activated. It is also easy to perform because all the motions can be performed with your arms down by your sides, reducing exhaustion in long game play sessions.

Lessons Learned

You hear it all the time but it is critically important to prototype ideas when it comes to creating Kinect controls.  They simply don’t work as well as you would like in reality as they do in your head. Here’s a demonstration video of the end result,

Click here to view the embedded video.

Anthropometry (Greek anthropos (άνθρωπος – "man") and metron (μέτρον – "measure") therefore "measurement of man") refers to the measurement of the human individual.

Ask any Kinect developer what the hardest problem is developing a game or application that uses the Kinect – or any other depth camera.  The answer you’ll get most often will be creating something that works well for 95% of target users.

This is something you have to consider for all your gestures, poses, and UI interaction.

• Is this gesture too difficult for your average user?
• Does this pose require too much flexibility?
• Is this UI interaction comfortable and easy to perform?

One area that can benefit from anthropometry data is UI interaction.  When you think about UI interaction with Kinect you’ve got to picture it as a real world space (box, sphere, cylinder, other) located somewhere around the body that you’re mapping the hand position in that space to the 2D or 3D UI coordinate plane.

When determining the size and location of this real world space and how it maps user hand locations onto the UI coordinate system the largest question you need to consider is:

Where will they be most comfortable?

Generally speaking you want a space that minimizes upper arm movement – as that is much more strenuous compared to forearm movements.

However, since the 3d position we’re mapping into our 2d/3d UI coordinate system varies based upon user skeleton size we can’t choose a single set of real world dimensions that will work for everyone.  We’ll have to make educated guesses about the size and location of our real world UI coordinate frame based on size and location of the users bones.

So how does anthropometry fit in?

Because the skeleton you get from Kinect – and other SDKs can be unreliable in certain poses you often find yourself heavily filtering any kind of data you’re tracking about the user.  Especially things like the user’s arm length – which can vary dramatically over a session.

So one thing I prefer to do is use anthropometry tables to ensure a more consistent size and location and doesn’t fluctuate as much as the user’s skeleton.  Using anthropometry tables we can estimate the users arm length or hand size based on other bones in their body, bones that are more stable in your skeleton SDK of choice (Kinect, OpenNI, Iisu, Omek…etc).

You can also use anthropometry tables to estimate the size of body parts that the skeleton SDK you’re using doesn’t provide – such as the size of the users hand.

But where do you find that kind of anthropometry data?

Luckily such a resource has already been painstakingly cataloged for us by the FAA – The Human Factors Design Guide.  The HFDG was put together so that planes could be constructed so that almost anyone would fit and be able to operate anything from their seat.

The anthropometry data that’s valuable to us starts in chapter 14, page 791.  For example, these lovely tables from page 818 show the functional reach and the extended functional reach of men and women broken down by population percentiles.

If two artists touch the same level, have your tools programmers made enough of an effort to make merging possible and if possible as painless as can be?

Here’s a handy checklist,

1 – Use XML/JSON (or some other text based file format)

It’s slower, it’s bigger but it’s going to make merging possible.  If your level files are binary blobs merging without a custom tool just isn’t going to be possible.  Using XML or JSON are the simplest text based alternatives because there are already many libraries for reading and writing them.

2 – For XML – Give Each Attribute a Line

If you’re using XML, you should make sure to write out the files such that every element and every attribute receives its own line.  If you don’t do this it will make conflicts a lot more likely if two users touch the same object.

If you’re using JSON, having each attribute on its own line is the norm unless the library is attempting to keep the text compact.

Here’s a quick example, note that after separating the attributes the merge tool handles the conflict correctly while it fails to do so when they are on the same line.

Figure 1 – Attributes all on the same line

Figure 2 – Attributes on different lines

If you’re using .Net this can be achieved very easily by changing the default XmlWriterSettings with the NewLineOnAttribute property set to true.

var settings = new XmlWriterSettings()
{
    NewLineOnAttributes = true
};
 
using (XmlWriter writer = XmlWriter.Create(filestream, settings))
{
    // Write out document...
}

3 – Maintain Order

Always write out the order of the data the same.  This one is pretty easy, the only real pitfall is to make sure your data structure and undo/redo system are working correctly.  For example, if you happen to store your objects in a list and someone removes and object at the front, but when it’s undone it’s inserted at the end of the list (Even if the UI doesn’t reflect this) this could affect your output order.

4 – Don’t Add New Objects At The End

When you go to save the entities don’t write them out in the order they were created.  This will definitely result in a merge conflict.  Because it ensures that if two users edit the file and both add an entity they’ll both show up in the same location and confuse the merge tool.

One thing you could do is to sort them based on a GUID that is stored as part of the objects data.  Sorting based on GUID ensures a lower probability of collisions occurring when two users both add objects.

Alternatively a sorting based on a string containing the machine name of the original creator of the object is another idea.  It would ensure every user touching the file would be inserting to their own section of the file instead of everyone inserting to the tail.

While wrapping up my post for generating simplified occluders for Hierarchical Z-Buffer Occlusion Culling, I was pointed to a paper called Complete Polygonal Scene Voxelization.  Afterwards I found time to read it thoroughly and implement it as a replacement for my existing ray casting based solid voxelization method.

The problem with the solid voxelization technique I was using previously was that it used ray casting; making it impossible to perform solid voxelization unless the mesh is watertight in addition to having no anomalies like intersecting geometry.

However, that restriction makes it an unrealistic solution in the real world because game art typically has holes in the locations players never see; such as the bottom cap on a building, which is rarely modeled.

The New Solution

The Complete Polygonal Scene Voxelization paper’s solution to voxelizing a scene is pretty clever; It applies a heuristic model to the problem of determining the inside/outside status of each voxel or octree cell.  Allowing it to overcome the problem of holes and intersecting geometry making it suitable as a real world solution.

How It Works

You can download the paper and read it for yourself, but let me go ahead and summarize the algorithm for brevity’s sake so that the rest of the article makes sense.

The algorithm takes place in 3 stages:

1. Create Octree
2. Find Seed Cell
3. Propagate Seed Cell

Create Octree

First you create an octree around the mesh that continues to subdivide each cell until either the cell no longer intersects with any triangle or a maximum depth is reached.  A typical maximum octree depth that will work for most meshes is 5.  If the mesh has some exceptionally thin walls that you want the cells to be small enough to fill you may need to go as high as 7 or 8.

I was having some problems with the GPU AABB/Triangle overlap test I used for voxelization in the Hierarchical Z-Buffer Occlusion Culling article and so I ended up porting the Möller implementation of AABB/Triangle overlap test to C# and just used it instead.

Also if you ever need to lookup how an intersection is performed I highly recommend the gigantic matrix of intersections over at realtimerendering.com.  It was a handy resource since I don’t keep the algorithm for AABB/Triangle overlap stored in my brain.

After you’ve created the octree we need to process each cell that wasn’t intersecting with a triangle to determine if it is inside or outside.

Find Seed Cell

Before we can determine if a cell is inside or outside the mesh we need to find a seed cell.  The seed cell is sort of the ground truth example cell that we use to propagate its status to the other cells that it can see.  The seed cell’s status is determined by rendering a cube map centered inside the cell with the near plane placed at the cell edge.

When rendering each side of the cube map, you render the scene such that all front facing polygons are blue and all back facing polygons are red.  You then read back each cube map surface from the GPU and determine the percentage of red and blue pixels seen at each face.

If at least 4 sides of the cube map contain red pixels, the cell is determined to be inside the mesh.

The paper says that NO red pixels can be seen for a seed cell to be determined to be outside however I found this problematic since occasionally a red pixel can be seen just through tiny rendering artifacts.

So I feel a better solution is one like the following:

MIN_INSIDE_FACES = 4;
MIN_INSIDE_PERCENTAGE = 0.03f;
 
int cubemap_sides_seeing_inside = 0;
 
for (int i = 0; i < 6; i++)
{
    RenderCubeMapSide(i);
    float backfacePercentage = CalculateBackfacePercentage(i);
 
    if (backfacePercentage > MIN_INSIDE_PERCENTAGE)
        cubemap_sides_seeing_inside++;
}
 
if (cubemap_sides_seeing_inside >= MIN_INSIDE_FACES || cubemap_sides_seeing_inside == 0)
{
    if (cubemap_sides_seeing_inside >= MIN_INSIDE_FACES)
        cell.Status = CellStatus.Inside;
    else // cubemap_sides_seeing_inside == 0
        cell.Status = CellStatus.Outside;
 
    // Propagate cell status...
}
else
{
    // Unable to solve status exactly.
}

Where you don’t count a face as inside unless at least 3% of the total red and blue pixels are red.  The percentage is just something I picked out of thin air, it feels like a number small enough to be easily overcome by any truly inside cube face, but high enough to allow me to ignore tiny artifacts.

Propagate Seed Cell

The last step is to propagate the seed cell’s status to other cells.  After classifying a seed cell you need to test every unknown cell against each the depth map and frustum of each cube map surface.

You’re performing a test to see if any of the 8 corners of the octree cell when projected into the camera space of each cube face are closer than the depth value at that pixel.  If it is, then then the entire octree cell likely visible.

If the cell is visible from the seed cell, then in all likelihood the cell has the same status as the seed cell.  However because having holes means 2 seed cells (one inside and one outside) can potentially see the same cell you want several seed cells to confirm the status of a cell before committing to it.

So once you’ve determined a cell is visible from the seed cell you’ll increment a counter on the cell for that status.  Once one of the statuses reaches a threshold, like for example 16 you’ll change the status of the cell from Unknown to whatever status counter overcame the threshold and no longer process that cell.

It should be noted that only seed cells propagate their status.  Cells that you propagate to do not propagate their own status.

Repeat

After you’ve found a seed cell and propagated its status you’ll continue to repeat finding a seed cell and propagating the seed cell until all cells have a status of inside, outside or intersecting.  You can rarely end up with some cells whose status simply can’t be determined so make sure your code can handle that scenario and not loop forever.

Improvements

While implementing the paper I made some additional improvements to the proposed solution.  I sped up the process by taking advantage of hardware improvements to render the scene using a single pass.  I also improved the conservativeness of the algorithm in situations where you’re using square voxels. When a mesh is wider than it is tall in those sitations there will be padding below the mesh; if the bottom of the mesh is uncapped it can lead to inside cells ‘leaking’ their status outside.

Single Pass Rendering

The paper was published back in 2002 and due to limitations at the time the simplest method of rendering front faces one color and back faces another was to render the scene twice and flip the winding order and the color of the triangles being rendered.  However this method is slower than just using a simple pixel shader to change the color of front/back faces.

In OpenGL you can use gl_FrontFacing and in DirectX 11 you can use SV_IsFrontFace.

void main()
{
     gl_FragColor = gl_FrontFacing ? vec4(0,0,1,1) : vec4(1,0,0,1);
}

Intersect Mesh Bounds and Clip To Bounds

One problem I found is that when a mesh (like a building) is uncapped at its base but is wider than it is tall there will be several cells below the base of the mesh.  Cells that will have the status of inside spread to them, even though a human could easily see those cells are outside.

So one improvement I ended up adding is that when testing for triangle intersection, you should also test against intersection against the mesh bounds.  Additionally you immediately mark a cell as Outside if is outside the mesh bounds, since it simply is not possible for that cell to be inside the mesh, but don’t treat that cell like it’s a seed cell; just mark it as outside and move on.

I needed some real game art to properly test the solution so I exported a roof structure from the Necropolis map from UDK’s UTGame sample.  Here you can really see the difference it makes to clip to the bounds of the mesh.  Note how many additional voxel/octree cells (purple lines) are determined to be ‘inside’ because of how many backfaces (red triangles) they can see.

Figure 1 – Roof Inner Voxelization Not Bounded (Before)

Figure 2 – Roof Inner Voxelization Bounded (After)

Future Improvements

When the cube map for each seed cell is processed it’s read back from the GPU and each pixel is checked on the CPU.  This is wildly inefficient when all we care about is the percentage of red to blue pixels on each face.

So that processing can be moved to OpenCL to improve the performance significantly.  I would prefer to have all cells be seed cells since that makes it easier to define the rules of what cells are inside vs. outside.  Allowing the cells to propagate has a higher potential to cause problems I suspect on meshes with very nasty artifacts.  Giving each cell the ability to individually determine their status will be more stable and more predictable.

Currently my cube map rendering is performed in 6 passes.  Shifting over to a single pass method using the geometry shader will likely add additional speed improvements, but I don’t know for sure.

If I move enough of the processing to the GPU it may allow me to make more cells seed cells (perhaps all?) and still maintain an acceptable performance footprint for tool time usage.  For offline usage though this method is already very acceptable (a few seconds for an average mesh and a maximum depth of 5) even with all the CPU read-backs I’m performing.

Sample Code

I’m still working on an improved version of the Generating Occluders for Hierarchical Z-Buffer Occlusion Culling sample.  So the code is a bit tied up at the moment.  However the next post I do on generating the occluders will contain it, which should be soon.

Robust Inside and Outside Solid Voxelization @ altdevblogaday.com

Tiled Resources for Xbox 360 and Direct3D 11 – Matt Lee

This talk was about mega-texturing in DirectX 11/Xbox 360.  Matt Lee was showing a new DirectX SDK sample that’s coming in the next SDK release giving a reference implementation of a mega-texturing run-time.  I’ve only skimmed mega-texturing papers so I got a lot out of this talk since he walked through all the steps in the run-time.

The sample shows off how you begin by creating different tiles for different resource formats.  Each pool is dedicated to a different texture format.  The tiles in the pool are all the same size; However the tiles may vary in size depending upon the texture format to maximize cache efficiency.  When you render the scene you have a shader that can write out texture look-up failures.  When the UV coordinates and mip level are not found to be resident in memory a failure is added to this list.  After the shader completes you read back the failures and proceed to load the tiles that will fit in your established pools.

Unlike most texture streaming systems you’re not loading an entire mip level or the entire mip chain of the texture.  You’re only ever loading into the tiles a sub-region of a texture (like a 64×64 pixel region), which overcomes one common texture streaming problem, texture memory fragmentation.  Because the tile pools you create are never deallocated you don’t have to worry about fragmenting your texture memory because of different sized textures being streamed in and out.

Now the sample is not without its short comings, but that is mostly due to hardware limitations.  Ideally the virtual texture system would be transparent, you wouldn’t need to write a shader that recorded look-up failures.  The GPU and DirectX would simply report when a failure occurred and allow you to handle it.  Maybe some day…

Gesture Detection Using Machine Learning – Claude Marais

If you have ever been interested in machine learning this is a worthwhile presentation to check out when the slides are posted.  Claude Marais talked about a case study they performed to try and use machine learning to detect a Punch and a Kick.  For their experiment they used Adaboost which is a machine learning technique that combines thousands of weak classifiers that ‘boost’ each other and provide you with a high degree of accuracy in the results.

The classifiers are all extremely simple things, for example you may have a classifier like:

if (elbow_joint_angle > ANGLE)
    return 1;
return -1;

Then simply create a macro and have 180 variants of this classifier one for each ANGLE.  If you can imagine all the different things you could measure about the skeleton, creating simple variants of the kernels for each of the possible test cases will explode the number of weak classifiers you have; Claude had around 21,000 weak classifiers for his system.

The training phase looks at labeled data sets to know what examples of punches look like (positive examples) and what -not- punches look like (negative examples).  It uses the +1/-1 scores each weak classifier provides to determine the weights to apply to each classifier.  After it has determined the best weak classifiers to detect a punch and not detect a negative example as a punch on accident you can use the classifiers at run-time with the weights applied to detect a punch.

The results were undeniable; they had a demo setup the the expo area that was really good at detecting a punch and kick.

The only real drawback to this solution is the data collection; they needed something on the order of 70,000 examples of punches and 7x that in negative not a punch examples before the training produced accuracies over 90-95% from the chart they had; if my memory is correct.

In training the system they had 70,000 frames worth of recorded training data.  The actual number of recorded punches used to train the system was 25 different people doing 10 punches, so around 250 punch examples.  Then they had about 7x that number in negative training examples, which might be things like waves, or other actions that SVM can use to differentiate between random movement and an intentional punch. (Thanks to Claude for clarifying this)

Kinect and Kids: Pitfalls and Pleasantries – Deborah Hendersen

If you had asked me to make a Kinect game for kids (ages 3-6) before seeing this presentation I likely would’ve designed something with a dumb-me as the target audience.  What I quickly realized is how wrong I would’ve been to make that assumption.  At that stage of development kids are not capable of interacting with games I’m used to playing.

Something as simple as a menu of options is an impossibility since they are illiterate.  How many games have you seen that you could play without knowing how to read?

When interacting with an onscreen character, the kids ignore social norms of waiting for the person to finish talking.  They may just jump the gun if they already know what is expected and get frustrated if they can’t do it when they want to.

Kids are distracted very easily and will make their own games out of game behavior.  Deborah mentioned one story where a kid stopped playing the game because he realized he could get the game to react to leaving the play area and Kinect could no longer detect him the game would do something.  So he made up his own game of jumping in and out of the play area to activate this condition; utterly boring for adults, completely entertaining for this kid.

You almost have to design the game like passive experience like a children’s TV show.  Where on TV because there is no feedback, the TV show host asks the kid, “Can you find _______?” and the kid at home says something, and expecting this the show simply pauses while he waits for the response.  The game has to function in essentially the same way, regardless of the kid participating in the expected fashion the game has to move forward.  If it functions like a state machine that requires proper actions to move forward the kid may become bored and simply want to move on.  If the game refuses to let them move on, they’ll just walk away.

I really enjoyed this presentation because it was very clear how difficult the problem space is and it was interesting to hear how they tried to solve each one.

Kinect Hands: Finger Tracking and Voxel UI - Abdulwajid Mohamed and Tony Ambrus

This presentation was broken into two completely different parts, the first part was on finger tracking with Kinect.  This is one area I’ve been playing around in for awhile so it was interesting to see someone else’s attempt to solve the problem.  Because the Kinect is a structured light depth camera you don’t necessarily have depth at each pixel like you would on a time of flight depth camera.  Structured light cameras build a topology of depth using the light pattern it projects into the scene, viewed from a different angle it can discern depth, but a single dot does not give you depth.  It connects groups of them when determining the depth of a surface.  This means that even though your hand can be seen by Kinect, the further you back away from the sensor, the more like a mitten your hand becomes.  The gaps between your fingers disappear until they are just clumps on your wrist.

Because of this limitation you can’t go past 10 feet, there simply isn’t enough data.  Ideally the user is at 6 feet or closer, past 6 feet the accuracy begins to break down.

The way Microsoft tackled the problem was to first capture lots of hand examples and then to train an SVM (Support Vector Machine) against a curvature analysis of the hands.  So once you know all the pixels that make up a persons hand you find the points on the hand that result in the largest changes in curvature.  On an open hand these curves are your fingers and if you’re close enough to the camera that it can see the gaps between fingers it’s a very large change in curvature.  A closed hand has more or less a uniform curvature change viewed from any angle.  By training the SVM against a set of closed hand curvature examples vs. open hand curvature examples they were able to get pretty accurate results at about the 6-8 foot range for an adult, 5-7 feet for kids.

Because the detector is instantaneous i.e. it can tell you in a single frame is the hand open or closed, you need some something to counteract a single/couple misinterpreted frame.  So they trained an HMM (Hidden Markov Model) on examples of a flaky transition where the system is quickly switching between 2 states because the hand is at an odd orientation confusing the SVM; I thought it was an interesting solution to the problem.  I’ve only ever tried something simple like requiring 3 contiguous frames of agreement to have a state change.

The second half of the presentation was on a 3D (not stereoscopic) UI for Kinect.  One of the problems with navigating a ‘push to click’ interface is that it’s hard to correct for user drift.  When a user pushes forward they may do several things,

• Push toward the TV
• Push toward the sensor
• Push forward (wherever forward happens to be at that moment in time)

Depending upon what you’re expecting them to do there’s going to be drift away from the thing on the screen they are trying to click.  To attempt to correct this Abdulwajid presented a UI where the hands are visualized as voxelized clumps of boxes in a 3D environment with 3D buttons that could be mashed.  Seeing the hand in the same space as the button appeared to make it much easier to perform the click.

One thing I noticed that was not called out was his use of 2 directional shadow casting lights.  By having 2 directional lights facing each other both casting shadows, the resulting effect is a focal point.  As the hand gets closer to a surface the eye perceives the two shadows heading towards each other and can see the point where they will meet.  I thought that was and additional powerful indicator of where your hand was moving in the space and made it much easier to correct drift.

The September 2011 issue of Game Developer Magazine contains an article of mine!  They asked me to write an article for the “Inner Product” section summarizing the Hi-Z GPU Culling algorithm as well as the related research I started working on to automatically generate occluders for the runtime.

My copy hasn’t yet arrived in the mail so I don’t know how the final version turned out, but the last draft I saw after the good folks at GD Mag spruced up the language in my article was looking really good.

It was a lot of work putting the article together, more than I actually thought going into it.  Especially since I already had 3 blog entries to pull from.  However, you find yourself second guessing everything you write because it’s not like a blog entry that you can go back and edit if there’s a mistake.

It also becomes a bit more of a challenge to summarize things that you simply can’t add a link to and tell the reader, go read this.  Also, not being able to just insert a huge chunk of code makes explaining something simple often a lot harder to explain in words.  Hopefully you’re good at making diagrams in those situations.

It was an interesting experience.  Really glad I got the opportunity to contribute an article.

I ended up coming up with a solution that I thought would be good to do a post on, since there doesn’t seem to be much out there on SWIG and these types of problems.

Some Background

For those unfamiliar with why it’s difficult, let me explain. When your API returns a native pointer to be wrapped, of lets say class Foo, and your function returns Foo* but the instance it returns is actually the more derived version Bar. Sadly SWIG has no way of knowing this, so the object it creates in C# would be a C# Foo class. Therefore, if you tried to cast the C# Foo to the more derived C# Bar you’d be unable to because as far as .Net is concerned the instance of Foo is just a Foo and nothing more.

Alright now that you understand the problem, lets talk about how we can solve it.

How We Can Solve It

The solution to the problem can be solved in essentially 3 ways,

1 // Walk Away

Simply don’t do anything that would require it. Don’t expose classes that users will ever need to downcast to for any reason. If this option is available to you, take it. This however means that your base class needs to support all the functionality a user will ever need, something not always possible or desirable. But if you can, do it, because once you get to script land, down casting becomes a expensive process.

2 // Manual Cast Wrapper

Write or generate C++ functions for every downcast that you would like to perform and SWIG those functions. This would allow you to take object Foo as a parameter, returns Bar, does the dynamic cast in C++ and returns the pointer to Bar.

This option is manpower intensive and is truly brute forcing a solution. It also causes potential holes where nasty aliasing occurs and can become a real problem if the source object is ever deleted. Imagine an API with a factory function that returns object Foo, but Foo is actually Bar due to the way the factory works. You need to access a Bar specific only function on the object so you downcast Foo to Bar.

Now in C++ these objects are the same object, but SWIG doesn’t know this, so two completely different C# objects are created one wrapping the native pointer to Foo, and one wrapping the native pointer to Bar. If the SWIG layer was properly authored, the factory function notes that the object returned was created with new memory and thus SWIG is responsible for deleting the object.

So we call our Bar specific function and we decide to hang onto Bar and we let Foo on the stack pass out of scope, because why hang onto it, it’s the same object as Bar…right? But Foo was responsible for managing the memory for the common native object they both point to. So the underlying native object is destroyed when the garbage collector runs next after Foo is no longer referenced and now our Bar object points to totally bogus memory.

3 // Custom Solution

The solution I ended up coming up with requires you to have a custom RTTI system that allows you to check if an object is a specific type by name.

The first step is to extend the C# wrapper for your base class with RTTI information. If you don’t have a base RTTI object this will be a bit harder. The reason we have a new m_castedSource member variable is to solve the aliasing problem. After the cast we can store the source here so that we never have to worry about the garbage collector accidentally cleaning up memory we’re still using.

%typemap(cscode) YourBaseRTTIObject
%{
    internal YourBaseRTTIObject m_castedSource;
 
    public bool IsKindOf<T>()
    {
        // Use your C++ RTTI system to test if this wrapped C# object's
        // native object is a 'typeof(T).Name'
    }
%}

The next step is to create a function that can create the C# method needed to factory the C# object. We could use reflection, but that method is slow and we can do a far better job by emitting IL at runtime to specifically initialize one type and just cache those created methods.

private static Dictionary<Type, Func<IntPtr, bool, object>> constructorCache =
    new Dictionary<Type, Func<IntPtr, bool, object>>();
 
private static Func<IntPtr, bool, object> CreateConstructorDelegate<T>()
{
    // We need to first grab the reflection information for the more derrived type we're casting to.
    // SWIG has protected constructors we'll be able to call that take 2 parameters, the native pointer
    // to the type and whether or not SWIG owns the memory
    ConstructorInfo ctor = typeof(T).GetConstructor(
        BindingFlags.Instance | BindingFlags.NonPublic, null,
        CallingConventions.HasThis, new Type[] { typeof(IntPtr), typeof(bool) }, null);
 
    // Lets emit some IL that will allow us to construct our type much
    // faster than using reflection.
    DynamicMethod dm = new DynamicMethod("Create", typeof(object),
        new Type[] { typeof(IntPtr), typeof(bool) }, typeof(T), true);
    ILGenerator il = dm.GetILGenerator();
    il.Emit(OpCodes.Ldarg_0);
    il.Emit(OpCodes.Ldarg_1);
    il.Emit(OpCodes.Newobj, ctor);
    il.Emit(OpCodes.Ret);
 
    // Create the delegate for the dynamic method, which will further improve the calling speed.
    return (Func<IntPtr, bool, object>)dm.CreateDelegate(typeof(Func<IntPtr, bool, object>));
}

After that we need to add a function that can perform the down cast. This is just a matter of checking if the object is actually the desired casting type. If it is we get or create the factory delegate for the type. We construct the new wrapper object and hand it the pointer for the type. (NOTE: This is only viable if the new type isn’t offset in the vtable, as would be the case with multiple inheritance).

public static T CastTo<t>(YourBaseRTTIObject self) where T : YourBaseRTTIObject
{
    if (self == null)
        return null;
 
    // Check if the object is actually the type we're trying to cast to.
    if (self.IsKindOf<t>())
    {
        Type type = typeof(T);
 
        // Check if we've already cached the factory function.
        Func<IntPtr, bool, object> factory;
        if (!constructorCache.TryGetValue(type, out factory))
        {
            factory = CreateConstructorDelegate<t>();
            constructorCache.Add(type, factory);
        }
 
        // Call the factory function and set the casted source to the
        // current object.
        T castedObject = (T)factory(YourBaseRTTIObject.getCPtr(self).Handle, false);
        castedObject.m_castedSource = self;
 
        return castedObject;
    }
    return null;
}

Sadly it never went past stage one.  It’s harder than you might think to hold a T-pose for 3 minutes while someone circles you twice snapping pictures at 10 degree intervals.

I don’t have any pictures of the results; I came out looking like the elephant man.  We’ll probably try again at some point, but in the meantime I made another scan of a pair of static objects that was turning out pretty good until I got to the back of the monkey.

Click here to view the embedded video.

All in all Autodesk’s Photofly software is pretty cool.  It’s still lacking in the area of iteration and debugging.  You can try manually tagging photo matchup points between images to give it a better idea on how the images fit together but it takes awhile for the data to be processed in the cloud.  It’s also unclear where some data comes from, or why portions of the background become part of the foreground mesh.  If it had better feedback for how that data became part of the mesh cleaning up the results would be a lot easier.

Also, if you own a camera with a sports video mode that captures at 60 FPS you can just slowly circle the subject and then dump all the frames using ImageGrab.  Which is way easier than snapping individual pictures.

I wonder if I could generate 3d art for a game jam…