NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to make Unity's Humanoid take the same pose as Kinect V2's skeleton recognized in real time

2017.08.06: created by
2017.08.07: revised by
2017.08.10: revised by
2017.08.11: revised by
Japanese English

In this topics, Humanoid's face does not move. If you want to move humanoids face as well as its skeleton, please refer " Recognizing Skeleton and Face with Kinect V2 and move Unity's Humanoid Character in Real Time ".

To Table of Contents

Prerequisite knowledge

Make Unity's Humanoid take the same pose as Kinect V2's skeleton recognized




  1. Start using the Unity project CheckNtKinectDll5.zip of " NtKinect: How to make Kinect V2 skeleton recognition as DLL and use it from Unity" .

    2. Expand the above zip file and change the folder nane as "CheckNtKinectDll6/".

  2. Start Unity and Open the project "CheckNtKinectDll6".
  3. Start "New Scene" and save the scene as "Assets/Scenes/humanoid.scene".
    4. Menu bar -> File -> New Scene  

    Menu bar -> File -> Save Scene as ... -> humanoid.scene
  4. Create a folder "Assets/Models" and import Humanoid (Human Character ) data into the assets.

    5. Now, we use the data makehuman.zip created in " MakeHuman: Create Humanoid Model to use in Unity5 " .

    Expand the above zip file, then you will find the "AsianBoy.fbx"" and "textures/"" under the "exports/" . Please import them intto the "Assets/Models/" の下にimportします。




    [CAUTION]You can do the above operation through 

    Assets -> Import New Asset... -> AsianBody.fbx
    , but in this way, some textures may not be imported and the model became white. If you meet the situation, you must import all the textures in "Assets/Models/Materials/" manually.




  5. Select "AsianBoy" in the Project window, Change the Animation Type into "Humanoid" inthe Inspector's "Rig" window, click "Apply" button while Avatar Definition is "Create From This Model".



  6. When you apply the Humanoid Bone correctly, you can see the checked "Configure" in the Inspector window. Confirm to click "Configure" button.



  7. Let's check the bone assignment by clicking "Mapping" tab in the Inspector window.

    8. If required bones are not assigned, you can see no "check" sign before the "Configure". In this project we use optional bones. Please make sure that the "Neck" and "UpperChest" are correctly assgned. 。




  8. Click "Muscles & Settings" in the Inspector window and move the slider to see if each bone works properly. When you can confirm, click "Done".



  9. Drag "AsianBoy" from Assets/Models into Hierarchy window. Select "AsianBoy" in the Hierarchy window, and select "Reset" from Transform in the Inspector windows to reset Position (x,y,z)=(0,0,0).






  10. Create C# Script "RigBone.cs" at Assets/Scripts/ in the Project window. This "RigBone" class does not extend the MonoBihavior class.

    11. Given Humanoid and Bone, the RigBone class extract Bone's transform from the Animator component attached Humanoid and manages it.

    RigBone.cs
    /*
 * Copyright (c) 2017 Yoshihisa Nitta
 * Released under the MIT license
 * http://opensource.org/licenses/mit-license.php
 */

/* http://nw.tsuda.ac.jp/lec/unity5/ */
/* version 1.1: 2017/08/05 */
/* version 1.0: 2017/08/02 */

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class RigBone {
  public GameObject gameObject;
  public HumanBodyBones bone;
  public bool isValid;
  public Transform transform {
    get { return animator.GetBoneTransform(bone); }
  }
  Animator animator;
  Quaternion savedValue;
  public RigBone(GameObject g, HumanBodyBones b) {
    gameObject = g;
    bone = b;
    isValid = false;
    animator = gameObject.GetComponent<Animator>();
    if (animator == null) {
      Debug.Log("no Animator Component");
      return;
    }
    Avatar avatar = animator.avatar;
    if (avatar == null || !avatar.isHuman || !avatar.isValid) {
      Debug.Log("Avatar is not Humanoid or it is not valid");
      return;
    }
    isValid = true;
    savedValue = animator.GetBoneTransform(bone).localRotation;
  }
  public void set(float a, float x, float y, float z) {
    set(Quaternion.AngleAxis(a, new Vector3(x,y,z)));
  }
  public void set(Quaternion q) {
    animator.GetBoneTransform(bone).localRotation = q;
    savedValue = q;
  }
  public void mul(float a, float x, float y, float z) {
    mul(Quaternion.AngleAxis(a, new Vector3(x,y,z)));
  }
  public void mul(Quaternion q) {
    Transform tr = animator.GetBoneTransform(bone);
    tr.localRotation = q * tr.localRotation;
  }
  public void offset(float a, float x, float y, float z) {
    offset(Quaternion.AngleAxis(a, new Vector3(x,y,z)));
  }
  public void offset(Quaternion q) {
    animator.GetBoneTransform(bone).localRotation = q * savedValue;
  }
  public void changeBone(HumanBodyBones b) {
    bone = b;
    savedValue = animator.GetBoneTransform(bone).localRotation;
  }
}
  11. Creat C# Script "CharacterSkeleton.cs" at Assets/Scripts/ in the Project window. This class does not extend the MonoBihavior class.

    12. The CharacterSkeleton class manages data of Humanoid for one person, and changes the pose using given joints data.

    In the set() method, the joint data acquired by Kinect V2 is passed. As the joint data for up to 6 person are handed together, the index of the skeleton that is currently being focused is specified bye the 3rd argument offset, an integer from 0 to 5.

    The direction of Humanoid reflects the direction of the human body. The orientation of the human pelvis (rotation about the y-axis) is the direction of the whole humanoid, and the orientation of both shoulders is the direction of the upper body of the humanoid.

    CharacterSkeleton.cs
    /*
 * Copyright (c) 2017 Yoshihisa Nitta
 * Released under the MIT license
 * http://opensource.org/licenses/mit-license.php
 */

/* http://nw.tsuda.ac.jp/lec/kinect2/ */
/* version 1.3: 2017/08/11 */
/* version 1.2: 2017/08/10 */
/* version 1.1: 2017/08/07 */
/* version 1.0: 2017/08/06 */

using UnityEngine;
using System.Collections;
using System.Collections.Generic;

class CharacterSkeleton {
  public const int
    // JointType
    JointType_SpineBase= 0,
    JointType_SpineMid= 1,
    JointType_Neck= 2,
    JointType_Head= 3,
    JointType_ShoulderLeft= 4,
    JointType_ElbowLeft= 5,
    JointType_WristLeft= 6,
    JointType_HandLeft= 7,
    JointType_ShoulderRight= 8,
    JointType_ElbowRight= 9,
    JointType_WristRight= 10,
    JointType_HandRight= 11,
    JointType_HipLeft= 12,
    JointType_KneeLeft= 13,
    JointType_AnkleLeft= 14,
    JointType_FootLeft= 15,
    JointType_HipRight= 16,
    JointType_KneeRight= 17,
    JointType_AnkleRight= 18,
    JointType_FootRight= 19,
    JointType_SpineShoulder= 20,
    JointType_HandTipLeft= 21,
    JointType_ThumbLeft= 22,
    JointType_HandTipRight= 23,
    JointType_ThumbRight= 24,
    // TrackingState
    TrackingState_NotTracked= 0,
    TrackingState_Inferred= 1,
    TrackingState_Tracked= 2,
    // Number
    bodyCount = 6,
    jointCount = 25;

  private static int[] jointSegment = new int[] {
    JointType_SpineBase, JointType_SpineMid,             // Spine
    JointType_Neck, JointType_Head,                      // Neck
    // left
    JointType_ShoulderLeft, JointType_ElbowLeft,         // LeftUpperArm
    JointType_ElbowLeft, JointType_WristLeft,            // LeftLowerArm
    JointType_WristLeft, JointType_HandLeft,             // LeftHand
    JointType_HipLeft, JointType_KneeLeft,               // LeftUpperLeg
    JointType_KneeLeft, JointType_AnkleLeft,             // LeftLowerLeg6
    JointType_AnkleLeft, JointType_FootLeft,             // LeftFoot
    // right
    JointType_ShoulderRight, JointType_ElbowRight,       // RightUpperArm
    JointType_ElbowRight, JointType_WristRight,          // RightLowerArm
    JointType_WristRight, JointType_HandRight,           // RightHand
    JointType_HipRight, JointType_KneeRight,             // RightUpperLeg
    JointType_KneeRight, JointType_AnkleRight,           // RightLowerLeg
    JointType_AnkleRight, JointType_FootRight,           // RightFoot
  };
  public Vector3[] joint = new Vector3[jointCount];
  public int[] jointState = new int[jointCount];

  Dictionary<HumanBodyBones,Vector3> trackingSegment = null;
  Dictionary<HumanBodyBones, int>  trackingState = null;

  private static HumanBodyBones[] humanBone = new HumanBodyBones[] {
    HumanBodyBones.Hips,
    HumanBodyBones.Spine,
    HumanBodyBones.UpperChest,
    HumanBodyBones.Neck,
    HumanBodyBones.Head,
    HumanBodyBones.LeftUpperArm,
    HumanBodyBones.LeftLowerArm,
    HumanBodyBones.LeftHand,
    HumanBodyBones.LeftUpperLeg,
    HumanBodyBones.LeftLowerLeg,
    HumanBodyBones.LeftFoot,
    HumanBodyBones.RightUpperArm,
    HumanBodyBones.RightLowerArm,
    HumanBodyBones.RightHand,
    HumanBodyBones.RightUpperLeg,
    HumanBodyBones.RightLowerLeg,
    HumanBodyBones.RightFoot,
  };

    private static HumanBodyBones[] targetBone = new HumanBodyBones[] {
    HumanBodyBones.Spine,
    HumanBodyBones.Neck,
    HumanBodyBones.LeftUpperArm,
    HumanBodyBones.LeftLowerArm,
    HumanBodyBones.LeftHand,
    HumanBodyBones.LeftUpperLeg,
    HumanBodyBones.LeftLowerLeg,
    HumanBodyBones.LeftFoot,
    HumanBodyBones.RightUpperArm,
    HumanBodyBones.RightLowerArm,
    HumanBodyBones.RightHand,
    HumanBodyBones.RightUpperLeg,
    HumanBodyBones.RightLowerLeg,
    HumanBodyBones.RightFoot,
  };

  public GameObject humanoid;
  private Dictionary<HumanBodyBones, RigBone> rigBone = null;
  private bool isSavedPosition = false;
  private Vector3 savedPosition;
  private Quaternion savedHumanoidRotation;

  public CharacterSkeleton(GameObject h) {
    humanoid = h;
    rigBone = new Dictionary<HumanBodyBones, RigBone>();
    foreach (HumanBodyBones bone in humanBone) {
      rigBone[bone] = new RigBone(humanoid,bone);
    }
    savedHumanoidRotation = humanoid.transform.rotation;
    trackingSegment = new Dictionary<HumanBodyBones,Vector3>(targetBone.Length);
    trackingState = new Dictionary<HumanBodyBones, int>(targetBone.Length);
  }
  private void swapJoint(int a, int b) {
    Vector3 tmp = joint[a]; joint[a] = joint[b]; joint[b] = tmp;
    int t = jointState[a]; jointState[a] = jointState[b]; jointState[b] = t;
  }
  public void set(float[] jt, int[] st, int offset, bool mirrored, bool move) {
    if (isSavedPosition == false && jointState[JointType_SpineBase] != TrackingState_NotTracked) {
      isSavedPosition = true;
      int j = offset * jointCount + JointType_SpineBase;
      savedPosition = new Vector3(jt[j*3],jt[j*3+1],jt[j*3+2]);
    }
    for (int i=0; i<jointCount; i++) {
      int j = offset * jointCount + i;
      if (mirrored) {
	joint[i] = new Vector3(-jt[j*3], jt[j*3+1], -jt[j*3+2]);
      } else {
	joint[i] = new Vector3(jt[j*3], jt[j*3+1], savedPosition.z*2-jt[j*3+2]);
      }
      jointState[i] = st[j];
    }
    if (mirrored) {
      swapJoint(JointType_ShoulderLeft, JointType_ShoulderRight);
      swapJoint(JointType_ElbowLeft, JointType_ElbowRight);
      swapJoint(JointType_WristLeft, JointType_WristRight);
      swapJoint(JointType_HandLeft, JointType_HandRight);
      swapJoint(JointType_HipLeft, JointType_HipRight);
      swapJoint(JointType_KneeLeft, JointType_KneeRight);
      swapJoint(JointType_AnkleLeft, JointType_AnkleRight);
      swapJoint(JointType_FootLeft, JointType_FootRight);
      swapJoint(JointType_HandTipLeft, JointType_HandTipRight);
      swapJoint(JointType_ThumbLeft, JointType_ThumbRight);
    }
    for (int i=0; i<targetBone.Length; i++) {
      int s = jointSegment[2*i], e = jointSegment[2*i+1];
      trackingSegment[targetBone[i]] = joint[e] - joint[s];
      trackingState[targetBone[i]] = System.Math.Min(jointState[e],jointState[s]);
    }

    Vector3 waist = joint[JointType_HipRight] - joint[JointType_HipLeft];
    waist = new Vector3(waist.x, 0, waist.z);
    Quaternion rot = Quaternion.FromToRotation(Vector3.right,waist);
    Quaternion rotInv = Quaternion.Inverse(rot);
 
    Vector3 shoulder = joint[JointType_ShoulderRight] - joint[JointType_ShoulderLeft];
    shoulder = new Vector3(shoulder.x, 0, shoulder.z);
    Quaternion srot = Quaternion.FromToRotation(Vector3.right,shoulder);
    Quaternion srotInv = Quaternion.Inverse(srot);

    humanoid.transform.rotation = Quaternion.identity;
    foreach (HumanBodyBones bone in targetBone) {
      rigBone[bone].transform.rotation = rotInv * Quaternion.FromToRotation(Vector3.up,trackingSegment[bone]);
    }
    rigBone[HumanBodyBones.UpperChest].offset(srot);
    Quaternion bodyRot = rot;
    if (mirrored) {
      bodyRot = Quaternion.AngleAxis(180,Vector3.up) * bodyRot;
    }
    humanoid.transform.rotation = bodyRot;
    if (move == true) {
      Vector3 m = joint[JointType_SpineBase];
      if (mirrored) m = new Vector3(-m.x, m.y, -m.z);
      humanoid.transform.position = m;
    }
  }
}
  12. Create C# Script "RigControl.cs" at the Assets/Scripts/ in the Project window.
    13. RigControl.cs
    /*
 * Copyright (c) 2017 Yoshihisa Nitta
 * Released under the MIT license
 * http://opensource.org/licenses/mit-license.php
 */

/* http://nw.tsuda.ac.jp/lec/kinect2/ */
/* version 1.0: 2017/08/06 */

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.Runtime.InteropServices;

public class RigControl : MonoBehaviour {
  [DllImport ("NtKinectDll")] private static extern System.IntPtr getKinect();
  [DllImport ("NtKinectDll")] private static extern int setSkeleton(System.IntPtr kinect, System.IntPtr data, System.IntPtr state, System.IntPtr id);
  int bodyCount = 6;
  int jointCount = 25;
  private System.IntPtr kinect;

  public GameObject humanoid;
  public bool mirror = true;
  public bool move = true;
  CharacterSkeleton skeleton;
 
  void Start () {
    kinect = getKinect();
    skeleton = new CharacterSkeleton(humanoid);
  }
  void Update () {
    float[] data = new float[bodyCount * jointCount * 3];
    int[] state = new int[bodyCount * jointCount];
    int[] id = new int[bodyCount];
    GCHandle gch = GCHandle.Alloc(data,GCHandleType.Pinned);
    GCHandle gch2 = GCHandle.Alloc(state,GCHandleType.Pinned);
    GCHandle gch3 = GCHandle.Alloc(id,GCHandleType.Pinned);
    int n = setSkeleton(kinect,gch.AddrOfPinnedObject(),gch2.AddrOfPinnedObject(),gch3.AddrOfPinnedObject());
    gch.Free();
    gch2.Free();
    gch3.Free();
    if (n > 0) {
      skeleton.set(data,state,0,mirror,move);
    }
  }
}
  13. Add the Empty Object into the Hierarchy, and rename it as "RigController". Select "RigController" in the Hierarchy window, click the "Reset" from Transfrom in the Inspector window to set Position (x,y,z)=(0,0,0).



  14. Drag "RigControl.cs" from Assets/Scripts/ in the Project window onto the RigController object in the Hierarchy window. Select RigController in the Hierarchy window, you can see the "Rig Control (Script)" Component in the Inspector window. To the "Humanoid" item of the component, assign "AsianBoy" in the Hierarchy window.






  15. Change the position of Main Camera in Hierarchy window.
    16. Transform Position (x,y,z) = (0, 1, -2)
  16. Execute the program, and you can see the Humanoid takes the same pose as the human recognized with Kinect V2.

    17. If the "mirrored" variable is checked in the "Rig Control (Script)" component of RigController object, The humanoid takes the mirrored pose of the human. When you uncheck "mirrored" variable, Humanoid turns his back on the camera and follows the movement of human beings as it is. (CAUTION) The screen currently being recognized by Kinect V2, which is displayed small on the right of the running Unity window, is already a mirrored image.

    Since the value of the transform.rotation of Humanoid is changed by the program, the initial value of Rotation of AsianBoy's Transform can be anything.

    The captured screen of the running program is here CheckNtKinectDll6d.mp4. The mirrored variable was unchecked in the middle of execution.

    Also, if "move" variable is unchecked, the Humanoid will not follow the movement (place change) of the human.







    [Notince] We generates an OpenCV window in DLL to display the skeleton recognition state. Note that when the OpenCV window is focused, that is, when the Unity game window is not focused, the screen of Unity will not change. Click on the top of Unity's window and make it focused, then try the program's behaviour.

    [Notice 2] In the "CharacterSkeleton.cs" program, the optional bones like HumanBodyBones.Neck and HumanBodyBones.UpperChest are accessed. If the bones cannot be set with your Humanoid data, an error will occur.

    [Notice 3] In this program, the orientation of Humanoid's face is not changed. To do so, use face recognition and set the orientation of HumanBodyBones.Head.

  17. This sample project of Unity is here CheckNtKinectDll6.zip.

Recognize Multiple Skeletons and Reflect Them on the Movements of Multiple Humanoids

We will create a project to move up to 6 Humanods simultaneously.

  1. Exapnd the above Unity project CheckNtKinectDll6.zip, and rename the folder as "CheckNtKinectDll6b/"
  2. Save the scene with new name.
    3.   File -> Save Scence as -> humanoid2.unity
  3. Import 5 types of Humanoid to the "Asserts/Models" in the Project window.

    4. Here we usemakehuman2.zip data created with makehuman . Import AfricanBoy, AfricanGirl, AsianGirl, CaucasianBoy, CaucasianGirl, and texture/. Change Animation Type of each data as "Humanoid".




  4. drag 5 models into the Hierarchy from the Assets/Models/ to add.

    5. Please set the position and rotation of the Humanoid. Set the z-coordinate not to apear on the camera.

    Model NamePositionRotation
    XYZXYZ
    AsianBoy00-10000
    AsianGirl00-10000
    AfricanBoy00-10000
    AfricanGirl00-10000
    CaucasianBoy00-10000
    CaucasianGirl00-10000



  5. Create C# Script "Assets/Scripts/RigControl2.cs".
    6. RigControl2.cs
    /*
 * Copyright (c) 2017 Yoshihisa Nitta
 * Released under the MIT license
 * http://opensource.org/licenses/mit-license.php
 */

/* http://nw.tsuda.ac.jp/lec/kinect2/ */
/* version 1.1: 2017/08/10 */
/* version 1.0: 2017/08/06 */

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.Runtime.InteropServices;

public class RigControl2 : MonoBehaviour {
  [DllImport ("NtKinectDll")] private static extern System.IntPtr getKinect();
  [DllImport ("NtKinectDll")] private static extern int setSkeleton(System.IntPtr kinect, System.IntPtr data, System.IntPtr state, System.IntPtr id);
  int bodyCount = 6;
  int jointCount = 25;
  private System.IntPtr kinect;

  public GameObject[] humanoid = new GameObject[] {null, null, null, null, null, null};
  public bool[] mirror = new bool[] {true, true, true, true, true, true};
  public bool[] move = new bool[] {true, true, true, true, true, true};
  CharacterSkeleton[] skeleton = new CharacterSkeleton[] {null, null, null, null, null, null};

  void Start () {
    kinect = getKinect();
    for (int i=0; i<bodyCount; i++) {
      if (humanoid[i] != null) {
	skeleton[i] = new CharacterSkeleton(humanoid[i]);
      }
    }
  }
  void Update () {
    float[] data = new float[bodyCount * jointCount * 3];
    int[] state = new int[bodyCount * jointCount];
    int[] id = new int[bodyCount];
    GCHandle gch = GCHandle.Alloc(data,GCHandleType.Pinned);
    GCHandle gch2 = GCHandle.Alloc(state,GCHandleType.Pinned);
    GCHandle gch3 = GCHandle.Alloc(id,GCHandleType.Pinned);
    int n = setSkeleton(kinect,gch.AddrOfPinnedObject(),gch2.AddrOfPinnedObject(),gch3.AddrOfPinnedObject());
    gch.Free();
    gch2.Free();
    gch3.Free();
    for (int i=0; i<bodyCount; i++) {
      if (i < n && skeleton[i] != null) {
	skeleton[i].set(data,state,i,mirror[i],move[i]);
      } else {
	humanoid[i].transform.position = new Vector3(0,0,-10);
      }
    }
  }
}
  6. Select RigController in the Hierarchy window, and Remove Component "Rig Control (Script)" in the Inspector window.
  7. Drop Assets/Scripts/RigControl2.cs onto the RigController in the Hierarchy window. Select the RigController in the Hierarchy window, set the "Humanoid" variable of "Rig Control 2 (Script)" component in the Inspector window with 6 Humanoids.



  8. When executing the program, up to 6 skeleton of people are recognized and Humanods will follow the pose of human.
  9. The sample project of Unity is here CheckNtKinectDll6b.zip.
  10. For the way how to move Humanoid using face recognition as well as skeleton recognition, please see here.

http://nw.tsuda.ac.jp/