NtKinect

NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to get and record audio with Kinect V2

2016.07.18: created by

To Table of Contents

Prerequisite knowledge

NtKinect: How to get RGB camera image with Kinect V2 (Fundamental Settings)

Getting and Recording Audio

If you define USE_AUDIO constant before including NtKinect.h, the functions and variables of NtKinect related to audio become effective.

If you call the setAudio() function, Audio data is acquired, and saved to the file if recording is in progress. To end recording, call closeAudioFile() function.

NtKinect

NtKinect's function for Audio

type of return value	function name	descriptions
void	setAudio(bool flag = false)	Acquire audio and save it in a file during recording Determines the direction of the speech and sets it to the variable "beamAngle". Argument flag When Calling this function with the argument set to "true" after setSkeleton() function call, the speaker's skeletonTrackingId[i] is set to the member variable "audioTrackingId".
void	drawAudioDirection(cv::Mat& image )	Draw the direction of audio beam on image.
bool	isOpenedAudio()	Returns whether or not recording is in progress.
void	openAudio(string path )	Open path as a recording file
void	closeAudio()	Close the recording file.

NtKinect

NtKinect's member variable for Audio

type	variable name	descriptions
float	beamAngle	Direction of audio (angle to the left and right)
float	beamAngleConfidence	Confidence value of beamAngle (0.0 ... 1.0)
UINT64	audioTackingId	Speaker's skeletonTrackingId

How to write program

Start using the Visual Studio's project KinectV2.zip of "NtKinect: How to get RGB camera image with Kinect V2 (Fundamental Settings)"
Add WaveFile.h to the project.

Download a little modified WaveFile.h of "AudioCaptureRaw-Console C++ Sample" by Microsoft, and place it in the folder where other souce files (such as main.cpp) are located. Then, add it to the project.

Change the contents of main.cpp as follows.

Define USE_AUDIO constant before including NtKinect.h.

Call the openAudioFile() function to start recording, and closeAudioFile() to end recording. Audio is acquired with setAudio() function, and audio is automatically saved in the file during recording.

In the example program below, we create a WAV file whose name is current time like "2016-07-18_09-16-32.wav".

main.cpp

#include <iostream>
#include <sstream>

#define USE_AUDIO
#include "NtKinect.h"

using namespace std;

#include <time.h>
string now() {
  char s[1024];
  time_t t = time(NULL);
  struct tm lnow;
  localtime_s(&lnow, &t);
  sprintf_s(s, "%04d-%02d-%02d_%02d-%02d-%02d", lnow.tm_year + 1900, lnow.tm_mon + 1, lnow.tm_mday, 
	    lnow.tm_hour, lnow.tm_min, lnow.tm_sec);
  return string(s);
}

void doJob() {
  NtKinect kinect;
  bool flag = false;
  while (1) {
    kinect.setRGB();
    if (flag) kinect.setAudio();
    cv::putText(kinect.rgbImage, flag ? "Recording" : "Stopped", cv::Point(50, 50),
		cv::FONT_HERSHEY_SIMPLEX, 1.2, cv::Scalar(0, 0, 255), 1, CV_AA);
// rename CV_AA as cv::LINE_AA (in case of opencv3 and later)
    cv::imshow("rgb", kinect.rgbImage);
    auto key = cv::waitKey(1);
    if (key == 'q') break;
    else if (key == 'r') flag = true;
    else if (key == 's') flag = false;

    if (flag && !kinect.isOpenedAudio()) kinect.openAudio(now() + ".wav");
    else if (!flag && kinect.isOpenedAudio()) kinect.closeAudio();
  }
  cv::destroyAllWindows();
}

int main(int argc, char** argv) {
  try {
    doJob();
  } catch (exception &ex) {
    cout << ex.what() << endl;
    string s;
    cin >> s;
  }
  return 0;
}

When you run the program, RGB images are displayed. Exit with 'q' key.

Recording starts with 'r' key, and stops with 's' key. Recording status is displayed as "Recording" or "Stopped" at the upper left of the RGB image.

Please click here for this sample project KinectV2_audio.zip。

Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.

http://nw.tsuda.ac.jp/