Motion Tracking of an object moving with Six Degrees of Freedom

What is Six Degrees of Freedom (6 DoF)?

6 DoF (Image from: http://forums.eidosgames.com/showthread.php?p=1377717)

In simple words, 6 DoF is ability of an object to, move along and rotate around, the three axes of a Cartesian Coordinate System.

Motion Tracking for Feel3D

There are many systems developed for  augmented reality projects which track motions But almost all of them are focused on tracking motion of human face or hand, and those systems use unique features of those body parts to track motion. So ability to use those mechanisms for this project is limited.

Forum post on stackoverflow.com generated lot of information on the current situation of this area.

Keeping the knowledge gained in mind we decided to try to design a mechanism to calculate motion of a 3D object.

The proposed approach for motion tracking

We are using a static background. So detecting the moving object can be done easily using a background subtraction mechanism. Then we will apply methods described below on the foreground (object) pixels to calculate needed information.

First we will track motion of the object giving it 2 DoF on X and Y axes of Cartesian Coordinate System. This is expected to be pretty much straight forward. There are many methods available to do this, including blob tracking using OpenCV library. We assume that movements are restricted to X,Y plane.

Secondly, we will add another degree of freedom to the object. We will calculate rotation of the object around Z axis. We propose, first find the line connecting mid points of front and back of the object. Then calculate the difference of rotation angle of this line at two time points. At this phase we assume that movements are restricted to X,Y plane and rotation only happen around Z axis.

Thirdly we will try to address the more challenging problems, tracking motion of the object with other 3 degrees of freedom. To tackle this we propose a method that use edges. We plan to detect edges of the object and then infer rotation angles based on distance between two parallel edges and the intensity values of pixels near the edges. We can also use surface features to infer rotation direction of the object.

Next steps of the project will be directed towards implementing above proposed approach.

An Introduction to Feel3D

Free-viewpoint video

Idea of making free-viewpoint video is relatively a new topic which surfaced few years back. So far, there have been several attempts to make free-viewpoint videos. The most common techniques involve recoding some event with several video cameras and using these video footage to build a video that can be watched from any viewpoint.

Free-viewpoint video has a great potential in revolutionizing entertainment industry. This will be a whole new experience to viewers. Videos will become realistic than ever before. Free-viewpoint videos will give the control to the viewer sitting in the living room holding the remote controller of the home theatre system. Viewer will be able to watch the video from the point of view of a character in his or her favourite cinema stripe, or feel the game from the point of view of a boxer in a boxing-ring or make a critical decision from the point of view of the umpire in a cricket match. Adopting free-viewpoint videos in to entertainment industry will require innovative approaches in disciplines such as video recording, film directing, video transmission, video player software etc. Applications of free-viewpoint video are not limited to entertainment.

Education, Medical science and Forensic Science will also benefit from free-viewpoint video technology. Students will be able to analyse some physics practical test done in the laboratory from different angles and researchers will be able to study their experiments in more comprehensive manners. In medical science surgeons will be able to perform and participate in remote operations with a better idea on what they are doing. With free-viewpoint video, understanding the crime scene and how it may have happened will be simpler than ever before. Investigators can validate evidence and testimonies by comparing them with a free-viewpoint video of the crime scene.

Uses of free-viewpoint video will not be limited to above applications. With advances in technology and way of life more and more applications will emerge. Has video making technology reached its peak in terms of improvements and new features?

Free-viewpoint 3D video

Enabling walk-through with in the video will be a totally new addition to watching experience. This will require making complete 3D videos, videos which will contain 3D models of the actors and objects rather than a set of 2D images taken from multiple angles. Such a technique will also require certain amount of planning, preparing and pre-processing from the video maker's side. From the viewer's side special hardware and software will be required for video rendering and watching. But the most significant requirement will be a transmission medium for broadcasting 3D data. Failing in transmission aspect may significantly limit applications of 3D videos. And this could even remove the commercial viability of 3D videos with regard to entertainment. Based on these facts, a free-viewpoint 3D video system should have following characteristics.

1. Use minimum bandwidth possible which will enable transmission of 3D video using existing infrastructure.
2. Provide good quality videos capable of providing an overall viewing experience.

First characteristic will define the format of the video and the technique used
for transmission. The second will define the video making, rendering and data
compression techniques.


Feel3D 

Methodology

3D video recording mechanism proposed here intends to achieve above characteristics through a unique approach. The system will follow a modular approach during all its main phases; 3D video making, storing and transmission.
The system proposed here is mainly targeted towards entertainment industry where video recording occurs within a pre-specified environment and characters that participate are known before hand. Recording an expedition program which will feature unknown environment and unexpected objects will not be feasible with the proposed system. The first requirement for video making is a 3D model of the environment where action will occur. Although 3D modelling of indoor environments with images still remain a tricky business, there are few researches that has successful modelled 3D environments integrating laser scanners and image sequences. Then 3D models of the characters or moving objects in the environment is acquired. With these models prepared we can move on to video making.

Before making the actual 3D video the recoding camera, real environment (RE) and virtual environment (VE) has to be calibrated. The calibration can be done by measuring the exact position of the camera in the real environment and applying that to VE which will be proportional to the real environment. Another optional methodology is processing images of a known object in the RE to find its relative size to RE and using this information to calibrate VE. For this to succeed pre-built 3D model of the object is needed.

Then the character or the object movements are recoded. The action must happen within the environment. The recorded video is processed for building the 3D video. During video processing, first an object detection algorithm is run and secondly movements of that object are tracked. Motion tracking involve displacement tracking and pose tracking. Data gathered from motion tracking is used to render the 3D model of the environment and 3D model of the object together to form the free-viewpoint video. 3D nature of the models allow having free-viewpoints and video nature can be achieved by updating object's position and pose according to the changes in the video.

Significance of Feel3D

3D video making in the proposed system will happen almost at real-time. Since 3D models are already built, only processing tasks that have to be done during recording are motion tracking and rendering of 3D models. This is a computationally light weighted process compared to building complete 3D models and 3D video during recording. Importance of camera position during recording is much higher in the latter system than the proposed system.

Proposed 3D videos will take less bandwidth and storage space. Conventional free-viewpoint 3D videos have to store the video as a whole. The format of these videos does not allow separation of objects and environment. But in the proposed system objects and environment are treated separately. This allows storing them as individual components, instead of a complete 3D video. Addition to 3D models, time relative motion updates calculated during motion tracking of the object are also stored. This will consume much less memory compared to other systems.

When it comes to transmission of the data, high bandwidth usage will only occur during the initial phase where 3D models have to be transmitted to the client system. After that phase, transmitted data will only limit to updates of the positions of the objects. In fact after initial phase bandwidth usage will be lesser than other systems. And the extra bandwidth space can be used to send the 3D models of the next scene. This will be a much efficient approach compared to other systems which have to send 3D data all the time.

Validation

For proving the success of the approach we will generate a simple video of a toy car moving around a room with several obstacles placed at arbitrary positions. These obstacles will be of basic shapes. 3D models of the object (the car) and the environment (the room) are built prior to actual video making. The purpose of this research is making a free-viewpoint 3D video using models. Based on that fact, we consider that focusing on a highly detailed environment and similar complex objects is considered beyond the scope.