System Integration, Configuration, and Calibration¶

These installation steps connect the additional system components, specifically the 3D camera (Kinect or other camera) and the digital projector, physically align them with the sandbox for optimal image quality, and calibrate the camera with respect to the projector so that real and projected features in the sandbox line up precisely.

Step 1: Connect and Configure the 3D Camera¶

Instructions for this step start at 28:52 in the walk-through video.

Plug in your 3D camera and download intrinsic calibration parameters directly from its firmware. In a terminal window, run:

sudo /usr/local/bin/KinectUtil getCalib 0

This might ask you for your password again; if so, enter it to continue.

Step 1a (Optional): Calculate Per-pixel Depth Correction¶

First-generation Kinect cameras (Kinect-for-Xbox-360) have a certain amount of non-linear depth distortion. If you point such a camera at a flat surface, the 3D reconstruction will not be flat, but slightly bowl-shaped. This distortion can prevent getting a good alignment between the physical sand surface and the topographic map projection in Step 5.

RawKinectViewer has a built-in calibration utility to correct for this distortion. This step is not necessary for operation of the ARSandbox, but it usually improves alignment quality. Again, this issue only applies to first-generation Kinect cameras (models 1414, 1473, and 1517). It is easiest to perform this calibration step with the Kinect camera removed from the ARSandbox, so if your Kinect is difficult to remove after assembly, it might be a good idea to do this step before putting it in place.

To perform per-pixel depth correction, place your Kinect camera such that it faces a large flat non-shiny surface, like an empty wall. Then start RawKinectViewer as administrator, as it will need to write a calibration file to the system location /usr/local/etc/Vrui-8.0/Kinect-3.10. In a terminal window, run:

sudo /usr/local/bin/RawKinectViewer -compress 0

Enter your password if/when asked.

To begin calibration, move your camera towards the flat surface until black blotches start appearing in the depth image stream on the left side of the display window of RawKinectViewer, and then pull back until there are only small and isolated black spots. It is best to have the camera face the flat surface head-on; rotate the camera left/right and up/down until the displayed depth color is uniform between the left, right, top, and bottom edges.

Next, assign a calibration tool to two arbitrary keys. Press the first key, say A, and move the mouse to highlight "Calibrate Depth Lens" in the tool selection menu that pops up, then release the key. This will close the tool selection menu and open a "Creating Calibrate Depth Lens Tool..." dialog prompting you to press another key for tool function "Calibrate." Pick a key, say S, press and release it. This will close the dialog, and finish tool assignment.

To capture a calibration tie point, keep the camera very still and press the A key once. This will show a "Capturing average depth frame..." message. Do not touch or move the camera while this message is being displayed. After it disappears a few seconds later, move the camera a little distance further away from the flat surface, and press A again to capture the next tie point.

Repeat this process a few times to collect anywhere between five and ten tie points, from distances between about 0.5m and 1.5m. Once done, press the S key. This will calculate per-pixel depth correction factors and write them to a calibration file in the /usr/local/etc/Vrui-8.0/Kinect-3.10 directory. RawKinectViewer will print "Writing depth correction file /usr/local/etc/Vrui-8.0/Kinect-3.10/DepthCorrection-<camera serial number>.dat" when it is finished. If any error messages appear at this point, close RawKinectViewer and redo the entire process. Otherwise, close RawKinectViewer, install the camera in the ARSandbox, and continue with the next installation step.

Step 2: Align the 3D Camera¶

Align your camera so that its field of view covers the interior of your sandbox. Use RawKinectViewer to guide you during alignment. To start it, run in a terminal window:

cd ~/src/SARndbox-*.*
RawKinectViewer -compress 0

Ignore the color video stream on the right side of RawKinectViewer's display window and focus on the depth image stream on the left (the ARSandbox does not use the color video stream). Move and/or rotate the camera until it can see the entire interior of your sandbox.

Step 3: Measure Sandbox's Base Plane Equation¶

Instructions for this step start at 0:00 in the ARSandbox Calibration video.

Calculate your sandbox's base plane by following the beginning of the video linked above. You can use the already-running instance of RawKinectViewer from Step 2.

To recap, you need to bind an "Extract Planes" tool from the root tool menu to some unused mouse button or keyboard key (see the note on "Tool Assignment" in Step 3 in "Software Installation")for details, then select "Average Frames" from RawKinectViewer's main menu, wait for the depth image display to freeze, and then draw a rectangle on the depth image by pressing and holding the button/key to which you assigned the "Extract Planes" tool. This will print the plane equation best fitting the selected rectangle to the terminal window from which you ran RawKinectViewer. After extracting the base plane, you should turn "Average Frames" back off in RawKinectViewer's main menu.

You need to enter the base plane equation (and the 3D sand surface extents in the next step) into the BoxLayout.txt file in etc/SARndbox-*.* inside the SARndbox source directory. To edit the file using xed, run in a terminal window:

cd ~/src/SARndbox-*.*
xed etc/SARndbox-*.*/BoxLayout.txt &

The ampersand ("&") at the end of the second command will keep the terminal window usable while the text editor is running. Now enter the base plane equation as described in the video. To copy text from a terminal window, highlight the desired text with the mouse, and then either right-click into the terminal window and select "Copy" from the pop-up menu that appears, or press Shift+Ctrl+C. To paste into the text editor, use the "Edit" menu, or press Ctrl+V. Or, highlight the desired text in the terminal window with the mouse, and then move the mouse into the desired position in the text editor window and press the middle mouse button to copy and paste.

RawKinectViewer prints two plane equations when a depth plane is extracted: the first in depth space, the second in camera space. The ARSandbox needs the second, camera-space, plane equation. After copying it, the equation has to be reformatted slightly. RawKinectViewer will print:

Camera-space plane equation: x * <some vector> = <some offset>

where <some vector> is a three-component direction vector defining the "up" direction in camera coordinates, typically close to (0.0, 0.0, 1.0). <some offset> is the vertical position of the measured plane underneath the camera, which is in centimeters and negative. BoxLayout.txt needs that plane equation in the following format, where the direction vector and the offset are separated by a comma:

<some vector>, <some offset>

Step 4: Measure Sandbox's 3D Box Corner Positions¶

Measure the 3D extents of the sand surface. As of version 3.2 of the Kinect package, this can be done inside RawKinectViewer as well by following the instructions at 4:10 in the ARSandbox Calibration video. Make sure to measure the box corners in the order: lower-left, lower-right, upper-left, upper-right.

To recap, you need to bind a "Measure 3D Positions" tool from the root tool menu to some unused mouse button or keyboard key (see Step 3 for details), and then click on the corners of your sandbox in the depth image using the button/key to which you assigned the "Measure 3D Positions" tool. This will print a sequence of 3D positions to the terminal window from which you ran RawKinectViewer.

After you have copied the box corner positions into the text editor as described in the video, save the file (via the "File" menu or by pressing Ctrl+S), and quit from the text editor (via the "File" menu or by pressing Ctrl+Q or by closing the window).

After Steps 3 and 4have been completed, the contents of BoxLayout.txt should look like the following, with different numbers depending on your installation:

(-0.0076185, 0.0271708, 0.999602), -98.0000
(  -48.6846899089,   -36.4482382583,   -94.8705084084)
(   48.3653058763,   -34.3990483954,   -89.3884158982)
(   -50.674914634,    35.8072086558,   -97.4082571497)
(   48.7936140481,    36.4780970044,     -91.74159795)

Ensure that this text starts in the first column of the first line, and that the file contains no other text at all.

Step 5: Align the Projector¶

Align your projector such that its image fills the interior of your sandbox. You can use the calibration grid drawn by Vrui's XBackground utility as a guide. In a terminal window, run:

XBackground

After the window showing the calibration grid appears, press F11 to toggle it into full-screen mode. Ensure that the window really covers the entire screen, i.e., that there are no title bar, desktop panel, or other decorations left. Press the Esc key to close XBackground's window when you're done.

Ensure to disable all digital image distortion features of your projector before alignment, and only use optical features, i.e., optical focus adjustment and optical zoom. Specifically, disable any kind of digital keystone correction, and check that the projector maps the incoming video signal 1:1 to its display pixels. The best way to check for 1:1 matching is to look at the central horizontal bar in XBackground's test image. It should be a clear pattern of alternating white and black one-pixel-wide vertical lines with no smearing or stair steps.

Slight overprojection outside of the sandbox, and any remaining keystone distortion of the projected image, will be taken care of by the following projector calibration step.

Step 6: Projector or Camera Calibration¶

Instructions for this step start at 10:10 in the ARSandbox Calibration video.

Calibrate the Kinect camera and the projector with respect to each other by running the CalibrateProjector utility:

cd ~/src/SARndbox-*.*
./bin/CalibrateProjector -s <width> <height>

./bin/CalibrateProjector -s 1024 768

Ensure that the given image size exactly matches the size of the projector's input video signal.

To recap, you need to bind a "Capture" tool from the root tool menu to two unused mouse buttons or keyboard keys to capture tie points.

The calibration program expects a disk of diameter 120mm (4.7"), which is a standard CD, CD-ROM, or DVD. The easiest way to create a calibration disk is to glue a sheet of white paper to the data side of a CD or DVD, carefully cut around the edge of the CD, and draw a cross onto the paper that intersects exactly in the center of the CD's hole.