LookOut! | Supplemental Videos

Welcome to the LookOut video gallery! We've captured hours of LookOut footage and associated telemetry data. we've loosely organized that footage here so you can get an idea of what LookOut can do and critique raw telemetry data.

Most videos will have two versions. "Star Footage" will contain footage from our main star camera, the Sony RX0, albeit at a reduced bitrate for consumption via web. "Illustrated footage" will contain a video output from the guide camera used to drive the tracking system. The footage is saved to disk and later (offline) overlayed with telemetry data that we store during LookOut operation. Note that the operator doesn't need video output for filming, and indeed only a very rudementry version of the video is output to a display for debug purposes. The illustrated footage is slowed down to make taking in telemetry easier. Not all LookOut videos will have illustrated footage and some will have missing telemetry data. An explanation of the telemetry is at the bottom of the page. Please also note that some videos were shot with an older version of our tracker that doesn't have a recovery stage; these videos will be marked with a bold asterisk (*) and are listed at the bottom of the page.

1. Detections from the tiny-YOLOv3 detector used in our tracker in opague blue.
2. Raw track bounding boxes from our tracker colored with the associated actor's distinct identifying color and labeled with the actor's identifier. The dashed colors of the boxes will change to dark orange when a target is in recovery and red when a target is being searched for.
3. Camera frame radial velocities (pitch and yaw velocities) over time.
4. Translational acceleration (front/back and side to side) experienced by the camera frame due to external forces.
5. Instantaneous camera frame radial velocities (yaw and pitch). Note that these are not the gimbal's motor velocities, but the radial velocities of the camera relative to the world.
6. Values for the orientation of the camera yaw and pitch relative to the world.
7. Actor process variances. As discussed in Sec. 6.1 in the paper, raw track values are augmented using a bank of Kalman Filters whose process variances are modulated to achieve areas of agnosticism around an actor's location. In these area, visualized using ellipses here, actor movement doesn't result in the actor's contribution to frame control to be changed. The vairable red bar (full for Eric here) illustrates the process variance ratio for movement in the X-axis. Green bars do the same, but for Y-axis movement. There are two of these green-red pairings, one for each actor. Some footage was filmed without agnostic areas, but with fixed variance Kalman Filters; in those cases, these panels will be greyed out.
8. Actor Control Weights. The weights each actor have in the controller are modulated depending on whether the actor is in frame (and being tracked) or involved in a scripted camera behavior. Here a script change requires the Car to also be centered, so the control weight for the Car is in the process of being increased. This mechanism is described in Sec. 5.2 of the paper.
9. Visualization for what the tracker's status is for each target. Each tracker update will run from right to left (incoming from the future and disappearing into the past). An OK from the tracker is shown in green, a recovery attempt in dark orange, and a searching state in red.
10. Visualization of the current camera behavior. Note the last heard cue showing what the last speech command or audible cue heard by the system was. In the white canvas, actor location requirements are shown with colored markers.
11. Leniency ellipses defined by the user for allowing areas of agnosticism around which the camera shouldn't move if an actor moves. Note how here the X-axis of Eric's ellipse boundaries are red, indicating that the smoothed augmented point is leaving this zone (or the zone is leaving the filtered point behind) and an update should occur to the smoothed augmented point by ramping up the Kalman Filters for this actor in the X-axis.
12. Smoothed Augmented actor points. These are the P^a points defined in Sec. 5.1 of the paper. These are a product of the leniency mechanism and are passed to the weighted procrustes component for final control.

Tracker Version

As mentioned above, some scenes were filmed with an earlier version of the LookOut tracker without recovery. They are listed here.