PEP: Performance Evaluation Platform for Object Tracking Methods
The goal of this project is to develop a complete performance evaluation platform for object detection tracking systems. We implemented all of the conventional metrics as well as MERL's own novel matrix based measures. We standardized the data representation and automated the evaluation of various configurations (different detector/tracker parameters) and tracking scenarios (multiple people/fast moving people/etc.).
Background & Objective: The issue of evaluating the performance of video surveillance systems is becoming more important as more and more research effort is drawn into object detection and tracking. It's a natural question to ask whether there has been quantifiable progress in the form of robust, commercial- grade video surveillance systems as a result of past and ongoing research in this direction. We address the issue of comprehensive performance evaluation of automatic object detection and tracking systems. We designed several performance evaluation metrics for quantitative assessment of the performance of video surveillance systems. Initial efforts towards performance evaluation of video detection and tracking systems began with the workshops dedicated to the topic, namely VS (visual surveillance) and PETS (performance evaluation of tracking and surveillance) series of workshops.
Technical Discussion: Frame-based metrics are used to measure the performance of surveillance system on individual frames of a video sequence. This does not take into account the response of the system in preserving the identity of the object over its lifespan. The results from individual frame statistics are then averaged over the whole sequence. This represents a bottom-up approach. On the other hand, the object-based evaluation measures take the whole trajectory of each object into consideration. The various ways of finding the best correspondence (association) between individual ground truth tracks and tracker result tracks are analyzed. Finally, based on a particular association, success and error rates are computed and accumulated for all the objects. This represents a top-down approach. We propose metrics for both approaches.
Publications:
Technology Area: Computer Vision
Modification Date: September 12, 2007