Autonomous and Scalable Crowd Surveillance


Consider a wide-angle camera is installed at an airport for human activity surveillance or in a forest for wildlife observation. The wide-angle camera can provide large, low resolution coverage of the scene. However, recognition and identification of humans and animals usually require close-up views at high resolution which needs PTZ cameras. The resulting autonomous observation system consists of a fixed wide-angle camera with multiple PTZ cameras as illustrated in Fig.~\ref{fig:Architecture}. The wide-angle camera monitors the entire field to detect and track all moving objects. Each PTZ camera selectively covers a subset of the objects.

Fig. 1. System Architecture

Camera Planning

However there are usually more moving objects than the number of PTZ cameras. With these competing spatiotemporal observation requests, the major challenge is the control and scheduling of the PTZ cameras to maximize the "satisfaction" to the competing requests. The system design emphasizes the "satisfaction" to the requests which takes into account the 1) camera coverage over objects, 2) camera zoom level selection, and 3) camera traveling time. We approach the control and scheduling problem in two steps. First, a subset of the requests/objects is assigned to each PTZ camera. Second, each PTZ camera selects its PTZ parameters to cover the assigned objects. We formulate the problems in both steps as frame selection problems and propose an approximation algorithm [3] to solve them in real time.



We have implemented and validated the system through both simulation and physical experiments. Our camera is mounted on the 6th floor of the Evans Library of Texas A&M University to monitor the crowd entering and leaving the library. The system is capable of partitioning the objects and assigning each PTZ camera with a subset of the objects. Each PTZ camera selects the parameters that ensure the assigned objects are covered for the duration of the observation. We attach a video clip that records a representative observation operation which contains two consecutive observation cycles at 17:25 on May 4th, 2009. (Video 1500K).

Fig. 2. An example of two surveillance cycles. Green rectangles are tracked subjects. Yellow rectangles are predicted locations and sizes of subjects. Red rectangles are PTZ camera frames.


  1. Yiliang Xu and Dezhen Song, Systems and Algorithms for Autonomous and Scalable Crowd Surveillance Using Robotic PTZ Cameras Assisted by a Wide-Angle Camera, Autonomous Robots (Conditionally accepted – minor revision) [ pdf 640K].
  2. Yiliang Xu and Dezhen Song, Systems and Algorithms for Autonomously Simultaneous Observation of Multiple Objects Using Robotic PTZ Cameras Assisted by a Wide-Angle Camera, The 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), St. Louis, USA, Oct. 11-15, 2009 [pdf 510k] .
  3. Yiliang Xu, Dezhen Song, Jingang Yi, and A.Frank van der Stappen, An Approximation Algorithm for the Least Overlapping p-Frame Problem with Non-Partial Coverage for Networked Robotic Cameras, IEEE International Conference on Robotics and Automation (ICRA), Pasadena, CA, May 2008. [pdf 310k]