A Distributed Virtual Vision Simulator

Wiktor Starzyk and Faisal Z. Qureshi

Synopsis

Realistic virtual worlds can serve as laboratories for carrying out camera networks research. This unorthodox "Virtual Vision" paradigm advocates developing visually and behaviorally realistic 3D environments to serve the needs of computer vision. Our work on high-level coordination and control in camera networks is a testament to the suitability of virtual vision paradigm for camera networks research. The prerequisite for carrying out virtual vision research is a virtual vision simulator capable of generating synthetic imagery from simulated real-life scenes. Here we present a distributed, customizable virtual vision simulator capable of simulating pedestrian traffic in a variety of 3D environments. Virtual cameras deployed in this synthetic environment generate synthetic imagery—boasting realistic lighting effects, shadows, etc.—using the state-of-the-art computer graphics techniques. Our virtual vision simulator is realized as a collection of modules that communicate with each other over the network. Consequently, we can deploy our simulator over a network of computers, allowing us to simulate much larger networks and much more complex scenes then is otherwise possible.

How to get it for your own research?

We have released the our virtual vision simulator to the community under GNU General Public License V3.0 in order to support virtual vision and camera networks research. Please feel free to download the code from Github repository and use it for your own research. Please send us bug fixes and improvements, so we may integrate them into the virtual vision simulator. Notice that if you want to use parts of our code in a commercial product they will need to comply with the GPLv3 license (all derived products are required to be open source).

Thanks to (annevanrossum@gmail.com) for helping us with the above wording.

Instructions about how to download, install and use our virtual vision simulator are provided below.

Caveat

We purchased pedestrian models (textures and motion capture data) from aXYZ Design, which hold the copyright to the pedestrian models. Licensing prevents us from distributing these models. You will have to purchase these models for our simulator to work properly.

If you are brave you may be able to use your own pedestrian models to work with our simulator. The simulator simply needs a textured human with animation data. If you plan to use your own pedestrian models, let us know. Time permitting we might be able to work together to get things rolling.

Citation

Please cite the following publications when using our simulator for research. Thank you!

"Smart Camera Networks in Virtual Reality," F.Z. Qureshi, D. Terzopoulos, Proceedings of the IEEE, 96(10), October, 2008, 1640—1656, (Special Issue on "Smart Cameras").

"Software laboratory for camera networks research," W. Starzyk, F.Z. Qureshi, IEEE Journal on Emerging and Selected Topics in Circuits and Systems (special issue on "Computational and Smart Cameras), vol. 3, iss. 2, pp. 284-293, 2013.

Publications

This simulator is introduced in the following publications.

"Software laboratory for camera networks research," W. Starzyk, F.Z. Qureshi, IEEE Journal on Emerging and Selected Topics in Circuits and Systems (special issue on "Computational and Smart Cameras), vol. 3, iss. 2, pp. 284-293, 2013.

Projects that used this Simulator

We used the current simulator to carry out camera networks research, which lead to the following two publications.

"Learning Proactive Control Strategies for PTZ Cameras,'' F.Z. Qureshi, W. Starzyk, Proc. 5th ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC 11), Ghent, Belgium, August, 2011, 1—6.

"Multitasking Smart Cameras for Intelligent Video Surveillance Systems,'' W. Starzyk, F.Z. Qureshi, Proc. 8th IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS 11), Klagenfurt, Austria, August, 2011, 1—6.

Videos

Multi-tasking PTZ Cameras

A video showing our simulator in action tracking pedestrians using a Vision Processing Client developed by Adam Domurad. [multitasking.wmv (49MB)]

Camera Handoffs

A video showing a Visual Processing Client connected to a static wide field of view camera handing off a pedestrian to another Visual Processing Client connected to an active pan\tilt\zoom camera. [handoff.wmv (3.7MB)]

Installation

Get the code from https://github.com/vclab/virtual-vision-simulator.

Linux

The system is developed on Ubuntu 10.10 using packages from the ubuntu repository. The following packages have to be installed:

Windows

Usage

All of the provided applications are located in the src folder.

Simulator

Execute: python 3D_Simulator.py [options] Where options are the options listed below.
NOTE: specifying a configuration directory is mandatory.

Options: 
-d directory    Set the directory where the config files are located
-p port         Set the port the server should run on (default is 9099)
-a              Set the mode of the simulation controller to automatic (only used when server not needed)
-s              A Sync session is used to control the time
-h              Print this help message and exit
--debug         Show debug messages
- left and right arrow keys switch between different cameras
- escape exits the applications
- i displays information about the simulator including ip address, port and camera types

Sync Client

Execute: python Sync_Client.py [options] ip_address:port ip_address:port ...
Where ip_address:port is replaced with the ip address and port of all the simulators

Options: 
--debug         Show debug messages

Sample Client

Execute: python sample_client.py [options]

Options: 
- p port        Set the port of the virtual world
- a ip_address  Set the ip address of the virtual world
- s             Save the images received from the server
- h             Print this help message and exit
--debug         Show debug messages

A screenshot of our sample client which gives a user control over a camera.

Getting Started Guide

Please see the Getting Started Guide for step by step instructions on how to use our Virtual Vision Simulator.

Configuring a Scenario

A scenario is made up of three config files which allow you to configure the scene, the cameras and the pedestrians. These three files must be called scene.xml, cameras.xml and pedestrians.xml and must be placed in a single folder which can be specified using the -d flag.

scene.xml

The scene.xml file is used to configure the 3D models and lights that make up the scene. A sample is provided below. 
<?xml version="1.0" encoding="UTF-8"?>
<scene>
  <models>
    <model path="../../media/scenes/office_floor/office_floor.egg" scale="25" pos="0 0 0" hpr="0 0 0" has_lighting='1'/>
    <model path="../../media/scenes/office_floor/skybox.egg" scale="25" pos="0 0 50" hpr="0 0 0" has_lighting='0'/>
  </models>
  <lights>
    <light type="ambient" color="0.8 0.8 0.8 1.0"/>
    <light type="spotlight" casts_shadow="1" pos="-575 150 100" fov="170" exponent="1.5"/>
    <light type="spotlight" casts_shadow="1" pos="-575 -150 100" fov="170" exponent="1.5"/>
  </lights>
</scene>

Model

The following attributes must be set for all model tags: Our simulator supports three types of lights: ambient, directional and spotlight.

Ambient Light

Mandatory Attributes

Spotlight

Mandatory Attributes Optional Attributes

Directional Light

Mandatory Attributes Optional Attributes

cameras.xml

The cameras.xml file is used to configure the cameras. The sample provided below shows all of the options that are currently available. Most of these options should be self explanatory however, we would like to point out that the id of each camera must be unique and the default_direction is the direction the camera faces when the pan and tilt values are 0. 
<?xml version="1.0" encoding="UTF-8"?>
<cameras>
    <camera type="rc_camera">
        <color>0 255 0</color>
        <id>1</id>
        <position>-575 90 -320</position>
        <up_vector>0 1 0</up_vector>
        <near_plane>1</near_plane>
        <far_plane>100000</far_plane>
        <constraints>
            <fov_limits>30 100</fov_limits>>
            <pan_limits>-30 30</pan_limits>>
            <tilt_limits>-30 30</tilt_limits>
        </constraints>
        <default_direction>0 -2 4</default_direction>
        <default_fov>100</default_fov>
        <fov>100</fov>
    </camera>
</cameras>

pedestrians.xml

The pedestrians.xml file is used to configure all of the pedestrians and their trajectories. 
<?xml version='1.0' encoding='UTF-8'?>
<pedestrians>
    <pedestrian character='CMan0011' texture='CMan0011_3.png' pos='-780 0 -50' scale='25' hpr='0 0 0'>
        <commands start_time='10'>walk walk l90 walk walk 180 walk walk stand</commands>
    <pedestrian>
</pedestrians>

Pedestrian Attributes

Commands

The commands tag contains a list of the commands that the pedestrian will execute. The available commands are: Additionally, you can use the start_time attribute to specify when the pedestrian will start executing its list of commands.

Custom Client

Our simulator allows researchers to write their own clients which can get images from cameras and tune their parameters using a light weight packet structure. The messages that a custom client can send and receive are provided below. Please see sample_client.py to see how these messages can be implemented.

Messages that a client can send

VP_SESSION (100)
message type int
connection id int
type char This value should be set to 1
camera type char 0 for static or 1 for ptz

VP_CAM_PAN (105)
message type int
connection_id int
camera id int
angle float The pan angle in degrees. Pans right if positive, left if negative value

VP_CAM_Tilt (106)
message type int
connection_id int
camera id int
angle float The tilt angle in degrees. Tilts up if positive, down if negative

VP_CAM_ZOOM (107)
message type int
connection_id int
camera id int
angle float The angle to change the field of view by in
degrees. Zooms out if positive, in if negative

VP_CAM_DEFAULT (108)
message type int
connection_id int
camera id int

VP_CAM_IMAGE (109)
message type int
connection_id int
camera id int

VP_CAM_RESOLUTION (110)
message type int
connection_id int
camera id int
width int
height int

Messages that a client will receive

VV_ACK_OK (0)
message type int
server ip string
server port int
client id int

VV_VP_ACK_OK (9)
message type int
server ip string
server port int
camera id int

VV_REQ_VIDEO_ANALYSIS (5)
message type int
server ip string
server port int
camera id int
image width int
image height int
jpeg bool

VV_IMG (2)
message type int
server ip string
server port int
camera id int
image width int
image height int
image depth int
color code char The color codes are RGB(0), BGR(1), RRRGGGBBB(2), BGRA(3)
jpeg bool
frame timestamp long
image data string


Before a custom client can begin to control a camera, it must first complete the connection handshake shown in the diagram bellow. Once connected, the custom client will automatically be assigned a camera. If a camera of the specified type is not available, the connection will fail.

The communication between a custom client and a Virtual Vision Simulator

Acknowledgments

We would like to thank Adam Domurad for his contributions to the Visual Processing Client which allowed us to test our simulator with a real pedestrian tracker.