Crowd Safety with AI/ML Object Detection

CrowdTrek is a software application that uses AI/ML-based object detection models to process video footage to count crowd sizes and track the movement and location of crowds across a wide area, for public safety applications. Created by Ardent’s Data Science & Analytics practice, CrowdTrek is a tangible demonstration of how Ardent can bring the latest AI/ML applications to solve real-world problems for mission-driven organizations.

Explore the Dashboard Directly

The Challenge

Since 2006, Ardent has supported DHS and multiple federal, state, and local law enforcement and public safety agencies with our technology solutions. We operate small Unmanned Aerial Systems (UAS) and other sensor platforms and develop custom geospatial tools, and intelligence sharing software applications. We have helped these agencies provide security for major public events, including Presidential inaugurations, meetings of heads of state at the United Nations General Assembly, the Boston Marathon, and the Super Bowl.

In September 2020, the Department of Homeland Security’s Science and Technology Directorate (S&T) Operational Experimentation (OpEx) Program recently sought technologies for “Crowd Count Analysis” for first responders to “determine and monitor counts, movement, location, and status of a crowd to help manage an event”. In managing crowd security, answering the fundamental question of how many people are in a crowd often requires manually counting people with tally counters. Other technologies, such as triangulating mobile device locations from cellular towers or using mobile apps, risk privacy concerns and miscounting individuals.

We noted how aerial observation data paired with AI/ML-based technologies and geospatial location tracking could automatically count crowd sizes.  Within the last few years, proven, pre-trained AI/ML object detection models have emerged to make this vision a practical reality.

The following major events have accelerated the urgency of knowing how large crowds are and what risks threaten them or are posed by them:

• The January 6th, 2021 attack on the US Capitol demonstrates the threat posed by crowds to facility security.
• The civil unrest and demonstrations from protestors and counter-protestors underscore the need to preserve First Amendment rights and civil liberties (including privacy), while also protecting crowds from the outbreak of violence.
• COVID-19 highlights the public health threat to a crowd, especially in places where social distancing is not in effect.

The Solution

CrowdTrek is an Artificial Intelligence/ Machine Learning (AI/ML)-based object detection model that processes stored or live video footage to determine and monitor counts, movement, location, and status of a crowd for a wide variety of public safety applications including crowd safety, facility security, orderly movement and evacuation, and event management. CrowdTrek’s AI/ML platform can be adapted to detect other objects besides people.

Value Proposition

• CrowdTrek algorithms easily adapted to count and track other objects besides people (e.g., vehicles for traffic management);
• First dedicated AI/ML product to measure crowds in large, outdoor spaces—existing models are for indoors/narrow field of view or require special sensors
• Computer vision-based object detection model avoids miscounts from tracking mobile device signals or signatures
• Model is agnostic to sensor—processes aerial images from UAS, manned aircraft, satellite images, aerostats, stationary cameras, or mobile devices
• Crowd size and movement data = clear metrics = many applications, easy integration, foundation for behavior analysis
• Anonymized data collection—avoids privacy, ethics issues (e.g., no facial recognition or PII)

CrowdTrek’s modular ML model and architecture can easily adapted to detect other objects, such as cars. CrowdTrek’s output metrics data can be displayed in a variety graphing formats, to more easily visualize activity patterns over time.

How Does Object Detection Work?

An object-detection model is a computer vision-based technique that identifies and locates object in an image or video. It is used to count objects, label them, and track their location.

It does this by using  a type of supervised machine learning model based on convolutional neural network (CNN) algorithms to classify labeled data sets. In this case, the labeled datasets are image files. Features of images such as pixel edges, shapes, and colors can be deduced into visual patterns and hence quantifiable data. An algorithmic model can be ‘trained’ to recognize patterns in this dataset of image features. The more data input into the model, the more accurate the model will become in recognizing objects, even with lower image quality or ambiguity and variety in the shape, sizes, closeness of the target object.

Objects are ‘identified’ and labeled as distinct entities (e.g., people or cars). Our model does not identify individual people (e.g., facial recognition), which would require a different, more intensive type of model as well as higher resolution images.

Once objects are identified, they can be tracked through time and space. Motion and flow simulation models can determine their location in an observed area. The objects can be counted. From there, other data such as density, direction, and speed can be derived.

Tools and Approach 

• AWS SageMaker ML Service
• AWS S3 & EC2 for infrastructure
• PyTorch ML framework
• Plot.ly Dash for front-end dashboard and map display