A game-changing transformation in how governments use video surveillance is getting underway as emergency responders discover they can combine AI-fueled analytics with real-time streaming to stay current with events extending to the farthest reaches of their jurisdictions.
The new approach contrasts with traditional reliance on ground- and drone-based camera feeds reaching operations centers over high-latency streaming and routing systems, which prevents responders from getting anything close to a comprehensive real-time understanding of what’s happening. Aided by increased 5G mobile coverage through private and public networks along with other advances pushing visibility into remote areas, surveillance becomes far more useful when responders can put advanced analytics tools to work processing frame-synchronized mixes of multi-camera views delivered to their screens within a few hundred milliseconds of capture from anywhere in the field.
Leveraging high-resolution cameras, they can spot virtually anything of interest down to the details of things like the license plate of a getaway car, unauthorized drones in crowded air traffic, signs of movement in search-and-rescue missions or newly igniting hot sports in a wind-driven wildfire. Moreover, initial experiences with real-time surveillance streaming are opening the door to new strategies and use cases.
For example, local, state and federal government administrators will be able to adjust operations to enable cross-agency sharing of live camera streams from large numbers of widely dispersed assets aloft, on land and at sea. And they can make more effective use of video output from emergency responders’ cellphones, body cams and other mobile equipment.
One new use case in particular stands out as a dramatic example of how these advances could impact law enforcement operations everywhere. We’re told an unnamed municipality is using real-time streaming to capture output from police bodycams as operations are underway. This is a big step beyond the use of recorded bodycam footage for post-event analysis.
Limited but Eye-Opening Revelations of What’s in Store
While there’s been a trickle of press releases and news reports about the use of AI with analytics related to surveillance, especially in the private sector, there’s been practically no publicity about the role real-time streaming is playing with AI in emergency responses, ground and air traffic control, military operations and other government tasks executed by local, state and federal agencies. Amazingly, most of the information about all this activity comes from real-time streaming technology supplier Red5.
Judging from what other providers are highlighting as use cases for their platforms, Red5, which like its competitors is seeing growing use of its platform in many fields, appears to be the only real-time networking supplier that has a significant stake in government-run surveillance operations. As described in our article on the real-time interactive streaming (RTIS) ecosystem, there are about a dozen suppliers competing in this space, most of which are focused on supporting M&E and e-commerce applications in the consumer market.
Only one other real-time streaming vendor, Wowza, mentions anything approaching meaningful involvement in the surveillance market on its website, in this case in conjunction with an affiliation with Blueforce Development, a supplier of mobile edge compute software supporting intelligence related to various IoT- and sensor-related applications. In its latest post on the engagement, dated January 30, 2019, Wowza indicated that Blueforce was using Wowza’s support for streaming at 2-second latencies with a switch to WebRTC support for 500ms latencies to follow. There’s been no mention of lower latency or anything else about the engagement by either Wowza or Blueforce since then.
The Red5 Solution Set
Meanwhile, most of what Red5 has shared has to do with surveillance operations under management of two California agencies, the California Department of Transportation (Caltrans) and the San Diego County Sheriff’s Department. In an interview with UltraMedia Pipeline, Red5’s senior vice president of global sales Brett Fasullo expanded on previously published information and explained how the company’s support for surveillance works.
“We have a system that can be used in the cloud or on premises to do all things that are needed to put real-time streaming to work in the surveillance environment,” Fasullo says. “Our platform’s ability to ingest any number of camera feeds into the thousands is the key.”
Several processes supported by the Red5 Pro software stack go into enabling real-time multi-camera surveillance operations, he notes. Together they allow video feeds from multiple cameras, typically transmitted via the Real-Time Streaming Protocol (RTSP), to be combined and transported as a single multi-screen stream over WebRTC for viewing and analysis by surveillance operators – all within 250ms of camera outputs. Any number of such aggregations from groups of cameras in different locations can be synchronized for real-time streaming to surveillance operations centers.
Red5’s Mixer software running on dedicated commodity processors allows users to configure exactly how they want multiple camera feeds to be combined for ingestion into XDN Origin Nodes. Fasullo says the Mixer can perform overlays, resizing and synchronization combining dozens of live RTSP streams into a grid layout that’s converted for streaming over WebRTC to end devices, which allows browsers to process the video without the use of plugins.
Operators can dynamically adjust grids to accommodate variations in the number of feeds and how they’re laid out in the grid. Adding to surveillance efficiency, Red5’s TrueTime MultiView toolset provides customers the option to instantly pull from the Mixer-generated multi-camera grid any view for full-screen display with the ability to switch back and forth among camera outputs as often as necessary.
The Role Played by AI-Assisted Analysis
It's up to users to choose how the consolidated real-time camera feeds are processed for analysis. But there’s less work involved if the analytical components have already been integrated to work with XDN architecture, Fasullo notes.
One case in point is the actionable intelligence platform provided by Accenture Federal Services (AFS) through the Novetta technology it acquired in 2021. Red5 customers in defense, federal law enforcement and government intelligence are employing the AFS platform for analysis in surveillance operations.
For example, U.S. Army and Border Patrol units are using the real-time multi-intelligence solution to deliver actionable insights derived from sensors and radar as well as video feeds from drones, Humvees, underwater vehicles, body cameras and even canine patrols. The cross-platform web application enables multiple functions essential to coordinating operations across dispersed command posts. Through integrated chat services, event/report sharing, and remote sensor control, operators can collaborate in real time to quickly understand what’s happening around them in their areas of operation.
AFS has designed its data processing platform to work with a wide range of generative-AI suppliers. Separately, there are a number of vendor solutions touted for surveillance analytics that use predefined algorithms to support detection of objects and their attributes, movement and other behavioral characteristics, often with utilization of facial recognition to focus on specific individuals. In most cases, these insights can be used to create historical records recounted textually and in charts, graphs and heat maps.
Fasullo also notes that Red5 has integrated with several suppliers of application-specific AI technologies as it works with customers to implement support for real-time surveillance. For example, one AI solution integrated to work with the company’s Experience Delivery Network (XDN) architecture executes facial recognition on a per-frame per-video basis across the aggregated camera feeds. Another solution can be used to automatically anonymize content in the video feeds as prescribed by privacy regulations.
Expanding Agendas in California
What we’ve reported so far adds up to quite a picture of what we can expect as emergency responders move in this direction, regardless of whether Red5 competitors choose to get involved. But there’s more to the story, some of which can be seen in the expansive approaches Caltrans and the San Diego Sheriff’s Department are taking with their use of Red5’s XDN architecture.
Caltrans, which is responsible for highway, aviation, rail, bus and subway operations across the state, has been using Red5’s real-time streaming platform over the past few years to stream aggregated feeds from about 800 highway surveillance cameras operated by Caltrans District 7 in Los Angeles and Ventura Counties. Now, according to Fasullo, Caltrans is exploring plans to extend use of real-time streaming to drones and thousands more cameras positioned along highways and other locations throughout the state.
With a live real-time surveillance ecosystem operating statewide, Caltrans would be able to support a wide range of goals beyond its own highway and air traffic management agendas, Fasullo says. These include the possibility of extending use of the surveillance system to the state’s port authority, railway administration, air traffic controllers and highway patrol.
If preliminary planning comes to fruition, he adds, responders to emergencies under the purview of any department would be able to call up whatever camera streams are relevant to their needs for real-time aggregation on their dashboards. And they’d be able to instantly record and archive all or selected segments of those feeds for future investigation.
Red5 has also brought to light the San Diego Sheriff’s Department’s thinking about what it needed from a real-time streaming platform to accommodate creation of a drone command center that could be used by multiple agencies across its vast jurisdiction. Apparently, the department relied on Nomad Media, a content management platform supplier now listed as a Red5 partner, to recommend a supplier who could meet the San Diego County requirements.
The department wanted a secure, reliable, scalable real-time streaming solution that was also device agnostic, Red5 says. It required support for “a virtually unlimited numbers of cameras and users with an easy-to-use, customizable interface,” along with “a future-proof solution that could grow and change over time as their needs evolved.”
Moving Real-Time Surveillance into Remote Areas
Adding to the benefits derived from the use of real-time streaming in surveillance, there’s now an opportunity to expand coverage into previously unreachable territory. Much faster, more knowledgeable responses to emergency and battlefield scenarios can be executed when it’s possible to aggregate and analyze surveillance camera feeds from aerial and maritime drones, body cams, mobile phones and remote fixed locations in relatively close proximity to the action.
Red5 has integrated its XDN management system with two distinct approaches to extending the reach of real-time streaming infrastructure, Fasullo notes. One approach involves the use of Oracle’s Roving Edge Infrastructure (REI) devices, which lend portability to the global Oracle Cloud Infrastructure (OCI) computing platform now operating in datacenters spanning 50 geographic regions on six continents. Deploying REI modules, Red5 customers can support surveillance use cases where the speed-of-light latencies imposed with use of fixed OCI resources would be too high for remote emergency operations coordinated from mobile command centers in the field.
An even more versatile approach to conducting surveillance-supported emergency operations remotely is available with use of appliances and a cloud software platform supplied by Canadian cloud networking software provider Dejero. The company brings another dimension to meeting the remote surveillance challenge by providing cameras network connectivity that would otherwise be unavailable or subpar in hard-to-reach areas.
Dejero leverages relationships with mobile, fiber, broadband and satellite network providers to create software-defined cloud-managed network-of-networks solutions that enable internet access over whatever link can be used moment to moment to deliver the best throughput without incurring interruptions switching from one network to another. As explained by Fasullo, emergency responders can use Dejero’s field deployable appliances to provide network aggregation and real-time encoding support in the delivery of remote camera feeds to Red5’s XDN infrastructure for compilation into real-time streams to remote command centers.
Obviously, the next phase in mass surveillance operations has just begun. We’ll keep tabs with Red5 as the saga unfolds. And we’ll keep an eye open for signs that other real-time streaming providers are awakening to the opportunity.
How could they not? Unless, of course, the extensive list of capabilities enumerated here are beyond their reach.
The new approach contrasts with traditional reliance on ground- and drone-based camera feeds reaching operations centers over high-latency streaming and routing systems, which prevents responders from getting anything close to a comprehensive real-time understanding of what’s happening. Aided by increased 5G mobile coverage through private and public networks along with other advances pushing visibility into remote areas, surveillance becomes far more useful when responders can put advanced analytics tools to work processing frame-synchronized mixes of multi-camera views delivered to their screens within a few hundred milliseconds of capture from anywhere in the field.
Leveraging high-resolution cameras, they can spot virtually anything of interest down to the details of things like the license plate of a getaway car, unauthorized drones in crowded air traffic, signs of movement in search-and-rescue missions or newly igniting hot sports in a wind-driven wildfire. Moreover, initial experiences with real-time surveillance streaming are opening the door to new strategies and use cases.
For example, local, state and federal government administrators will be able to adjust operations to enable cross-agency sharing of live camera streams from large numbers of widely dispersed assets aloft, on land and at sea. And they can make more effective use of video output from emergency responders’ cellphones, body cams and other mobile equipment.
One new use case in particular stands out as a dramatic example of how these advances could impact law enforcement operations everywhere. We’re told an unnamed municipality is using real-time streaming to capture output from police bodycams as operations are underway. This is a big step beyond the use of recorded bodycam footage for post-event analysis.
While there’s been a trickle of press releases and news reports about the use of AI with analytics related to surveillance, especially in the private sector, there’s been practically no publicity about the role real-time streaming is playing with AI in emergency responses, ground and air traffic control, military operations and other government tasks executed by local, state and federal agencies. Amazingly, most of the information about all this activity comes from real-time streaming technology supplier Red5.
Judging from what other providers are highlighting as use cases for their platforms, Red5, which like its competitors is seeing growing use of its platform in many fields, appears to be the only real-time networking supplier that has a significant stake in government-run surveillance operations. As described in our article on the real-time interactive streaming (RTIS) ecosystem, there are about a dozen suppliers competing in this space, most of which are focused on supporting M&E and e-commerce applications in the consumer market.
Only one other real-time streaming vendor, Wowza, mentions anything approaching meaningful involvement in the surveillance market on its website, in this case in conjunction with an affiliation with Blueforce Development, a supplier of mobile edge compute software supporting intelligence related to various IoT- and sensor-related applications. In its latest post on the engagement, dated January 30, 2019, Wowza indicated that Blueforce was using Wowza’s support for streaming at 2-second latencies with a switch to WebRTC support for 500ms latencies to follow. There’s been no mention of lower latency or anything else about the engagement by either Wowza or Blueforce since then.
The Red5 Solution Set
Meanwhile, most of what Red5 has shared has to do with surveillance operations under management of two California agencies, the California Department of Transportation (Caltrans) and the San Diego County Sheriff’s Department. In an interview with UltraMedia Pipeline, Red5’s senior vice president of global sales Brett Fasullo expanded on previously published information and explained how the company’s support for surveillance works.
“We have a system that can be used in the cloud or on premises to do all things that are needed to put real-time streaming to work in the surveillance environment,” Fasullo says. “Our platform’s ability to ingest any number of camera feeds into the thousands is the key.”
Several processes supported by the Red5 Pro software stack go into enabling real-time multi-camera surveillance operations, he notes. Together they allow video feeds from multiple cameras, typically transmitted via the Real-Time Streaming Protocol (RTSP), to be combined and transported as a single multi-screen stream over WebRTC for viewing and analysis by surveillance operators – all within 250ms of camera outputs. Any number of such aggregations from groups of cameras in different locations can be synchronized for real-time streaming to surveillance operations centers.
Red5’s Mixer software running on dedicated commodity processors allows users to configure exactly how they want multiple camera feeds to be combined for ingestion into XDN Origin Nodes. Fasullo says the Mixer can perform overlays, resizing and synchronization combining dozens of live RTSP streams into a grid layout that’s converted for streaming over WebRTC to end devices, which allows browsers to process the video without the use of plugins.
Operators can dynamically adjust grids to accommodate variations in the number of feeds and how they’re laid out in the grid. Adding to surveillance efficiency, Red5’s TrueTime MultiView toolset provides customers the option to instantly pull from the Mixer-generated multi-camera grid any view for full-screen display with the ability to switch back and forth among camera outputs as often as necessary.
It's up to users to choose how the consolidated real-time camera feeds are processed for analysis. But there’s less work involved if the analytical components have already been integrated to work with XDN architecture, Fasullo notes.
One case in point is the actionable intelligence platform provided by Accenture Federal Services (AFS) through the Novetta technology it acquired in 2021. Red5 customers in defense, federal law enforcement and government intelligence are employing the AFS platform for analysis in surveillance operations.
For example, U.S. Army and Border Patrol units are using the real-time multi-intelligence solution to deliver actionable insights derived from sensors and radar as well as video feeds from drones, Humvees, underwater vehicles, body cameras and even canine patrols. The cross-platform web application enables multiple functions essential to coordinating operations across dispersed command posts. Through integrated chat services, event/report sharing, and remote sensor control, operators can collaborate in real time to quickly understand what’s happening around them in their areas of operation.
AFS has designed its data processing platform to work with a wide range of generative-AI suppliers. Separately, there are a number of vendor solutions touted for surveillance analytics that use predefined algorithms to support detection of objects and their attributes, movement and other behavioral characteristics, often with utilization of facial recognition to focus on specific individuals. In most cases, these insights can be used to create historical records recounted textually and in charts, graphs and heat maps.
Fasullo also notes that Red5 has integrated with several suppliers of application-specific AI technologies as it works with customers to implement support for real-time surveillance. For example, one AI solution integrated to work with the company’s Experience Delivery Network (XDN) architecture executes facial recognition on a per-frame per-video basis across the aggregated camera feeds. Another solution can be used to automatically anonymize content in the video feeds as prescribed by privacy regulations.
What we’ve reported so far adds up to quite a picture of what we can expect as emergency responders move in this direction, regardless of whether Red5 competitors choose to get involved. But there’s more to the story, some of which can be seen in the expansive approaches Caltrans and the San Diego Sheriff’s Department are taking with their use of Red5’s XDN architecture.
Caltrans, which is responsible for highway, aviation, rail, bus and subway operations across the state, has been using Red5’s real-time streaming platform over the past few years to stream aggregated feeds from about 800 highway surveillance cameras operated by Caltrans District 7 in Los Angeles and Ventura Counties. Now, according to Fasullo, Caltrans is exploring plans to extend use of real-time streaming to drones and thousands more cameras positioned along highways and other locations throughout the state.
With a live real-time surveillance ecosystem operating statewide, Caltrans would be able to support a wide range of goals beyond its own highway and air traffic management agendas, Fasullo says. These include the possibility of extending use of the surveillance system to the state’s port authority, railway administration, air traffic controllers and highway patrol.
If preliminary planning comes to fruition, he adds, responders to emergencies under the purview of any department would be able to call up whatever camera streams are relevant to their needs for real-time aggregation on their dashboards. And they’d be able to instantly record and archive all or selected segments of those feeds for future investigation.
Red5 has also brought to light the San Diego Sheriff’s Department’s thinking about what it needed from a real-time streaming platform to accommodate creation of a drone command center that could be used by multiple agencies across its vast jurisdiction. Apparently, the department relied on Nomad Media, a content management platform supplier now listed as a Red5 partner, to recommend a supplier who could meet the San Diego County requirements.
The department wanted a secure, reliable, scalable real-time streaming solution that was also device agnostic, Red5 says. It required support for “a virtually unlimited numbers of cameras and users with an easy-to-use, customizable interface,” along with “a future-proof solution that could grow and change over time as their needs evolved.”
Adding to the benefits derived from the use of real-time streaming in surveillance, there’s now an opportunity to expand coverage into previously unreachable territory. Much faster, more knowledgeable responses to emergency and battlefield scenarios can be executed when it’s possible to aggregate and analyze surveillance camera feeds from aerial and maritime drones, body cams, mobile phones and remote fixed locations in relatively close proximity to the action.
Red5 has integrated its XDN management system with two distinct approaches to extending the reach of real-time streaming infrastructure, Fasullo notes. One approach involves the use of Oracle’s Roving Edge Infrastructure (REI) devices, which lend portability to the global Oracle Cloud Infrastructure (OCI) computing platform now operating in datacenters spanning 50 geographic regions on six continents. Deploying REI modules, Red5 customers can support surveillance use cases where the speed-of-light latencies imposed with use of fixed OCI resources would be too high for remote emergency operations coordinated from mobile command centers in the field.
An even more versatile approach to conducting surveillance-supported emergency operations remotely is available with use of appliances and a cloud software platform supplied by Canadian cloud networking software provider Dejero. The company brings another dimension to meeting the remote surveillance challenge by providing cameras network connectivity that would otherwise be unavailable or subpar in hard-to-reach areas.
Dejero leverages relationships with mobile, fiber, broadband and satellite network providers to create software-defined cloud-managed network-of-networks solutions that enable internet access over whatever link can be used moment to moment to deliver the best throughput without incurring interruptions switching from one network to another. As explained by Fasullo, emergency responders can use Dejero’s field deployable appliances to provide network aggregation and real-time encoding support in the delivery of remote camera feeds to Red5’s XDN infrastructure for compilation into real-time streams to remote command centers.
Obviously, the next phase in mass surveillance operations has just begun. We’ll keep tabs with Red5 as the saga unfolds. And we’ll keep an eye open for signs that other real-time streaming providers are awakening to the opportunity.
How could they not? Unless, of course, the extensive list of capabilities enumerated here are beyond their reach.