Guest Post: Understanding the Impact of Storage Solutions on Data Lakes

As business products and systems, including those related to physical and cyber security, continue their migration to the network, new offerings have sprung up to manage and leverage the billions of data points they collect and produce. The sheer quantity of data that needs to be available to a range of different platforms has given rise to the concept of the data lake: a giant pool of useful data which has not been sorted, categorized or placed in any kind of folder hierarchy.

For surveillance and security, data in the data lake includes raw video and analytics information from cameras, VMS and other integrated devices. The need to archive and preserve all of this data makes storage a foundational layer of the data lake architecture. This storage layer must be fully scalable and must support an open platform capable of managing disparate data sets from multiple devices.

In selecting the right storage solution for your application, it is important to know the factors which play a role in the effectiveness of your surveillance system and the usability of your data. One primary factor which must be considered is your surveillance architecture.

There are three major surveillance architectures in use today: distributed, centralized and cloud. Some companies have distributed-only environments. Some have only centralized environments. Some use both on-premise and cloud architectures for different purposes, while others go cloud-only. Here are the differences between these environments with regard to how they affect – and are affected by – the storage option you select.

Distributed Architectures

Distributed architectures store video and surveillance data locally and then periodically transfer the digital data set to a central platform. One example of this might be a satellite police station that stores data in the office, but from time to time transfers the data over to headquarters.

Distributed architectures often integrate the data with applications and other systems, such as access control and intrusion detection, without engaging a central server. This reduces the potential for single points of failure and distributes processing requirements over many smaller sites.

When choosing the right storage vendor for distributed surveillance architectures, ask potential vendors the following questions: Is high bandwidth available at a low cost per GB? Can the configurations be described as “plug and play,” that is, simple and straightforward to deploy? Is virtualization easy for future growth?

Centralized Architectures

Scale is the primary consideration when choosing a storage vendor for centralized surveillance architectures. Commonly used by police headquarters, schools, governments, airports and energy companies, centralized surveillance architectures host high device-count environments and are able to support large amounts of surveillance data.

Storage must be made efficient in centralized architectures, and utilization rates must be high to prevent price creep. Since retention times and pixel/resolution quality are forever changing, migration time to apply these changes must be extremely low, if not non-existent.

Some companies use a converged centralized architecture when they need a total, extendable solution that consolidates systems. Components of a converged surveillance infrastructure may include servers, data storage devices, networking equipment and video management/surveillance software for IT infrastructure management, automation and orchestration.

Converged and non-converged centralized architectures solve different storage challenges. Both are ideal options for public safety organizations that need to scale. Both also commonly exploit video monitoring and analytics to increase security and opportunity on the same platform, which is highly attractive to companies looking to simplify their business.

Cloud Architectures

The cloud has been causing some confusion lately. One example can be found in the case of body-worn cameras. Body-worn data has very different storage requirements from state to state depending on the offense. Video of routine traffic stops may only be kept for 30-45 days, while DUIs may be kept for three-plus years, and federal crimes may need to be kept for the length of imprisonment, or in some cases, forever.

Most states have laws that require evidence used in a case to be kept a minimum of seven years. This means that, overall, video from body-worn cameras has a long shelf life, which results in big storage needs.

Organizations must consider these long-term storage and data management challenges and think beyond three-year or five-year buying cycles, or they could end up with an inflexible and costly solution. While the cloud is affordable at the start, it is important to understand the cost implications when storage exceeds 1PB, and organizations are paying monthly storage and access fees for 25-plus years.

Going the pure cloud route, therefore, is not always the best option for public safety organizations. Choosing a vendor that offers both cloud and on-premise storage options is a better bet as it will safeguard an organization’s assets and allow for future growth. Many companies opt to go on-premise first with bulk of their “cold,” or long-term, storage, and then go to the cloud for deeper storage. This approach often is more cost effective, provides greater security, and simplifies application integration.

Some vendors offer cloud storage bundled with cameras, enabling customers to go cloud first and then go on-premise to save on long-term storage. This bundled option can be much easier for those new to surveillance as the process of purchasing is made simple. However, it may not be the best option for organizations with high retention requirements or that need to frequently move data from local to storage and back.

It is important for organizations to weigh all of their storage options as there is no one-size fits-all solution. Ultimately the smart move is to work with an expert who can help you sift through the choices, ask the right questions and find the solution that works best for the data lakes within your unique organization.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.