The amount of data generated, processed and stored – with every passing day – is growing at an unprecedented pace. Data is created not just by humans, but also applications, algorithms and IoT devices – in different types and formats, for different purposes. This veracious appetite for data keeps growing and it’s leading to rapid innovation in storage technologies for consumer devices and, even more dramatically, IT data centers that power businesses across industries.
“In 2020, 64.2ZB of data was created or replicated, defying the systemic downward pressure asserted by the COVID-19 pandemic on many industries and its impact will be felt for several years,” said Dave Reinsel, Senior VP, IDC's Global DataSphere.
“The amount of digital data created over the next five years will be greater than twice the amount of data created since the advent of digital storage. The question is: How much of it should be stored?”
That is an assertion The Forecast made in an article titled “Diamonds might be forever, data isn’t.”
“Data is different from physical assets in that the value can actually increase over time”, said Lee Caswell, SVP at Nutanix.
“Applying new analytics to data can lead to new insights and better decision making.”
As a result, the value of storage systems is shifting from mere storage to active data processing for insights, which in turn drives the need for high-capacity, superfast and reliable flash devices.
“Organizations should consider preparing now to store more data as they seek to achieve digital transformation milestones and improve business metrics by accelerating innovative data analytics initiatives,” said John Rydning, Research VP at IDC.
Fortunately, advances in memory and storage technology have enabled data centers and cloud service providers (the so called “hyperscalers”) to store and transfer massive volumes of data. It pays to know how ubiquitous cloud technology uses cutting-edge storage resources – both hardware and software – for large-scale data processing.
Enter Flash-Powered SSDs
When it comes to cloud (or cloud data center) storage for everyday applications and data, the enterprise (and just about everyone else) has transitioned from hard disk drives (HDDs) to solid-state drives (SSDs). SSDs provide critical advantages in every aspect over HDDs:
- Higher storage capacity per unit
- Faster read, write and erase times due to lower inherent latency (less moving parts in the hardware) and a software-defined management plane
- Lower power consumption
- Greater reliability, endurance and longer life
- Storage efficiencies that allow a competitive cost per bit with slow hard disk drives
The building block of these SSDs is a simple (OK, very complex semiconductor-based cell) called flash. Flash memory today powers smartphones, laptops, servers, storage arrays and storage chips in IoT and other devices.
“Disk drives used to be common in cost-sensitive consumer devices like PCs, DVRs, and even MP3 players. All of consumer devices moved to flash as costs decreased, because flash offers near-instant response times along with low power, cooling, space, and noise,” said Lee Caswell, SVP of Product and Solutions Marketing at Nutanix.
“IT storage systems from the data center to the cloud are taking exactly the same path wherever processing speed and analytic insights are valued.”
3D NAND flash, the current iteration of flash technology, involves silicon chips that are arranged in a planar matrix as well as on top of each other in multiple layers to make a 3D stack. This way, the available storage per unit surface area is multiplied while the proximity between individual cells enables faster communication without adversely affecting the performance or reliability.
Source: EE World Online
Contemporary SSDs use the Non-Volatile Memory express (NVMe) interface to access the flash storage chip via a Peripheral Component Interconnect express (PCIe) bus. NVMe allows for thousands of parallel requests over a single connection, significantly eliminating performance bottlenecks faced by the older SATA interface.
Today’s enterprise data centers and cloud providers have complex architectures designed for speed and availability. In order to enable optimal application performance in multicloud environments, data centers use all-flash arrays – sets of multiple, high-capacity SSDs that provide block-based data storage to multiple servers across the organization.
Source: PAC Storage
While consumers simply choose an SSD by reading a few PCWorld and ZDNet articles, cloud architects and product managers need to carefully select each unit of storage hardware on the basis of application performance requirements in production workloads, scalability, cost per TB, and a ton of other factors.
“Flash SSDs are a critical device in modern scale-out infrastructure, whether deployed on-premises or in the public cloud, because for the first time, storage latencies are less than network latencies,” explained Caswell.
“As the price of flash SSDs has decreased and capacity has increased, it’s become practical to build out server-based infrastructure suitable for the highest performance applications.”
Flash-powered SSDs are increasingly forming the core of enterprise and hyperscale storage systems running on hybrid multicloud, high performance computing (HPC) or hyperconverged infrastructure (HCI) architectures.
Software-Defined Storage Delivers the Benefits of Flash to the Cloud
Buying storage at cloud platforms and infrastructure providers obviously doesn’t involve bulk or batch orders from enterprise SSD vendors. Proprietary storage systems didn’t meet the scale-up needs of the cloud which is built, like hyperconverged infrastructure, using high-volume servers.
As a result, it’s now possible to access the latest flash technology using servers that are common across the hybrid cloud. The local storage devices on these servers are then aggregated, protected, and shared across virtual compute instances using software-defined virtualization technologies that create of large-scale block, file or object virtual storage arrays.
This gives organizations the freedom to prioritize performance, availability or reliability according to the requirements of its workloads. Some of the primary benefits include:
- Scalable I/O operations per second (IOPS)
- A “try hard or fail fast” per I/O retry policy
- Lower tail latency
- Granular access to disk analytics and telemetry on a per-block basis
- Ability to prioritize requests by overriding the disk firmware schedule
- Control over timing of background tasks
These features, taken together, serve to expand critical storage capabilities wherever the software defined architectures are deployed, from edge to data center to the cloud. Standardization of performance-intensive storage hardware leads to lower TCO, reduced data fragmentation, higher availability, and simple, no-lock-in management.
Hybrid Cloud and HCI Environments Are Defaulting to Flash Storage
As more workloads move to the cloud, as more applications execute more data processing tasks, cloud data centers, HPC and HCI environments need to employ more storage acceleration methods such as parallelization and shuffling. NVMe SSDs fit the bill perfectly for these intensive data processing needs.
No surprise then, that NAND flash overtook hard drives as the most popular means of storage back in 2019, per a Wikibon study.
The price of flash SSDs vs that of HDDs used to be a huge concern – until recently. That is changing rapidly: prices of NAND flash are nosediving even as this article is being written. In fact, SSDs are projected to be cheaper than HDDs on a cost per unit of capacity ($/TB) basis by 2026.
Source: Blocks & Files
HCI and hybrid cloud environments need to process huge amounts of data every day. Several cloud operators and enterprises are looking for storage arrays with 50 TB drives that can cater to their complex applications and databases.
This is where the price-to-performance advantage of flash storage comes into play. Flash SSDs deliver I/O performance several orders of magnitude better than HDDs due to their DRAM-level read/write speeds and bandwidth. Legacy HDDs and tapes are not built to handle the I/O-intensive requirements of present-day cloud-native apps and databases.
Flash storage not only lowers the cost-per-IOPS for the organization but also makes it easier to move workloads between on-prem and cloud environments – and predict the impact of such shifts on operations more accurately.
“Today’s users are incredibly sensitive to performance,” Caswell said.
”If an application is slow to respond, customers are quick to move on and flash is especially good for accelerating workloads like databases, data warehousing, data analytics, and virtual desktops. Flash acceleration makes it possible to run these applications on server-based scale-out infrastructure that can offer a consistent experience for applications running in the cloud or on-premises.”
No wonder all major cloud providers now offer underlying SSD storage as part of their SaaS, PaaS and IaaS services:
- Amazon: AWS offers SSD storage for both its block storage EBS, GP2 and IO1 as well as file storage FSx Windows and FSx Lustre volumes, NC2 block storage with Nutanix.
- Microsoft: Azure has the Managed Disks block-level storage option for its VMs. It supports partner solutions including NetApp files, premium files and storage accounts on SSD, Azure VMware Solutions, and Nutanix NC2.
- Google: GCP has two flash storage offerings – the premium Local SSD storage for high-performance VM instances and Persistent Disk for less demanding workloads.
With such variations and options in architecture and deployment, the cloud storage industry needs performance standards. Microsoft and Facebook have attempted to do exactly that by developing and ratifying the Open Compute NVMe Cloud SSD specification. It defines interoperable, unified design and performance standards for throughput, latency, endurance and security that hyperscale and hyperconverged environments typically need. This ensures organizations have on-demand availability of storage and manufacturers know exactly what they want and for what purpose.
Data in a Flash
The ultimate test of any storage technology (hardware or software) lies in the way it enables cloud applications – and by extension, users – to access and process data. Flash has passed this test with flying colors by speeding up everything from productivity to telemetry apps, to read/write-intensive scripts and databases.
And in doing so, it has become the de facto standard for enterprise cloud architecture.
“Once users experience a fast application, they can never go back to slow performance,” said Lee, explaining the UX aspect of high-performance application deployment.
“It’s simply human nature that the fastest experience immediately becomes the norm against which all other experiences are compared. It’s becoming a critical advantage for businesses that increasingly compete for customer loyalty using performance applications.”
Which is why cloud storage is increasingly finding favor with applications in AI and ML (where large datasets need to be processed over extended periods of time), hi-res streaming media (with high availability for uninterrupted access), high-end gaming and AR/VR (which both mandate extremely high read speeds and very low latency) as well as high-performance computing (HPC) (which need isolated data processing, maximum capacity and ultra-low latency).
