Databricks, DataPelago, DDN Lead 2026 Data Innovations

Databricks, DataPelago, and DDN have been highlighted as key innovators shaping the future of data infrastructure in 2026, with advancements designed to meet the escalating demands of artificial intelligence. These companies are addressing critical challenges in AI development, including the need for faster data processing, efficient resource utilization, and scalable storage solutions.

Databricks has been recognized for its significant contributions to bringing databases up to AI speed. The company's platform has seen a dramatic shift, with AI agent creation of test and development environments growing from a mere 0.1% two years prior to 97% today. Traditional database architectures have struggled to keep pace with this dynamic. In response, Databricks developed Lakebase, a fundamental rearchitecture of databases engineered to spin up and shut down environments at AI speeds. This capability is crucial for supporting the multitude of small, rapid workloads inherent in AI-driven development. In August, Databricks secured $5 billion in a strategic funding round, valuing the company at $190 billion. The company's revenue run rate has experienced an 80% year-over-year increase, underscoring its rapid growth and market impact.

DataPelago is being acknowledged for its transparent routing of code to optimal processors, a critical function in the AI landscape. While GPUs and NPUs are primary architectures for AI, high-performance computing also leverages CPUs and FPGAs. Efficiently directing computations to the most suitable processors can significantly reduce overhead. DataPelago's Nucleus product modernizes traditional CPU-centric designs and serves as a vital bridge between data lakes and query engines. It intelligently routes computing requests to the most efficient processor type available without necessitating any code modifications. The company claims its Accelerator for Apache Spark can deliver up to 10 times faster performance while reducing compute costs by 80%. In August, DataPelago was acquired by NetApp, which has integrated Nucleus into the core of its comprehensive AI offerings.

DDN is being cited for its advancements in scaling storage to accommodate supercomputer workloads. The development of increasingly larger and more sophisticated data models has been historically constrained by slow data path throughput, often leading to underutilization of GPUs. DDN's EXAScaler is a parallel file system designed to overcome these limitations. The article does not provide further details on the specific innovations of DDN's EXAScaler beyond its role in addressing data path throughput challenges for large-scale AI model training and its parallel file system architecture.