CXL-Based Hybrid SSDs

Like a traditional SSD, a CXL-based hybrid SSD features a combination of NAND flash as its backing storage and DRAM that is employed as a cache. The key difference is that these devices offer a CXL-based host interface in addition to the asynchronous and block-based interface found in present-day commercial SSDs. This hybrid interface design enables new opportunities but also brings along a set of new challenges to make such SSDs work effectively. What makes these devices further interesting is a CXL feature known as Global Persistent Flush (GPF). With GPF, data in volatile caches (e.g., CPU or on-device caches) can survive a sudden power loss by utilizing residual energy in small buffers to flush the data to a persistent medium. Our research offers approaches to leverage GPF in practical scenarios.

Enabling efficient Direct Access (DAX) is considered a key requirement to achieving performance and energy efficiency improvements with CXL-based hybrid SSDs. DAX is used in Linux and Windows to allow userspace applications to access files on a synchronously addressable storage medium without going through the operating system kernel or complex block-based abstractions. Instead, file contents are mapped into the virtual address space of a process and can be retrieved and manipulated via load/store instructions. 

Unlike earlier DAX-capable technologies such as Intel Optane DCPMM, a Flash-based SSD cannot possibly guarantee uniform access latencies across all addresses. Synchronously accessing the device may therefore lead to prolonged CPU core stalls, with potentially catastrophic impacts for tail latencies.

In the following, we present two different approaches that our group is pursuing in order to make CXL-based hybrid SSDs fit for real-world use cases. For both of them, we rely on our own homegrown hybrid SSD prototype. If you are interested in learning more about our FPGA-based prototype design, contact Yussuf Khalil.

OS-Centric Hybrid SSD Management 

As existing DAX support was built under the assumption of a uniform PMem device, current operating system support, i.e., Linux’s DAX subsystem, fails to accommodate hybrid SSDs. Our analysis of Linux DAX identified two core problems: 

The first problem is that slow, synchronous accesses (i.e., ones that miss the on-device cache) cannot be deferred. Consequently, when applications unknowingly access uncached file ranges through a DAX mapping, the resulting memory access incurs the aforementioned CPU stalls impacting CPU throughput and tail latencies. 

The second problem concerns the granularity of DAX. As Linux determines DAX use on a per-file basis, applications cannot differentiate between data with and without frequent synchronous writes within a single file. Thus, latency-critical loads and stores compete with other I/O operations for scarce cache resources, further exacerbating the first problem.  

To address these limitations, we propose deeply integrating the operating system into the management of hybrid SSDs. Additionally, we introduce new abstractions for byte-addressable storage that are centered around fine-grained control over device resources. A core concept of our approach is the persistence-aware page cache that reflects the on-device cache state within the host’s page tables.  By marking storage pages absent from the on-device cache as inaccessible, the operating system can defer the synchronous load/store access through the page fault handler while fetching the data from the backing storage. 

We further extend the POSIX API with support for page-granular DAX mappings, granting applications more fine-grained control over DAX usage. While applications can request arbitrarily large DAX mappings, the kernel governs cache allocation and serves as the central arbiter. By leveraging the kernel’s global view of all DAX mappings, the kernel can ensure fair resource allocation while also enforcing process priorities and providing strong performance isolation between DAX users.  For applications that require guaranteed low-latency access, we provide an additional pinning API that is subject to strict resource limits enforced by the kernel. 

In our preliminary testing using an emulated hybrid SSD, we examined how the key-value store Valkey can be adapted to our proposed DAX interface and evaluated the benefits of using a hybrid SSD over conventional storage. Our testing showed promising results for application performance, CPU utilization, and energy efficiency (up to 4 times higher throughput while using 78% less CPU time and maintaining low cache usage for SET requests). For future work, we plan to evaluate our design on Samsung’s CMM-H as well as our in-house hybrid SSD prototype, and further explore the design space for operating system abstractions. Learn more in our DIMES'24 publication Fundamental OS Design Considerations for CXL-based Hybrid SSDs.

Contact: Daniel Habicht

Transparent DAX Mappings 

To achieve meaningful performance improvements with CXL-based hybrid SSDs in a general-purpose setting without risking harmful impacts for CPU performance, we propose Transparent DAX Mappings (TDMs) as our second design approach. With TDMs, the operating system is put in charge of managing the SSD’s cache in order to determine the expected performance of storage accesses: a block available in the cache is guaranteed to be readable and writable within hundreds of nanoseconds, whereas a reference to the underlying storage is generally considered too slow for DAX. 

However, userspace applications should not be burdened with the complexities of dealing with device cache management, neither can fairness between processes be guaranteed this way. Here, the other central design idea of TDMs comes into play: instead of requiring applications to be DAX-aware, the operating system automatically promotes traditional POSIX file handles with blocks present in the device’s cache to DAX mode. This enables userspace applications to benefit from kernel bypass for file operations without requiring explicit support for DAX or hybrid SSDs. This is what makes Transparent DAX Mappings transparent.

Although our research is still in progress, early results prove that TDMs are a viable approach: using a Samsung CMM-H hybrid SSD prototype, we were able to demonstrate up to 96.2% increased throughput in the YCSB benchmark using Valkey as a database backend. Learn more in our DIMES'25 publication Transparent DAX Mappings: Towards Automatic Kernel Bypass with CXL-Based Hybrid SSDs.

Contact: Yussuf Khalil