GPU4FS: The Fully GPU-Accelerated File System

In modern computer systems, accelerators like GPUs take over an increasing amount of increasingly complex computations. One major obstacle when growing GPU usage is found in the data transfer between the graphics card and the remaining system. Challenges include both data movement orchestration and organization, which we address with the GPU4FS project.

Traditionally, the complete data set to be worked on needs to be transferred to GPU memory before the actual computations starts. Additionally, most contemporary GPU software uses as much GPU memory (VRAM) in a computation as possible, which prevents background transfers for later computations. Therefore, while the actual computation happens, the limited bandwidth between CPU and GPU remains unused. With fine-grained control over the data movement and GPU progress, only the data used at a particular moment could dynamically be moved the GPU. However, in traditional systems where the CPU orchestrates data movement, high communication latency prohibits this optimization.

An example for a modern, complex, GPU-side application is large-scale text and image search. Here, the input data is quite varied, with inputs ranging vary from few kilobytes of text, to multiple terabytes of image or movie collections. When searching, the application may only need to load parts of the data corpus, e.g., a single frame from a movie. Additionally, to facilitate the search effort and the result sorting, the applications requires ways to organize the data, e.g., hierarchical in a tree. Lastly, an underlying system needs aid programmers in developing such applications, i.e., by supporting data discovery primitives and data passthrough for applications structured in pipelines.

To solve these problems, we propose GPU4FS, a full-scale file system running completely on the GPU. First, using the GPU as the orchestrator enables the fine-grained data transfer required for low latency accesses and maximizing both bandwidth and computational utilization. Second, a file system offers hierarchically organized data that can efficiently manage differently sized chunks of data and dynamically share persistent storage. In combination, we achieve an easy to use and deploy system that offers high performance without manual program and data flow optimizations while being familiar to programmers. Additionally, we can use the information from the file system to further inform the data movement orchestration, increasing the value of this combination.

We separate our research activity into four compartments that are integrated into our GPU4FS system: First, we explore the application behavior of GPU applications and the opportunities that GPU4FS offers here. Second, we investigate the modifications and optimizations that we need to employ to run the file system efficiently on the GPU. Third, we develop the necessary infrastructure to even run a file system on the GPU, like enabling direct NVMe storage access from the GPU. And fourth, we seek to broaden the use of GPU4FS for other fields, like offloading the file system from the CPU for CPU-heavy but storage-light application when a GPU is available.

Subprojects