The Google File System

Shoeb Ilyas - September 1, 2026 - 2061 Words

Throughout various Google services, thousands of users create, edit, and delete their files every single instant. The problem we face is how do we ensure availability and consistency of the data that the user is working with at any given instant. In this article we breakdown Google's file system design and help you, the reader, understand how simple yet efficient this google file system is.

Before understanding Google file system design, let's understand how Linux handles file storage and retrieval. Understanding Linux file system design in this regard will help us in our future discussion.

Whenever we create a file in Linux, it breaks down our file into 4 KB chunks. Each chunk is stored at random available locations inside our disk instead of being stored linearly. During retrieval, the chunks related to any file are combined and served and the user sees the full file. While there can be more nuances and details in how files are handled, this much of discussion helps us form a good mental model.

Like any other system, GFS also makes some design assumptions. Here we list out the following:

GFS is composed of many inexpensive commodity hardware. Therefore, at any given time, one or more of the available systems are bound to fail. Failure is the norm rather than the exception.
The system stores a modest number of large files. On average, a file is usually 100 MBs or more and multi-GB files are common. Therefore, we do not optimize for smaller KB-sized files.
High sustained bandwidth is more important than latency. Since we are working with large multi-GB files, a little lack of performance is forgiving.
Atomicity guarantee is a must since it's possible for multiple clients to try to append to a single file simultaneously.
Workloads usually append to file sequentially instead of performing random writes. Writes at random offsets are supported but we do not optimize for it.

It's possible that the you might not understand all of the design assumptions at this point of your reading. However, I request you to read further and these concepts shall become clear as you read through.

Basic Architecture and Interface

GFS provides the functionality to create, read, open, close and write to files. In addition, it also provides two additional functionalities - snapshot and record append.

Snapshot: creates a copy of a file in directory at low cost using CoW Copy-on-Write method. Record Append: appends to a file while guaranteeing the atomicity of each client's append operation.

Any GFS cluster consists of one master server and many chunkservers. Each chunkserver is a commodity linux machine running a user-level process.

Files are divided into fixed size chunks. At the time of their creation, each chunk is assigned a unique 64 bit chunk handle. Chunkservers store these chunks locally as linux files and get be read or written to provided the chunk handle and the byte range. For reliability, each chunk is replicated over N number of chunk server where N denotes the replication factor. Replication factor is set to 3 by default.

The master server stores the file metadata about each chunkserver in it's cluster. It stores the namespaces, access control information, mapping of files to chunks and the location of those chunks. It manages chunk lease management. It also helps migrate chunks between chunk servers when needed. It also talks with the chunkservers in heartbeats to give them instructions and get their state.

The client first talks with the master to get the information about the chunks and the location of the related chunkservers. The client then talks directly with the chunkservers to read or write to files.

The client also caches information about the chunkservers for a set interval of time. Clients interacting directly with the chunkservers for read and write operations prevents the master from becoming a bottleneck. The master is solely responsible for chunk placements using global knowledge, replication decisions and therefore remains relatively lightweight.

Architecture Deep Dive

Now that we have a solid understanding of GFS works on a high level, it's worth taking out some time to understand GFS's technical implementation in detail. In the following sections, we will go through the various cogs that build GFS.

Chunk Size

The chunk size is set to 64 MB. This is unusually greater than the size of linux file chunk size. The reason for such a large file chunk size is that it reduces the number of requests made to the chunk server for read or write operations. It also reduces the size of the metadata needed to store the chunk information on the master. Since the client is most likely to work with a given chunk, we can keep the TCP connection open for longer communication instead of going through multiple TCP handshakes back and forth.

A Theoretical Problem

Large chunk size even with lazy space allocation has a disadvantage. If a small file has let's say one or a few chunks, the chunk itself can become a hotspot when accessed by many client's at once. However, 2003 GFS paper states that this was a theoretical problem and that this issue of hotspots had not been observed then. Please check the latest updates from Google, in case Google addresses this issue at

Master Node

Master stores three types of metadata. First, information about the files and their chunk namespaces, mapping between the file and it's chunks and each chunk's chunkserver location. At startup, the master polls the chunkservers for chunk metadata. The master also polls to fetch the metadata when a new chunkserver joins the cluster.

In-Memory Storage

The metadata is stored in-memory on the master server. Therefore, the master operations for fetching chunk and file metadata are blazing fast. Master also scans through it's state in the background whenever it is free for garbage collection, chunk replication and handling chunk server failures. The only bottleneck is the master's memory but this is a theoretical bottleneck since the master only needs somewhat around 64 bytes of metadata per 64 MB of chunk. The cost of adding extra memory to the master server is a small price to pay for simplicity, reliability and scalability.

System Interactions

Let's understand the flow of the entire system in detail. For this we have divided it into two sections. The first is the current one - system interactions. In this section we will go through how the master, the client and the chunkservers interact to perform mutations on file. In the following section on master server, we will go understand how the master handles garbage collection, chunk replication, and master related processes. As we have already stated, we try to keep the master as lightweight as possible.

Now we illustrate the control flow of a write operation through these numbered steps.

The client requests the master server the chunkserver locations of the replicas. The master assigns one chunkserver as the primary if no one is granted that lease.
The master replies with the location of the chunkservers and the client caches this data for a set interval of time for future operations. The client only interacts with the master again the when lease of the primary ends.
The client then pushes the data to all the chunkservers. The chunkservers hold this data in an LRU buffer cache. Once the client receives acknowledgement from all the chunkservers that they have received the data to be mutated, it can proceed to the next steps.
The client then makes a write request to the primary replica. The primary replica assigns a serial order to all the mutations and applies it locally. Then it forwards this write request to all the chunkservers. Once all chunkservers have applied the given order of mutations as specified by the primary, they return to the primary.
The primary reports back to the client with any errors that any chunkserver may have encountered during the mutations. If the mutation fails at primary, it is never forwarded to the secondary replicas in the first.
The lease duration is by default 60 seconds but the primary can request for an extension in specific scenarios such as when a mutation is in progress.

We decouple the flow of data from the control flow for efficiency. The data is pushed linearly through the chunkservers instead of being pushed in a tree like manner. Each replicas network bandwidth is fully utilized to push the data across the chunkservers as fast as possible. To avoid bottlenecks and high-latency links, each chunkserver pushes the data to it's closest chunkserver.

Atomic Record Append

Snapshot

GFS can create snapshots of a file almost instantaneously without interfering with the any client mutations that might be in process. When a client requests to create a snapshot of a file to the master, it revokes any existing chunk leases. Once that is done, it logs the operation to disk. The master then duplicates the metadata for the source file. The newly created file points to the same source file in the disk. When any client intends to write to chunk C, master points out that for chunk C, there are more than one references and defers replying to the client. Meanwhile, it tells all the chunkservers to create a copy of chunk C called C'. From here the rest of the flow is same as before.

The Master

Many master operations can take a while to execute. Therefore, to mitigate master from becoming a bottleneck and blocking other operations, GFS allows multiple active operations to co-run through by ensuring locks over regions of namespaces. Furthermore, unlike other Linux or Unix filesystems, GFS does not have a per directory data structure. Metadata for every file is maintained in a lookup table by mapping it's entire file location to the metadata. By using prefix compression this table can be efficiently represented in memory. Whenever you want to mutate a file in a directory, a set of locks are acquired. Let's assume, we intend to modify d1/d2/d3.../dn/leaf. GFS will acquire locks on d1/, /d1/d2, d1/d2/d3 and so on and either a read or a write lock on the entire directory - d1/d2/d3/d4..../dn/leaf.

Replica Placement and Chunk Replication

Chunk replicas are created for three reasons: chunk creation, re-replication and rebalancing.

While creating chunk replicas the master must choose an initial chunkserver. It tries to choose a server based on the following criteria:

Choose the chunkserver with the least number of recent writes.
Choose the chunkserver with below-average disk space utilizatoin
Try to spread out our chunk replicas as much as possible The master re-replicates the chunk as soon as the chunk replica numbers falls below replication factor. It prioritizes chunk replication based on various factors - first of which being the diff between the current replicas vs the desired end state. It prioritizes a chunk that has lost two replicas over the one that has lost only one.

The master picks the highest priority chunk and 'clones' it by giving a signal to a chunkserver to clone the chunk from an existing replica chunkserver.

Finally, the master re-balances replicas periodically.

Garbage Collection

When a file is deleted, it's deletion is logged but the space is not immediately reclaimed. The file is simply given a hidden deletion name along with a deletion timestamp. During the regular system scan by the master unlinks the metadata of these files. That is removes it from the metadata. During the regular heartbeat communication, the chunkserver reports to the master with a subset of chunks it has and the master also sends back a list of deleted chunks that the chunkserver is free to delete from it's system.

The stale replicas holding stale chunk version numbers are also garbage collected during the regular background scan process. Until then, GFS treats such a replica as practically non-existent.

Conclusion

This was my breakdown of the GFS article. I have intentionally left out some concepts and topics in this article. However, having read this article, you should now have a solid understanding of GFS and should be able to read through the official GFS research paper. I will be referencing the various