Git Packfiles

TL;DR: While Git initially stores files as complete zlib-compressed “loose objects,” it eventually consolidates them into highly optimized binary Packfiles (.pack) and index files (.idx) using delta compression. This process compresses history by storing only the differences between versions, dramatically reducing repository disk usage and network transfer sizes.

If you’ve spent any time reading about Git’s architecture, you’ve probably heard that Git stores full snapshots of your files rather than file deltas. As we discuss in how-git-is-different and git-objects, this snapshot model is what makes Git clean, fast, and conceptually simple.

However, storing a full copy of every single file on every single edit has an obvious downside: disk space. If you have a 10MB source code file, and you change a single semicolon in it, Git’s porcelain commands will write a new 10MB blob to your .git/objects folder. Change it again tomorrow, and you write another 10MB blob. If you did this over years, your local repository would balloon to hundreds of gigabytes.

But it doesn’t. A decades-old repository with millions of commits can often fit in a few hundred megabytes.

How is this possible? The answer lies in Git Packfiles and the garbage collection system (git gc). Packfiles are Git’s primary compression layer, transforming individual loose objects into highly optimized binary packages using a technique called delta compression.

What Do Git Packfiles Actually Do?

To understand packfiles, you must understand the two distinct states in which Git objects exist: Loose Objects and Packed Objects.

graph TD
    Loose1[Loose Blob: version1.txt <br> 10MB zlib compressed]
    Loose2[Loose Blob: version2.txt <br> 10MB zlib compressed]
    GC[Run git gc / repack] --> PackFile[Packfile: pack-xxx.pack <br> Stores version2.txt in full <br> Stores version1.txt as delta]
    IdxFile[Index File: pack-xxx.idx <br> Maps SHA-1 to Packfile byte offset]

Loose Objects

When you run git-add or git-commit, Git takes your workspace files, prepends a metadata header, hashes them, compresses them using standard zlib compression, and writes them into .git/objects/. These are called loose objects.

• Loose objects are fast to write, which keeps staging and committing operations instantaneous.
• However, because each file version is a standalone zlib-compressed file, Git cannot compress duplicates across history efficiently in this state.

Packed Objects

Eventually, Git decides to clean up. It takes hundreds or thousands of loose objects and consolidates them into a single binary file called a Packfile (ending in .pack), alongside a corresponding Index File (ending in .idx).

Inside the packfile, Git switches from its snapshot storage model to a Delta Compression model.

Delta Compression: Inverting the Traditional VCS

Delta compression is the process of looking through your repository history, finding files that are similar to each other (usually older versions of the same file), and storing only the differences between them.

However, Git does delta compression in a very clever way. Traditional version control systems (like SVN) store the oldest version of a file in full, and then apply differences forward. To check out the latest version, they have to apply years of diffs step-by-step.

Git does the exact opposite:

• The latest version of the file is stored in full. This is because Git assumes you are most likely checking out or compiling the latest branch tip. Accessing the latest version is a fast $O(1)$ read.
• The older versions are stored as reverse deltas. To construct a version of a file from two years ago, Git reads the latest version and applies instructions backward (e.g., “remove this line, insert that line”) to reconstruct the historical file.

What Happens Inside .git/ (Hands-on Exploration)

Let’s build a repository, watch loose files pile up, pack them manually, and inspect the resulting binary files using plumbing commands.

Step 1: Create a Repository and Build Loose Objects

Let’s initialize a repository:

$ mkdir git-packfile-demo
$ cd git-packfile-demo
$ git init
Initialized empty Git repository in /path/to/git-packfile-demo/.git/

Now, let’s create a text file containing some lines of text, and modify it in several consecutive commits.

$ echo -e "Line 1: Git Internals\nLine 2: Understanding Packfiles" > content.txt
$ git add content.txt
$ git commit -m "First revision"

$ echo -e "Line 1: Git Internals\nLine 2: Understanding Packfiles\nLine 3: Added some new info" > content.txt
$ git commit -am "Second revision"

$ echo -e "Line 1: Git Internals\nLine 2: Understanding Packfiles\nLine 3: Added some new info\nLine 4: A third revision" > content.txt
$ git commit -am "Third revision"

Let’s look at the loose files inside .git/objects:

$ find .git/objects -type f | sort
.git/objects/06/de3410657519a4e41ff34ab7987e9154a4f8df
.git/objects/09/b35e29fa757fb1e4a6435c249a5b3992850cd3
.git/objects/42/bdbe49dbb0ffb8b209b55502c77d9cbb28d09f
.git/objects/59/cdbc68fbfbdcbe812bb1e24749f90263f350c3
.git/objects/a5/d87bc0ee734a7809622d99214b72605f624d77
.git/objects/ba/92b678fa42cb869ab12be3f5bc41a9bc9c3b88
.git/objects/c9/791c2fc886c99c8bf8a89ee1be2a09c2a382e2
.git/objects/d6/f90659d007c6f2df83f4b48bc4a5b060d4b9bf
.git/objects/fa/01d43d1a6015bd67634f19b8813a7c64a3d4f8

We have nine loose objects (three commits, three tree objects, and three blobs).

Step 2: Repacking Loose Objects

Let’s run Git’s garbage collector, which repackages these loose objects:

$ git gc
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (9/9), done.
Total 9 (delta 1), reused 0 (delta 0), pack-reused 0

Let’s check the .git/objects folder again:

$ find .git/objects -type f
.git/objects/info/commits
.git/objects/info/packs
.git/objects/pack/pack-98e9a2b8e398188fc836859846cd4d34eb51897d.idx
.git/objects/pack/pack-98e9a2b8e398188fc836859846cd4d34eb51897d.pack

All the individual loose object subdirectories are gone! They have been cleaned up and consolidated into two files:

1. pack-98e9a2....pack (the actual data pack)
2. pack-98e9a2....idx (the index mapping hashes to their offset locations inside the .pack file)

Step 3: Inspecting the Packfile contents

We can verify the contents of our new packfile using the plumbing command git verify-pack -v. This reads the .idx file to report on the state of the packfile.

$ git verify-pack -v .git/objects/pack/pack-98e9a2b8e398188fc836859846cd4d34eb51897d.idx

Output:

ba92b678fa42cb869ab12be3f5bc41a9bc9c3b88 commit 172 110 12
09b35e29fa757fb1e4a6435c249a5b3992850cd3 commit 224 140 122
06de3410657519a4e41ff34ab7987e9154a4f8df commit 224 140 262
fa01d43d1a6015bd67634f19b8813a7c64a3d4f8 tree   64 45 402
42bdbe49dbb0ffb8b209b55502c77d9cbb28d09f tree   64 45 447
d6f90659d007c6f2df83f4b48bc4a5b060d4b9bf tree   64 45 492
59cdbc68fbfbdcbe812bb1e24749f90263f350c3 blob   92 68 537
a5d87bc0ee734a7809622d99214b72605f624d77 blob   62 53 605 1 59cdbc68fbfbdcbe812bb1e24749f90263f350c3
c9791c2fc886c99c8bf8a89ee1be2a09c2a382e2 blob   43 38 658 2 a5d87bc0ee734a7809622d99214b72605f624d77
non-delta: 8 objects
chain length = 1: 1 object
chain length = 2: 1 object

Let’s dissect this output:

• Columns read: [SHA-1] [Type] [Decompressed Size] [Compressed Size] [Byte Offset] [Delta Depth] [Parent Base SHA-1].
• Look at blob 59cdbc68.... Its decompressed size is 92 bytes, compressed size is 68 bytes, and it is located at byte offset 537. It is stored as a non-delta (base object). This represents the latest revision of our file.
• Now look at blob a5d87bco.... It is a delta (depth 1) pointing to parent base 59cdbc68.... It only uses 53 bytes.
• Look at blob c9791c2.... It is a delta (depth 2) pointing to a5d87bc0.... It uses a mere 38 bytes!

By storing only the delta differences instead of whole new versions, Git compresses history recursively, saving massive amounts of storage space.

Detailed Command Options & Advanced Usage

Git provides underlying mechanics and parameters to tweak how aggressively objects are repacked.

1. The underlying plumbing: git repack

While developers use git gc to clean repositories, git gc is actually a wrapper that calls git repack under the hood. You can use git repack directly:

• git repack -a -d: Packs all loose objects in the repository into a single packfile and deletes (-d) the loose files that were just packed.
• git repack -f: Forces Git to ignore all existing delta associations and re-evaluate every object from scratch. This takes more time and processing power but can shrink your repository significantly if history was modified.

2. Aggressive Garbage Collection

If your repository is getting sluggish or has accumulated a large amount of history, you can trigger an aggressive repack:

$ git gc --aggressive --prune=now

This flag tells Git to spend more CPU time analyzing file histories to construct deeper and more efficient delta chains. It adjusts the window size and depth configurations.

3. Understanding Pack Windows and Depth

Two Git configuration values govern how delta compression behaves:

• pack.window (Default: 10): When packing, Git sorts objects by filename, extension, size, and type, and loads them into a sliding queue (the “window”). Git only compares an object against other objects currently in this window. A larger window allows Git to compare more files to find better deltas, but uses more memory.
• pack.depth (Default: 50): Specifies the maximum length of a delta chain (how many diffs are layered on top of a base object). If the depth is too shallow, files aren’t compressed as much. If it is too deep, Git has to execute too many backward instructions to reconstruct older versions of a file, slowing down commands like git checkout or git log -p.

You can modify these configurations globally or for a specific run:

$ git -c pack.window=250 -c pack.depth=250 repack -a -d -f

Network Efficiency: Clone and Fetch

Packfiles are not just designed for local storage. They are also the mechanism that makes commands like git-clone and git-fetch incredibly fast compared to old-school VCS.

sequenceDiagram
    participant Client as Local Client
    participant Server as Remote Server
    Client->>Server: git clone / fetch
    Server->>Server: Determine missing objects
    Server->>Server: Generate single Packfile in memory
    Server-->>Client: Stream Packfile (.pack)
    Note over Client: Runs git index-pack
    Client->>Client: Generate local Index File (.idx)

1. Missing Objects Calculation: When you run git fetch, your local Git negotiates with the remote server. It sends the hashes of commits it currently has, and the server calculates exactly which objects the client is missing.
2. On-the-fly Packing: The server doesn’t send loose files. It takes the missing objects, runs a fast repack in memory, generates a single .pack stream, and sends it over the network.
3. Indexing on Arrival: To save bandwidth, the server does not transmit the .idx file. Instead, when the client receives the .pack file, it runs the plumbing command git index-pack to scan the packfile locally and generate the corresponding .idx index file.

Real-World Scenario

Scenario A: Rebuilding a Lost Index File

A developer’s computer crashes during a write operation. The file .git/objects/pack/pack-xxx.idx becomes corrupted or is completely deleted. The developer runs git status and gets a crash error because Git can no longer read the objects stored inside the corresponding .pack file.

The Fix:

Since the .idx file is simply an index map of the objects inside the .pack file, it can be fully rebuilt!

1. Locate the .pack file path:

   $ ls .git/objects/pack/
   pack-98e9a2b8e398188fc836859846cd4d34eb51897d.pack

2. Reconstruct the .idx index file using git index-pack:

   $ git index-pack .git/objects/pack/pack-98e9a2b8e398188fc836859846cd4d34eb51897d.pack

3. Git scans the binary packfile, extracts the hashes, and regenerates the .idx file. The repository is immediately restored to a working state.

Scenario B: Shrinking a Bloated Repository

A repository has ballooned to 2GB because developers frequently commit compiled build assets. Even though these assets were deleted in the latest commits, they are still taking up space in the history.

1. Clean up history using a tool like git-filter-repo (recommended over the deprecated git filter-branch).
2. Run standard git gc — but notice that the repository size doesn’t shrink immediately. This is because Git keeps dangling commits around for 2 weeks by default to allow recovery.
3. Force Git to immediately purge all loose objects and repack everything with maximum compression:

   $ git reflog expire --expire=now --all
   $ git gc --prune=now --aggressive

   This clears out the reference logs, deletes all dangling blobs, and repacks the entire codebase into a fresh, highly compressed packfile, immediately reclaiming disk space.

Common Pitfalls and Mistakes

1. Believing Deleting a Large File and Committing Solves Storage Bloat

The Mistake: Deleting a large 1GB movie file, committing the deletion, and expecting the .git folder size to drop. Why it fails: Git preserves your complete history. Since the commit containing the 1GB file is still part of the repository graph, the 1GB blob remains inside your packfile forever. The Fix: You must rewrite history to remove all references to the large file from every commit, expire the reflog, and run git gc --prune=now.

2. Manually Deleting Packfiles

The Mistake: Opening .git/objects/pack/ and deleting large .pack files to save disk space. Why it fails: Deleting a packfile deletes the actual object data. If you delete a packfile, you permanently corrupt your repository history. Git will fail to check out files or run commits because the required history is missing. The Fix: Never touch packfiles manually. If you need to reduce space, use git gc or prune operations.

3. Creating Too Deep Delta Chains (pack.depth too high)

The Mistake: Configuring pack.depth to a very high number (like 1000) hoping to get the smallest possible repository size. Why it fails: While this might save a few extra megabytes, checking out older files or running git log -p requires Git to compute a chain of 1000 deltas sequentially. This degrades performance significantly, making the developer experience slow. Stick to reasonable defaults unless you have specific constraints.

Command Quick Reference

FAQ (People Also Ask)

What is the difference between loose and packed objects in Git?

Loose objects are individual files stored under .git/objects/??/ that represent single, zlib-compressed snapshots of files, trees, or commits. Packed objects are compressed together into a single binary packfile using delta compression, which stores only the differences between files to optimize disk storage and network performance.

What is delta compression in Git packfiles?

Delta compression is an optimization process where Git compares similar files inside its database and stores older versions as a set of differences (deltas) relative to a newer base version. This dramatically reduces the space needed to store version history.

Can you open or edit a packfile directly?

No. Packfiles are binary files composed of zlib-compressed blocks, offsets, and delta instructions. To read or inspect them, you must use plumbing commands like git verify-pack or git cat-file. Editing them directly will corrupt the entire package.

How does git index-pack work during clone?

During a clone, the remote server streams a single .pack file containing all the objects you need. To avoid downloading an index file (which would waste bandwidth), the client reads the incoming .pack stream and calculates the offsets of all objects locally, writing a fresh .idx file onto the disk.

Test Your Knowledge