Illustration displaying different files types moving to the cloud

Git Large File Storage

7 Feb 2017 by Matt Fedorovich

Being a Unity3D developer, I'm used to some of the hindrances to Unity, including compression qualities, native embedding issues and the topic of this article: large files. Our Git clients were also not impressed with these files.

For our purposes, we usually use a GitLab setup in-house, but for the purpose of this system, we tested it with GitHub and Sourcetree.

The issue

As my scenes and assets grew, so did the file sizes: 3D assets, movies files, large 360-degree photos. All of these files are multiple megabytes, or gigabytes. As I would push them to our GitLab, I would come across broken pipes, timeout issues and failure errors due to file size.

After doing a bit of research, I came across documentation that GitHub and GitLab wouldn't accept general files over 100MB. Most of our files reach well over that.

The solution

In April of 2015, GitHub introduced an open-source tool called Git Large File Storage (LFS). It was developed specifically for large files such as audio, video, data sets and graphics that exceed the maximum file size.

Instead of forcing the file up onto the server or simply bypassing the max setting, Git LFS actually pushes the large file to another server (large file storage) and creates a text file that points to it on that server. The text file is then pushed to your Git as a placeholder. 

Drawing displaying files moving to large storage server

How to

The process of getting these files to the large file storage is fairly simple to set up. I'm going to use Sourcetree to show you how to do it because it has Git LFS built right in.

For your Git repository to push to the LFS, it needs to know which files are going to be more than 100MB or ones that are already more than 100MB. Luckily, you don’t need to provide the exact file name but just the file extension it's associated with (i.e., a *.PSD file).

First, let’s open Sourcetree and pick a repo that has large files in it. Go to the top menu bar and select Repository > Git LFS > Initialize Repository.

Display of Git LFS repository

Then this popup shows up. Click Start using Git LFS.

Display of opening pop up screen for Git LFS

Next, add the file extensions you think, or know, are going to be large files. When it comes to Unity, there are four files that are usually pretty big: *.a, *.asset, *.resS, and *.assetbundle. The first three are generated based on your scenes, scene links and the assets associated with those scenes. AssetBundles are like ZIP files with assets compressed inside them, making them huge. Once in game, Unity is able to decompress and use those assets on the fly, making them a great resource for assets that need to be downloaded (i.e., large skyboxes, high-poly models).

Display of acceptable types of Git LFS extensions

Once you hit OK, Sourcetree will create, or add to, a file called .gitattributes. Be sure to commit and push that file up to GitHub before pushing the rest of your files.

And that's it. Now you can commit your large files, and GitHub will automatically track which ones get sent over to LFS.

Quick note: Git LFS will also warn you when it notices a file that may be too large for the Git. It gives you a list of files, and you can add their extensions by going to Repository > Git LFS > Track/Untrack Files.

Headshot of Matt Fedorovich

About Matt Fedorovich
App Dev Consultant

Matt is a mobile developer with experience in iOS and Android native development. He also has extensive experience in game development with Unity3D and is integral in Insight's VR/AR application development. 

Read more posts by Matt
Connect with Matt on LinkedIn