Item — HN Reader

While I feel like this is generally true for most programmers and knowledge workers, Git is absolutely not suited to the workflow of several industries, including the one I work in: games.

Working with an engine like Unreal Engine for a project of any reasonable size requires working with both hundreds of thousands of active files (my current project's repo's HEAD has ~400k) and hundreds of gigabytes of active files, many of which are many GB on their own. Our repository (in perforce) is currently in the order of 10TB.

Git, even with LFS and partial clone and shallow copies and fsnotify just falls apart at this scale. Add to that the necessity for less-technical people (artists, designers, VFX, audio, etc) to use it, and it is really just a non starter.

I absolutely loathe Perforce (having used and occasionally admin'd it professionally since 2007), but I begrudgingly admit that is is currently the only publicly available VCS that can fulfill all of the requirements of this industry in a practical way. Plastic and the others just aren't there yet.

4 comments

This is a solved problem with git. I’ve worked on bigger projects than yours with histories that are so big you can’t clone them if you wanted to. I’ll make a note to drop what to google for tomorrow, but basically the history is fully available but git knows how to query it over the network. When you open a file, it loads it over the network. Remote build systems do your builds.

Most of this was built by Microsoft to work on Windows with git.

We moved from SVN in the late 2010’s, and holy crap man, did it change our workflows. No longer were we passing around patch files in a team for code reviews, but actually reviewing code somewhat like on GitHub. It was magical.

Microsoft had a bunch of solutions to handle their massive Windows repo: VFS for Git (GVFS), Scalar, and now it has a bunch of MS specific patches on top of the official git client, but apparently that one is also not required any more as partial clone is now supported on azure as well (which is another such implementation from Microsoft employees that made it to both GitHub and upstream git).

So yeah, solved problem thanks to Microsoft. Solved multiple times in fact, and because it's Microsoft, only not all had Linux installers: https://github.com/microsoft/scalar/issues/323#issuecomment-...

https://github.blog/2020-01-17-bring-your-monorepo-down-to-s...

https://devblogs.microsoft.com/devops/introducing-scalar/

https://github.com/microsoft/git

https://devblogs.microsoft.com/devops/git-partial-clone-now-...

Thanks!

>Most of this was built by Microsoft to work on Windows with git.

Hacking git into something it is not does not qualify as learning to cope with git in my book.

Apparently git does not solve all problems and there is indeed value in exploring other options and building new systems that fill other niches.

umm. it's still git. Git is basically content-addressable storage with a couple of layers on top (heads/tags/trees) which is itself content-addressable storage. If you can offload that storage to a remote, you still have git... and can use git as you've always used git. The tricky part is making it feel like the storage is local and your own, instead of shared.

You only need this when, well, you need it. You can go a surprisingly long way (nearly a decade of daily commits by hundreds of programmers) and stick to vanilla git.

> Add to that the necessity for less-technical people (artists, designers, VFX, audio, etc) to use it

How is that solved with Git?

Maybe they just need to follow carefully written instructions created by more technical people?

When has that ever worked? And why does this count as a solution rather than a poor workaround for you?

What has ever worked, reading and following manuals? Like, forever? Stop pretending there's a simple solution. People just need to use their brains. Both people who write those manuals and those who read them.

There literally already is a simpler solution for non-technical people, it’s called Perforce and tons of game studios use it

But I’m sure you’re smarter than the whole industry so you probably know better

Hey smart man, this is a thread about git, not perforce. Just fyi.

Since forever? It's not like people are born with an innate knowledge of how any software works.

Manuals/guides/tutorials are the most scalable way to teach people how to do literally anything.

I shed a tear as I remembered the pain of doing that with SVN

That’s why you need a binary repository like Artifactory and not store your large files in Git. But you can still track them in Git with the large files in a binary repository like Artifactory.

This could be solved with references rather than copies if your tools integrated them. e.g. dependencies can be ref'd with source/name and version. To ensure availability all such used blobs could be stored efficiently elsewhere and versioned.

It's great that Perforce works and I've heard others in the graphics field using it, so it satisfies a need. Don't know if/when it would be a general need for say GitHub users.

This sounds like a case of using excel as a db. Holding a hammer by it's head a declaring it unfit for normal people to paint with.

I also work in games and everyone I know uses Perforce for the reasons mentioned above

Have you considered that maybe they just have a different use case than you?

What I mean is, no one ever said that git was meant to hold bulk assets and resources directly, and so doing so is painting with a hammer, not an insufficient pain brush.

Yes of course they have a different use case. That is just another way to say the same thing.