Git isn’t bad at storing large binaries, it’s just not always what you want. It’s also not true that if we store binaries in Git, that users have to download them when contributing to projects. It’s possible to fetch only the source code and not the binaries.
However, it doesn’t mean you always want to store binaries directly in the repo. You can imagine something like git-annex, where the checksums and tree are stored in the repo, but the actual file is stored somewhere else. This model would work if you prefer to use a CDN or specialized server for serving binaries.