What should be done to address FAF patch download issues?

There have been a number of complaints from a variety of people regarding downloading the patch, particularly with excessively slow download times for some (including some people with fast internet speeds). So, what can be done to address this/to prevent it from happening with future patches?

If those with the relevant technical knowledge could chime in, it would be good to know more details about why this is occurring and what could be done to fix it. For example, is this something that money (ie: Patreon funds) could reasonably address, or is it from a reasonably solvable code issue, or should FAF just create a bandage-style solution such as an additional way to download new patches, etc.

A really lousy bandage is the gamefiles and some of the bin folder files being hosted here by me: http://95.216.153.185/gamedatafiles/
For some that download has worked faster.

Other ways to improve the downloads would be to have some CDN/cloudflare service or get another server and make some system that redirects user to the closest server based on their geolocation. These both require manhours to do, server admins can probably explain these better and more in-depth regarding pros/cons and required amount of work.

Another alternative is to use something like windows with updates, users sharing update files with others, or something like bittorrent...

The math game:
Our server has a bandwith of 1Gbit/s = 125mb/s. This is a theoretical optimum. With tcp/http overhead and other services still running we might be able to provide 80mb/s for downloading patches.

That means 8 players can download a patch at 10mb/s in parallel. Or 80 players can download a patch in parallel at 1mb/s. Unfortunately there is 1500+ players who want to download that. Of course not exactly all at the same time, but you get the problem.

The server network speed is one potential limitation.
This doesn't explain why this particularly affects players outside of the EU. Maybe there are other effects taking place which outside of FAFs control.

We have a similar situation with client downloads. I offloaded these to a private webspace mirror a long time ago which improved the situation for some users and not for others.

Load balancing and CDN:
So why not just put more servers up containing the files?
Well it doesn't just work. Putting in a mirror server with the same files doesn't make it automatically download from it. You need a load balancer in front. Also then you have some backup server capabilities that are not used 99% of the year. So you want to provision a super small one. But still you have to keep the files in sync with the main server, the operating systems regularly updated yadadada. Also doesn't solve conflicts due to the region.

The "enterprise" solution: content delivery networks. Have big companies such as Google, Amazon (AWS) or Cloudflare cache your files in multiple regions and distribute it their.
That is guaranteed to work. But also not a thing we are experienced with at all (this has nothing to do with running server or developing software anymore). So if I look out Cloudflare claims you get it very cheap while Google or Amazon price calculators say it's about 90-400$ a month depending on a lot of variables. I barely believe that Cloudflare can offer it so cheap, so the cheap offers must have caveats we are not aware about. The whole topic is very complex and very intransparent and nothing you just enable you with just a button click (even though providers like Cloudflare try to make this impression).

Fact is we have traffic of 15-20TB a month and traffic just costs money, it can't be free without major pitfalls.

The patch bloat:
Each patch is roundabout 50mb to download.
Do you want to know what the average binary change of a patch is? It's roundabout 100kb.
I know this because from 2017-2019 I spent a 100-150 hours trying to build a better patch system. I actually have a working one, but it went into crazy complexity and exchanged the download speed for speed in applying a patch. I did not take into account that 30%+ of our playerbase runs FAF on 10-15 year old notebooks with the power of a toaster.

There were many discussions about other ways of doing it. But creating game patches in FAF is super complicated consisting of source code from git repository paired with binary files and different setups for different featured mods. Basically it's hell and touching it will most probably break other parts of FAF (especially backwards replay compatibility).

So the "correct" solution would be a proper patching mechanism, bringing down patch size to justifiable levels (maybe 1-5mb with decent patching speed or similar...)

Summary

So overall speaking it is a very complex topic on a problem that occurs a) only to subset of the playerbase and b) only on rare occurrences (game patches, partially client updates) and c) it can be solved by the user showing patience and/or trying again at a later time.
Furthermore d) the problem is as old as FAF and doesn't break anything.

My personal opinion on that matter is: FAF has other issues and topics that have a higher priority. Not all of them are [intended to be] known to the public.

We can try to apply some of these proclaimed "cheap and easy" solutions, but my common sense super powers tell me they won't work on the scale of FAF and/or will be unreliable or shut down by the service provider because we excessively use it.

The best bang for the buck solution is probably doing what Giebmasse or FtX were doing: Uploading the files somewhere so users that consistently have problems can try it from there. Yes, it's ugly, but game patches don't happen so often and as Brutus explained, it is basically only a problem for a few days after a patch where literally everyone tries to download the files.

I was working from home the day of the patch release, saw it take about 3 hours and not finished, canceled and moved on with my life.

The next day after work, got home, downloaded the patch in like 10 minutes.

I don't think a few days of hectic patch download, coupled with issues is abnormal even for a lot of AAA title games, so expecting a free-service game community do better is a little unrealistic.

Problem isn’t really solved by just saying it’s a patch day problem. I had to spend like 30 minutes downloading stuff to watch fafdevelop replays one day and then the exact next day I had to spend yet another 30 minutes downloading fafdevelop to join a 2v2 lobby. Both of these were before the patch and well before any level of undue stress should have been put on the server.

I’m going to guess that new players downloading coop files also takes up this bandwidth? Because if so that could explain why fafdevelop always takes forever too, but it also means this problem is never really going to stop.

There is no way to break it down into files any further, so that would be guesswork.

Yet again waiting 30 minutes is not acceptable? How did I survive the 00s with only ISDN (64kb/s max) and paying per minute? A 5mb patch took 1.5 hours and cost me around 2€.

It really isn’t, I could download and play Witcher 3 faster. If I treated FAF like any random game I would have uninstalled it and probably came back somewhere in the next 6 months once the 2nd 30 minute download happened because I’d think this was a constant occurrence.

If it’s going to be like this then you might as well as automatically turn on file caching for players.

We already do offer caching for files. Develop just changes enough that it needs to be redownloaded.

Yes, but it isn’t enabled by default. If this is going to keep being a problem I’m saying you might as well as enable it and if people have a memory problem they can disable it instead. People might grudgingly tolerate the first mega slow download but it consistently happening might just make them decide to stop bothering with the client.

The caching is only useful if you go back and watch old replays. For most players this is never any issue. The download happens when a patch is released and then it is over.

It is a much bigger issue to consume all the memory with the cached files as people may have limited space on their C: drives.

Is it not useful for joining/watching FAFdevelop games and then having to join/watch normal games and then potentially going back to FAFdevelop? I mean it isn’t like tons of FAFdevelop lobbies are up, I guess.

I don't think the average user really plays fafdevelop at all

@sheikah said in What should be done to address FAF patch download issues?:

The caching is only useful if you go back and watch old replays. For most players this is never any issue. The download happens when a patch is released and then it is over.

It is a much bigger issue to consume all the memory with the cached files as people may have limited space on their C: drives.

I thought we added a file (size) limit and that it is on by default. We should really do that as it seems stable.

I think this issue is actually quite a large problem. I somewhat regularly get PMs from users with (allegedly) fast internet that are running into extremely slow downloads trying to download coop/mods outside of patching periods. I have no idea how often this is actually due to the server's bandwidth being saturated, but I suspect it is somewhat often. Considering that the percentage of people who will message a mod (and me specifically) about an issue is typically an extraordinarily small percentage of people who experience this issue, this problem may be widespread outside of patching periods. (Someone with server analytic access can tell me if I'm totally off base if the bandwidth actually isn't saturated outside of patching periods)

We should also strive to provide the best user experience possible during patching periods. Creating a 1-2 day window where FAF is basically unusable for users outside of Europe every few months should be avoided if possible. We should not expect the average user to use some random mirror that no one knows about. If players can't play FAF, they go somewhere else and they might not come back. If we have someone willing to set up a CDN that costs $400/month to avoid this issue, we should do it, we can afford it. Will it eat into the the tournament budget? Yes, but ensuring the best possible UX is most important.

@brutus5000 said in What should be done to address FAF patch download issues?:

@sheikah said in What should be done to address FAF patch download issues?:

The caching is only useful if you go back and watch old replays. For most players this is never any issue. The download happens when a patch is released and then it is over.

It is a much bigger issue to consume all the memory with the cached files as people may have limited space on their C: drives.

I thought we added a file (size) limit and that it is on by default. We should really do that as it seems stable.

I am not aware of any file size limit. Right now there is just an option for users to specify how long the cache exists for.

I agree with Skwoll, for die hard fans of FAF it might not be that big of a problem. But for newer or more casual players this might be something that is really off putting, even more that it doesn't happen only on patch periods.

I myself while not suffering as hard as others still get mildly annoyed when I have to spend 5-10 minutes downloading older patches just to download the current one in about half an hour later, and that's not even on the patch days. Considering that some might even spend 10 times the amount of time downloading few rather small files seems like something that we should look into.

I feel lucky as my patch downloads usually take 10-15 seconds. Also at the same time when many others experience download issues on several occasions, making me believe we are not saturating the server bandwidth but rather networks/ISP's along the way have routing issues or are throttling traffic.

I'm always able to download the patch instantly while other people have been having the issue for several days in a row, so it seems to be more complex than just bandwidth. I also live in the US.

Yeah I can count on one hand the number of times I've had issues downloading patches or maps and none of those occurrences were recent. I am also US based. I am also wondering if what Gieb said is a part of the issue, I remember when myself and other burgers stuck with Comcast as their ISP basically couldn't play FAF without awful lag for a month or whatever it was.

Edit: Also I completely agree with Swkoll. Sure I'm not going anywhere if I can't download a patch for a day or two, but that's not the case with new people.