administrators

Well we had the same problem with Github downloads which has a worldwide CDN, so I'm still not sure if CDN can actually fix it.
So apart from just throwing money at some company, how would we measure that we actually improved the situation?

@dontstealecorock said in Client v1.6.0:

I think u can try better. I dont mind if you will be rude(actually your last post is toxic so whats the difference) There should be a reason for switch to non blocking approach so just write it. Maybe u can share perfomance numbers. How many memory blocking code was consuming per call?

Ok, so here you go:
You take the time to spit out unqualified critics on a topic by reading a short summary of the changes obviously without a) even have the slightest understanding what reactive is about or b) how it affects the client architecture overall. Furthermore if you would have reviewed the changes you would have spotted that this change goes beyond just introducing reactive patterns.

Less memory consumption due to less threads is one of many benefits, but it is not the main driver behind moving to reactive patterns.

I will not argue that in general replacing classic synchronous blocking code with reactive code is increasing the code complexity. I have seen it. I work with it every day. And in some cases it's worth it and in others it's not and I would not introduce it without good arguments.

But that "reactive is complex"-bashing you do here doesn't apply at all, because the client already tries to be asynchronous by applying the crappy Java 8 CompletableFuture logic everywhere it touches API calls. This code was already pretty ugly and complex to understand and using Project Reactor actually can make the code more consistent and easier to read. It also allows additional improvements such as better unit test supports and better handling and mixing of separate async operations than the Java 8 library can.

Furthermore the client by itself is per definition already forced to run stuff in background thread and schedule it elsewhere because it has a GUI with a main thread and we need to be particularly cautious not to ever block this thread. Therefore reactive programming actually is a good match for UI programming which is the reason why web frameworks such as React and Angular was ahead of the Java ecosystem years before (well apart from the fact that JavaScript only allows one thread they needed a better solution).

Looking at actual lobby-server tcp connection: reactive programming here allows us to move away from a constantly polling io thread which eats a lot of CPU waiting (RAM is almost completely irrelevant here). But again this refactoring brought along some other changes and Sheikah can't mention them all along.
We split off the lobby protocol into a dedicated module and wrote it in Kotlin which results in much fewer code. We now have protocol mesage classes with correct null/not-null annotations which makes it easier for future developers to work with it. We tried to decouple the lobby protocol from the client state so we can adapt to protocol changes more easily also enhancing the overall architecture.

Soo essentially we did these changes for different kind of reasons.
You cannot know these reasons. That's completely fine. You can ask for it though and we'd be glad to explain them within our time constraints.
But just picking one argument and bashing on it is just buzzword trolling.

I feel lucky as my patch downloads usually take 10-15 seconds. Also at the same time when many others experience download issues on several occasions, making me believe we are not saturating the server bandwidth but rather networks/ISP's along the way have routing issues or are throttling traffic.

@sheikah said in What should be done to address FAF patch download issues?:

The caching is only useful if you go back and watch old replays. For most players this is never any issue. The download happens when a patch is released and then it is over.

It is a much bigger issue to consume all the memory with the cached files as people may have limited space on their C: drives.

I thought we added a file (size) limit and that it is on by default. We should really do that as it seems stable.

@dontstealecorock said in Client v1.6.0:

Sounds like you higly increased code complexity in return for nothing. JVM allocates 1mb per thread. How many parallel blocking request client does? I assume that number is not big. You can even reduce memory per thread using jvm option. 512kb most likely is enough.

I literally cannot answer this seriously without getting sarcastic and rude, so I will not comment on this any further.

There is no way to break it down into files any further, so that would be guesswork.

Yet again waiting 30 minutes is not acceptable? How did I survive the 00s with only ISDN (64kb/s max) and paying per minute? A 5mb patch took 1.5 hours and cost me around 2€.

The math game:
Our server has a bandwith of 1Gbit/s = 125mb/s. This is a theoretical optimum. With tcp/http overhead and other services still running we might be able to provide 80mb/s for downloading patches.

That means 8 players can download a patch at 10mb/s in parallel. Or 80 players can download a patch in parallel at 1mb/s. Unfortunately there is 1500+ players who want to download that. Of course not exactly all at the same time, but you get the problem.

The server network speed is one potential limitation.
This doesn't explain why this particularly affects players outside of the EU. Maybe there are other effects taking place which outside of FAFs control.

We have a similar situation with client downloads. I offloaded these to a private webspace mirror a long time ago which improved the situation for some users and not for others.

Load balancing and CDN:
So why not just put more servers up containing the files?
Well it doesn't just work. Putting in a mirror server with the same files doesn't make it automatically download from it. You need a load balancer in front. Also then you have some backup server capabilities that are not used 99% of the year. So you want to provision a super small one. But still you have to keep the files in sync with the main server, the operating systems regularly updated yadadada. Also doesn't solve conflicts due to the region.

The "enterprise" solution: content delivery networks. Have big companies such as Google, Amazon (AWS) or Cloudflare cache your files in multiple regions and distribute it their.
That is guaranteed to work. But also not a thing we are experienced with at all (this has nothing to do with running server or developing software anymore). So if I look out Cloudflare claims you get it very cheap while Google or Amazon price calculators say it's about 90-400$ a month depending on a lot of variables. I barely believe that Cloudflare can offer it so cheap, so the cheap offers must have caveats we are not aware about. The whole topic is very complex and very intransparent and nothing you just enable you with just a button click (even though providers like Cloudflare try to make this impression).

Fact is we have traffic of 15-20TB a month and traffic just costs money, it can't be free without major pitfalls.

The patch bloat:
Each patch is roundabout 50mb to download.
Do you want to know what the average binary change of a patch is? It's roundabout 100kb.
I know this because from 2017-2019 I spent a 100-150 hours trying to build a better patch system. I actually have a working one, but it went into crazy complexity and exchanged the download speed for speed in applying a patch. I did not take into account that 30%+ of our playerbase runs FAF on 10-15 year old notebooks with the power of a toaster.

There were many discussions about other ways of doing it. But creating game patches in FAF is super complicated consisting of source code from git repository paired with binary files and different setups for different featured mods. Basically it's hell and touching it will most probably break other parts of FAF (especially backwards replay compatibility).

So the "correct" solution would be a proper patching mechanism, bringing down patch size to justifiable levels (maybe 1-5mb with decent patching speed or similar...)

Summary

So overall speaking it is a very complex topic on a problem that occurs a) only to subset of the playerbase and b) only on rare occurrences (game patches, partially client updates) and c) it can be solved by the user showing patience and/or trying again at a later time.
Furthermore d) the problem is as old as FAF and doesn't break anything.

My personal opinion on that matter is: FAF has other issues and topics that have a higher priority. Not all of them are [intended to be] known to the public.

We can try to apply some of these proclaimed "cheap and easy" solutions, but my common sense super powers tell me they won't work on the scale of FAF and/or will be unreliable or shut down by the service provider because we excessively use it.

A really lousy bandage is the gamefiles and some of the bin folder files being hosted here by me: http://95.216.153.185/gamedatafiles/
For some that download has worked faster.

Other ways to improve the downloads would be to have some CDN/cloudflare service or get another server and make some system that redirects user to the closest server based on their geolocation. These both require manhours to do, server admins can probably explain these better and more in-depth regarding pros/cons and required amount of work.

Another alternative is to use something like windows with updates, users sharing update files with others, or something like bittorrent...

We had a server update since March so it should be fixed? But I guess nobody checked

Setup
Ryzen 7 3700X (mildly PBO tuned)
ROG STRIX X570-F Gaming
32 GB DDR4 running at 3726, 16-17-17-35-60

CPU score old
135

CPU score new
122