@foodlfg said in Unexpected chain of events: How a users cpu load issue paralysed the server:
interesting. nice find!
but it feels like especially from the java client side that the technologies that it is using were not properly selected. it does unnecessary things that the programmers have no control over. neither simple, nor fast. working with pre-made packages glued together.
I can't defend each and every choice made in the Java client, but I can tell you if we had a choice, we use battle-proven standard libraries used by thousands of developers. Your statement goes more into a general critic against Spring framework with all its "magic" involved. It can be quite difficult, but that doesn't mean you don't have the control over it. The general problem is complexity, as Spring components try to cover everybodys use case and can be configured in dozens of ways.
However, some libraries turned out to be a real stick in the mud. But you can't always know that beforehand. JavaFX e.g. is a pretty solid library in general. But using the JFoenix library for "nicer" ui controls was a bad choice as the developers stopped updating it to newer versions. Or the current IRC library used served us well for 5 years, but seems to make problems (not sure yet) if we hit 2k+ users in a single channel (who would have known FAF would ever reach that numbers?).
If you would write a new client from scratch in whatever languages you would have the same problems. Again. Which libraries are stable and well maintained for the next 5 years? Should I use the trendy libraries or the stable one? What if the trendy one supersedes the stable one?
as for the server side, i was like wtf when i saw that the API can receive database query like parameters which are a big feature and convenient to use, i guess. but do you guys really need this?
Do we really need this? Yes we do. The library that we use allows us to do basically permissions based proxy access to the FAF db. This includes:
We can expose everything that we want (not all tables of the FAF db are public but all that are used).
We can put security filters on what we need to protect. (e.g. some fields or some tables are only visible to privileged people)
We can also create and update stuff (also permission based)
And some things are just free: sorting, filtering, including other files
We had a different API before where we wrote individual queries for each use case and it was a huge amount of work even for the starting use cases back then. Do you want to rewrite the sorting and pagination rules each time on your own? Also the include concept would be lost then, as it is very generic and you would need to make dozens of api calls instead.
By now we have the whole moderation client built around this API. The java client uses it extensively. The website uses it (for leaderboards and the league things) and even 3rd party tools make use of it. So I think the whole API concept is a huge success.
the most simple solution would be to write a simple database auery in the DATABASE and an API that calls that one query. the further the query is from the database, the more control you lose and you have less chance to operate the database efficiently.
But it's not one query. It's hundreds of queries. I don't want to write and test these manually if there is a library that can build almost optimal SQL queries if you don't misuse it.
you cannot really prepare for the queries this way for example. there are queries that needs up to date data, there are others that need only a few hours fresh data. you can store the last ones in memory in the database if you know them so you just serve the already queried data.
just ideas, i know it is not a paid project but it is something to learn from...
That is the downside of it yes, indeed. Flexibility comes at a price. We need to use it with care and we need to think about cancelling too expensive queries somehow in the long run
Eventually it always comes down to this: We are a very small core team and there is rarely more than 1 core developer per service. Thus we need to make the right abstractions to keep the code size small enough to handle. If I write hundreds of queries manually to squeeze out maximum performance, I can no longer make changes to the database without fixing dozens of them (apart from the fact that I couldn't do anything else for FAF as I'm busy just with that). We made a trade-off here and choose flexibility over performance.
