Replay server fixing - volunteers needed!

Hi all,

I'm the guy that's responsible for FAF replay server, here to talk about stuff. Tl; dr if you have issues watching live replays and want to help out, scroll down to the bottom.

A long time ago there was the original replay server. That thing had a bunch of issues and had to be restarted every once in a while. Some time back I rewrote it and the new version has been running ever since. The replay server does two things: it streams live replays and it saves them to FAF vault. Right now saving replays works perfectly fine, but streaming doesn't quite work.

Many of you know that there's a bug with live replays where if you're behind on replays and the game ends, the replay ends abruptly. I know about this bug, but never managed to fix it. I can't reproduce it myself and I have absolutely no clue what causes the bug. Maybe it has something to do with how Python deals with TCP connections, maybe it's the FA executable acting up, or maybe there is some bug in the server I missed. I don't know. Here's a possible way to fix it.

Not long ago I rewrote the replay server again. This version is IMO simpler, more tidy and it's done in Rust rather than Python, meaning it's faster, multithreaded and easier to maintain. I spent some time polishing it, making tests and so on, the last bit that's left is testing it with a bunch of real games, then deploying it. Now, we held off for a while on that last step. From our (dev) perspective, current server has been working fine, in a sense that replays were saved properly and nothing crashed. Here's where you come in.

  • First, I need to hear your opinion on live replays so us devs can gauge how serious of an issue this is.
  • Second, I need a bunch of people for a testing session on the test server, maybe a mini-tourney of sorts, with some people playing and other people watching. This way I can verify that the server works fine, and with a little luck the live replay issue will be fixed as well.

I don't have any timeframe for the testing session, but we'll figure something out, maybe on some weekend. Come help out pls!

I've been using the live replay functionality since I joined FAF, and a couple of times I've run into problems.

Mainly, sometimes the replay desyncs due to the host leaving or a player getting killed which could subsequently lead to a crash (didn't save any logs unfortunatly) or replays simply stop (possibly because the host leaves).

They are not big issues for me personally, since they don't occur every game, but I'd say there is a 1 in 5 chance that problems may arise in a replay.

I can be available for the testing session, just tell me when, I'll gladly help you out!

Right, forgot to mention - if anyone else is interested in helping test the replay server, please comment in this thread.

Hello, just to confirm i have that issue pretty often on livereplay.
I think it happen most of the time when you are just behind the 5 min delay, and the game end.
Ill try to be more focus on the exact pattern and come back here if that can help.

A major use for live replays is tournaments. It's pretty bad when people are trying to cast tournament games w/ delay and they have to stop casting it 10 minutes before the game ends. It makes us look like idiots and amateurs to people who might be dropping in to watch a tourney. People who don't know about FAF might get a negative impression of it.

@arma473 cool story. How about commenting something constructive tho. You volunteering to help test the new replay server?

Jesus @Askaholic relax. I don't care about complaints or people being annoyed, I just want to get something done.

Alright alright xD

@mazornoob said in Replay server fixing - volunteers needed!:

  • First, I need to hear your opinion on live replays so us devs can gauge how serious of an issue this is.

@arma473 said in Replay server fixing - volunteers needed!:

A major use for live replays is tournaments. It's pretty bad when people are trying to cast tournament games w/ delay and they have to stop casting it 10 minutes before the game ends. It makes us look like idiots and amateurs to people who might be dropping in to watch a tourney. People who don't know about FAF might get a negative impression of it.

@askaholic said in Replay server fixing - volunteers needed!:

@arma473 cool story. How about commenting something constructive tho. You volunteering to help test the new replay server?

🙄

@Th3-11 🤷♂

Many of you know that there's a bug with live replays where if you're behind on replays and the game ends, the replay ends abruptly. I know about this bug, but never managed to fix it.

@arma473 Even if this issue was fixed, the preferred method for tournament casting is still likely to be live observer because there is no way to sync 2 casters watching a live replay.

Not to say this isn't a big problem in general. I'd like to be able to use live replay to watch random ladder games and such without it breaking horribly.

Interested into helping testing the live replay function.

I am at discord contact me anytime under same username.

As for my feedback with the current one it does break 1 in 5 games that i watch live with minimal input on my end. (Sometimes i leave it as a background noise and replay ends the game with no acu dead.)

Analyze, Adapt, Overcome...

@askaholic You complained about someone honestly responding to Mazor literally EXACTLY AS HE REQUESTED (as Th3-11 explicitly showed). Based on your pointless response to Th3-11, are you honestly incapable of seeing why that is not just a dick move, but also really stupid? Come on.

@swkoll said in Replay server fixing - volunteers needed!:

@arma473 Even if this issue was fixed, the preferred method for tournament casting is still likely to be live observer because there is no way to sync 2 casters watching a live replay.

A mod does exist to do this I believe? At least I heard about it, never actually tested it out myself.

@ftxcommando
It worked really well in the past, haven't used it in years though, so idk what the state is.

https://github.com/fafafaf/livereplayserver
https://forums.faforever.com/viewtopic.php?f=41&t=5774

@aulex Despite it's name, I believe this is only for syncing replays that already finished, ie where a full replay file exists not live replays. I remember hearing this was broken too, but I haven't tested it recently.

Is there a way to use the synced replay feature to "step into" a match at a particular point in time? E.g. if tex and swkoll had a match on Point of Reach, could I start a synced replay with a pal, and at 8 minutes, I take Tex's position (his units, his buildings, including all existing orders) and my friend takes Swkoll's and we play the game out from there?

This would be a lot of fun and it would let people settle disputes like "if I didn't lose connection, I totally would have won" or "If I didn't forget to build TMD, I totally would have won" or "I think at 15 minutes, blackheart had a stronger position, but at 22 minutes, Blodir is in the stronger position" -- we don't have to just guess, we could play the match out multiple times and see what happens.

@arma473 If this was possible, I would run a "relay race" tournament where a team of players swaps control of a 1v1 game every few minutes.

If I'm available I'd be happy to help test this. Not sure about my availability though because I'm traveling a lot atm. I've had all of the issues mentioned in this thread, the most annoying for me has been the live replay randomly ending when the game ends.

Also, out of curiosity is there any possibility of being able to go back in time in a replay? I'd love that feature but have no idea how easy or even possible it would be to do that.

Edit: I'd like that feature enough that if it's possible to do I'd consider personally putting in some dev time to accomplish that if I can find some bandwidth in me to do some more dev work outside of my main job 🙂

@arma473 @Swkoll If it's possible I love those ideas