How long should FAF keep old replays?

Defiant · 23 Jan 2021, 22:50

Oh please keep all replays. Can we throw money at buying more storage?

TheVVheelboy · 23 Jan 2021, 22:54

Imo, no need to keep replays older than 3 years no matter what they are. If they were good they are already casted w/e.
Even then I wouldn't really keep replays older than 2 years unless they are from ranked games. It just seems pointless to keep so much when it's useless.

Eternal- · 24 Jan 2021, 12:44

I would suggest to keep only the special old games. But I think there are no stamps like "Tournament". I also think that no one will sort their favorite replays. I guess you can just save replays of certain players, purely for memory?
But in general, I do not see any point in these replays, the game is growing up, the old tactics and methods of the game can not be applied to the current realities.
I am in favor of removing all replays since the last old patch.

And further sorting them with some parameters

Removing replays with one player
Removing replays with a sandbox
Removing of replays of the company
Removing "blacklisted" maps (gap, astro, etc)
Removing replays with desync
Removing replays after (optional) 4/8/12 months

Cascade · 24 Jan 2021, 13:01

@ZLO I've been searching for this parser for ages! The old link on google 404s
edit: still seems to be broken for me

Eternal- · 24 Jan 2021, 13:08

@Cascade You should try to open it with vpn. Works fine for me without vpn

PattogoTehen · 24 Jan 2021, 16:35

Hi,
Bitbucket deleted my old mercurial repos (https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket).
Moved what i could save to Github.

Please update your links to: https://fafafaf.github.io/

Btw. sometime the CORS header got removed from https://content.faforever.com/faf/vault/replay_vault/replay.php hence it cant load the files directly by replay id.

Brutus5000 · 24 Jan 2021, 21:17

@PattogoTehen We changed a lot of urls, but I couldn't make a pull request against bitbucket as it was gone. I'll do one against your Github repository.

The CORS header issue is something we raised as a bug ticket in our reverse proxy, it might vanish in the near future (or already does if you use replay.faforever.com, not sure).

Zokora · 25 Jan 2021, 05:24

I can offer to archive all replays and make them accessible to others as well. Size and traffic do not matter for my google drive.

edit: Others can make backups as well

nemir · 1 Feb 2021, 14:03

Is it possible to create criteria by which you could keep some replays for longer... IE if it was a game with stronger players it gets kept longer... Or if it was a game with a certain number of downloads... Or 5 star reviews...

That might be a cool thing. It would be a decent way to raise the overall value of the replay library.

Brutus5000 · 6 Mar 2021, 10:07

So due to the server disk running full causing an outage we have prioritized recompressing all replays to Zstandard without base64 encoding. This reduced total replay size from 677 GB to 406 GB (-40%).
With the next server update we'll also activate the new compression for the replay server. So also the growth of new replays will be decreased by 40%.

This was actually much more than we expected.
Also we found a lot of broken replays in terms of corrupt files that need to clean up.

So as of now the server has 280 GB usable disk space free which should suffice for another 3-5 years.

Auriko · 6 Mar 2021, 11:27

Good news

tatsu · 12 Mar 2021, 18:32

@brutus5000 you rock!