Update about C&C3 online bugs

An update was posted earlier this week about the problems with stats in Command & Conquer 3: Tiberium Wars. The developers say they are working hard on it but that it's not very straightforward and that therefore it's taking longer than expected to fix all the problems.

However, they say that most problems are already fixed:

The backend stats processing infrastructure is quite sophisticated, but unfortunately there were a couple of components that did not scale as well as we had hoped. This led to a backlog of match results that, at times, could be as long as three or four days. While these results were never in danger of being outright lost, when you don't see the matches showing up in your stats for days on end, it's almost as bad as if they never showed up.

This problem has now been fixed and the system is now processing stats at a pretty good clip. The back log has been cleared so I'm afraid that if you are still missing stats they may fall into one of the categories described below and are probably lost. Our preliminary testing has shown that generally match results are being processed in close to real-time; most results are making it into the backend database in about 15-30 seconds, with the website usually receiving them within just a few minutes.

There were a few website bugs that were preventing stats and match history from displaying correctly. One of these bugs caused the “match time” stat in the history list to be set to the time that the match was processed, and not the time that the match ended. When our backlog was a few days, this meant that matches in your history had timestamps with little relation to when the match actually occurred. This problem has been fixed and all new matches should be getting the correct timestamp.

It doesn't have anything to do with stats, but it’s definitely worth mentioning. With the 1.04 patch, lobby stability has been much improved. We’ve opened up many more lobbies because the crux of the stability problem – crashes and disconnects -- was due to the strain of having four or five hundred people in a single lobby. We now load-balance people at login, and limit the maximum number of people in a single lobby to a little over a 100. Check out the new podcast, where the Producers talk about this issue in more detail. (BTW, when Amer talks about people on the Internet being crazy, of course he’s not talking about you.)

However, the following problems still need to be fixed:

There is still a known problem where a very small percentage of stats submissions – less than half of one percent – are being outright lost. We’re still investigating, but the fact that this so difficult to reproduce is definitely gating how quickly we can track it down. We can detect when it happens, so if nothing else we will probably make changes in the “Big Patch” that will give players some indication at the beginning of the match that the match stats will not be submitted. If you are still seeing a significant number of dropped stats with new games, and especially if your loss of stats has high reproducibility, it would be a big help if you could please respond to this thread and describe exactly what you are seeing.

There are still a few people – at last count we’ve identified about ten people through forum posts – that have no stats history at all. They are definitely playing games, and even broadcasting those games fine, but not a single stat submission from them has come through. I know this is making some of you guys go, “WTF!” -- us too. We have already begun to contact these individuals and will be working with them to get this bug figured out. BTW, I don’ t mean to imply that only 10 people are being affected by this bug. There are certainly more folks who fall into this category but haven’t contacted us or haven’t posted on the forums. Still, it’s clearly a very small percentage of everyone playing, which makes tracking the bug down more difficult. Hang in there – we’ll get it.

We, or in most cases our ISPs, have been hit by several denial- of-service hacker attacks over the past month. Unfortunately many (most?) stats submissions that occurred during this downtime are lost forever and we do not have any way to retrieve them. I’m afraid that when bandwidth gets saturated to the point where no network traffic can get to our servers, we obviously cannot receive and process stats submissions. Hopefully we’re out of the woods on this one.

As noted in the podcast, we will be making some changes in the upcoming “Big Patch” that will return the themed lobby names, while retaining our load- balancing strategies and lobby population caps.