End of the apocalypse, and lag fixes today
April 8, 2018
Boy, did that wake everyone up or what?
For those of you who have played 100+ hours and are so mad after one day of change that you're thinking about asking for a refund....
Remember: this is a game that is being actively developed. By one person. Working alone. Doing everything. 12-16 hour days. It's Saturday. My family needs me. But here I am.
So, think for a minute before you jump on the review button and call me LAZY in all caps, please.
I must have the freedom to try things, dangerous things, game-breaking things, in my endless quest to make the game better and more interesting.
I appreciate that you love the game as-is and don't want it to change. But the numbers that I see on my end tell a different story. Yes, there are an impressive number of concurrent players at peak times (150 - 200). But that number has started to slide, and is nowhere near where it was a few weeks ago. In the mean time, 14,000 people own the game. They are not playing. For a reason.
And it has nothing to do with the apocalypse.
It has to do with the game not being quite good enough yet. The game is interesting and compelling up to the point where established villages achieve a steady, perpetual state. If you have limitless food, there is no challenge, no danger, no drama. Griefers are a symptom, not a cause. If you are struggling to survive, you have no time for griefing.
And this game should always be about struggling to survive, at some level. It should always be possible to fail, both at the individual and village level.
But villages were everywhere. You could always wander into a deserted one and pick right back up. Failure meant nothing.
Thus, the game sorely needed a hard reset. I decided it would be more interesting to put that power into your hands and see what you did with it. I also wanted to create a shared collective event.
Those who witnessed the apocalypse waves first-hand will never forget them. It's over now, but the reset happened.
And the result, for the time being, is a game that is much more interesting again.
Building a village from scratch is the interesting part, and making a contribution that really matters is the most meaningful way to leave a legacy. Making another bearskin rug in a village that already has 20 rugs, because there's nothing else to do, is far less interesting.
In the place of the apocalypse, I have added a new placement algorithm for Eves that will have a similar periodic cleansing effect. Not server-wide, but at the lineage level. Your chance to continue living and working in a given village will end when the lineage in that village dies out. No more wandering back later and starting over in the same spot with everything already done/built for you. Each new line will start in the wilderness.
That said, pilgrimages to the old village locations are still possible, but they will require a concerted group effort to pull off, Oregon-Trail style.
But after I implemented this new Eve placement, which involved only a few lines of code, a strange thing happened.
Server CPU and disk usage rose steadily over the next 16 hours, eventually getting to the point where the servers were so bogged down and laggy that the game was almost unplayable.
If you experienced this today, I'm sorry about that. I've fixed it now, but the source of the problem was surprising.
The underlying databases are hash table based. As more entries are added to these tables, collisions occur, effectively creating a chain of "pages" in the hash table. Lookups for these later entries thus have to step through several pages before finding the matching item.
The general pattern here is that as more of the map is explored and modified, the servers become slower and slower, as hash table collisions become more common, and multi-page lookups are needed.
That has always been the case, throughout the history of the game.
But now suddenly, with the new far-flung Eve placement, it became a serious problem.
It turns out that all those far-flung Eves were exploring more and more of the world than ever before (whereas previous Eves were in the same area, so they kept wandering through already-visited places on the map). This made the underlying databases grow and fill with collisions.
As an example, one of the databases had such long collision chains than the average lookup would need to hop through 175 hash table pages. Not good.
Even worse, the newer entries in the hash table go at the end of these chains. As Eves were placed farther and farther away, this meant that the quickest-to-access entries in the table (the oldest entries) were never being needed again, while the latest entries---the tiles we were looking at around the latest Eves---were at the end of very long chains.
The game uses an existing database module called KissDB that is very fast, but probably not designed with this usage pattern in mind.
The long-term solution is to re-write the database from scratch as a stack, so that the most recently-accessed elements are the fastest to access, while the forgotten parts of the map slide to the ends of the chain. I'll be doing that work next week.
In the mean time, I changed the usage patterns for some of the largest databases, resulting in a huge performance increase and reduced RAM footprint.
The servers are lag-free again.
And once I write a new database engine, performance should be even better, allowing me to raise the player caps per server.
So I hope you'll stick with me as I continue working to improve the game.
It's not over yet. We have years to go, together.
Jason
|
|
|
