Why have any more than One Server?

CrazyEddie · 2019-01-19 02:07:21

Agreed that avoiding lag is the higher priority.

Just to keep clarity on the other issues: With the previous load balancing algorithm, a significant chunk of the playerbase would find themselves with no children when the global population fell, due to the servers being removed from the active pool. With the current load balancing algorithm, everyone experiences a modest reduction in fertility and no one experiences a dramatic reduction in fertility (which I think is a big improvement); but at the same time, server populations get low enough that each server has only one or two active towns.

Anecdotally, I think the change has dramatically improved the subjective impression that towns often fail because there just aren't any children born. I think the fix to the temperature-based weighting has helped a lot with that as well; experienced players (such as those on the forum and discord) know to stay warm and generally have plenty of children, while inexperienced players stay cold and childless - but we don't hear from them because they aren't on the forum or discord, and they were probably going to die anyway regardless of fertility because they tend to starve or get bitten.

The low-population / few-towns issue is new. I think it's a lesser issue, but it's still an issue. It's worst when there's only a single town on a server; when you get lineage banned from that one town, you will spend a long time playing in Eve camps (either as Eve or a low-gen child). The more towns there are on a server, the fewer Eve spawns there will be and the fewer Eve camps you will play in, as you will instead bounce between towns with each life. If the server population is low enough, it's even possible that the only town on the server will die from infertility simply because every player on the server is lineage-banned from that single town simultaneously.

If server performance were improved, you could raise the upper add-a-new-server threshold. And if you raised the lower remove-an-existing-server threshold, the not-enough-players-to-keep-towns-alive problem would be ameliorated. And if you had a large enough spread between the upper and lower thresholds, you'd reduce the frequency of adding and removing servers, which would ameloriate the everyone-on-a-server-becomes-sterilized problem.

Last edited by CrazyEddie (2019-01-19 02:08:47)

CrazyEddie · 2019-01-19 02:14:31

In this post Gederian speculated about using memcachedb (etc) to make a scalable db backend that could be shared by all the servers, making the whole thing effectively a single server.

In this post I speculated that eliminating sticky sessions in the reflector could bring about results similar to having a single server.

ryanb · 2019-01-19 02:17:43

jasonrohrer wrote:

Long ago, someone suggested that we should at least be storing map cells in blocks together (like blocks of 100x100 x-y locations) so that lookups of map chunks can hit the same location in the file and then exploit cache locality. That makes sense for the base cell values, but I'm not sure how that would work for expandable containers. As containers grow, you can't just "make room" in the middle of the file, which sounds like you'd end up adding extra room at the end of the file. I.e., back to a random-access linked list. I.e., it's very hard to have the whole growing container in the same place in the file, and thus in the same cache block.

Along these lines, would it be possible to scope the database by the location? Basically make a separate database for each given chunk of tiles. Is the issue that containers cannot be scoped by location?

mrslax · 2019-01-19 02:21:49

It could be built as a cluster environment where multiple servers for the database and end-user nodes But, (and this is a big but) the technical work to do this and the environment cost will go though the roof. So to keep the game simple enough for one man dev team the current setup is the best option.

It would be nice if the server is going down you could get like a portal to be transfer to another server so the lineage can live on.

jasonrohrer · 2019-01-19 02:36:11

Ryan, yeah, that could work somewhat. I mean, if each DB was small enough that it was one cache block or something, then I sususpect it would result in a substantial performance improvement for chunk reads. And I don't think containers would affect this (they'd be in the same chunk DB file along with everything else in that regional chunk).

However, it would certainly add a great deal of complexity to the implementation.

Eddie, yes, I think I will turn off the so-called "sticky" session on the reflector. Its easy enough to do, and my initial reasons for building it that way weren't that strong, so why not?

As for doing it with separate DB backend servers.... I'm pretty sure that would make DB access even slower (network requests between servers has to be slower than SSD access).

And this still wouldn't solve the problem of player interaction on multple servers (yeah, they would share the same underlying map DB, but that doesn't mean they'd see other players walking around from other servers, because that's not in the map database). In other words, the actions of other players on other servers would initially be invisible and then "pop in" the next time you re-fetched that portion of the map.

Unless he means one central "interaction" server managing all users connecting to multiple DB backends..... maybe that would help, somewhat (ignoring network latency vs straight local SSD access), but that certainly wouldn't scale to millions or billions of users!

jasonrohrer · 2019-01-19 03:20:19

Okay, Eddie, this change is live:

https://github.com/jasonrohrer/OneLife/ … 44dcbc169b

Still thinking about a RAM-based solution to the back-end database.

Anyone have any bright ideas here?

How would you efficiently store and retrieve a 4-byte value V based on a 16-byte key (x, y, s, b), where the number of values to store is unknown up front, and the thing will operate with a growing/changing value set for months?

Actually, consider the following code interface:

int LINEARDB3_open(
    LINEARDB3 *inDB,
    const char *inPath,
    int inMode,
    unsigned int inHashTableStartSize,
    unsigned int inKeySize,
    unsigned int inValueSize );



/**
 * Close database
 *
 * @param db Database struct
 */
void LINEARDB3_close( LINEARDB3 *inDB );



/**
 * Get an entry
 *
 * @param db Database struct
 * @param key Key (key_size bytes)
 * @param vbuf Value buffer (value_size bytes capacity)
 * @return -1 on I/O error, 0 on success, 1 on not found
 */
int LINEARDB3_get( LINEARDB3 *inDB, const void *inKey, void *outValue );



/**
 * Put an entry (overwriting it if it already exists)
 *
 * In the already-exists case the size of the database file does not
 * change.
 *
 * @param db Database struct
 * @param key Key (key_size bytes)
 * @param value Value (value_size bytes)
 * @return -1 on I/O error, 0 on success
 */
int LINEARDB3_put( LINEARDB3 *inDB, const void *inKey, const void *inValue );

How would you implement that, if RAM were not a concern?

jasonrohrer · 2019-01-19 03:21:35

Also, I should probably do some more profiling first, at some point....

apereason · 2019-01-19 03:34:18

tell me if I got this right, players will be randomized between servers?

jasonrohrer · 2019-01-19 03:36:44

Yes, randomized between the currently active servers. Current, that's servers 1-4:

http://onehouronelife.com/reflector/ser … ion=report

apereason · 2019-01-19 03:43:18

cool! now we just need to get like 50 people on a less popular server!

Last edited by apereason (2019-01-19 04:03:01)

CrazyEddie · 2019-01-19 04:06:31

jasonrohrer wrote:

And this still wouldn't solve the problem of player interaction on multple servers (yeah, they would share the same underlying map DB, but that doesn't mean they'd see other players walking around from other servers, because that's not in the map database). In other words, the actions of other players on other servers would initially be invisible and then "pop in" the next time you re-fetched that portion of the map.
Unless he means one central "interaction" server managing all users connecting to multiple DB backends..... maybe that would help, somewhat (ignoring network latency vs straight local SSD access), but that certainly wouldn't scale to millions or billions of users!

Hm.

You could have a three-tier architecture, one not too dissimilar from the two-tier architecture you have now. Tier One would be a reflector like you have now but with a bit more logic, Tier Two would be a pool of gameplay servers like you have now but without persistent data store, and Tier Three would be a pool of database servers. Move the spawning logic from the current server tier to the reflector, so that the reflector can make decisions about where and under what circumstances the players spawn. The reflector sends players to a specific gameplay server depending on their spawn location. Each gameplay server is responsible for a given set of regions of the map. Each server uses a networked database, locally cached in RAM, for both the map data and the player data. The network database is itself divided up between multiple db servers. That allows you to scale up by adding servers to the gameplay tier and/or the datastore tier as needed, depending on where the bottlenecks are. Only the reflector would be a single server, but it would only be used once each new connection, not constantly during each game.

Most of the time everyone would be localized on their own gameplay and datastore servers, and most of the time player/map interactions would be in the local server's RAM cache. Cross-region interactions involving players from one region and map objects from another region would have to hit the network, and cross-region player-player interactions would hit it even more. But those will be rare. You want them to be possible ("everyone is playing in the same one big world") but even so they'll still be rare.

You could even set it up so that the gameplay servers are also the datastore servers, so that each server keeps its assigned map and player data in RAM and serves it up locally to the local gameplay instance (common) and remotely to the remote gameplay instances (rare). Gameplay servers would have to know which of the other servers to query for which map regions.

All very complicated. But possible!

Whether it's feasible or valuable is an entirely different question.

CrazyEddie · 2019-01-19 04:46:05

|--> server1.onehouronelife.com : 8005 ::: 45 / 120

|--> server2.onehouronelife.com : 8005 ::: 92 / 160

|--> server3.onehouronelife.com : 8005 ::: 37 / 160

|--> server4.onehouronelife.com : 8005 ::: 34 / 160

|--> server5.onehouronelife.com : 8005 ::: 1 / 160

I suspect that Server2 is where the pros are. First off, the manual connections by pros to Server2 will increase the player count on that server above what would be sent there by the reflector as an even share of non-manual connections. Secondly, the pros are going to do a better job of keeping everyone alive, which increases the server's total share of the population. The longer the average lifespan, the larger the resulting player count.

This is the opposite of what we had before Jason's change this evening. All servers would have (roughly) the same population regardless of how quickly or slowly the players on each one died.

CrazyEddie · 2019-01-19 08:32:12

jasonrohrer wrote:

Still thinking about a RAM-based solution to the back-end database.
How would you implement that, if RAM were not a concern?

(I am not an expert, nor even a real developer)

Am I correct in thinking that the current database stores the values on disk, and stores a hashtable in RAM that contains pointers to each key's value on the disk, but that the values themselves are not stored in RAM, and thus every value read requires a disk access?

Since you've determined that disk reads are your bottleneck, it seems like using some kind of caching of the keys' values in RAM would speed things up. Here is a short article about some different caching strategies and their trade-offs. You'll also need to consider things like cache invalidation, time-to-live, and cache eviction when out of memory. You could write your own caching mechanism, or you could integrate an off-the-shelf product like redis or memcached.

Alternately, instead of keeping the "source of truth" on disk and caching whichever results happen to get queried, and thus having to deal with issues like cache invalidation, you could instead keep the entire database in RAM and use the disk only for long-term persistence. It seems to me that for OHOL you could use a very lazy flushing-to-disk strategy, only updating the database (or even rewriting it entirely) on disk every now and then, perhaps every few seconds. As long as the server is running, you don't need to worry about the data on disk being stale, since the RAM copy will be the source of truth. If the server is shut down gracefully, then you'll have ample opportunity to flush the RAM copy to disk after the server stops writing to the database. And if the server is ever halted ungracefully, you'll lose a few seconds of data - which won't matter, because everyone in play will have been summarily disconnected anyway, with no expectation of resuming where they left off.

Keeping the db in RAM means you'll have to decide what to do when you run out of memory. Or you could monitor the db size out-of-band and just make sure it never grows larger than the amount of RAM you've purchased, knowing that if it ever does the server will probably crash ("If you choose not to decide you still have made a choice.").

If you use a cache, you can write your own or use something existing. The leading candidates appear to be redis and memcached. Here is a nice post describing the differences between them and showing several different ways that you can use them. It sounds like redis could be a good fit for you - you can use it to store all of your data, it will store it all in memory for fast access, and it will write it to disk so that you won't lose it if the system halts.

A quick-and-dirty solution would be to keep using your database code exactly as you are, but store the database file on a ramdisk. Have something that periodically copies the file from the ramdisk to the SSD. You could even get away with not trying to make the copy operation atomic, if you do some kind of integrity check on the database when the server starts up and just bluntly delete any records that aren't consistent. In most cases probably everything would be just fine, and in the worst case you might occasionally lose an object here or there if the server ever gets halted ungracefully.

Last edited by CrazyEddie (2019-01-19 09:04:00)

Tarr · 2019-01-19 10:37:15

CrazyEddie wrote:

I suspect that Server2 is where the pros are. First off, the manual connections by pros to Server2 will increase the player count on that server above what would be sent there by the reflector as an even share of non-manual connections. Secondly, the pros are going to do a better job of keeping everyone alive, which increases the server's total share of the population. The longer the average lifespan, the larger the resulting player count.
This is the opposite of what we had before Jason's change this evening. All servers would have (roughly) the same population regardless of how quickly or slowly the players on each one died.

Honestly the mass grouping could be for all sorts of things along with what you're suggesting. If you look at the reflector before playing would you rather go to the server that is most lively (server two in this case) or one of the less popular servers? I normally go where the people are since the Eve changes and what not so now instead of specifically going to server 1 I'll be hitting 2/3/4 depending on the player counts.

While yes I don't think one completely dominant server should be the goal I think it's better than keeping players stuck on the low pop servers in an "unplayable" state. I still think he should just move us to three servers on these early hours where we're at 27/52/40/17 as server 4 is in a terrible state, server 2 is great, server 3 is good, and server 1 is good enough.

Greep · 2019-01-19 11:55:41

Definitely agreed, on 4/5 servers being too much on dead hours, and retuning them so two or 3 is good enough. It's a good compromise from server1 being the only one able to survive nocturnal infertility, and experienced players can always just choose server2 which wasn't a great option before the population changes.

But really the issue of people thinking "not enough servers" boils down to "not enough substantative towns" or "towns dying out to lack of babies", and these are issues that do not require massive overhauls of code. The death of eve chaining is a bigger concern than server number. I think if this behavior isn't desired to be intentional, then something in it's place needs to be added.

It would be interesting if servers were divided into high and low tech (probably just even and odd), which makes everyone happy. High tech servers intentionally spawn eves into exactly one area (distant towns would naturally arise as people break off). Low tech servers are the status quo.

Throwing players into random servers keeps newbies satisfied with a variety of lives this way, and as much as I like the growing numbers of One City Server, I would like having real towns on the main server because I sometimes miss teaching pure newbies and the chaos of it. But I like some sort of progression at least over a few days, and that's never going to happen with the current system.

Last edited by Greep (2019-01-19 11:56:44)

lionon · 2019-01-20 08:44:56

jasonrohrer wrote:

Anyone have any bright ideas here?

Not so bright, but have you considered instead of using a simple key/value store to use a document based database like mongodb? As far I get it, your data is "document" based, as in one map cell is one "document" (of unknown size).

I mean there are so many years of research and optimization in database and implementation details, like matching reads/writes with harddisc sectors etc.

Disk usage may go up tough, quite a bunch, as they optimize speed over everything else.

One Hour One Life Forums

#26 2019-01-19 02:07:21

Re: Why have any more than One Server?

#27 2019-01-19 02:14:31

Re: Why have any more than One Server?

#28 2019-01-19 02:17:43

Re: Why have any more than One Server?

#29 2019-01-19 02:21:49

Re: Why have any more than One Server?

#30 2019-01-19 02:36:11

Re: Why have any more than One Server?

#31 2019-01-19 03:20:19

Re: Why have any more than One Server?

#32 2019-01-19 03:21:35

Re: Why have any more than One Server?

#33 2019-01-19 03:34:18

Re: Why have any more than One Server?

#34 2019-01-19 03:36:44

Re: Why have any more than One Server?

#35 2019-01-19 03:43:18

Re: Why have any more than One Server?

#36 2019-01-19 04:06:31

Re: Why have any more than One Server?

#37 2019-01-19 04:46:05

Re: Why have any more than One Server?

#38 2019-01-19 08:32:12

Re: Why have any more than One Server?

#39 2019-01-19 10:37:15

Re: Why have any more than One Server?

#40 2019-01-19 11:55:41

Re: Why have any more than One Server?

#41 2019-01-20 08:44:56

Re: Why have any more than One Server?

Board footer