|Update: Black Gold|
December 8, 2018
"And then one day he was shootin' at some food, and up through the ground come a bubblin' crude. Oil that is. Black gold. Texas tea." Californy here we come!
My father was an oil man. At least he tried to be, for a while. When I was young, he invested in a local oil exploration company in Ohio. Everyone in the family got free ball caps with the drilling outfit's logo. I remember climbing up the steps on the side of a huge oil tank, where he wrenched open the porthole cover and we peered inside as the oil poured in. Was a sight! And more importantly, what a smell! With tiny me perched up there by his side, my father reached his arm down inside with an empty peanut butter jar and filled her up directly from the gushing pump stream. (What the hell was he thinking?) That jar sat---and settled---on his office desk for many years. Sludge would be a naive appraisal of what was in that jar. Brown sludge. A reminder of an investment in oil that never turned a profit.
But your investment in oil will turn a profit!
The more time you spend around crude oil, the more trouble you will have believing that people actually used to rub this noxious stuff on their bodies as a medicine.
But what about the very best stuff? Light, sweet crude, with a very low sulfur content---really top notch. If you're brave enough to take a sip, it actually tastes sweet.
So you want that oil. But oil prospecting and refining requires a lot of equipment along the way. And that's why this update is one of the largest and most complicated in the history of the game so far. Machines galore.
But when you finally see that black plume shoot toward the sky, you too will dream of striking oil before you die.
|Update: Newcomen Atmospheric Engine|
November 30, 2018
This update paves the way for the forthcoming industrial revolution. After banging my head against the actuality of human history in this area (much of which is, strangely, shrouded in uncertainty) for four straight hours on Monday, I realized that bootstrapping is hard. This is the point where the true mystery of human civilization comes to a head: how do you make a lathe without a lathe?
We've come a long way so far, and bootstrapped a whole bunch of things in this game. But these are all things that I myself know how to make from scratch, in principle. This is the kind of stuff that is made on the Primitive Technology YouTube channel. If it looks hand-made, it can probably be made by hand, right?
But how do you bootstrap metal machines? I'm starting to suspect that this is something that no one now living actually knows how to do. So it's my job to figure it out, or at least make up a reasonable approximation. I was thinking that we'd be skipping right over the age of steam---that steam was just an unnecessary side-branch on in the inevitable path toward internal combustion. But now I think it's actually about tolerances. Internal combustion requires extremely tight tolerances, and metal gaskets, because there's fire right there in the cylinder. Steam engines can be made with much lower tolerances, and leather or rubber gaskets, because the fire is kept outside the cylinder. I.e., crude steam machines can be made without the process of machining. And with steam machines, we can make the machines with which we can actually "machine" parts with tighter tolerances. Steam lathe, here we come.
And speaking of steam lathes, this video of a 1900's era steam-powered machine shop is pretty amazing. But none of those belt-driven machines contain parts that were made by a blacksmith:
But there's no lathe in there yet. What can you do with this Newcomen Atmospheric Engine? Pump water! Why would you want to do that? You'll find out soon enough.
This update also dramatically improves the mouse interface when dealing with stacks of things that should also be moveable as a stack (stacks of plates, for example). Left click grabs the whole stack, and right click removes an item from the stack. In other words, the stack now behaves similar to a container when you left or right click on it.
New wild wounds (or sickness) no longer replace your current wound. You can't heal from a snake bite by contracting yellow fever or getting hog cut.
|Update: Turkey Hunt|
November 22, 2018
Yes, this update is a bit obvious. It's Thanksgiving tomorrow in the US. But separate from that, the turkey is a majestic animal.
The wattle! The snood! The caruncles! The beard! What other animal, of feather or fur, has facial anatomy more distinguished or beautiful than the turkey? Furthermore, it is a true native of our new world. An American original. Imagine what the early European settlers must have thought when they first laid eyes on it.
To prevent turkeys fighting over who has the longest, reddest, and most engorged snood, farmers often desnood the turkey chicks at a young age. That sounds like a barbaric practice to me. There will be no desnooding in this game, at least not yet, because turkeys cannot even be domesticated. They are wild turkeys, thank you very much, and you just can't pin a wild turkey down long enough to desnood it. Maybe someday, a future update will feature domestication, desnooding, and the all-important Presidential Pardon.
The only other content change involves yellow fever. Getting bitten multiple times will no longer auto-kill you, but will instead just extend the duration of your illness. The side-effects of the illness have also been adjusted to reduce the chances of survival without help from friends. You're sick. They need to nurse you back to health. This should help balance the Jungle a bit more.
Those of you who have been having trouble with mouse cursor alignment on ultrawide displays will now hopefully find that the problem has been fixed. On such displays, I now hide the system mouse pointer, which is off, and draw an emulated mouse pointer at the true click location. This emulated pointer can also be scaled to your heart's content from the SETTINGS screen, which also should be helpful for low-vision and high resolution users.
You can now access your personal family trees, which will open in a web browser, from a button on the main menu screen. No more typing your email or logging in via Steam to access your lineage page.
|Update: Jungle Flora|
November 19, 2018
Wow, what a week! Steam, of course, had the expected effect: an absolute flood of new players, and the resulting chaos.
This is the first weekly update post-Steam, and as you can tell, it's a bit later than usual. And the release of the update, for any of you playing on Sunday evening when it came out, was quite a bit bumpier than usual. I'll explain more about that in a bit, but first, the update itself.
As you can see from the GIF, there's a brand new biome, the first new one in a while. If you spend some time there, you'll find that it has a pretty good balance between benefits and drawbacks, though that balance will be tweaked over time. It also contains raw ingredients that will help you push transportation tech into even more efficient and durable realms.
Beyond the new content, the biggest change client-side is that the crafting hint filter system has been overhauled to align better with people's expectations. In general, the crafting hint system shows you hints for what you can do with what you are holding, or what you just touched. It does not show you specifically how to make an end product, but steps along the way that you might want to explore. The filter system (/hatchet) is meant to cull down the list for what you are holding to show you only steps along thew way that lead to a hatchet.
First, an easy fix. The old filter system would show you any steps along the way AND anything you could do with the hatchet once you got it. This isn't very helpful---you're only filtering by /hatchet because you don't have one yet. Who cares what you can do with it? So those are hidden now, when filtering.
One point of confusion: What happens when a hatchet itself can be used to make some of the end-products for a hatchet? This may not be true for a hatchet, but it's true for an axe, which requires kindling to make, but can also make kindling. Still, it's best not to show this when filtering for /axe. There have to be other ways to make kindling, after all, because you don't have the axe yet.
Finally, what happens when none of the steps for what you're holding are relevant to /hatchet? Let's say you grab a piece of flint. In the old system, the full, unfiltered list for flint would be shown, and "NONE RELEVANT" would be shown next to the filter. But a lot of people didn't see this, and it was confusing. Why is it showing that I can skin a wolf with flint? I'm trying to make a hatchet! Now it shows no steps at all, and spells it out for you. "MAKING HATCHET (FLINT NOT RELEVANT)"
Also, I found and fixed some mistakes in the filtering logic that led to the display of extra, irrelevant steps. To summarize, the whole filtering thing works much better than it used to. Try it. You'll like it.
Okay, so what happened during the update process?
First, an old version mismatch issue which had bitten me before reared its head. The client makes sure that it is at or ahead of the server's version, and if it is ahead, it makes sure that it's data version at least matches the server's version. This works fine most of the time. For the weekly update, I can decide to either update the client binary first, or update the data an server first.
But during weeks where the client and server receive several updates, while the data version remains untouched (like this week, when I was fixing all kinds of steam issues), we can get into a dangerous situation. The server and data version MUST be updated together, first, before the client is updated.
But this week, there were protocol changes requiring that the client be updated first. The server was going to send newly-formatted messages that the client wouldn't be able to parse. So, I blindly updated the client first, without realizing that we were already several versions ahead of the data version.
On next startup, the client checks if its version matches the server. Nope. Server is v167. Client is v168. Okay, well, is the client data version at least matching the server's version? Nope. Data is v164. Client displays a version mismatch error.
This was put in place to make sure that people connecting to private servers wouldn't experience content-mismatch crashes if their private server lagged behind the official update schedule. But actually, the above logic is too strict. If the client is really at v168, we are ahead of the server, but we can see that our data version number is behind the server, so we're actually okay. What we really need to worry about, in the case of private servers, is if both the data version and binary version number are ahead of the server's version number. So, the client logic has been fixed to prevent this kind of version mismatch in future updates.
So that was fire #1. It caused about 34 minutes of downtime for people who were downloading the update and trying to reconnect. You can see the first huge downward spike in the above graph.
After that got sorted, and everyone had a working client again, it was time to push out the content update. That all went according to plan, and the Steam clients got the update correctly. Everything went fine, until server2 tried to start up. It got through its init process, and started accepting connections, but then got bogged down into a huge processing load. 99% cpu, even with 0 players connected. And still accepting connections.
The reflector, the bit of server code that tells you which server you should go to, had a 3-second timeout in place for a server. So, when checking on server2, it would wait 3 seconds, and then timeout. But with hundreds of client requests coming in, 3 seconds of waiting for each one was too much. And server2 was at the top of the list, meaning that the reflector would consider it first, for everyone, before considering other servers. 100s of stalled web requests means that nginx starts running out of processes, eventually triggering the dreaded 504 gateway timeout. This results in no one getting successfully assigned to a server, and error messages getting displayed in the client.
Fortunately, I was able to track the problem to server2 very quickly, and take it off the list. I think there was something like 15 minutes of downtime from this, fire #2.
But I also realized that server1, which was in the queue to update next, would likely have a similar problem. Other servers didn't seem to be affected.
Initial debugging showed that server2 was bogged down with 200,000 moving animals. Hmm.... maybe I made mosquitoes too common in this update. That must be it.... the Mosquito Meltdown.
But further investigation showed that it wasn't mosquitoes at all, but wild sheep and boars. 200,000 of them, all trying to move at the same time, right after server startup.
Well, thank you Steam, for exploring more of the map on server2 than had ever been explored, because you unearthed a horrible, lurking issue. Animals aren't supposed to move unless they are seen by a player. So, even if there are 200,000 of them known to be in the map (seen by some player at some time in the past), at server startup, this should be no different than a map containing 200,000 trees or rocks.
However, a particular bit of server map processing code was inadvertently "looking" at these moving animals, causing them all to get put in the queue of moving things that need active updating.
This has apparently been a lurking problem for a while, but historically small enough that the server could simply brute-force its way through it at start up, and get back to processing requests. Not this week. 200,000 moving animals.
The first step was to fix the reflector so that it would timeout more quickly. These servers are all connected by high-speed networks. 3 seconds is overkill. I was actually able to bring the timeout down to 1/8 second (125 ms). If a sever is too bogged to respond in that time, better think of it as offline anyway, for now, right?
And with that fix in place, when server1 restarted and had exactly the same issue as server2, there was no widespread outage triggered.
With everyone back online and happy, I spent the rest of a very long night refining the server startup code to avoid "looking at" any of the map objects that are being processed. I got server1 down to the point where it could startup, process the entire map, including all 200,000 animals, but not result in a single animal waking up and moving, at least not until a player spawned in to see them.
This also dramatically sped up the server startup and shutdown process. The other servers will get this improved code during the next update, later this week.
|Update: Games of Chance|
November 3, 2018
This week I developed an extension of the category system which allows crafting transitions to have more than one possible outcome. This is obviously needed for things like fishing, which is included in a minor way in this update. The old "do this and get that, guaranteed" just doesn't feel right when dipping a hook in the water. You'd be hauling 'em out one after another.
All transitions in this game are A + B = C + D. In the past, A and B could be categories, which might mean that an axe can chop anything in the "kindling source" category into kindling.
The new probability system now allows C and D to be categories as well, but a new kind of category with weights for each member. Whenever the transition is invoked, one of the members is picked at random, according to the weights. The weights can be any probability distribution. Some outcomes can then be very rare, when that's appropriate (like catching a special fish, or finding a diamond in a mine).
But I didn't stop at fishing. To fully put this new system through its paces, it made sense to explore the human activity that led to the development of probability theory in the first place: games of chance. Also known as gambling. Thus, I didn't just stick in a probability set with a few outcomes, like fishing. I included one set with 21 outcomes (for a pair of rolled dice), and another with 52 outcomes (for a deck of cards). Good thing I did this, because including categories this big unearthed a serious, long-term crash lurking in the category system.
And come to think of it, fishing and gambling really do go hand in hand.
The tutorial was also updated with some improved wording and a "hint hallway" for people who are really stuck (age-gated at 35, so you have to have been trying for 20 minutes before you get any hints).
Those of you who saw moving ice holes related to penguins, that bug has finally been found and fixed. And a new protocol is in place (PING/PONG) to help the client detect true cases of the bouncing-forever bug, as opposed to just genuine network outages. This should result in far fewer false reports. And one new cause of bouncing-forever, this time caused by the new reconnect system, has been fixed.
And stacks, and partially-used objects, no longer revert to their full state on server restart. Their true states are correctly remembered. This also means no more weird universal locks after server restart.
All that said, I still have quite a long list of mini-bugs on GitHub to fix this week before the Steam release on Thursday. That will be my focus, and there will be no new content this week (it would be weird to ship a content update on the very day that the game goes live---the new players will need a week to settle in). I hope to push out the final bug fixes (including content fixes) on Tuesday, to allow time for some last-minute fixes on Wednesday as well.
The plan is to put out a nice-sized content update the following week, and then continue with weekly content updates thereafter.