Update: Sweet and Spicy
March 29, 2019
Back from GDC with a bang. This is the largest content update in a while, with over 100 new objects to grow, catch, cook, and eat. And all this from only three new naturally-occurring objects.
What else? Spring loaded doors for your kitchen. That whole GDC basket horse cart right-click bug has been fixed for real client-side. Email-based family tree searches are no longer public (access your personal family trees from the button on the login screen or on your personal download page), so people can't spy on you anymore if they know your email. Animals no longer spawn outside their home biome along biome boundaries. Goodbye, arctic snakes and desert penguins. It was fun while it lasted.
And now for a tale of a funny lurking bug situation. Some of you may have noticed that recently, the whole shebang has been going down like clockwork at 5 am EST for about five solid minutes. Website, forums, login server, everything. I'm asleep during that time, so I didn't notice, and no one bothered to email me about it. Not sure how long this had been going on.
Turns out that 5am is the time when the various backup processes run, and one of these accesses the main MySQL database with mysql dump to back it up. That backup might be locking out other database requests, which would affect the website any everything else. But five minutes is a long time. Turns out that the review/stats server log table had ballooned to over 2 GB in size, with 220 million entries. Uh, yeah, that would take a while to extract with mysqldump. Turns out that it was filled to the brim with bad requests by other servers for non-customers. What other servers? From the IP addresses, these were Chinese servers. To the tune of 300,000 bad requests per day, and each one getting logged. The mobile developers apparently forgot to switch this stat-reporting feature off in their Chinese servers, which means that every time a Chinese player lives a life, I get hammered with a bad "log stats" request from their servers.
And yes, they are the biggest fish in this pond, but there are other bad requests getting logged from small-time servers too, most likely private servers being run by individuals.
First of all, I probably shouldn't be logging every bad request that comes in, since it's a waste of disk space, and I have no control over how many bad requests come in. So, that's fixed, and these logs have been cleared, making them tiny again. Now the backup mysqldump takes 3 seconds instead of 5 minutes, and it doesn't seem to block anything (it's supposed to set a read-only lock anyway, which should at least allow the website and such to load).
I also reached out to the mobile devs to ask them to knock it off. Their Chinese servers are being run by a third party, but I'm hopeful that this deluge of requests will end soon. Anyway, 300,000 lives a day is a lot. Apparently the game is popular in China.
But the fact that so many servers, in general, are submitting requests to me by accident is a bad sign. Yeah, I guess I made "reporting" the default behavior in the public server code on github, which means that anyone who is not paying attention when they set up a server will end up hitting me with reporting requests forever. Because these other servers don't have the secret key, the requests will be rejected, of course, but they're still a waste of time and bandwidth for everyone involved. I can imagine a nightmare scenario in the future where there are 1000s of private servers, and their collective accidental reporting traffic combined results in a DDOS attack. So, the public server code has been fixed to not report by default. Update your server code, folks.
|
|
|
