Public data available again (separate server)

jasonrohrer · 2019-03-25 21:08:26

The public life log and name data has been moved to a separate server here:

http://publicdata.onehouronelife.com/

There's nothing else on this server, so if you bog it down, you only bog down other people who are trying to get this data, not the whole OHOL website.

There was also an oversight in the old cronjob that exported this data---it skipped the curse log export! So that data will be included in the future (for now, it only goes back 3 days, because I didn't want to bog down the server with processing all that curse history).

Still trying to figure out how to ensure privacy of email addresses on the family tree server.

jasonrohrer · 2019-03-25 21:33:36

Yeah, the privacy of email addresses on the family tree server is a tough one...

It's so convenient to be able to see all of your own lives, instead of remembering names and searching by those.

The only way around this is if you MUST access your own lives via your download page, instead of searching for your own email in the form. This is a bit of a pain, but will mean that no one but you will be able to search for you in the family tree server.

wondible · 2019-03-26 02:21:29

How about the client program link? Could it use the game key in some way to provide an additional proof of authenticity?

Whatever · 2019-03-26 05:30:11

Cool that it is up again.

About the privacy, people like to see their stats. Please dont change the way the data is saved on http://publicdata.onehouronelife.com/
But i think you mean the lineage browser?
You could give people a small password, that is added to the email address, for example:
the password: 1234
the email: myemail@host.com
and then you sha1: myemail@host.com1234
But then you need to remember this password, you could show it in the client somehow and or the download page.

I personally dont mind the way it works right now.
I am not sure how many people see this as a privacy issue.

Whatever · 2019-03-26 05:47:26

About the data:

I can see you added some curse files. Which is cool, but i think its a bit strange that they are separated from the other data.
Would it not make more sense to save it together with the other data, like this:

2018_03March_31_Saturday.txt 04-Jul-2018 19:31 1191956
2018_03March_31_Saturday_names.txt 04-Jul-2018 19:31 30304
2018_03March_31_Saturday_curses.txt 04-Jul-2018 19:31 30304
2018_04April_01_Sunday.txt 04-Jul-2018 19:31 1364079
2018_04April_01_Sunday_names.txt
2018_04April_01_Sunday_curses.txt

But i am also fine if you keep saving it like it is now, its not a big deal.
Will you later add more curse files from the past?

Another thing, if we look at these lines:

D 1522454434 158660 1580c7bc3970613cbb9d5a194a0f488c6b1b597b age=0.38 F (-429,-672) disconnect pop=35
B 1522454435 158661 8d4a91c6a8f153edb45e2b7fe333f57b7d68cf05 F (-552,-949) parent=158565 pop=37 chain=5

We can see "age=", "parent=", "pop=", "chain="
which is nice if a human reads it, it makes it more understandable, but i assume most of the time programs will read it.
for a program its not necessary to have these words. You are saving the same words over and over again which wastes a bit of memory.
But this is also not a big deal, i just wanted to mention it and see what you think about it.

Chard · 2019-03-26 07:14:41

Whatever wrote:

Another thing, if we look at these lines:
D 1522454434 158660 1580c7bc3970613cbb9d5a194a0f488c6b1b597b age=0.38 F (-429,-672) disconnect pop=35
B 1522454435 158661 8d4a91c6a8f153edb45e2b7fe333f57b7d68cf05 F (-552,-949) parent=158565 pop=37 chain=5
We can see "age=", "parent=", "pop=", "chain="
which is nice if a human reads it, it makes it more understandable, but i assume most of the time programs will read it.
for a program its not necessary to have these words. You are saving the same words over and over again which wastes a bit of memory.
But this is also not a big deal, i just wanted to mention it and see what you think about it.

The convention with log files is that they should be consistent and unambiguous and human readable. Programs don’t care about the extra data and humans do so there’s no sense not having it, it just makes the log file more useful to humans and no less useful to programs.

Regarding the extra file size. Firstly it will compress well which is good to know. Secondly we’re talking about approx 10% file size. Personally I’d wait for it to look like an issue before compromising the utility of the logs. Profile first etc.

Whatever · 2019-03-26 07:34:00

Chard wrote:

The convention with log files is that they should be consistent and unambiguous and human readable. Programs don’t care about the extra data and humans do so there’s no sense not having it, it just makes the log file more useful to humans and no less useful to programs.
Regarding the extra file size. Firstly it will compress well which is good to know. Secondly we’re talking about approx 10% file size. Personally I’d wait for it to look like an issue before compromising the utility of the logs. Profile first etc.

Ok, i see, i just wanted to ask.
I also dont want that jason changes it now, since my scripts would stop working.

jasonrohrer · 2019-03-26 19:37:48

This week I'm rolling out a change to the lineage server so that you can't just search by a raw email address.

You must either click the FAMILY TREES button in your client, or the "Family Trees" button on your personal download page. Both of these methods of access include an HMAC based on your (secret) account key, which is checked before showing you your own personal family trees.

If this is sniffed, then it is vulnerable to replay attacks (since there's no sequence number), but I figured that it's good enough for now, and I don't want to bother with a full blown challenge-response implementation. Most likely, a person who is interested in spying on your family trees won't be sniffing your traffic.

(The main game authentication protocol is immune to replay attacks.)

This reminds me that there is a sniffing vulnerability in your personal download page, which is not served over https. It would be simple enough to switch over... but here's the problem:

The family tree server itself is served over http, which means that the "Family Trees" button (a form) on the download page triggers a browser warning if the download page itself is https.

And why not serve the family trees over https? Because they are on a different domain name, which (I think) would require them to have their own unique cert. I'm not 100% sure about this.

But in general, I see https as much more fragile than http, so I'm very hesitant to build long-term, resilient systems around it. Certs expire, taking whole websites down with them. Ugg...

The main reason I installed a cert at all on here is to protect your forum passwords.

Anyway, the other issue is that the client code is NEVER gonna support https (it currently rolls its own http calls, which are easy). That would require shipping with updated root cert bundles and a whole bunch of other heartache. So, the client is sending the game server stuff in the clear, which means it needs a replay-proof protocol anyway (challenge response).

jasonrohrer · 2019-03-26 19:41:52

Actually, I just checked my cert, and it seems like it can be installed on lots of servers....

Still, the more places it is installed, the more places I have to remember... and the more things will break when the cert expires.

It's actually overkill for protecting forum passwords. It's not that hard for forum software to do replay-proof challenge-response logins, with a bit of client-side javascript. Of course, to set the password in the first place, it needs to be sent from client to server. Though maybe the client could be trusted with computing the original salted hash too, so at least the raw password never goes on the wire.

Jk Howling · 2019-03-26 20:11:23

I presume this is unfinished since right now I quite literally cannot see my family tree data whatsoever. Not via email, as you blocked that. Not via clicking the "Family Tree" button from the client. The family tree page on the download link works, but that's the only thing that works atm.

jasonrohrer · 2019-03-26 20:14:10

Hmm... should be working... let me see what's going on.

jasonrohrer · 2019-03-26 20:30:46

It's working for me from the real client, both on and off steam.

Can you test it again on your end?

Did it ever work for you from inside the client? Like, before I changed this?

Is there any chance the capitalization of your email address in your client is off from the one you have registered?

I just tried it with your email address from inside the client, and I can see you lives.

Chard · 2019-03-26 21:14:27

jasonrohrer wrote:

Actually, I just checked my cert, and it seems like it can be installed on lots of servers....
Still, the more places it is installed, the more places I have to remember... and the more things will break when the cert expires.
It's actually overkill for protecting forum passwords. It's not that hard for forum software to do replay-proof challenge-response logins, with a bit of client-side javascript. Of course, to set the password in the first place, it needs to be sent from client to server. Though maybe the client could be trusted with computing the original salted hash too, so at least the raw password never goes on the wire.

Jason, I recommend using letsencrypt/certbot. I literally just grab a new certificate for every single domain and certbot automatically keeps them up to date. In fact I think the most recent Debian/Ubuntu packages include that cronjob themselves but the renew certificates command is just "certbot renew", not a hard cronjob to add. Plus its free, which is cool.

jasonrohrer · 2019-03-26 21:17:10

letsencrypt would be one more thing to figure out....

Chard · 2019-03-26 21:17:49

Whatever wrote:

Chard wrote:
The convention with log files is that they should be consistent and unambiguous and human readable. Programs don’t care about the extra data and humans do so there’s no sense not having it, it just makes the log file more useful to humans and no less useful to programs.
Regarding the extra file size. Firstly it will compress well which is good to know. Secondly we’re talking about approx 10% file size. Personally I’d wait for it to look like an issue before compromising the utility of the logs. Profile first etc.
Ok, i see, i just wanted to ask.
I also dont want that jason changes it now, since my scripts would stop working.

Likewise!

Chard · 2019-03-26 21:23:46

jasonrohrer wrote:

letsencrypt would be one more thing to figure out....

I just mean that its made my life a lot easier personally and professionally. I think it might be an area where a small amount of learning could remove a good amount of hassle. I certainly consider it safe for production use. Equally of course if what you have works then no sense changing it up.

jasonrohrer · 2019-03-26 21:45:09

JK: not sure if this was your problem, but I added some lower-case-forces client-side for the email addresses. That will ship later this week.

Jk Howling · 2019-03-26 21:51:40

jasonrohrer wrote:

It's working for me from the real client, both on and off steam.
Can you test it again on your end?
Did it ever work for you from inside the client? Like, before I changed this?
Is there any chance the capitalization of your email address in your client is off from the one you have registered?
I just tried it with your email address from inside the client, and I can see you lives.

It's never worked for me from the client- tbh I didn't even know it was supposed to work that way. I just thought it was a quick link to the front page of the family trees.

I figured out why it wasn't working though!

Fsr there was an extra @ symbol at the start of my email. It must've been there for literal months and I never noticed... wow. It doesn't help that the email box is too small for my email address, so it just shows me the end of it, not the beginning. It's always logged in just fine, and I never knew about the Family Tree button linking directly to your own lineages, so I never questioned if there was something wrong.

I must've accidentally hit the "enter @ symbol" button months ago without knowing... damn.

Edit: Forgot to clarify, it does work fine now. Sorry for the trouble and my own oversights lol.

Last edited by Jk Howling (2019-03-26 21:54:27)

Léonard · 2019-03-27 17:55:51

Jason, I don't know if you've read my reply to you in the other thread but I know it's this thread that made you move this data in the first place.
However as far as I can tell from the changes nothing was done to prevent the fact that people can track you NSA style given your hash.

It doesn't really matter if you can or not use the browser to check your lives, you can still do it in the logs.
As soon as someone knows your hash (which can easily be achieved if you share even a single one of your lives) they can know all your other lives.

I hadn't read the data logs raw by myself before, but from what I can tell even births are written, meaning someone could potentially find out who you are in the game in real time depending on if the name is written in the logs at death or birth.
And even if the name is written at death only, it isn't hard to imagine certain people continuously using their tokens to curse a specific individual simply because they dislike him.
The token cooldown might prevent them from sending said individual to DT but their lifetime score sure as hell would increase. Continuously in fact.
Even without knowing the individual specifically, someone could still work out a stranger's hash if they shared a lineage with them (since the logs state motherhood link) and use that to continuously curse a stranger forever simply because they did something someone personally disliked.

I always thought your game purposefully relied on the element of anonymity.
But as long as you keep logging lives using the exact same hash to identify someone for the totality of their lives anonymity can be breached.
And if anonymity is breached in a game that relies on it, things can go wrong as I described above.

People can find out who you are, what you did and even make up stats about you and share those.
Do you intend to fix this issue?
I don't really see this problem going away as long as the hash is the exact same for each individual's lives.
Even if you intend to let people make up global stats and personal stats for themselves there should still be solutions which allows that while keeping the hashes different per lives.
I even suggested a solution to you in the other thread.

I was also not aware that there were curse logs which as far as I can tell use the exact same hash.
So people can find out your current curse score and how often you get cursed.
It even shows the "curser's" hash meaning a griefer could work out who exactly cursed them and perhaps harass them (perhaps even outside of OHOL).
Though I can see why you would want those to stay consistant per account (as curse score is fundamentally an account-tied thing and not tied to single lives).
Perhaps simply making a separate per-account hash for the curse logs would solve this.

Whatever · 2019-03-27 18:20:11

Léonard wrote:

I hadn't read the data logs raw by myself before, but from what I can tell even births are written, meaning someone could potentially find out who you are in the game in real time depending on if the name is written in the logs at death or birth.
And even if the name is written at death only, it isn't hard to imagine certain people continuously using their tokens to curse a specific individual simply because they dislike him.

The logs are only updated once a day, so you cannot find out who you are playing with while you are playing.
I dont know how curses work, if you play with someone, curse him and remember his name can you then curse the same person with the same name again in one hour?

I think you are seeing this all to extrem. There are many games that show your account name in game. (for example minecraft)
In ohol your name changes every life, that is very unusual.
Only because you know a players hash doesnt mean you know anything about this person outside of ohol.
Even if you know someones hash here in the forums or on discord they could just easily change their name.
Knowing their hash only gives you their stats, which doesnt really matter in my opinion.

Léonard · 2019-03-27 19:01:46

Whatever wrote:

The logs are only updated once a day, so you cannot find out who you are playing with while you are playing.

Unless you're playing at the same time the logs get updated I guess.
If the point was really to prevent gathering realtime information about who is who ingame then the logs would be lagged behind by a day rather than written once a day.

Whatever wrote:

I dont know how curses work, if you play with someone, curse him and remember his name can you then curse the same person with the same name again in one hour?

Yes, you can curse dead people.
But I suppose the update rate does make this unviable.
Still, people can find out who cursed who specifically and I still dislike that.

Whatever wrote:

I think you are seeing this all to extrem. There are many games that show your account name in game. (for example minecraft)
In ohol your name changes every life, that is very unusual.

This doesn't really add anything to the discussion at hand.
Of course minecraft has no concept of anonymity.
And people can be harassed both inside and outside the game by griefers because of that.
If we can avoid that in OHOL then it is my opinion that we should.
I don't want to see some stupid griefer that I cursed a day ago start PMing me on discord or on the forums.

Whatever wrote:

Only because you know a players hash doesnt mean you know anything about this person outside of ohol.

Unless, which was the argument made in my post, this person has shared information about at least ONE life.
If someone makes a link between one of your lives and your identity, then they can make the link for every single one of your lives without needing your permission and THAT's what I dislike.

This is kind of a hypocrite argument to make from your position by the way, considering you used this very fact to make stats about a bunch of people without needing their permission.

I mean don't get me wrong, I like the tools you came up with and I am thankful for them.
But I would much rather have my stats be private than have you dig through my posting history to find one of my lineages and make up stats about me without me even knowing.

Whatever wrote:

Even if you know someones hash here in the forums or on discord they could just easily change their name.
Knowing their hash only gives you their stats, which doesnt really matter in my opinion.

So now this is another kind of argument to make.
You're basically saying "if you get harassed, simply change your name online or ignore it".
Which obviously goes without saying.
The thing is, again, if we can avoid having to deal with this in the first place, then we absolutely should!

Whatever · 2019-03-27 20:06:10

Léonard, when a player gets born it will be logged, with an id (this id is server specific) and a salted hash.
Now lets say jason would remove this hash but keep the id and then give all players their ids.
So only the player would know what ids he has. This means only the player could get his own personal stats because he needs his id list to do so.

Then it would be no longer possible to see how many people played on a certain day. You could see how many lives were lived but not to how many players they belong.
You can no longer create stats for the average player, because you dont know what lives belong to whom.
These are all anonymous stats that have nothing to do with a specific player but you need player ids (unique hashes for each player) in order to make them.

So what i want to say is that this change would remove opportunities / make the log data less valuable.
It would mean that jason has to invest work to make something worse. (therefore more anonymity)
In the same time he could work on new features or improve other things. He already wasted some time to make the lineage email thing more secret.

All of this for a problem that might not even be a problem.
You say that people might pm you on the discord or on the forums because you cursed them, but this has never happened before.
The logs already exist since half a year and no one ever got harassed because of it, as far as i know.
Maybe this will change now since i made some tools? But i dont think so, i believe only very few people will ever use those tools and probably not to harass people with it.
If someone curses you in game it is very likely that you will never find out who his hash belongs to.
Or if you curse someone it is also very unlikely that that person even knows about how the logs work and its even more unlikely that he is crazy enough to try and find out who the hashes that cursed him belong to in order to harass these people.
But i can understand that you want it to be impossible to do and unlikely is not good enough. Than dont share your lineage links. Dont tell people who you are in what lives. You want to play anonym? So play anonym. This log system is not preventing you from being anonym.
Most people dont care so much about playing anonym, we even see people here on the forums that are proud about griefing and post about it.
They openly admit to all the bad things they have done and make themselves vulnerable to harassment.

Last edited by Whatever (2019-03-27 20:07:20)

Léonard · 2019-03-27 22:04:37

Whatever wrote:

Léonard, when a player gets born it will be logged, with an id (this id is server specific) and a salted hash.

I know. Did you even bother to read my post in your own thread?

Whatever wrote:

Then it would be no longer possible to see how many people played on a certain day. You could see how many lives were lived but not to how many players they belong.

We're just talking about public logs here.
Jason might very well be keeping track of this privately and share those type of numbers with you.
You don't need to share the entire data of everyone to make up global stats..

Whatever wrote:

You can no longer create stats for the average player, because you dont know what lives belong to whom.
These are all anonymous stats that have nothing to do with a specific player but you need player ids (unique hashes for each player) in order to make them.

So your argument is that you loose the possibility to make up stats about random people who we have no idea who they might be and what type of play they have?
That doesn't sound very useful.
You might as well make global stats or take random sets of data to make up stats at this point..

I will happily trade this ability for proper anonymity.

Besides, like I said above, you could still keep the public logs anonymous but have Jason's lineage server calculate stats and report them to you without sharing the complete data set of individuals or identifying anyone.

Whatever wrote:

It would mean that jason has to invest work to make something worse. (therefore more anonymity)
In the same time he could work on new features or improve other things. He already wasted some time to make the lineage email thing more secret.

This is a terrible argument and I'm honestly tired of seeing it everywhere.
Yes, Jason could use the time he'd take to implement my suggestion to cater to your preferences instead.
It just boils down to "in my opinion, x is better than y" so of course "you're wasting your time by doing x instead y" is part of that.
You're not adding anything to the discussion by saying that. It's just your opinion.

Whatever wrote:

this has never happened before.

That's also a poor argument to make.
There is nothing wrong with preventing bad things from happening.

Did anyone ever steal your car IRL? No? Then why do you bother putting it in a garage and locking it?

Whatever wrote:

Than dont share your lineage links. Dont tell people who you are in what lives. You want to play anonym? So play anonym. This log system is not preventing you from being anonym.

People can both be concerned about their anonymity/data and at the same time enjoy sharing stories with others.
Compromising this for the sake of you making a bunch of random stats about a bunch of random people is a very poor choice if you ask me.
What's wrong with saying "I don't want people to be able to access all my data simply because I shared a single life".
I still remember this post which shows people were already concerned with this sort of stuff.
The argument also applies with sharing a single lineage link. Why should that allow you to make a bunch of stats about me or watch my playtimes/whatever?

Whatever wrote:

Most people dont care so much about playing anonym

I don't think I'm the only one here who thinks you making up stats about random people without asking them seems a bit iffy.
Here's an example post, just for the sake of backing what I say.

fragilityh14 · 2019-03-27 23:27:03

wait was the in text "Family trees" always supposed to let you browse within the app? I also thought it was just a link to the website, cause it just went to a web browser for me before

Anyway, this is all kind of tedious and annoying, as I very often later check lines on my phone and just searching my email address is quite convenient, but I suppose all the more I have to do is bookmark the download link.

It seems like kind of a problem in search of a solution, as it seems very unlikely that someone is going to go to the effort to find the email address of a character and later harass the person, and at that beyond the extent which can be easily fixed by blocking.

still, i suppose if you think it's a problem better safe than sorry. Though if I had a friend who played i potentially would want to be able to look at his or her lives.

Also if you're messing around with lineage server I'd REALLY like to be able to see # of descendants and/or generations. It's soooooooooo tedious browsing around to figure out which family lines survived. Either would serve the same basic purpose (knowing which family lines to look at) but I imagine by the time you do one doing the other is no problem.

jasonrohrer · 2019-03-27 23:55:48

Well, the problem occurred for real in that other thread where they were cross-referencing forum members with various stats. Stats that are supposed to be private.

However, I realize now that this was NOT based on search for email addresses, but instead based on cross-referencing "my life" family tree link posts made in the forums. So at least those people wanted to share.

I do worry that someone who does not want to share at all could have their private data leaked. Maybe they play more than they want to admit, or play during work, or whatever. What if their boss looks them up by email address? If people don't want to share, they should be anonymous.

So, with the new change, that will be impossible.

There will still be a cross-reference possible if someone posts a family tree link. I can't really see a way around that.

Yes, Léonard, a public/private key thing is possible, making the stats for a given play only interpretable by the player who holds the matching key. But that cuts off a bunch of interesting data analysis opportunities. Having a sequence of lives connected together in the logs is important. Like, after being murdered, how likely is a player to quit for the day? Having unique, repeating hashes in the logs allows us to answer those questions.

(And by the way, I have full, un-hashed logs on my end, complete with email addresses, so I can answer those questions myself.... I'm talking about other people being able to run deep analyses.)

I also understand that some people might have shared family tree links before the log data was posted, so they may have leaked in the past without realizing it. Not sure what to do about that.

One Hour One Life Forums

#1 2019-03-25 21:08:26

Public data available again (separate server)

#2 2019-03-25 21:33:36

Re: Public data available again (separate server)

#3 2019-03-26 02:21:29

Re: Public data available again (separate server)

#4 2019-03-26 05:30:11

Re: Public data available again (separate server)

#5 2019-03-26 05:47:26

Re: Public data available again (separate server)

#6 2019-03-26 07:14:41

Re: Public data available again (separate server)

#7 2019-03-26 07:34:00

Re: Public data available again (separate server)

#8 2019-03-26 19:37:48

Re: Public data available again (separate server)

#9 2019-03-26 19:41:52

Re: Public data available again (separate server)

#10 2019-03-26 20:11:23

Re: Public data available again (separate server)

#11 2019-03-26 20:14:10

Re: Public data available again (separate server)

#12 2019-03-26 20:30:46

Re: Public data available again (separate server)

#13 2019-03-26 21:14:27

Re: Public data available again (separate server)

#14 2019-03-26 21:17:10

Re: Public data available again (separate server)

#15 2019-03-26 21:17:49

Re: Public data available again (separate server)

#16 2019-03-26 21:23:46

Re: Public data available again (separate server)

#17 2019-03-26 21:45:09

Re: Public data available again (separate server)

#18 2019-03-26 21:51:40

Re: Public data available again (separate server)

#19 2019-03-27 17:55:51

Re: Public data available again (separate server)

#20 2019-03-27 18:20:11

Re: Public data available again (separate server)

#21 2019-03-27 19:01:46

Re: Public data available again (separate server)

#22 2019-03-27 20:06:10

Re: Public data available again (separate server)

#23 2019-03-27 22:04:37

Re: Public data available again (separate server)

#24 2019-03-27 23:27:03

Re: Public data available again (separate server)

#25 2019-03-27 23:55:48

Re: Public data available again (separate server)

Board footer