Letter from the Encyclopedist

A long, long time ago I used to write a "Letter from the Encyclopedist" from time to time, to keep people informed about the latest developments in the Encyclopedia. To my great shame I must admit that the last one was over 4 years ago. I must apologize for the lack of formal updates, but it's not like the Encyclopedia has been standing still either, far from it. Data has been added. Occasional features and improvements have been added; sometimes announced in the forum, sometimes slipped in quietly. More data has been added. Editors have been busy validating information and addressing error reports. Even more data has been added. Flame wars and spam wars have broken out. And through it all, data has been added and added and added. A lot. In fact people from all over the world have contributed so much information that the Encyclopedia is now quite out of the reach of mortal minds to grasp its entirety. Or at least out of the reach of my mortal mind.

The ANN Encyclopedia was always meant to hold structured information and statistics rather than a mass of informative but unstructured text like a wiki. That means the content of each page would be fleshed out by adding information rather than editing it. This brings us to the one flaw I regret most in the Encyclopedia design, a fundamental problem eating away at its coherency.

The Troubled Years

I assumed that most users would be fundamentally "good" and interested in submitting valid information. That turned out to be true. I also assumed that the few errors that slipped through could be quickly cleaned up by a few administrators. That turned out to be a big miscalculation. Even well-intentioned people can easily commit mistakes, from honest misunderstandings about Encyclopedia guidelines to the propagation of misinformation found on other sites. And when there's so many people submitting so much info... even a small percentage of incorrect data can turn out to be an insurmountable mountain to our small team of Encyclopedia editors. And because I had only anticipated the necessity for a few administrators, the admin tools I created were like industrial power tools: heavy and unwieldy and dangerous. It was impossible to enlarge the Encyclopedia staff to a size that could swiftly handle the number of submissions.

The submission process was able to scale infinitely, but the editorial process was not.

So far we've trundled along thanks to the heroic — and I must underline HEROIC — efforts of the Encyclopedia staff, but the fact remains that the Encyclopedia has had issues of growth and accuracy because the people who can add data outnumber those who can fix it by orders of magnitude.

Today this will change. At least I hope so. Today we launch Encyclopedia Audits, a system to automate the systematic peer-review and peer-correction of Encyclopedia information.

We're starting small, in part to test the waters and in part because we are fixing a very ingrained structural flaw; it will take some time until I can modify all areas of the Encyclopedia to use this system.

So what are we starting with?

Genres

The ability to search the Encyclopedia by genre has been by far the most often requested feature in the past few years, but I've held off because I wanted to update the genre definitions to something more sensible before making that data searchable. The old genres have now been divided into a few general and well-recognized Genres (action, adventure, drama, etc) plus Themes for more specific details (mecha, ninja, samurai, etc). And of course to go with the updated genres & themes we now have a genre search.

Now, it would be pretty nice to make sure that the results of the genre search are actually accurate. For that purpose I hope you'll consider lending us a few minutes of your time to participate in audits. As of this writing there is a grand total of 17,231 genres & themes to validate, and that number is only likely to increase. Can you imagine the impossible burden this would be for the handful of active editors we have? Not to mention that sometimes it's ambiguous whether a title belongs to a given genre or not, and so it might be better to simply defer to the majority opinion. If enough people lend a hand I believe we can get all that data double-checked pretty fast. If you wanted to contribute but didn't know where to start, audits are dead easy. And to sweeten the deal, participation in audits is counted in the contributors page.

Audits

How do audits work exactly? The basic idea is that any piece of information can be put through a process to determine if it belongs in the Encyclopedia or not. Like a mini-trial, the information's authenticity and accuracy will be evaluated by a jury of randomly selected users. Why randomly? In order to prevent spammers from inviting all their friends to an audit and promoting their favorite Naruto fake episode title to truthhood status. And in a wider sense, sortition favors neutral participants and minimizes emotional stakes and conflicts of interest which lead to nastiness like edit wars on Wikipedia.

But what if you really want to participate in a particular audit because you really know the information is incorrect? You can participate indirectly. Reporting an error has the effect of opening an audit and recording that you are a "sponsor" for that audit. From that point, when you volunteer for a random audit, the priority of your sponsored audits will increase so that other users will participate in them (so it's not completely random). In other words, you can't participate directly in a particular audit but by volunteering you can cause someone else to participate. It's tit for tat!

I hope I have convinced you to help us with the audits; the Encyclopedia has grown so large that we really need all the help we can get in managing it. Validating genres & themes may not seem like much, but this is only the first step on a very long road. Eventually audits will allow us to peer-edit plot summaries, accept new titles, and enter a myriad of information that currently requires pre-verification by staff. This is more than a new feature; it's a completely new direction for the Encyclopedia. I'm really excited about it and I can't wait to see what the future brings.