Forum - View topicDid something cause a lot of genres and themes to need auditing?
|
|
| Author | Message | |||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Calathan
 SubscriberPosts: 9117 |
|
|||||||||||||||||||||||||||||
|
Usually when I click the link to audit genres and themes, it either has none for me or only a few. However, over the last couple days, it has had tons of them (I stopped volunteering before it ran out, but I did more than 70 of them over the last couple days). While some of the ones it had me audit seemed like bad entries that would legitimately come up on the audits, others seemed like well established genres or themes that I wouldn't expect it to ask me to confirm. Some of the ones it asked me to audit seemed really odd, such as asking if Azumanga Daioh and Aria the Animation are action or adventure (neither had either of those genres when it asked me to audit them, and they are obviously wrong for those anime). I'm kind of worried that a lot of genres/themes that have already been established as belonging or not belonging were somehow accidentally added back to the auditing queue. Or did something else happen to cause a lot of genres and themes to need auditing?
|
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
|
Dan42
Chief Encyclopedist
 Posts: 3817 Location: Montreal |
|
|||||||||||||||||||||||||||||
|
About 10 days ago I fixed some problems with the audit system and as a result a few closed audits were re-opened. As long as the audit tasks you get are for anime you saw and you never answered them before, then everything's working fine and it just means the system is able to find more suitable audits for you. There's no such thing as genres or themes that you shouldn't expect to audit. *Everything* can legitimately come up in audits since there's no way to tell a priori if a genre is obviously wrong or not.
|
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
|
Calathan
Subscriber Posts: 9117 |
|
|||||||||||||||||||||||||||||
|
Ok, thanks for the information.
When I said I wouldn't expect some things to come up, what I meant is that they were for obviously right genres/themes for well known anime, where I would have expected enough people to have already voted on them. My assumption is that it stops having people audit a given genre/theme for a given anime once enough people have confirmed that genre/theme is correct or incorrect for that anime. But it makes sense that even genres/themes that have been in place for a long time on well known anime would come up if the vote on those genres/themes was reopened. Anyway, everything it is having me audit is something I have watched, so it seems like it is working correctly. I guess it just has so many for me because I have watched a lot of anime. |
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
Devil Doll
 Posts: 656 Location: Germany |
|
|||||||||||||||||||||||||||||
|
Given that the users are offered quite a lot additional audits now I wonder whether the result of these audits could be made more accessible to the users.
I'm thinking along the lines of using something like different text attributes (bold, normal, italics) for genres/themes with high, average resp. low percentage of votes in favor for, thus visualizing the reliability of each keyword to the visitor of each anime page. Additionally, the list of keywords might be visibly tagged as "audited" or "unaudited", similarly to the different colors of "info" widgets depending on whether an error report was filed, or whether the logged-in user was the one who contributed this information. Each keyword offers a "source" and an "audit" link on the anime page. But the audit link is offered regardless whether this user has already audited this keyword, and performing the audit and then "volunteering for another audit" offers a link to continue auditing keywords for random other anime. What I was looking for (but didn't find) was a link "audit all keywords for this anime", offering a sequence of audits for keywords only for this anime, and only those which the logged-in user didn't audit already. What's more, the link "find similar anime based on genres & themes" leads to a page with a genre search, listing the results ordered by the total number of matching keywords of the anime from whose page this search was started (which I'll name "reference anime" from here on). This page might be a good place to show the exact voting behavior for each keyword of the reference anime (i. e. number of votes pro/contra each keyword), and as such the significance of each keyword for this anime. Based on this information, a GUI could be offered to allow the user to refine the "similarity search" for this anime, either by offering each keyword with a check-box (allowing the user to specify any subset of keywords of the reference anime) or offering some input form where the user could enter
So as an alternative approach, the genre search routine itself might take these audit votes into account more actively (and visibly). Currently the result of the "similarity search" is grouped into sections, each of which is titled by the number of matching keywords (and even listing these keyword for each anime found as search result), with these results being ordered by anime quality within each group. But a user interested in anime with the same keywords as the reference anime might actually be more interested in results that show the most similar anime, thus taking into account the validity of the keywords, i. e. the audit votes for each of these, rather than the quality of the anime. Now taking the votes for the keywords of the reference anime into account might be relatively simple - use them as weight for this keyword when calculating the number of matching keywords. But what about the votes of the keywords for the query results? For the best possible "similarity" these should be taken into account as well. Perhaps multiply the audit percentages of matching keywords to get their "combined weight"? If "romance" has an audit quota of 70% for both anime the combined weight would be 49%; if it has an audit quota of 90% for anime A but 50% for anime B the combined weight would be 45%. This might then require the search routine to collect a lot of data that it currently doesn't have to, and thus require some re-implementation of this routine, which might be beyond what Dan42 would be willing to invest time and effort for, unless given a significant feedback by the users. As such, for the time being this posting can only serve as a source of discussion for users of this potential feature-to-be:
But perhaps the most interesting discussion would be whether the requirement for a keyword to be set is too high. There are many keywords, mostly amongst the "themes", that are obviously matching the anime and even significant for the whole story yet not enough to qualify for "I would watch this show because of this theme alone". If I take this literally then I have to vote down about 75% of all genres and themes. Thus I wonder whether a tri-state logic would be more appropriate:
|
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
|
Dan42
Chief Encyclopedist
Posts: 3817 Location: Montreal |
|
|||||||||||||||||||||||||||||
|
As usual, Devil Doll, your detailed suggestions are interesting and constructive, but too verbose and difficult to decipher.
The thing I can tell you for sure is that there will never be an "audit all keywords for this anime". The fundamental principle behind audits is sortition, which requires that the auditors (the jury) be selected at random in order to ensure impartiality. Clicking the "audit" link for a specific genre/theme has a different function; it flags the keyword to indicate it requires an audit. Visualizing the reliability of each keyword seems doable but a little tricky because there are two factors to consider: 1) is the keyword audited, or not, or the audit is in progress? 2) what is the balance of votes in the audit? Take for example these 5 scenarios when auditing the genre "comedy":
If I were to implement it, I would make it go from gray for 50% acceptance to black for 100%. But in addition there needs to be some kind of way to show that the 100% of c) is vastly different from the 100% of a), and that the 52% of d) is vastly different from the 50% of e) Sorting the search results by genre similarity requires to solve the same problem. Beyond the "acceptance ratio" itself, the number of audits votes (or lack thereof) indicates how uncertain is the acceptance ratio. I can't really think of a sane way to display that. As for the tri-state logic idea, I like it but I can't imagine how I would work it into the existing system. |
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
|
Devil Doll
Posts: 656 Location: Germany |
|
|||||||||||||||||||||||||||||
|
Thanks for your illuminative response; I wasn't aware of that sortition principle of the audit mechanism until now.
I consider the audit state more relevant than the exact percentage. So for visualizing I would suggest
Unaudited keywords would always get 100% as nobody challenged their validity (so far), based on the assumption that the majority of all keywords will never be audited. The number of audit votes processed for a keyword could be displayed numerically as a tooltip on the keyword but this information wouldn't help that much anyway (unless we're told what "10 votes processed" actually means in term of getting a final result - your examples a) and b) showed keywords in "processed" state with different numbers of total votes, so perhaps a percentage of the required quorum would be the relevant information here - or how exactly is an audit considered "decided"?). Knowing that an audit is in progress might suffice in this case - this information already ranks the keyword lower than it were without the challenge. As for the tri-state logic issue: I'm aware that implementing this would break compatibility with everything ANN currently has. My point was that the description text for voting a keyword sets an exceptionally high requirement by demanding "watching for this keyword alone". I would rarely ever consider more than two keywords per anime being of this quality ("hikikomori" is a perfect keyword for NHK ni Youkoso but I wouldn't watch this show because of it alone); at the same time I would consider an average of eight keywords perfectly normal for correctly describing an anime and allowing for a reasonable matching with "similar" anime. This is only about telling the voters how those keywords (that in tri-state logic would get a "half weight") should be voted for; your current description text implies that you want a "no" vote for them, resulting in too few keywords being positively audited (and then remaining usable for a reasonable genre search match). So for your (assumed) purpose a less restrictive description text might be an improvement. (I'm thinking into the direction of "essential for the anime" or "omnipresent in the anime" etc.; your formulation "central to the anime" is quite close to what I would use while "watch only for the X" is too much.)
Last edited by Devil Doll on Wed Mar 28, 2012 12:26 am; edited 1 time in total |
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
|
Devil Doll
Posts: 656 Location: Germany |
|
|||||||||||||||||||||||||||||
|
One issue that I noticed while auditing several dozen entries: There are some keywords that I consider "unusable". For example, ANN has prominent keywords "school" (347 hits) and "romance" (2990 hits), both used in many anime, and as such helpful for genre match. Then again, Amagami SS has the keywords "Love/Romance" and "School Life/Relationships", both of which are used only for Amagami and match nothing else.
- - - - - Right now, I voted for an audit and did not check the check-box "and immediately volunteer for another audit". As a response, ANN showed me a page with my recent votes but claimed that I had "given up" on auditing "comedy" for XxxHOLiC the Movie: A Midsummer Night's Dream. Which is not true - I just had never been asked to vote for this one. Trying to get rid of this entry, I decided to vote for it as the very next operation. Surfing to its audit page, I was offered to vote but at the end of the page I read: "This assignment will expire in -2 days, -15 hours". Huh? "The genre auditor who leapt through time"? Something mysterious appears to be going on here. |
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
|
Calathan
Subscriber Posts: 9117 |
|
|||||||||||||||||||||||||||||
I would definitely vote against "love/romance" as a theme . . . romance is a genre and there is no need to have the same information in a theme, plus love seems to be the sort of non-visual theme that isn't supposed to be included. I personally would also vote against "school life/relationships", as it is redundant with other themes and is useless to have in the encyclopedia since it won't help with searching. For combining redundant themes, I know Dan42 combined a few that I pointed out in another thread. I don't think they are regularly cleaned up though (I notice that there are once again a lot of different variations on shounen-ai). I think it would be useful if someone could try to clean them up every once in a while, but on the other hand if an encyclopedia editor has time to work on things I would much rather someone work on the anime submission queue (which has basically not moved in more than a year, as far as I can tell). |
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
|
Devil Doll
Posts: 656 Location: Germany |
|
|||||||||||||||||||||||||||||
|
theme: love triangle (28 results)
theme: love polygon (4 results) theme: love rectangle (4 results) I remember that some other anime database has implemented a concept named "keyword alias", meaning that
This is basically the same idea as translating kanji task credits to roumaji task credits. - - - - - - - Genres at ANN have their own description text. I would like themes to have their own description text as well. This would allow the creator of a new theme to specify both the intended meaning of a theme and the necessary requirements for assigning it to an anime. It's difficult to vote whether an anime fits a theme when the meaning of the theme itself isn't precisely known. Example: theme "school" (346 hits).
|
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
 |
All times are GMT - 5 Hours |
|
|
|
Powered by phpBB © 2001, 2005 phpBB Group