Forum - View topic
Release Scraper Completely Broken Unable to Add Releases




Anime News Network Forum Index -> Site-related -> Bugs & Technical Questions
View previous topic :: View next topic  
Author Message
BigOnAnime



Joined: 01 Jul 2010
Posts: 801
Location: Minnesota, USA
PostPosted: Sun Sep 17, 2017 2:46 am Reply with quote
See the following thread for more info on the extent of what's wrong with the release scraper.
animenewsnetwork.com/bbs/phpBB2/viewtopic.php?t=3053645

Simply, the release scraper has been broken for months now for multiple sites (it was a problem before the domain hacking), and now it's to the point where it's impossible to add any releases except manually, which is not able to be done by regular users. Months worth of releases are now currently missing from the encyclopedia as a result of this problem. Is it possible for this to finally be fixed? I sent Dan42 a PM about this issue about a month ago and didn't get a response like I did the last time I brought this issue up via PM in hopes it would finally be addressed.
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
BigOnAnime



Joined: 01 Jul 2010
Posts: 801
Location: Minnesota, USA
PostPosted: Fri Oct 06, 2017 6:41 am Reply with quote
Bump, the issue is still here, and the number of releases missing keeps growing.
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
Calico



Joined: 05 Jan 2013
Posts: 210
PostPosted: Sun Oct 22, 2017 2:44 pm Reply with quote
Yeah, I just tried to add another release, and it's still broken. Are there any plans to fix this? And are there any reasons why the release scraper just broke like it did?
Back to top
View user's profile Send private message
BigOnAnime



Joined: 01 Jul 2010
Posts: 801
Location: Minnesota, USA
PostPosted: Wed Nov 29, 2017 12:20 am Reply with quote
Bump, it's still broken, and now there's only 3 releases left on the upcoming list.
animenewsnetwork.com/encyclopedia/releases.php?format=video
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
Dan42
Chief Encyclopedist


Joined: 02 Jan 2002
Posts: 3549
Location: Montreal
PostPosted: Fri Dec 01, 2017 7:23 pm Reply with quote
BigOnAnime, I want to thank you a lot for the bumps and your politeness and patience. I keep thinking I have to get to this soon and I keep getting buried in other urgent things, and at the same time I sort of remember the last time I looked at the issue and it was ugly, with some sites blocking our scraper and others being javascript-only. I'll try to get to this as soon as I can. Please slap me if I don't.
Back to top
View user's profile Send private message Visit poster's website AIM Address My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7367
Location: Wales
PostPosted: Sat Dec 02, 2017 1:25 pm Reply with quote
Amazon should be the easiest fix since it is, at least sometimes, pulling data from the API after a release is added manually. The wizard still fails 100% of the time because it can't reach or parse the actually page, but the API data alone ought to be sufficient to at least build a partial entry (my personal API access is broken because they've changed the way partner accounts work, so I can't recall what, if anything, is only available via the web).
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Spoofer



Joined: 03 Aug 2003
Posts: 266
Location: NY
PostPosted: Wed Feb 07, 2018 8:46 am Reply with quote
I finally updated my ANN Collection list with all my purchases throughout the past year. I had to custom add placeholders for almost half of them. Really sad to see this happen to ANN, as I've said before, I used to rely on the ANN encyclopedia's release accuracy when deciding which releases to purchase (and regularly used the affiliate links), as well as to catalog my collection. Obviously this problem just gets worse and worse over time as more and more titles will never be accounted for.

Who's even manually adding titles to begin with? It's admirable that someone's even trying, I suppose, but it's so hit or miss. It would be great if, even if the scraper itself doesn't get fixed any time soon, some alternate solution could be found for the community to chip in and start trying to undo the gap year's damage as far as maintaining a consistent release database goes. Perhaps granting some trustworthy community members with a passion and a vested interest in the release database the ability to manually add entries as well, or, IDK, something so that this aspect of ANN doesn't just crumble away completely.

The scraper could never account for releases from projects such as KickStarter or Indiegogo anyway, and I feel the encyclopedia should chronicle those releases as well so that people know what's out there, especially when many of those items are still available via Slacker Backers or company websites. Same deal with rare hybrid items like the Tekken: Blood Vengeance cgi film, where the DVD was easy enough to be added since it contained just the film, but the BD was released as "Tekken Hybrid" and featured both the film and a couple of games on the same disc and couldn't be added via the scraper.
Back to top
View user's profile Send private message My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7367
Location: Wales
PostPosted: Wed Feb 07, 2018 11:24 am Reply with quote
I don't know who else, if anyone, is adding releases manually. I do most of it between calls in work, which means Funimation releases are difficult since the best source, their web store, is inaccessible from the UK without using either a VPN or fiddling around with Google Cache. Prior to the scraper going down I concentrated on releases that couldn't be added with the scraper like TRSI exclusives, although that was less of a problem after their site update. I also moved into ebooks since they were commonly neglected by users when adding the physical books. I also, obviously, concentrate on things that I own or that personally interest me, or just happen to be looking into at the time (I spent days adding Seven Seas releases after adding their latest new title announcements to the system). Yen Press is honestly the best because everything I need to build a release is right there on their business portal. It is also a full time job on its own with both the volume of releases, the fact that they have a habit of changing release dates without notice and the fact that you need to check back occasionally to see if they've added missing cover art, volume titles and descriptions etc.

If there's anything that can't be added with the scraper (when it is working) you can post to the encyclopedia forum and request it. If it is something you have, then providing a barcode (UPC, EAN or ISBN) at least is beneficial (and essential if it is something not sold by regular retailers). If something is Kickstarter-exclusive and doesn't have a barcode then it not possible to add with the current system, at least not without the direct intervention of someone higher up than me. While there is now a workaround for ebooks that don't have barcodes (yay, more work!), we also can't add BookWalker exclusives at present either.

If there is anything you want added as a priority, let me know, but try to keep the volume down or it will just become another endless task for me to put off till later...
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Spoofer



Joined: 03 Aug 2003
Posts: 266
Location: NY
PostPosted: Wed Feb 07, 2018 12:30 pm Reply with quote
No no, I mean it's not a personal issue for me in terms of wanting it merely so I can easily add items that I already own to a collection list. That would be pretty petty.

What my primary concern is, I'd just love for the database in general to once again become reliable, for me and everyone else. I regularly disappear from the fandom for months or years at a time, then I come back, and often go through my Seen or Want to See lists and click through various titles to see if any new releases have come out that I can pick up during sales. ANN's always delivered in this regard in terms of a high degree of accuracy of what's actually out there. And I spend equal time buying the latest and greatest on RightStuf, and trawling eBay for good deals on old limited edition collector's stuff I again know exist by browsing the release database. Or I can often use an encyclopedia page to compare releases, like for older ADV titles where some collections contain extras and some don't, etc. So it's always been a really handy database for me to base my buying habits on, and as I said I always used the referral links as much as possible, if that actually brought you guys any extra revenue.

But this was the first year that I missed out on ordering various stuff during sales because ANN hadn't had stuff listed and I had no idea new releases were out. For example, the Record of Lodoss War BD, which I remember being announced years back as on the horizon, but without seeing it listed under ANN's Encyclopedia when I bought a ton of stuff during a big RS sale this year, I never knew it was out to add it to my cart. etc.

I mean now I basically just cross-reference stuff I'm interested in with with RightStuf itself, but that only works with current releases. Or other sites like DVD Aficionado or DVD Profiler. I'd just love for ANN itself to maintain reliability out of principle, is all, or pride that you guys remain at the forefront of accuracy. I basically shun sites like MAL since its database is so piracy-oriented (as are the bulk of its users), so I always loved the contrast that ANN was always so reliable from a US retail standpoint.

And I know it's prolly a ton of work without the scraper working as it should, so yeah, obviously I don't expect a single person to do all of it. Sounds like a nightmare adding them all manually as well, so kudos for having done as much as you have.

TLDR, again, I just wish there was a way, any way, either via the scraper or some decision to allow the community to chip in to some degree once again, to get it back up and reliable again. I figure it's better to have barebones entries that at least list the title and general date of a release, even if the page it links to has very little manually-entered information at all, compared to having no record of a release at all. Which again, my earlier point was, of the stuff I'm interested in (which is probably more along the mainstream side of things), at least 50% seems to be missing from the database since this has been an issue, and other people like me who discover releases this way might likewise have no idea releases are available. But maybe I'm a rare individual who always relied on ANN for this, IDK, heh. Anime smile;;
Back to top
View user's profile Send private message My Anime My Manga
Display posts from previous:   
Reply to topic    Anime News Network Forum Index -> Site-related -> Bugs & Technical Questions All times are GMT - 5 Hours
Page 1 of 1

 


Powered by phpBB © 2001, 2005 phpBB Group
Loading next article...