×
  • remind me tomorrow
  • remind me next week
  • never remind me
Subscribe to the ANN Newsletter • Wake up every Sunday to a curated list of ANN's most interesting posts of the week. read more

Forum - View topic
Release scraper problem




Anime News Network Forum Index -> Site-related -> Encyclopedia
View previous topic :: View next topic  
Author Message
BigOnAnime
Encyclopedia Editor


Joined: 01 Jul 2010
Posts: 1219
Location: Minnesota, USA
PostPosted: Thu Mar 17, 2016 7:30 am Reply with quote
I've brought this up in the thread for release corrections a few times, but I might as well bring it up here too to bring more awareness to the problem.

Whenever I and seemingly others try to add Blu-ray and DVD releases to the encyclopedia now, it fails to retrieve the plot summary and extras from the Rightstuf listing. This results in many entries I'd consider "blank" like these.
release#30996
release#30884
release#29990
release#29989

I've seen entries that unfortunately failed to list extras in the past (which is now rather frustrating for OOP releases Rightstuf doesn't list anymore), but they at least had the plot summary. I've been hesitating a bit to add some releases (I've been noticing the encyclopedia is unfortunately starting to get a little out of date with DVD's/BD's, particularly non-FUNi releases) because of this problem. I don't want to bring up every single entry with this problem in the release corrections thread.

Is there any way to fix this problem so normal users adding releases works as it should? One of my main uses of the encyclopedia is for the release entries, so this problem is really frustrating.
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
BigOnAnime
Encyclopedia Editor


Joined: 01 Jul 2010
Posts: 1219
Location: Minnesota, USA
PostPosted: Fri Apr 22, 2016 5:32 am Reply with quote
Bumping in hopes this gets more noticed. I brought this up a month ago on this same page. I see the issue still hasn't been fixed, and it's gotten even more frustrating as I'm seeing more and more releases suffer from the issue. Like I added this just a few days ago, and it just looks awful and is rather uninformative. "What is it about? What special features can I find on it? What episodes does it contain?". If the Rightstuf listing gets removed in the future, you won't know. It already frustrates me with OOP releases I'm interested in when I see the extras were never listed (and since most of those are no longer listed on Rightstuf, I can't use them as a resource either), so that's where I'm coming from here.

Also many in these posts still look the same as that Love Live! S2 release.
animenewsnetwork.com/bbs/phpBB2/viewtopic.php?p=4737649#4737649
animenewsnetwork.com/bbs/phpBB2/viewtopic.php?p=4745396#4745396
animenewsnetwork.com/bbs/phpBB2/viewtopic.php?p=4753245#4753245

Also strangely, with this (which needs to be changed to a combo pack, whoever added this for some reason thought it was a DVD when it's not, I have not yet tried to add the DVD-only release as it would only cause confusion atm), it's getting the plot summary from the Barnes & Noble listing instead of the Rightstuf one (which is way more complete).

If it helps much, IIRC when Rightstuf first made the site change, you had different tabs on the product page, however that's no longer the case, the reviews, extras, plot summary, everything is on the same page, there's no tabs you have to click on to see more info.
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
Dan42
Chief Encyclopedist


Joined: 02 Jan 2002
Posts: 3781
Location: Montreal
PostPosted: Mon Apr 25, 2016 11:37 am Reply with quote
ShanaFan852 wrote:
Bumping in hopes this gets more noticed.

Good thing you did, because I hadn't noticed. I just fixed the scraper to extract the description from Rightstuf. Existing description-less releases will get updated over the next 24 hours or so.
Back to top
View user's profile Send private message Visit poster's website AIM Address My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7578
Location: Wales
PostPosted: Tue Apr 26, 2016 7:07 am Reply with quote
Thank goodness. I was hoping for a system fix because that was a manual job that was only ever going to get longer.
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
BigOnAnime
Encyclopedia Editor


Joined: 01 Jul 2010
Posts: 1219
Location: Minnesota, USA
PostPosted: Tue Apr 26, 2016 8:15 am Reply with quote
Indeed, and I already brought up a bunch of examples, just about anything that wasn't a FUNi release was looking blank. Also glad to see anything added prior to the fix will get updated.

One of my main uses of the encyclopedia over the years has been the release info, so this was a rather huge deal to me. Thank you for fixing the problem.
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
BigOnAnime
Encyclopedia Editor


Joined: 01 Jul 2010
Posts: 1219
Location: Minnesota, USA
PostPosted: Sun Jun 05, 2016 10:11 am Reply with quote
Looks like AnimeCastle underwent a site redesign recently, so it looks like now all the AnimeCastle links in the database for releases are now broken. An example of what a new URL looks like from the site. Tried adding this link for that release, and I got an error to no surprise.
http://www.animecastle.com/brynhildr-in-the-darkness-blu-ray-complete-collection/

So that may need to get fixed, though then again, lately AnimeCastle seems to no longer be carrying that much for newer releases anymore.

Also as I noted recently in the releases correction thread...
BigOnAnime wrote:
[Edit 3]And the scraper picked Amazon again for some of the releases I just added for the description, yay. I think I've figured out why this happens now. The scraper looks at Amazon first if an Amazon link is present, if there's no "Editorial Review" to be found, it skips that and then goes over to Rightstuf to get the description. Looks like the scraper is going to need some additional fixing as Amazon descriptions usually aren't nearly as detailed, and it fails to get the extras, episodes, languages in there most of the time. Get it to look at Rightstuf first instead (as some items for whatever reason are sometimes not on Rightstuf, such as Belladonna of Sadness atm).

Also maybe worth noting, like Sentai releases never have those kinds of descriptions on Amazon (the "Editorial Review"), and with Softenni, whoever added them didn't add the Rightstuf links, but the very second I added them, it got the description from Rightstuf right away.
animenewsnetwork.com/bbs/phpBB2/viewtopic.php?p=4775420#4775420
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
Dan42
Chief Encyclopedist


Joined: 02 Jan 2002
Posts: 3781
Location: Montreal
PostPosted: Mon Jun 06, 2016 10:44 am Reply with quote
BigOnAnime wrote:
Looks like AnimeCastle underwent a site redesign recently, so it looks like now all the AnimeCastle links in the database for releases are now broken. An example of what a new URL looks like from the site. Tried adding this link for that release, and I got an error to no surprise.
http://www.animecastle.com/brynhildr-in-the-darkness-blu-ray-complete-collection/

This is too much of a mess, I'm just going to remove AnimeCastle.
Back to top
View user's profile Send private message Visit poster's website AIM Address My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7578
Location: Wales
PostPosted: Wed Jun 22, 2016 2:11 pm Reply with quote
It's been a while since I've searched Rakuten for a UPC and got a result. Wondering why, I browsed to the anime section, sorted by release date and was surprised to see stuff like Sailor Stars and The Daichis on the first page (Shirobako and what look like a couple of Japanese imports being the only actually recent releases there).
It looks like they have gone marketplace only (IIRC Rakuten UK did the same before closing entirely), with all the most recent stuff I've found being sold by "Importcds".
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
bglassbrook



Joined: 29 Aug 2006
Posts: 1243
Location: Gaithersburg, MD
PostPosted: Sat Jun 25, 2016 7:20 pm Reply with quote
I was rather taken aback when it actually found some of the manga I was adding this/last week. The more I think about it, there may only have been a marketplace listing at the time for the one I had to do manually. A shame, for a while they were actually a fairly reliable hit on super-advance pre-order titles.
Back to top
View user's profile Send private message My Anime My Manga
BigOnAnime
Encyclopedia Editor


Joined: 01 Jul 2010
Posts: 1219
Location: Minnesota, USA
PostPosted: Sun Jul 17, 2016 3:25 pm Reply with quote
Well the scraper is now having issues with Rightstuf again. Rightstuf recently had a minor site redesign and now here's how it's broken the scraper.

First off, it can no longer find the UPC so I'm unable to add stuff with just the Rightstuf link (a problem for NISA, Ponycan USA, and Aniplex USA releases). The UPC for stuff is now listed as SKU. Note how these match.
http://www.rightstufanime.com/Punch-Line-Blu-ray
https://images-na.ssl-images-amazon.com/images/I/8114WEyrPTL._SL1280_.jpg

Second, it's now unable to pick up the description. Not sure why given the only main change the descriptions have gone under is now when you have longer descriptions, you have to click "see more" to see the rest of it.

Lastly, the scraper seems unable to tell if an item is out of stock or OOP. Like for example this is OOP, but the scraper thinks Rightstuf still has it in-stock.
release#28820
http://www.rightstufanime.com/Persona-4-The-Animation-Complete-Collectors-Edition-BD-DVD-Hyb
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
BigOnAnime
Encyclopedia Editor


Joined: 01 Jul 2010
Posts: 1219
Location: Minnesota, USA
PostPosted: Mon Aug 08, 2016 5:13 am Reply with quote
Bump, this issue still hasn't been fixed after nearly a month.
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
BigOnAnime
Encyclopedia Editor


Joined: 01 Jul 2010
Posts: 1219
Location: Minnesota, USA
PostPosted: Tue Sep 20, 2016 2:32 am Reply with quote
Bumping again, two months later and the scraper is still having those same problems with RightStuf.
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
Display posts from previous:   
Reply to topic    Anime News Network Forum Index -> Site-related -> Encyclopedia All times are GMT - 5 Hours
Page 1 of 1

 


Powered by phpBB © 2001, 2005 phpBB Group