Forum - View topic
One and only release scraper issues thread


Goto page Previous  1, 2, 3  Next

Anime News Network Forum Index -> Site-related -> Encyclopedia
View previous topic :: View next topic  
Author Message
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7261
Location: Wales
PostPosted: Thu Jul 20, 2017 9:36 am Reply with quote
BigOnAnime wrote:
Also mysteriously many entries now have missing covers when they had them before.
release#31461
release#33792
release#31770 -Volume number needs to be changed to a "2" BTW.

The editor-only option to "Pick image from stores" still worked for the first one. I don't know how the automatic adding of pictures works though.

---

I just manually added one print manga, and the system automatically populated the B&N and Amazon links. The B&N link says "could not extract price from page" and the Amazon link says "Could not find page" (the raw data was retrieved via the API, but it appears that if it can't confirm the page itself is accessible the wizard doesn't work). For the record, the system does not auto-populate the Amazon link for eBooks, only the B&N link.

---

For an example of the B&N stock/price issue see

Scum's Wish 2 GN - it is currently in stock at $13.17, but our system is showing the price as $7.73 and reporting

Quote:
out of stock
flag:!out_of_stock = true
flag:free_shipping = "25"
flag:out_of_stock = true
flag:stock_status = "false"


Scum's Wish 2 eBook - it is obviously in stock and at the list price of $6.99 but our system has the prices as "N/A" and is reporting

Quote:
out of stock
flag:!out_of_stock = true
flag:out_of_stock = true
flag:stock_status = "false"
could not extract price from page


OTOH I have checked a few anime releases and there are no errors, and the current price in the database matches that on the B&N site.

---

Updated top post.
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7261
Location: Wales
PostPosted: Wed Jul 26, 2017 1:06 pm Reply with quote
Struck out the Rakuten error. Forgot it was even on there (and with Rakuten being a marketplace they don't reliably list anything, and don't list pre-orders at all).
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Spoofer



Joined: 03 Aug 2003
Posts: 260
Location: NY
PostPosted: Tue Aug 01, 2017 1:38 pm Reply with quote
I've been away from the anime scene for the past half year or so, and I was inspired in the last few days of the Right Stuf birthday sale to pick up a ton of releases that have come out in that time. I've always used ANN's Encyclopedia to check to see what the latest releases of any given show are, but today I'm noticing just how many releases (in the hundreds) have not been listed under their respective encyclopedia titles from the past 6 months (which caused me to miss picking them up in the sale, but that's of minor concern). Is this related to the scraper issue (some releases have been listed, so it doesn't seem to be an issue of all or nothing), or has ANN simply given up keeping all releases up to date? I notice in the weekly release articles, links are now to encyclopedia title pages rather than to release listings themselves. Did releases always rely upon the community to update them, and now we've lost some of the most frequently contributing buyers, or otherwise what is the reason for this?

If ANN can no longer be counted on to keep track of what releases of any given title exist, there aren't many other anime-specific alternatives. =( I suppose I could revert to FilmAF, but always preferred ANN's system.
Back to top
View user's profile Send private message My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7261
Location: Wales
PostPosted: Tue Aug 01, 2017 6:25 pm Reply with quote
Users can submit releases and, problems like the above notwithstanding, if anything is missing it means it hasn't been submitted. I've personally been concentrating more on ebooks because they can be problematic and are often simply ignored when people are adding the print editions. That alone had led to hours crosschecking with Hachette's database...
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Spoofer



Joined: 03 Aug 2003
Posts: 260
Location: NY
PostPosted: Tue Aug 01, 2017 7:24 pm Reply with quote
Why doesn't whoever it is that does the weekly NA release news articles input the submissions? It only takes a minute or so per listing if they just stick to RS's scraper (when it's working), which would certainly be better than nothing at all. Shouldn't take more than half an hour per week I would think, maybe an hour at most, at least for the amount that are included in those weekly listings, unless I'm severely underestimating the amount of releases or remembering incorrectly how long a submission takes. I mean not that I have any idea how overworked all you guys are, but, clearly relying on user submissions alone isn't cutting it if I notice, again, literally hundreds of releases from the past 6 months alone that are missing.

For the longest time, I had 95~100% certainty that if a release existed, it was listed on ANN. But throughout the past couple of months at least, that number seems to be more like 10%. It's real disappointing to see, is all. As I said, the weekly release listings have been linking to titles rather than releases, and it seems it's been that way for quite a while, but I swear at one point they linked to actual releases.

I myself have far less free time than I used to, so I know it's easy to "complain" without offering much help in return, but I'm nowhere near as tuned in and invested in anime as I used to be, and I'm not on ANN's payroll. I just wish that whatever seemed to be working for ANN before, that kept the release listings as accurate as could be expected, and up to date, would make a return, is all. =/
Back to top
View user's profile Send private message My Anime My Manga
horseradish



Joined: 27 Oct 2015
Posts: 109
Location: California, USA
PostPosted: Tue Aug 01, 2017 9:31 pm Reply with quote
When adding anime#1458 -
Amazon: "could not fetch page"
Rightstuf : "This doesn't appear to be a valid page; could not find urlcode,refcode"
B&N worked. No picture or description.

When adding anime#17908 - (accidentally messed up title to display (BD+DVD) (DVD) Embarassed)
Amazon: "could not fetch page"
Rightstuf : "This doesn't appear to be a valid page; could not find urlcode,refcode"
DVD Empire: "This doesn't appear to be a valid page; could not find urlcode,refcode" (despite listed UPC Code 704400093654)
However when I added the B&N link, it worked and then auto generated the Amazon link too. There is a picture, but no description.

Barnes & Noble seems to be the only one working and sometimes in conjunction with the Amazon. I'll try to help fill in more gaps (and not mess up releases) in my spare time.
Back to top
View user's profile Send private message My Anime My Manga
BigOnAnime



Joined: 01 Jul 2010
Posts: 800
Location: Minnesota, USA
PostPosted: Wed Aug 02, 2017 2:00 am Reply with quote
Spoofer wrote:
Did releases always rely upon the community to update them, and now we've lost some of the most frequently contributing buyers, or otherwise what is the reason for this?

If ANN can no longer be counted on to keep track of what releases of any given title exist, there aren't many other anime-specific alternatives. =( I suppose I could revert to FilmAF, but always preferred ANN's system.
They've pretty much become more and more dependent on the community to add that sort of stuff, same goes for the Encyclopedia. In fact, I was like the main person adding most of the non-FUNi releases for like a year or so (there were times where I would submit like 30+ releases at once), though I got a bit lazy... (I'm behind on so much stuff and I lose motivation to catch-up) Though these release scraper problems are now going to make it much harder to add stuff. I can only add releases using the Barnes & Noble links now, and this leads to releases with missing retailer links and descriptions. This also means I can't add stuff that is available only on RightStuf (Aniplex USA, Ponycan USA, NISA). Who knows when or if these issues will get fixed as Dan42 has gone missing, he last made a post an entire month ago.

Also funny that you mention FilmAF, they're pretty much entirely community-driven, and they were also falling behind. In fact, there are a bunch of releases from the past few years that they're missing, a few of which I own. It takes a bit longer to add releases there than here unfortunately.
Back to top
View user's profile Send private message Visit poster's website My Anime My Manga
horseradish



Joined: 27 Oct 2015
Posts: 109
Location: California, USA
PostPosted: Wed Aug 02, 2017 4:59 pm Reply with quote
Barnes & Noble links seem to create an issue with combo packs. Combo packs (BD+DVD) are recognized as DVD only through the scraper.

The B&N url for 2017 Princess Mononoke BD+DVD from GKIDS is
https://www.barnesandnoble.com/w/?ean=0826663181548

Should I submit them anyway and list all of them for correction in the "Release corrections" thread, or wait for the issue to be fixed?

Edit: Similar problem with manga. Ebooks are only recognized as Graphic Novels. B&N link for GNs display as "out of stock".
Back to top
View user's profile Send private message My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7261
Location: Wales
PostPosted: Thu Aug 03, 2017 1:26 pm Reply with quote
You might as well submit them. Ebooks not being recognised is a longstanding issue (already on the list) and combo packs regularly have to be corrected also (although I only have it down for TRSI). Not retrieving stock status for B&N is also already on the list.
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7261
Location: Wales
PostPosted: Thu Aug 17, 2017 12:38 pm Reply with quote
B&N may now be an issue; it is giving "This doesn't appear to be a valid page; could not find urlcode,refcode" for the following ebook: https://www.barnesandnoble.com/w/strike-the-blood-vol-9-gakuto-mikumo/1126566423?ean=9780316442046
I'll confirm with other examples before adding to the list.

--edit--

Tried two ebooks and one print manga and none worked, so it is not looking good. Has anyone successfully added any releases since the domain change?
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7261
Location: Wales
PostPosted: Thu Aug 17, 2017 12:56 pm Reply with quote
Err.. looking at an existing, fully populated disc release it appears that the only store that is actually working at the moment is Rakuten; the rest are all returning "no product found!?" or "page missing/gone". Rakuten is also the least useful since it is now just a marketplace.

--edit--

It is still trying to work though; after I manually added release#34643 the system automatically populated the B&N link but with the following trace:

no product found!?
Extract from url: http://www.barnesandnoble.com/w/?ean=9780316346092
flag:!out_of_stock = true
flag:out_of_stock = true
flag:stock_status = "false"
could not extract urlcode from page
could not extract refcode from page
Does not appear to be a product page
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Double Mangekyo



Joined: 17 Aug 2011
Posts: 103
PostPosted: Thu Aug 17, 2017 1:14 pm Reply with quote
Shiroi Hane wrote:
B&N may now be an issue; it is giving "This doesn't appear to be a valid page; could not find urlcode,refcode"

Tried two ebooks and one print manga and none worked, so it is not looking good. Has anyone successfully added any releases since the domain change?

I added SAO 13, Umineko 17 and Juni Taisen (the latter just yesterday) after the domain change using B&N.

That message was something I got only when I tried to add ebooks before.

... I just tried adding the 6th print volumes of Asterisk War and Baccano!. I'm getting the same message.
Back to top
View user's profile Send private message My Anime My Manga
Shiroi Hane
Encyclopedia Editor


Joined: 25 Oct 2003
Posts: 7261
Location: Wales
PostPosted: Thu Aug 17, 2017 1:49 pm Reply with quote
I just started (again) going through Yen's upcoming releases making sure we had them (last time I did this only got through a page or two..) but so far I've only done the Corpse Princess ebooks (because I went to add vol. 16 and found we had none of them, despite the first vol coming out 2 years ago...)
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number My Anime My Manga
Double Mangekyo



Joined: 17 Aug 2011
Posts: 103
PostPosted: Thu Aug 17, 2017 2:02 pm Reply with quote
Shiroi Hane wrote:
ebooks (because I went to add vol. 16 and found we had none of them, despite the first vol coming out 2 years ago...)

Speaking of ebooks that are already released, methinks J-Novel needs some love.
Back to top
View user's profile Send private message My Anime My Manga
horseradish



Joined: 27 Oct 2015
Posts: 109
Location: California, USA
PostPosted: Thu Aug 17, 2017 5:08 pm Reply with quote
I have been adding releases from B&N almost every day this month, which worked until around 5 AM UTC today. That was around the time B&N product pages changed design. Also getting the following when adding https://www.barnesandnoble.com/w/?ean=9781626926875
"This doesn't appear to be a valid page; could not find urlcode,refcode"

Only DVDs, GNs, and sometimes Novels (when differentiated from manga versions of the same title) from B&N could be successfully added with the correct format. All other formats needed corrections. Now it seems to be broken.

Edit: When trying to add DVD release for anime#17448
https://www.barnesandnoble.com/w/?ean=0816726020013
"This doesn't appear to be a valid page; could not find urlcode,refcode"
Back to top
View user's profile Send private message My Anime My Manga
Display posts from previous:   
Reply to topic    Anime News Network Forum Index -> Site-related -> Encyclopedia All times are GMT - 5 Hours
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 


Powered by phpBB © 2001, 2005 phpBB Group