gScrape - Video Game Info Scraper

D

dustind900

Member
Supporter
RL Member
gScrape - Very Alpha Edition...

minimum .Net 4.5 required! Get it Here.

  • no error checking yet so it might crash (hasn't happened yet)
  • no App Busy notifications so just WAIT. depending on site speed info can be slow
  • only xml data can be saved for now. no images (yet)
  • theGamesDB is all (for now)

Getting Info from moby is hacky at best, but I am in talks with a very knowledgeable person with a much better plan of attack than my own.
I still have very little time to work on this, but I will continue because it's fun to learn new shit...

Download Here

Feedback please.!?
 
Last edited:
B

brolly

Administrator
Developer
Thanks. Was giving it a quick test, but got an error while loading the system list at startup:
XITz4ku.png


Get several of these, if I keep pressing No the system list will eventually show up, do you get this on your side as well?
Happens pretty much every time data is being read from moby.

Do I need to get the game info for each game one by one? Isn't there a way to read the game info for every single game for a system at once?
 
Last edited by a moderator:
D

dustind900

Member
Supporter
RL Member
Honestly Brolly I have not seen that error at all. And yes you have to get the info one by one. I had a feeling I would run into problems like these so I left a lot of features out including batch operations. It will be included in the final versions.

What version of windows are you running?
What version of IE do you have installed?
After the error is the app still fetching the information correctly?

I am running Windows 8 and IE11.


So I did a little digging and it turns out "COMSCORE" is , in short, an internet usage tracking service. Therefore if the application is still working correctly even after it displays the error, I would try putting a checkmark next to the "do not show this message again" and click no.
 
Last edited:
B

brolly

Administrator
Developer
I'm on XP, I don't use IE at all so I still have IE8 installed. I'm currently on an old laptop (15 years old) as I wait for my new machine. So you are using IE to scrap the info?

Yes everything seems to be working fine after that, I just get a bunch of these errors on each server call, it's not all COMSCORE related there are other error messages. I was going to click the checkbox, but wanted to get this info to you first in case you want to check what the problem might be.
 
D

dustind900

Member
Supporter
RL Member
I'm loading the page into a DOM document so technically yes I am using IE in a sense. Is it possible or are you willing to update to at least IE9? I'm almost positive this is a compatibility issue between some of the java libs on the page and IE8. There may very well be code on the page to detect older IE versions, but the way I'm loading the pages doesn't allow that code to execute causing errors.

I may have to add IE9+ as a requirement.

@sudo
That wouldn't be hard to add at all. Good suggestion.
Have you also tested this? Are you getting the same errors as brolly?
 
Last edited:
B

brolly

Administrator
Developer
I'll be able to test on another machine on Monday so I'll get back to you by then.

Btw do you guys need to press the thanks button to see this post's content as well? I'm seeing this a lot here lately so I wonder if this setting is enabled by default or something?
 
dragon57

dragon57

Member
Supporter
RL Member
The app seems to work well for me. I haven't seen any issues.

One thing that would save some time and effort would be to enable multi-select where it makes sense to do so. I would loved to be able to select all the games in the Game list box, then click on either the View Game or Get Game Info buttons, then let the utility iterate its little heart out. :)
 
D

dustind900

Member
Supporter
RL Member
There will be batch operations available in the future. But I'm not sure that I will allow it to do an entire game list. I cannot guarantee that the app will always fetch 100% of the information 100% of the time. This would leave users having to go back through manually to see if anything is missing and re-sync any games that are missing info. That and if you query moby's servers too often too fast ( especially the same page ) your IP will get blocked for about an hour or two possibly longer. Trust me I've found this out the hard way numerous times while testing.

A "nicer" UI is also in the works. I just kinda threw this one together in a hurry. It shows...
 
Last edited:
B

brolly

Administrator
Developer
Tested on a machine with IE10 and all worked fine, no error messages.
If you add batch processing, don't forget to also add a cancel button so you can cancel between scrapping or you will need to brute force close the application.

Honestly who cares about the UI? The output XML files are the important part :)
 
dragon57

dragon57

Member
Supporter
RL Member
dustind900 said:
There will be batch operations available in the future. But I'm not sure that I will allow it to do an entire game list. I cannot guarantee that the app will always fetch 100% of the information 100% of the time. This would leave users having to go back through manually to see if anything is missing and re-sync any games that are missing info. That and if you query moby's servers too often too fast ( especially the same page ) your IP will get blocked for about an hour or two possibly longer. Trust me I've found this out the hard way numerous times while testing.
Click to expand...

That is perfectly fine, and understandable. As long as the user knows what the constraints are, everything should work out great. As has been said, in the end, the xml is the most important thing.
 
Ninja2bseen

Ninja2bseen

Member
RL Member
So there seems to be a scraper that's working, cool but how do we utilize the info?


Sent from my iPhone using Tapatalk
 
Turranius

Turranius

Member
Supporter
RL Member
My understanding is that this is mostly a proof of concept right now.

I'd setup a database at your end and have the program first check that. If its not in there, check with mobygames and update your db with that info for the future.
That way, we can do a batch download from your db in the future and everyone that clicks on some game/games will help populate your db.
 
D

dustind900

Member
Supporter
RL Member
1. Next release will get rid of the "names.gS" and "names.gS.bak" files located in the Data folder. I am currently working on a small utility that will help with the update process.
2. New attribute "linkName" is being added to info xml files due to the removal of the "names.gS" file.
3. More info categories will be added to the xmls
4. App start up time is greatly reduced. Even the database update should be quicker.


Hopefully I will have this next release ready by Monday, but no promises. Sorry the releases are so slow, but between work and 4 kids I am one busy mofo.


Edit:
I just noticed they changed the some of the elements having to do with MobyScore. This is fixed and will be in the next release.
 
Last edited:
D

dustind900

Member
Supporter
RL Member
Ok so this next release is taking a lot longer than expected. But, on the bright side this delay brings many changes.

1. Changes to the output XML:
  • All node and attribute names will be converted to lowercase
  • "linkName" will be added as an attribute to the "<game/>" node
  • Publishers, Developers, and Release Dates will be listed by region
  • "region" will be added as an attribute to the above nodes children
  • "<subgenre/>" child nodes will now have a "name" and "value" attribute. This is because I found this game which has more than one subgenre category listed.
  • "<alternatetitles/>" child nodes
    was
    Code: 
    <title name="Japanese title" value="Virtual Horror: Norowarate Yakata"/>
    now is
    Code: 
    <title name="Virtual Horror: Norowarate Yakata" value="Japanese title"/>
  • "<mobyscore/>"
    was
    Code: 
    <MobyScore overall="4.2" totalVotes="3">
    <score name="Gameplay" value="4.3"/>
    <score name="Graphics" value="4.3"/>
    <score name="Personal Slant" value="4.0"/>
    <score name="Sound / Music" value="4.3"/>
<MobyScore/>
    now is
    Code: 
    <mobyscore>
    <score name="Gameplay" value="4.3"/>
    <score name="Graphics" value="4.3"/>
    <score name="overall" value="4.2"/>
    <score name="Personal Slant" value="4.0"/>
    <score name="Sound / Music" value="4.3"/>
    <score name="totalVotes" value="3"/>
</mobyscore>
  • same thing as above for "<mobyrank/>"

2. Batch operations have been added. 25 game limit, for now.
3. Speed improvements

Not sure when I will have this done. It's close, I just have to finish up the info viewer and fix the subgenre issue mentioned above.
 
Ninja2bseen

Ninja2bseen

Member
RL Member
On the hyperspin download website under tools other folder there is a money scraper made by rain. Not sure what we are trying to accomplish here but may be worth a look


Sent from my iPhone using Tapatalk
 
You must log in or register to reply here.
Top