Yacoby Posted August 12, 2008 Share Posted August 12, 2008 I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this: I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you. Link to comment Share on other sites More sharing options...
LHammonds Posted August 12, 2008 Share Posted August 12, 2008 I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this: I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you.Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds. The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID. You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus. RSS Feed: New Files TodayRSS Feed: File Updates Today LHammonds Link to comment Share on other sites More sharing options...
Yacoby Posted August 12, 2008 Author Share Posted August 12, 2008 Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds. The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID. You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus. RSS Feed: New Files TodayRSS Feed: File Updates Today LHammondsHeya The problem with using the RSS feeds is that it is updates today, and I don't think it is in the last 24 hrs, which would lead to this error:11:58 - Scan RSS11:59 - User uploads mod 00:00 - RSS reset mods upload missed Where as if I look at the previously uploaded entries once a day, I don't miss any mods Thanks for the suggestion though. Link to comment Share on other sites More sharing options...
LHammonds Posted August 13, 2008 Share Posted August 13, 2008 BEGIN TOTAL SPECULATION Maybe a query subscription service then? You call a page and pass the date as a parameter and it pulls all the mods uploaded that day in an XML format? Example: DayQuery.php?date=20081231 Returns and XML-formatted data set including the following data: File IDGameCategoryMod NameAuthorUploaderVersionSmall DescriptionDate UploadedDate Updated Maybe an additional query for further and updated information based on the info you already have collected (based on file ID) Example: ModQuery.php?fileid=8440 Returns the following data set: File IDGameCategoryMod NameAuthorUploaderVersionSmall DescriptionDate UploadedDate UpdatedTotal # of DownloadsTotal # of ViewsTotal # of CommentsTotal # of ImagesTotal # of FilesTotal # of RatingsAverage Rating I guess it would depend on how much this service would be worth for Dark0ne to invest into it. END TOTAL SPECULATION Link to comment Share on other sites More sharing options...
Michlo Posted August 13, 2008 Share Posted August 13, 2008 I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this: I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you.Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds. The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID. You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus. RSS Feed: New Files TodayRSS Feed: File Updates Today LHammonds Bloody hell, how did I miss that? Thanks, mate. :thanks: I'm now using the RSS feed as well. :) Cheers. And with that, I'm going home so no more posts from me tonight (I hear the sighs of relief). ;) Link to comment Share on other sites More sharing options...
Dark0ne Posted August 13, 2008 Share Posted August 13, 2008 I'm often skeptical of these things; I've had quite a few people ask if they can crawl the site with bots to make search engines and such things but I'm wary of them. I'd need to know exactly what the script was doing and what sort of calls it would be making to the server. Link to comment Share on other sites More sharing options...
Yacoby Posted August 13, 2008 Author Share Posted August 13, 2008 I'm often skeptical of these things; I've had quite a few people ask if they can crawl the site with bots to make search engines and such things but I'm wary of them. I'd need to know exactly what the script was doing and what sort of calls it would be making to the server.Heya Having taken a look at how the site works, this is the list of requests I want to make to your site For indexing already uploaded mods: (obviously the loop would be done over several months, and only needs to be done once)for every category{ for every page in that category{ Download page (http://www.tesnexus.com/downloads/cat.php?id=#&page=##&orderby=date&order=ASC) for every mod on that page{ Download description page: http://www.tesnexus.com/downloads/file/description.php?id=376 } } } For keeping the db updated. Run once a dayDownload Recent updates page (http://www.tesnexus.com/downloads/recent.php) for every new mod{ download mod page (http://www.tesnexus.com/downloads/file.php?id=###) download description page http://www.tesnexus.com/downloads/file/description.php?id=### } for every updated mod{ if updated description{ download description page http://www.tesnexus.com/downloads/file/description.php?id=### } } The data on mods that I would keep are:IDAuthorDate uploaded RatingNumber of RatingsDownloadsDate Rating/Downloads was retrieved DescriptionCategorySize of Download When/if the scripts written, I am happy to send you the scripts before I start running them :) EDITLet me know if you are, in theory, happy with the above requests Link to comment Share on other sites More sharing options...
Yacoby Posted September 9, 2008 Author Share Posted September 9, 2008 Just to let you all know, I have given my scripts to Dark0ne, so hopefully he should come back with some comments and/or some bandwidth/page request limits to stick to. :) Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.