General MW Mod Search Engine

[Yacoby]

I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this:

 

  Quote

I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).

I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).

Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you.

[LHammonds]

  Yacoby said:

I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this:

 

  Quote

I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).

I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).

Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you.

Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds.

 

The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID.

 

You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus.

 

RSS Feed: New Files Today

RSS Feed: File Updates Today

 

LHammonds

[Yacoby]

  LHammonds said:

Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds.

 

The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID.

 

You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus.

 

RSS Feed: New Files Today

RSS Feed: File Updates Today

 

LHammonds

Heya

 

The problem with using the RSS feeds is that it is updates today, and I don't think it is in the last 24 hrs, which would lead to this error:

11:58 - Scan RSS

11:59 - User uploads mod

00:00 - RSS reset mods upload missed

 

Where as if I look at the previously uploaded entries once a day, I don't miss any mods

 

Thanks for the suggestion though.

[LHammonds]

BEGIN TOTAL SPECULATION

 

Maybe a query subscription service then?

 

You call a page and pass the date as a parameter and it pulls all the mods uploaded that day in an XML format?

 

Example:

 

DayQuery.php?date=20081231

 

Returns and XML-formatted data set including the following data:

 

File ID

Game

Category

Mod Name

Author

Uploader

Version

Small Description

Date Uploaded

Date Updated

 

Maybe an additional query for further and updated information based on the info you already have collected (based on file ID)

 

Example:

 

ModQuery.php?fileid=8440

 

Returns the following data set:

 

File ID

Game

Category

Mod Name

Author

Uploader

Version

Small Description

Date Uploaded

Date Updated

Total # of Downloads

Total # of Views

Total # of Comments

Total # of Images

Total # of Files

Total # of Ratings

Average Rating

 

I guess it would depend on how much this service would be worth for Dark0ne to invest into it.

 

END TOTAL SPECULATION

[Michlo]

  LHammonds said:

  Yacoby said:

I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this:

 

  Quote

I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).

I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).

Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you.

Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds.

 

The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID.

 

You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus.

 

RSS Feed: New Files Today

RSS Feed: File Updates Today

 

LHammonds

 

Bloody hell, how did I miss that? Thanks, mate. :thanks: I'm now using the RSS feed as well. :)

 

Cheers.

 

And with that, I'm going home so no more posts from me tonight (I hear the sighs of relief). ;)

[Dark0ne]

I'm often skeptical of these things; I've had quite a few people ask if they can crawl the site with bots to make search engines and such things but I'm wary of them. I'd need to know exactly what the script was doing and what sort of calls it would be making to the server.

[Yacoby]

  Dark0ne said:

I'm often skeptical of these things; I've had quite a few people ask if they can crawl the site with bots to make search engines and such things but I'm wary of them. I'd need to know exactly what the script was doing and what sort of calls it would be making to the server.

Heya

 

Having taken a look at how the site works, this is the list of requests I want to make to your site

 

For indexing already uploaded mods: (obviously the loop would be done over several months, and only needs to be done once)

for every category{
for every page in that category{
	Download page
	(http://www.tesnexus.com/downloads/cat.php?id=#&page=##&orderby=date&order=ASC)

	for every mod on that page{
		Download description page:
		http://www.tesnexus.com/downloads/file/description.php?id=376
	}
}
}

 

For keeping the db updated. Run once a day

Download Recent updates page
(http://www.tesnexus.com/downloads/recent.php)
for every new mod{
download mod page
(http://www.tesnexus.com/downloads/file.php?id=###)

download description page
http://www.tesnexus.com/downloads/file/description.php?id=###
}

for every updated mod{
if  updated description{
	download description page
	http://www.tesnexus.com/downloads/file/description.php?id=###
}
}

 

 

The data on mods that I would keep are:

  Quote

ID

Author

Date uploaded

 

Rating

Number of Ratings

Downloads

Date Rating/Downloads was retrieved

 

Description

Category

Size of Download

 

When/if the scripts written, I am happy to send you the scripts before I start running them :)

 

EDIT

Let me know if you are, in theory, happy with the above requests

[Yacoby]

Just to let you all know, I have given my scripts to Dark0ne, so hopefully he should come back with some comments and/or some bandwidth/page request limits to stick to. :)