BSN is shutting down; files there could be lost.

[krlewis321]

Okay, I only found out about the disaster currently unfolding recently and it's like a sock in the gut. It's horrible so much fan content is going to be lost. If it's possible I want a copy of the backups everyone else has made to go along with the one I am independently working on of the BSN Mods, blogs and the Dragon Age Forums. I will financially compensate anyone willing to do this the amount they think fair. Tarshana, CreeperLava, Felexea? Please let me know! I'd also like a copy of the backup of DA posts made to Fextralife. I will mail anyone interested in providing me with their data a thumb or external drive and a return envelope/box and I will also provide my own scrape-content to anyone who asks though it might also need to be by mail given the size of it.

 

Concerning the mods:

 

Using an Autoit script (https://www.autoitscript.com/autoit3/docs/functions/InetGet.htm), I downloaded every page between

 

http://social.bioware.com/browse_bw_projects.php?page_num=1&project_search=Search&view=0&project_category_id=&status=&sort=1

 

and:

 

http://social.bioware.com/browse_bw_projects.php?page_num=397&project_search=Search&view=0&project_category_id=&status=&sort=1

 

to try to get a list of every project. I did it more than once to make sure, but the weird thing is that when I extracted the project urls it gave me a different number of projects each time. Looking at the individual .html files, I found that a project would appear on, say, page 15 one time but then be missing from page 15 another time. This makes me think this method for getting projects is very unreliable. So instead I downloaded every page between

 

http://social.bioware.com/project/1/

 

and

 

http://social.bioware.com/project/10000/

 

In fact, I also did 10000-20000 just to be safe but there were no projects over 10000. Adding zeros to the url doesn't change anything. For instance,

 

http://social.bioware.com/project/26/
http://social.bioware.com/project/026/
http://social.bioware.com/project/0026/
http://social.bioware.com/project/00026/

 

all lead to the same project.

 

Using a program called baregrep (http://baremetalsoft.com/baregrep/), I searched for any file containing "The project you are looking for has been deleted or does not exist," or "You do not have permission to view this project" and then removed them.

 

This left me with a list of 4211 projects:

 

https://www.dropbox.com/s/5iuvni4blttslwj/bsnprojectlist.txt?dl=0

 

Of these, 3973 can be accessed without being logged in but 238 require you to sign in with your account to view them:

 

https://www.dropbox.com/s/n4zwsbk2qgu9rqx/bsnprojectlistlocked.txt?dl=0

 

It took me awhile but I have downloaded all project pages in .mht file form using this Firefox add-on.

 

https://addons.mozilla.org/en-US/firefox/addon/unmht/

 

I also downloaded them in .html, extracting a list of 3997 download links (edit: restored! NWM!):

 

https://www.dropbox.com/s/ukyxwt81kbm1bn4/bsndownloadlinks.txt?dl=0

 

If these, eight are broken:

 

https://www.dropbox.com/s/qokkw5k0xfo4op3/bsnbrokendownloadlinks.txt?dl=0

 

And two have been removed since I started:

 

https://www.dropbox.com/s/iybujaypfsrlctj/bsndownloadlinksremoved.txt?dl=0

 

I have gotten all of them using Downthemall

 

https://addons.mozilla.org/en-US/firefox/addon/downthemall/

 

Posts will follow detailing my progress with the blogs and DA forum.

[moho25]

Uh, hello infodump, lol.

 

Hi krlewis321 :smile:

 

A few things:

 

1. Felexea isn't around these parts, unfortunately. Her forums and relocation effort of the existing BW forums are her own thing and separate from BCN (us). You'll need to register over on Fextralife to speak to her.

2. 4211 projects is close to the known count. When I log in and view all projects, it tosses the number 4170 at me. When I'm logged out, it tells me 3964.

3. Thanks for all the links. I may end up placing some on the website if you're okay with that.

4. Please make sure you don't actually include downloads of links for *mods*. Doing this will get you banned from Nexus. I don't think you've done that above, but it's a bit hard to tell, though. Just mentioning it in case.

5. BW's DA forum has already been copied over to Fextralife.

6. Many blogs are set to private and are not downloadable.

 

The best thing you can do right now to ensure continuity of project availability is to help with the inventorying and author contacting efforts. We have few people and it's tedious, time-consuming work. Having backups is all fine and great, but if the community loses access to the mod, the back up doesn't do much good since it can't be uploaded anywhere without the mod author's consent. Yes, random mediafire/dropbox links can always be shared privately between users, but that's also technically redistributing without the author's consent, and it isn't practical (or legal) at the community level.

 

EDIT: Fixed wording.

Edited August 25, 2016 by moho25

[Tarshana]

We have all of that on back up, but it is great to know we have a third source!

 

Please take down the links, thought, as they do contain mods that are still on the read only Legacy site. What we need is people to keep cataloging on the google spreadsheets. Thanks for wanting to help :)

[moho25]

@ krlewis321 --

 

This is really great data. I've added a new page to the spreadsheet with the complete list of project IDs. Projects that require the viewer to be logged in are highlighted.

 

One thing that would be handy is if you could use your data filtering skills to export a list of the name associated with the ID of each project. Then I can place that on the spreadsheet, associated with each ID. From there, I'd be able to create a formula that would recognize if the project has been catalogued, and fill in that data automatically.

[moho25]

We have all of that on back up, but it is great to know we have a third source!

 

Please take down the links, thought, as they do contain mods that are still on the read only Legacy site. What we need is people to keep cataloging on the google spreadsheets. Thanks for wanting to help :smile:

 

From what I can tell Tarsh, there are no downloads, short of lists of links. No actual projects.

 

What's great is we now actually have a list of all valid project IDs, so we can now actually know what is indeed present. That's pretty great :)

Edited August 25, 2016 by moho25

[krlewis321]

 

3. Thanks for all the links. I may end up placing some on the website if you're okay with that.

4. Please make sure you don't actually include downloads of links for *mods*. Doing this will get you banned from Nexus. I don't think you've done that above, but it's a bit hard to tell, though. Just mentioning it in case.

 

Yes, you have permission to post these anywhere. And I removed the download links to the actual files. I didn't realize it was against the rules. But anyone who wants them can ask me. (Edit: restored! NWM!)

 

We have all of that on back up, but it is great to know we have a third source!

 

Is there anyway I can get a copy of the back up you made?

 

@ krlewis321 --

 

This is really great data. I've added a new page to the spreadsheet with the complete list of project IDs. Projects that require the viewer to be logged in are highlighted.

 

One thing that would be handy is if you could use your data filtering skills to export a list of the name associated with the ID of each project. Then I can place that on the spreadsheet, associated with each ID. From there, I'd be able to create a formula that would recognize if the project has been catalogued, and fill in that data automatically.

 

I was thinking of doing something like this myself, and I'll get to work on it. I'm going to try to use Linux command line tools like grep and sed to extract info from each .html file (http://gnuwin32.sourceforge.net/packages.html). I don't think I can help with contacting mod authors, though. I have some social issues and would feel really uncomfortable e-mailing people out of the blue.

 

ETA: Okay, it's (mostly) done. Here's the list of project ids with their owners.

 

https://www.dropbox.com/s/6uukq4qu5ld6j78/bsnprojectidowner.txt?dl=0

 

These are the projects not included on the list because they don't list an owner in the proper spot. Some appear to be blank and empty but not others. I need to go through them manually.

 

https://www.dropbox.com/s/t5hod69i6vwcfss/bsnnownerprojects.txt?dl=0

 

ETAA:

 

This is a list with the project id, project name, and owner of each with a "$" delimiter between them. (This doesn't include the ones on the "no owner" list):

 

https://www.dropbox.com/s/q2xuccji3lfdbxx/projectid%24name%24owner.txt?dl=0

[moho25]

@Tarsh -- If the BW Forums are still up when you see this could you post in both the Archiving thread and BCN thread? Just to bump them up to the top of the page. I'm trying to make sure we don't fall off before the read only.

 

Would have been nice for BW to give us an ETA on forum closure. I'd expect EOB today EST... not that BW likes to do what we expect.

[Tarshana]

#krlewis - If your links do not contain mods, ignore what I said xD I thought you had made backups of the database. I apologize for misunderstanding. As to the copy, no. CreeperLava and myself will hold onto the copies for posterity's sake and for people have access to mods they can no longer get. If there are any mods you need that are not on Nexus or a personal website please let me know.

 

 

Edit: I rescind. I will zip a file for you as soon as it is ready. httrack likes to keep my log info >.< I will send you a link when completed.

 

MohoFish- Okay done :smile:

[Tarshana]

Ah well, the forums just went down :(

[moho25]

Wow, 9am Pacific time. They really didn't want to wait until EOB, did they?

 

--I'll post an update to the website that they are now officially closed.

--We're going to have to keep an eye on their twitter/blog/FB now regarding changes to BSN, which really, really sucks. People are going to miss things.

 

Yesterday when googling "bioware forums closing" BCN was on the 4th page of search results; now we're on 7 for some reason. If you use "bioware forums close projects bsn", then we're on the first page. I'm not sure what else to do to help web searches find us better. I've got tags on everything. I've tried to use strong keywords in post titles, etc.

 

Tarsh, now that things are closed, do you think we should put up a static home page with a brief summary and directing people about the site, or stick to everything on the home page as it is now?

 

Funny tidbit.

 

BW hasn't posted a thing on their FB or Twitter since the closure announcement. I thought that's how they *want* to communicate now?