CreeperLava

@gift Should we add an "Answer" column, to note the results of the contact ? I received an answer from Gibbed, he's going to look into organizing his repos on his website. https://twitter.com/gibbed/status/764100009214304256

I need help from an admin here. I've tried to contact a mod author named Dheuster on Nexus, but I get the error "The member Dheuster cannot receive any new messages". If this is because his inbox is full, is there any way to work around it ? I'm done with the pages 2-5. Page 1 has been done by someone else before. Some tips for other participants : - When searching for an author, use quotation marks in google. This ensures google will only give you results that contain exactly the author's name. Same goes for mod names. - To reduce the number of results, append the game's name next to the author/mod. eg "tmp7704" dragon age - For now, I've contacted authors on Deviantart, Nexus, Youtube, Twitter, ModDB and Bioware forums (when I didn't have another choice), so it's quite varied. @giftfish I added a DA2 tab in the spreadsheet, I didn't touch permissions though, I let you handle that. Also added a Planning tab to keep track of who does what, and if it is done or not.

I'll be doing pages 2 through 5 (ordered by downloads). If you want to work at the same time, please begin with page 6 :).

Got it, thanks. I'll work on it tomorrow. Almost 300 GB worth of forum data, with 240k/1064k.

You can actually jump multiple pages at once, but you have to edit the URL directly. eg : http:// social.bioware.com/browse_bw_projects.php?page_num=3&project_search=Search&view=0&project_category_id=&status=4&sort=1 Change the 3 to whichever page you want to go to. Should save some clicking :smile:. Edit : God I hate Nexus' auto formatting... Edit 2 : You should add that to the spreadsheet, gift :smile:. Edit 3 : If I can make suggestions for the spreadsheet : - Since the mods aren't really sortable by game, it would be easier for us to put them all in a single tab, and add a column for the game. - The first column should be the project's number on BSN (found here : eg social.bioware.com/project/1105), rather than a random index. There are 4168 mods, it's too easy to get lost the way it is setup right now. - Sorting by downloads is unreliable. There are loads of mods with 0 downloads, that are very popular, but host the mod files on other websites. Thus, an apparent 0 downloads. Question : If permissions are open, should we still try to contact the mod author ? Or just skip this step and directly upload on Nexus ? Edit 4 : Not sure why I see 4168 mods while you seem to see 3964. Weird.

It's not a "she" :P. @gift I don't think Fexelea downloaded the whole forums, only a few subforums afaik. My download is still going steadily : 290 GB, 230k/1027k (yup, over a million...) At this point I'm thinking some of the links must be duplicates that will get sorted by HTTrack later. A million pages seems excessive, even for such a forum. The WP is a good idea, but it needs to be linked to in every other post mentioning the BSN shutdown (as you said, ME3Explorer, Reddit, Nexus...). Otherwise it will never be seen. It would be the best way to centralize information, latest news will be there, in case the other posts aren't updated. I'd be glad to help contact mod authors, once you upload the gdoc. A sample message would be nice, too. How many of them are there still left to contact ?

In response to post #41271475. #41271555, #41271810 are all replies on the same post. Having the description below is at least an incentive to read it, removing that incentive won't ameliorate the situation in any way... What would you propose ?

In response to post #41271475. Agreed, I said the same thing a few posts below :).

The fact that we, as outsiders, are backuping their own stuff faster than they are, is not a good sign^^.

I like the new design a lot, good work ! A few suggestions : - Add a way to make advanced searches directly from the toolbar (with the possiblity of adding cumulative search filters, like eg search for a name and an author and a game). This could be done with a simple gear button at the right of the Search button. - I like GuerillaTech's idea of changing the color scheme of the website. Similarly to how the color scheme changes with each game in the current Nexus design, you'd be able to choose what you prefer. - The " Download Mod" feature, directly from a thumbnail on the main page isn't a good idea. People already don't read descriptions a whole lot, this encourages even more laziness. Mod authors will suffer the most from this, since they will be the ones users will come to complain when the mod (unsurprisingly...) doesn't work. Besides, how do you manage mods that have multiple main files ?

I see, not sure how well HTTrack can do with constant interruptions. Worth a try though :).

You can't leave your computer on when you're not there ?

That's pretty damn better than what I get. I'm at 200 GB, you'll catch up fast if you can run it 24/24.

190 GB, 146k/706k. 1 link done creates 5 more to do, great. Hopefully it's not infinite^^.

Thanks, hadn't thought of that. I still have about 800 GB free, plus 1 TB that is used by other programs. I will move these if needed, then only try the archiving method. Should be enough that way. 6 hours a day is not much, it depends how fast you can download there. Try running HTTrack for an hour on forums.bioware.com, see how much you download in that time.

FYI, forums backup is at 160 GB, 125k/611k links. The number of links is still climbing, worryingly, I hope I'll have enough space for everything.

Thanks, shouldn't impact the downloads though. HTTrack downloads everything it sees, so no worries about missing something :).

Just a quick note : to make sure your archive is as small as possible, you should use 7z's LZMA2 format on Ultra compression :smile:. I tested it, the size of the archive went from 387 MB (60% of 638 MB) down to 258 MB (39% of 638 MB). It's also faster to decompress.

Most of these are probably user profiles. I tried backuping social.bioware.com, had no issues with projects (which are the most important). If we can't backup user profiles, meh, that's not too bad.

That's a good one^^. This is important, and I suggest everyone here reads the rest of the post : Unless you can use different IP addresses, they should share the bandwidth allowed by the server. So yes, all would go slower. 900 GB should be plenty enough (I'm at 90GB), if you're following the settings (that is, no pictures, no data from external websites, only HTML/JS/CSS from forums.bioware.com). Once compressed, it should go down at least tenfold, since it's mostly text. Ah, so that's what you did to bypass it. Awesome, thanks ! HTTrack can resume downloads. However, I think it is counterproductive to have multiple people downloading the same thing, at the same speed. The only result is that we are putting a strain on the server, thus slowing down everything else. I'm downloading the whole forums.bioware.com, so unless your download speed (from HTTrack, not your regular bandwidth !!) is superior to 120 KB/s, there's no use in running it yourself. The same goes for the other projects. Tarshana downloaded the entirety of Bioware projects (BSN legacy). The most sensible thing to do would be to upload this archive and share it privately to make sure at least 2 backups are out there.

Tarshana, did you also backup the projects from the ME2/ME3 Texture Mods groups ? http://social.bioware.com/group/7065/#projects http://social.bioware.com/group/1547/#projects If not, I'll try to do that tomorrow. Also, I've contacted CDAMJC. A few of his mods are still only in his BSN thread.

The old BSN has already been downloaded, gift. I'm pretty sure they are indeed trying to download the forums. Correct me if I'm wrong, of course, Shades and Tarshana. 3088+185+174+517 = 3964, so there would be 3964 projects, not 8340. Doesn't change anything to the fact that we should backup all of them, though :smile:. Edit : That's the conclusion you make at the end, nevermind.

Made some screenshots to show the progress, and a preview of what it will look like once done : http://i.imgur.com/V1PqACb.jpg http://i.imgur.com/XDlLkAf.jpg http://i.imgur.com/peMkzVi.jpg http://i.imgur.com/35Nvcw9.jpg

I'm pretty sure the only way to make a backup is to download everything, from the root : forums.bioware.com. It may not be possible to reassemble bits of backups, eg per subforum. "Download website(s)" is what I use, too. The reason it displayed "Update existing download" is because it tried to continue the download that failed before. For the config, please follow what I had advised using, leave the rest as default. Max transfer rate at 25000 means you have 25 KB max bandwidth, which is not a lot^^. I would prefer not being the only one running HTTrack, in case something wrong happens, but we still have time. There's no reason to have more than 2-3 people running it either, it puts an unnecessary strain on the website. Edit : Just checked the file structure of HTTrack, it looks like we could in fact reassemble partial backups together. So that's good news, it's easier than downloading the whole thing at once.

Ok, if you want to help download forums.bioware.com with HTTrack, make sure you have a good bandwidth. There's no use in downloading the same thing multiple times if your download speed is slower than mine. My HTTrack is downloading at about 120 KB/s, Tarshana's 240 KB/s. Instructions to avoid the "too many links" error : In Mirror > Modify Options - Scan Rules : uncheck gif, jpg, jpeg... - Limits : Max external depth :0 Site size limit : something really big like 9999999999999999. We don't want the download to stop randomly because it thinks the site is too big^^. Same for max transfer rate and max number of links, just put many 9s. - Flow Control : Number of connections : 10 Torrents : Follow this guide I made : http://i.imgur.com/2kmDlZJ.png https://download.transmissionbt.com/files Then open transmission > File > New. In the window that opens, select a source folder ; add these trackers : udp://tracker.coppersurfer.tk:6969 udp://open.demonii.com:1337 udp://tracker.leechers-paradise.org:6969 udp://tracker.openbittorrent.com:80 And click OK. Once it's done, click Open and send us the .torrent. I can't really know which part HTTrack is downloading, the way it works isn't really linear. I set it to download everything in forums.bioware.com, but I don't think there's any particular order.