BSN is shutting down; files there could be lost.

[Shadesofnight]

Aha! After following the instructions, HTTrack is working again! And faster than before too thanks to the bandwidth speed option (I honestly hadn't even noticed the File, Preferences etc at the top... )

 

I'm not sure what bits everyone else is/is not doing so I decided to start working my way backwards on the Dragon Age sections. I've now done the Keep and Dragon Age Franchise... Onto DAO.

 

Edit: After becoming suspicious about the lightning quick download times I took a look and noticed that although the initial index page I visited was located on my computer, any other pages were in fact simply the online versions (as shown in the address bar), not ones saved. So I must have another setting wrong somewhere. But my previous downloading that failed was saving each page offline. I'll check to see which settings are different.

Edited August 4, 2016 by Shadesofnight

[Tarshana]

So I kinda wanna cry... but I decided it's not very Irish of me to cry at a computer.

 

Shades- we might have a similar issue. After pulling my files to my lap top I realized something: I hadn't captured my login information and nothing was saved properly... it was all "file access not granted." /facedesk . Here I thought my fibre cable was being a badass. It's so bizzare because when I was watching the download it would state "Project, Number" etc. So I don't know :sad: But hey, we got a wee bit of time to figure this out right?

I don't know what settings would be different but my downloads on the forums were the same like you >.< grruh. I'm going to recapture the BW forums with login information; login stuff is stored in a specific folder with the HTTrack stuff and not with the offline content so I think I'll be safe. I'll be swapping emails/pws after this endeavor to be on the safe side :smile:

http://httrack.kauler.com/help/CatchURL_tutorial
https://www.httrack.com/html/faq

[Shadesofnight]

Well, at least I'm not the only one having the problem, and it's certainly very good that we realized it this early :)

 

I suspect it may have something to do with what you set it to when you first start a new project (The Action: 'Download website(s)' etc bit just above where you put in the URL), but I'm really just guessing and unfortunately can't check what my failed attempt had selected because it's now changed to 'Update existing download'.

 

I can however put down the settings I had for that failed one though (keeping in mind it's the one that got the too many URLs error).

 

I don't think I changed many things at all, but this is what it had:

Scan Rules - nothing changed

Limits - Everything blank except max transfer rate at 25000

Flow Control - All blank, persistent connections ticked

Links - Attempt to get all links, and Get all non-HTML files both ticked

 

Pretty sure I didn't even look at anything else.

 

CreeperLava will likely be our best bet for figuring this out as I'm virtually clueless about how a lot of it works, I'm just used to trying to trouble shoot things.

 

Oh and I'd say it's certainly a good idea to do what I've been doing: When you back something up like say a forum section open it up, go to one of the last pages of topics in that section and randomly select a page to see if it's there, do the same in the middle and start topic pages, repeat for the sub-forums if your section has them. This way with minimal effort you can be fairly sure if it's captured everything or not. And as we now know, check the URL displaying to make sure it's on your computer and not the online version.

 

Also if you find something like I did with the DAO General discussion topics, which is a page of topics all called 'empty', don't panic. Check to see what it looks like on the actual forums. And as you can see here, there are indeed a lot of topics called 'empty' there for some reason, so it isn't a backing up problem.

[CreeperLava]

I'm pretty sure the only way to make a backup is to download everything, from the root : forums.bioware.com. It may not be possible to reassemble bits of backups, eg per subforum.

"Download website(s)" is what I use, too. The reason it displayed "Update existing download" is because it tried to continue the download that failed before.

 

For the config, please follow what I had advised using, leave the rest as default. Max transfer rate at 25000 means you have 25 KB max bandwidth, which is not a lot^^.

 

I would prefer not being the only one running HTTrack, in case something wrong happens, but we still have time. There's no reason to have more than 2-3 people running it either, it puts an unnecessary strain on the website.

 

Edit : Just checked the file structure of HTTrack, it looks like we could in fact reassemble partial backups together. So that's good news, it's easier than downloading the whole thing at once.

Edited August 4, 2016 by CreeperLava

[CreeperLava]

Made some screenshots to show the progress, and a preview of what it will look like once done :

http://i.imgur.com/V1PqACb.jpg
http://i.imgur.com/XDlLkAf.jpg
http://i.imgur.com/peMkzVi.jpg
http://i.imgur.com/35Nvcw9.jpg

Edited August 4, 2016 by CreeperLava

[Tarshana]

I was able to put the max xfer rate as blank and it went up by 10 fold. I've gone from having 8k pages to 52k and counting. Note: I have errors because some users I cannot see whatsoever no matter what I do. Because BW put limited access on the Legacy site, there's nothing I can do to circumvent their restrictions. I have HTTrack going on my laptop for the forums and will send download links in our private messages. I'll work on Wayback Machine today as well; it looks like they are back online.

[moho25]

I'm pretty sure the only way to make a backup is to download everything, from the root : forums.bioware.com.

 

Creeper, they're trying to download the old BSN, not the new forum. They're different URLs:

http://social.bioware.com/

That is the URL they need, that all projects, blogs, and the DAO Toolset and wiki are linked to.

 

----

 

@Shade -- You had a couple questions both on the BW forums and here.

 

1. Game Patches -- These are hosted on BSN for DAO, DAII, ME2, and ME3. They're accessible via the "Games" dropdown in the main menu. Here's the patch page for ME2, as an example. DLC are of course also in this same place, as you know, but are only accessible once you log in.

 

2. Mass Effect 1 -- This is the original Mass Effect site. A slew of stuff is here, including the ME1 Patches. We likely also want to archive this entire site, as I bet it will be removed with everything else that BW isn't mentioning.

main url  http://masseffect.bioware.com/me1/index.html

3. Blogs -- Creeper and I confirmed that there's no way to view all blogs right now. There are no page numbers associated with the results, and only the top 10 results are visible for "everyone's blogs". Same thing happens if you try to do a search. I have all accessible blog pages saved as HTML that have a modding purpose, and are not fan fiction or commentary.

 

4. Projects -- Could be my logic is wrong, but it's a bit hard to tell due to the poor layout of the site. Here's how I came up with the 8340 count via changing the menu options and looking at the results:

• Everyone + Uncategorized + No Specified Status = 3964
• Everyone + Uncategorized + Planning = 3088
• Everyone + Uncategorized + Building = 185
• Everyone + Uncategorized + Testing = 174
• Everyone + Uncategorized + Released = 517

That totals to 8340.

 

Right now it isn't clear to me when a dropdown is "blank", whether that means the author didn't fill in the field, or whether it means "show everything". I'm assuming the former and not the latter. If it means the latter, then there are 3964 projects.

 

Testing this right now in a different way adds some clarity, but still means our numbers don't match up.

 

Take a look at NWN: Everyone + NWN + Unspecified. This gives 439 projects. Then, if you change the Status, you find that plan=251, build=74, test=56, release=58. Those numbers add up to 439. That suggests then that the "empty" dropdown fields mean "show all". Which then means there are 3964 projects on BSN and not 8340. However, that doesn't line up with the "550 pages" number Tarshana provided. There are always 10 projects per page -- except for the last page -- which means approx 5500 mods. Seems we still have some type of numerical inconsistency.

 

EDIT: Clarification, final paragraph.

Edited August 4, 2016 by moho25

[CreeperLava]

The old BSN has already been downloaded, gift. I'm pretty sure they are indeed trying to download the forums. Correct me if I'm wrong, of course, Shades and Tarshana.

 

3088+185+174+517 = 3964, so there would be 3964 projects, not 8340. Doesn't change anything to the fact that we should backup all of them, though :smile:.

Edit : That's the conclusion you make at the end, nevermind.

Edited August 4, 2016 by CreeperLava

[moho25]

The old BSN has already been download, gift. I'm pretty sure they are indeed trying to download the forums. Correct me if I'm wrong, of course, Shades and Tarshana.

This quote by Tarshana and posted this morning specifically references "Projects", which as you know, are on BSN:

 

So I kinda wanna cry... but I decided it's not very Irish of me to cry at a computer.

 

Shades- we might have a similar issue. After pulling my files to my lap top I realized something: I hadn't captured my login information and nothing was saved properly... it was all "file access not granted." /facedesk . Here I thought my fibre cable was being a badass. It's so bizzare because when I was watching the download it would state "Project, Number" etc. So I don't know :sad: But hey, we got a wee bit of time to figure this out right?

 

I don't know what settings would be different but my downloads on the forums were the same like you >.< grruh. I'm going to recapture the BW forums with login information; login stuff is stored in a specific folder with the HTTrack stuff and not with the offline content so I think I'll be safe. I'll be swapping emails/pws after this endeavor to be on the safe side :smile:

 

http://httrack.kauler.com/help/CatchURL_tutorial

https://www.httrack.com/html/faq

Edited August 4, 2016 by moho25

[Tarshana]

I actually started on the Forums with my laptop last night/early morning :) So while that is going the Projects are also being downloaded. That was something Creeper and I mentioned in a PM- I cannot add more than one person for a PM :( sorry Moho-

 

Projects: 52gigs and counting (250k pages and counting; errors- 641, most of which are access denied because I am not allowed on the blog or project page)

Forums: 25gigs and counting (toolset)

Wayback Machine: DA Toolset GD completed.

Toolset Wiki: Completed database backup

I have pulled my daughter's machined to do some QA work while I configure the folders. I have Ubuntu and Win 7 on her machine so I am sending files to that computer and making sure they can be viewed without issues. I have an older rig set up with Linux RedHat to also check but I can only run so many computers before my head gets dizzy xD I don't think many people use Linux to run games but it does house stuff for me.

 

Once I have these all completed, links will be sent to you all to download and verify yourselves. Right now I do not think that giving people chunks will do much but I could be wrong. I can pause and zip what I have if anyone wants to see. I will be zipping the wiki shortly.