Downtime

Sianni · Post by **Sianni** » Thu May 04, 2006 7:12 am

Rollie,

Any idea how long the site will be down for maintenence?

Thanks!

Post by **Rollie** » Thu May 04, 2006 8:21 am

Well, already longer than I had hoped and the table I'm rebuilding still has a long way to go. I am going to try to enable certain parts of the site, but full functionality won't be online for a while yet.

DM. · Post by **DM.** » Thu May 04, 2006 8:31 am

At least we got the forums back. =)

*begins trolling*

Mather · Post by **Mather** » Thu May 04, 2006 12:10 pm

The downtime has been for quite awhile now, any idea when things will be operating fully again?

Post by **Rollie** » Thu May 04, 2006 1:04 pm

Honestly, not really. This table reconstruction is taking WAY longer than I had expected. Had I known it was going to take this long, I would have done it differently.

The time it has taken to get to the point where it is now is not linear and as it gets further along, the time to continue gets longer. At this rate, I would not be surprised if the table is not done for another 12 hours.

Ceto · Post by **Ceto** » Thu May 04, 2006 1:44 pm

One of my professors wrote a bit of code in each of C, bash, and Perl as a comparison of application speed. C was fast, bash was slightly slower, Perl was not able to finish by the time class started. Problem was, he had a slight error in his program logic and created a recursive loop, increasing his execution time exponentially. Good stuff coming from a professor. =)

Just an anecdote.

What sort of operation are you running on the database, if I may ask?

DM. · Post by **DM.** » Thu May 04, 2006 1:50 pm

Ceto wrote:What sort of operation are you running on the database, if I may ask?

I'm curious as well... WCR was very well laid out and running fine. What changes or new features are being done?

[Edit] grrr quotes not working... still.... I know! Rollie is doing all this just to fix the quotes, and those buttons and checkboxes at the bottom of the forum threads for quick reply which do nothing

Post by **Rollie** » Thu May 04, 2006 2:54 pm

It's actually a simple repair table call in mysql. The table in question is quite large and is taking quite a while to rebuild.

This is done because I am trying to track down the slow down in update processing. Something is up and this table came back with corrupt indexes so I am hoping after this is done that updates will be back up to speed.

It's about 72% done. It hit 50% last night before I went to bed and I expected it to be done by the time I got up, but it has slowed down considerably. I could stop the operation but at this point, since it has been running as long as it has, I am more inclined to just let it finish.

In the end, I am going to have to do some work. Either adding more hardware or changing some database options. The magnitude of character data that is being collected is huge and grows every day.

Sianni · Post by **Sianni** » Thu May 04, 2006 7:29 pm

Rollie wrote:The magnitude of character data that is being collected is huge and grows every day.

But that's a good thing, right!

Post by **Rollie** » Thu May 04, 2006 9:47 pm

I guess so, hehe. It does mean that character histories are more fleshed out, but actual concurrent user data and data submissions has actually slowed down. I could probably pare the data down a bit by pruning out all really low characters that haven't been seen in a certain amount of time (like 6 months or something), but I'm not ready to make that step yet.

I know I could improve things by throwing more hardware at the problem, but of course more hardware means higher costs and that is something I am trying to avoid.

WyriHaximus · Post by **WyriHaximus** » Fri May 05, 2006 3:24 am

Did you ever considerd InnoDB as table type instead of the standard MyISAM? I don't know how you exactly store your data in the database. Did you ever thougth up splitting it up in a US and a EU table or even a table for each server?

Post by **Rollie** » Fri May 05, 2006 8:33 am

I actually have a mix of InnoDB and MyISAM right now. I am debating moving everything over to InnoDB. I have thought also about breaking the data up by either server, region, month, year, or something. I already have some of the data split, but may just need to split it into smaller tables instead of the current 2 really big tables.

My only hesitation about moving over to all InnoDB is just file size and performance. I only have 1 InnoDB table right now and the table space is already over 20gig combined. If I add in the other large tables from InnoDB, it will be even bigger.

These are just not problems that a site the size of WR usually faces, hehe. We just track so much data!

WyriHaximus · Post by **WyriHaximus** » Fri May 05, 2006 1:07 pm

20GB? Auch!!! I must admit this kind of database sizes are normal for this type of sites. (Wonders how big thottbots database it.)

AFAIK InnoDB has better performence with bigger tables then MyISAM. You could try clone the while site and run some benchamrks. (It's a bit suggestion I know

.)

Another suggestion could be auto pruning in the UI mod of everythinglonger not seen the x days. That should take some process over head and database load away.

The splitting is a realy good thing to work out. If 1 table gets damaged you don't have to shutdown the entire site but only 1 server in the cencus view.