WarcraftRealms.com

More good news!

The good news is that the update process is now back up and running. The more good news is that whereas the last db was processing data files around 30 per hour, this one is processing around 180 per hour. Which is a very good thing because it's behind by about 6000 data submissions =x

AWESOME!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Good things come to those who wait!

Rollie wrote:More good news!

The good news is that the update process is now back up and running. The more good news is that whereas the last db was processing data files around 30 per hour, this one is processing around 180 per hour. Which is a very good thing because it's behind by about 6000 data submissions =x

Geesh, 6000? That's incredible XD guess a site can be too popular

Sweet and very good news. Guess that means we need to submit more and see if we can bring the server down?

J/K

BW

BWDemor wrote:Sweet and very good news. Guess that means we need to submit more and see if we can bring the server down?

J/K

BW

haha Nice one.

Rollie that is great news. I am glad that things are working out with the database.

From a purely morbid technological curiosity, may i ask what you did to the DB to increase performance by 6x?

(Feel free to get technical)

Well, basically I just moved the database to a much beefier machine.

Went from a dual AMD Athlon 2600 MP with 2GB of RAM and dual 80GB IDE HD to a dual Opteron 265 with 8GB of RAM and dual 160GB SATA HDs

The biggest improvement is probably the hard drives, but the increased ram is also very helpful.

This still isn't the optimal setup and over the next few months there will likely be another move when I get fully set up on a raid 5 machine with full replication.

Awesome, sounds like you had a fairly linear increase in performance.

I'm sure you're aware of this, but RAID 5 is horrible performance for writes, though great performance for reads. If the submissions are the slow point, you might want to consider raid 10 or some other raid to elminate disks as bottleneck.

Rollie wrote:Well, basically I just moved the database to a much beefier machine.

Went from a dual AMD Athlon 2600 MP with 2GB of RAM and dual 80GB IDE HD to a dual Opteron 265 with 8GB of RAM and dual 160GB SATA HDs

The biggest improvement is probably the hard drives, but the increased ram is also very helpful.

This still isn't the optimal setup and over the next few months there will likely be another move when I get fully set up on a raid 5 machine with full replication.

Wow thats a big improvement. Especialy the memory should speed things up !

I'm excited about this!

Yeah me to somehow I allways start thinking of creating heavy data systems for tracking and gathering data from wow to a website when I see such specs !

on that subject, what ever happened to the thing that showed the last file processed upload time? So you could "estimate" how far behind you were :-p

http://www.warcraftrealms.com/dbstats.php

I estimate we'll be caught up by the end of the weekend, if not sooner.

Excellent! Looking forward to being up to date again

Nice specs on the machine and also looking forward to the catch up. Then the question I have, since you are processing faster, would you be thinking of increaseing the file size? or maybe make a small unmentioned buffer incase people go over the file size?

Think i went over on my last Submit since the process time has passed it and it was never processed.

Thanks
BW

oeh yes, its cleary catching up now =)

RAM would make the biggest difference, at least up to 4GB depending on platform, provided tuning parameters for the database are set to utilize the RAM without messing with the OS space. HDD speed would be second, especially bigger HDD cache, a RAID5 array should help provided the logic is handled by the card and not offloaded to the cpu, if the RAID card has a decent cache that helps a lot too. (and yeah, it all boils down to handling as much as possible in RAM and dumping it as sequentially as possible on the HDD, new hybrid HDD's should really boost performance for a lot of stuff)

Nice to see the increase in performance, I'll likely not contribute much now that I'm employed but I'll still lurk and play on weekends at least

Work is overrated!

Getting paid is not =x