I was just searching for someone's history in my guild and than i saw that the char had no history , i also checked a few of my own chars and their history is also missing .
Also the page where my guilds guild exes (http://warcraftrealms.com/guildexes/402770) can be found is empty , and there were 1500+ guildexes the last time i visited that page .
History gone
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
All old data on the site that is connected to Character history is for awhile not available.
Beside what you listed user profiles are also not available.. [edit:mostly fixed]
SKIP THE WALL OF TEXT if not interested in the explainer
Here is what has happened and why (my belief not firm evidence.)
Over a period of time the Character history table has been tainted with corrupt character names.
This was little notice since only the people with those names would have wanted to look at the data.
Names based upon what is known in the computing world as Latin-1 characterset (for windows it is/was known as cp1252) having diacritic marks (such as Ç or ø or Ü for example) were being handled without much problems.
But names based upon for example Korean or Chinese or Russian just wouldn't work.
Then we moved from Rollie's older systems hardware and software to the modern versions on Metalbeast's system and we hit a problem that wasn't noticed, understood and handled well.
Back when Rollie started Warcraftrealms the computer world didn't handle languages of the world well.
First there was ASCII which handled English, then it was expanded to Latin-1 which could handle most of Western Europe.
After much false starts they settled on a new character mapping scheme called UTF-8.
With the older character mapping you could handle most languages by mapping them into 256 combinations or 1 byte of data per letter.
Considering that with Chinese and with Korean for example the number of unique characters(tokens) is in the 10's of thousands.
Since 10,000 is greater then 256 the mapping had to go from 1 byte to multiple bytes.
An example 'Unknown' takes 7 bytes in Latin-1, '알수없음'which is Korean for Unknown is 4 tokens and takes 12 bytes to store.
{Oddly enough my first sample attempt was with the word 'Server' the Korean equivilent took 2 tokens and only 6 bytes... so it wasn't a good example to show the encoding differences
}
If you can't see the Korean characters then your computer and browser doesn't know how to handle what has become the world standard encoding UTF-8.... I would be surprised if anyone reading this can't see the Korean.
UTF-8 opens up a world of opportunities.. but also a can of worms!
World of Warcraft is UTF-8 based which is why they can display the languages they do.
Unfortunately the site databases are almost all Latin-1.
Latin-1 is not equal to UTF-8.
In olden days if someone tried to upload non-Latin-1 based names they probably got dumped.. (I'm not aware of any Korean or Chinese named characters in the database.)
With the move to Metalbeast and his upgraded systems, Russian character names got into the system but got mangled and corrupt (remember UTF-8 is not Latin-1.)
I started noticing when I was given Forum Administration privileges and had to interact with more user names and character names then normal.
Metalbeast noted something was wrong during the two machine installs and lately while using database tools to do backups.
END OF EXPLAINER
The fix is in.... the western EU Realms, Races & Classes that Normally use Latin-1 have been fixed with changes in the Addon and website.
The Cyrillic Names used with the Russian client are all going away. When Russian WoW client users play on other language realms and are caught by census users. The name will be available but mostly unreadable in the game client, but the website will not be able to translate and will drop the character from the census.
Beside what you listed user profiles are also not available.. [edit:mostly fixed]
SKIP THE WALL OF TEXT if not interested in the explainer
Here is what has happened and why (my belief not firm evidence.)
Over a period of time the Character history table has been tainted with corrupt character names.
This was little notice since only the people with those names would have wanted to look at the data.
Names based upon what is known in the computing world as Latin-1 characterset (for windows it is/was known as cp1252) having diacritic marks (such as Ç or ø or Ü for example) were being handled without much problems.
But names based upon for example Korean or Chinese or Russian just wouldn't work.
Then we moved from Rollie's older systems hardware and software to the modern versions on Metalbeast's system and we hit a problem that wasn't noticed, understood and handled well.
Back when Rollie started Warcraftrealms the computer world didn't handle languages of the world well.
First there was ASCII which handled English, then it was expanded to Latin-1 which could handle most of Western Europe.
After much false starts they settled on a new character mapping scheme called UTF-8.
With the older character mapping you could handle most languages by mapping them into 256 combinations or 1 byte of data per letter.
Considering that with Chinese and with Korean for example the number of unique characters(tokens) is in the 10's of thousands.
Since 10,000 is greater then 256 the mapping had to go from 1 byte to multiple bytes.
An example 'Unknown' takes 7 bytes in Latin-1, '알수없음'which is Korean for Unknown is 4 tokens and takes 12 bytes to store.
{Oddly enough my first sample attempt was with the word 'Server' the Korean equivilent took 2 tokens and only 6 bytes... so it wasn't a good example to show the encoding differences
}If you can't see the Korean characters then your computer and browser doesn't know how to handle what has become the world standard encoding UTF-8.... I would be surprised if anyone reading this can't see the Korean.
UTF-8 opens up a world of opportunities.. but also a can of worms!
World of Warcraft is UTF-8 based which is why they can display the languages they do.
Unfortunately the site databases are almost all Latin-1.
Latin-1 is not equal to UTF-8.
In olden days if someone tried to upload non-Latin-1 based names they probably got dumped.. (I'm not aware of any Korean or Chinese named characters in the database.)
With the move to Metalbeast and his upgraded systems, Russian character names got into the system but got mangled and corrupt (remember UTF-8 is not Latin-1.)
I started noticing when I was given Forum Administration privileges and had to interact with more user names and character names then normal.
Metalbeast noted something was wrong during the two machine installs and lately while using database tools to do backups.
END OF EXPLAINER
The fix is in.... the western EU Realms, Races & Classes that Normally use Latin-1 have been fixed with changes in the Addon and website.
The Cyrillic Names used with the Russian client are all going away. When Russian WoW client users play on other language realms and are caught by census users. The name will be available but mostly unreadable in the game client, but the website will not be able to translate and will drop the character from the census.
Ouch. That makes a lot of sense, I'd wondered if Russian names in crossrealm groups would ever cause an issue; quite often see them on my server (clearly someone there has some Russian friends!), had hoped the system always dumps them, but if they're getting in after all then no wonder there's a problem.
Kinda funny looking at my character profiles now - there's some data... going up to about February 2007, then it stops dead. Nice blast from the past seeing which of my characters are old enough to have any data at all
Kinda funny looking at my character profiles now - there's some data... going up to about February 2007, then it stops dead. Nice blast from the past seeing which of my characters are old enough to have any data at all
Does this all tie into the following:
1) Several (over 20!) of my characters are gone, not just history, but completely erased from the site? As far as I can tell, several other people are missing, too.
2) Some of my characters are in thrice or twice? At least one (Neprionus, EU-Stormrage) has SEVEN entries, all at level 90? Some entries have IDs, all different, some do not... at least one does not have any actual info at all, just empty areas.
3) Both of my submissions from yesterday are listed as having failed (status E) despite being successfully uploaded?
1) Several (over 20!) of my characters are gone, not just history, but completely erased from the site? As far as I can tell, several other people are missing, too.
2) Some of my characters are in thrice or twice? At least one (Neprionus, EU-Stormrage) has SEVEN entries, all at level 90? Some entries have IDs, all different, some do not... at least one does not have any actual info at all, just empty areas.
3) Both of my submissions from yesterday are listed as having failed (status E) despite being successfully uploaded?
Although the announcement says that character history is back, and indeed it is for many of my characters, not all have it - http://www.warcraftrealms.com/char/193657531 is one of mine for example. Now, it wasn't among my claimed characters today (although I'm positive I claimed it some time back), perhaps its relative inactivity since Wrath days got it purged at some point, but I know I've logged it in in the last few months, while other relatively inactive characters of mine remain present.
I actually just logged onto a character in the guild to check last online times for my less active alts and I see 4 characters (quite possibly 5 if I include the one I logged onto!) who last logged on 5 months ago. Two of those remained in both the guild listing and my claimed characters, the others are nowhere to be found, searching for them on the site doesn't bring them up either. Given I always have Censusplus running, and Alliance side takes about 1 minute to run, there is no chance that they would not have been picked up by me.
If this is down to purging instead of a bug, then I'm not sure purging characters as high as lvl 70+ is exactly ideal? Fair enough on below lvl 10 bank alts and the like but real characters which had been max level at one point (and which do get logged in every few months in my case) losing their history feels a bit harsh. I'm hoping it's a bug rather than intentional!
Edit: further investigation finds six of my Horde characters are missing too. Alright, it takes an hour to census at peak times, so their logins might well have been msised, however I distinctly recall boosting one of them through two different dungeons late at night (when censuses take far less time!) no more than 2-3 months ago... so she should be there somewhere!
I actually just logged onto a character in the guild to check last online times for my less active alts and I see 4 characters (quite possibly 5 if I include the one I logged onto!) who last logged on 5 months ago. Two of those remained in both the guild listing and my claimed characters, the others are nowhere to be found, searching for them on the site doesn't bring them up either. Given I always have Censusplus running, and Alliance side takes about 1 minute to run, there is no chance that they would not have been picked up by me.
If this is down to purging instead of a bug, then I'm not sure purging characters as high as lvl 70+ is exactly ideal? Fair enough on below lvl 10 bank alts and the like but real characters which had been max level at one point (and which do get logged in every few months in my case) losing their history feels a bit harsh. I'm hoping it's a bug rather than intentional!
Edit: further investigation finds six of my Horde characters are missing too. Alright, it takes an hour to census at peak times, so their logins might well have been msised, however I distinctly recall boosting one of them through two different dungeons late at night (when censuses take far less time!) no more than 2-3 months ago... so she should be there somewhere!
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
First I must say that I know that Metalbeast has been very frustrated with the issue of database corruption.
The issue was bad enough that the standard database tools to backup the impacted table failed to work.
He has spent hours and hours trying to clean the old character history database to restore the integrity of the database.
What he had to do was recreate the character history section of the database using a different structure.
Then having moved the bad table out of the live database he manually pulled in data into the new character history area.
(and as a side note: if there isn't any character history, all of the rest of the data display for that character will not display.)
As you can see Metalbeast did a series of recovers selecting only the levels x0 and x5 levels starting with level 20 (this is from looking at my characters).
I too had missing characters.. and since I haven't been paying to play for a long time, I couldn't even remember some of my characters or even the guild names.
(Which is bad since I was GM for a couple of guilds).
I saw that Blizzard was offering me a 10 day trial reactivation so I pulled the trigger.
Looking at only characters level 20 or above and Death Knights at least level 60 my account has 24 characters that should have been censused at one time or another.
Of those 24 only 7 were recovered from old data! Needless to say I'm not very happy at this time
I am expecting Metalbeast to add some info directly here on the forums (instead of having it pass through me.) But I believe that he is back in real live make a living mode at this time.
One of the items I can point out is that we had around 117 Million character names, that has been reduced to 63 Million.
Some of which was intentional, all of the single character (bank/auction alt typically level 1-10) Guilds have been removed.
Since they aren't leveled it made sense to me also to remove them.
Even though I'm not a PVPer and don't do battlegrounds, I am hoping that Metalbeast will do more data recovery by doing another set of recovery runs targeting the 19,x4,x9 battleground level caps for those characters that became exclusively battleground PVP toons.
That is all from me for now.. my 10 day clock is ticking down.. back to rediscovering old pals (my toons) and seeing what they had missed.. [I'm questing on Mt. Hyjal at this point.]
The issue was bad enough that the standard database tools to backup the impacted table failed to work.
He has spent hours and hours trying to clean the old character history database to restore the integrity of the database.
What he had to do was recreate the character history section of the database using a different structure.
Then having moved the bad table out of the live database he manually pulled in data into the new character history area.
(and as a side note: if there isn't any character history, all of the rest of the data display for that character will not display.)
As you can see Metalbeast did a series of recovers selecting only the levels x0 and x5 levels starting with level 20 (this is from looking at my characters).
I too had missing characters.. and since I haven't been paying to play for a long time, I couldn't even remember some of my characters or even the guild names.
(Which is bad since I was GM for a couple of guilds).
I saw that Blizzard was offering me a 10 day trial reactivation so I pulled the trigger.
Looking at only characters level 20 or above and Death Knights at least level 60 my account has 24 characters that should have been censused at one time or another.
Of those 24 only 7 were recovered from old data! Needless to say I'm not very happy at this time
I am expecting Metalbeast to add some info directly here on the forums (instead of having it pass through me.) But I believe that he is back in real live make a living mode at this time.
One of the items I can point out is that we had around 117 Million character names, that has been reduced to 63 Million.
Some of which was intentional, all of the single character (bank/auction alt typically level 1-10) Guilds have been removed.
Since they aren't leveled it made sense to me also to remove them.
Even though I'm not a PVPer and don't do battlegrounds, I am hoping that Metalbeast will do more data recovery by doing another set of recovery runs targeting the 19,x4,x9 battleground level caps for those characters that became exclusively battleground PVP toons.
That is all from me for now.. my 10 day clock is ticking down.. back to rediscovering old pals (my toons) and seeing what they had missed.. [I'm questing on Mt. Hyjal at this point.]
Ouch, that makes sense, thanks for the explanation, and I'm glad it's not a purging policy, I'd have been annoyed if that had been the case! Database corruption, ok, stuff happens, irritating, but fair enough.
The x0 and x5 suddenly makes sense; I'd thought that one of my surviving characters was 43, but no, she's 45 - by pure coincidence all my high lvls are either 85 or 90, nothing in between, and nothing below till you reach the low 70s, so I hadn't noticed the connection there. However I do have three missing lvl 60s so not all were recovered...
The x0 and x5 suddenly makes sense; I'd thought that one of my surviving characters was 43, but no, she's 45 - by pure coincidence all my high lvls are either 85 or 90, nothing in between, and nothing below till you reach the low 70s, so I hadn't noticed the connection there. However I do have three missing lvl 60s so not all were recovered...
Is missing character and guild history still being worked on or not ?
My main character is still missing his history between 2006 and 2010 , other characters have gaps in other years , and my guild's history is still gone ; the only exes i can see are charcaters that left this year , evrything before this year is still gone .
My main character is still missing his history between 2006 and 2010 , other characters have gaps in other years , and my guild's history is still gone ; the only exes i can see are charcaters that left this year , evrything before this year is still gone .
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA