Missing data for eu realms with accents

Ody · Post by **Ody** » Tue Jan 21, 2014 5:07 pm

French Realms:
- Chants éternels
- Confrérie du Thorium
- La Croisade écarlate
- Marécage de Zangar

German Realm:
- Festung der Stürme

Italian Realm:
- Pozzo dell'Eternità

Post by **bringoutyourdead** » Tue Jan 21, 2014 9:50 pm

Missing on the website? the Server Submissions page shows that all of the mentioned realms have some recent submissions.

or missing in the addon??...

and how far out of date is your version of the addon?

I added a pair of tables long ago to insure that these realms were picked up correctly
( and this was very definitely tested by our EU beta tester.)

If it is something else other then basic census data that is missing please explain further.. with samples if possible.

Balgair · Post by **Balgair** » Wed Jan 22, 2014 2:41 am

Missing in the site by the looks of things: http://www.warcraftrealms.com/census.php?serverid=762 looks rather on the blank side, and I've personally uploaded data to that one during the past week (can't remember exactly which day, but I know the data was there alright in the addon). Didn't think to doublecheck all servers were receiving the data, given server submissions was showing it ok, I kinda assumed the rest worked!

Balgair · Post by **Balgair** » Wed Jan 22, 2014 3:21 am

A little poking through my lua file to find any affected realms: Timesplus has: EU-Confrérie du Thorium, EU-La Croisade écarlate, EU-Chants éternels, EU-Pozzo dell'Eternità, EU-Festung der Stürme - those all look fine, as expected by the fact server submissions updates ok.

Going into the main data section: EU-La Croisade écarlate, EU-Chants éternels, EU-Marécage de Zangar, EU-Festung der Stürme, EU-Pozzo dell'Eternità, EU-Confrérie du Thorium, unfortunately no data under any of these as I pruned recently. I'll clear the lua and run again when the servers come up and see if it's on the addon end, doesn't look like it at present though. (Unless, in the past the addon stripped out accents to make a form the servers understood - but then, the server submissions would also be broken, yet it's fine)

Incidentally, I see the addon is well capable of handling Russian characters, as I have stray entries from several Russian realms, presumably from mouseover. Wonder how much work it'd take to get the site capable of it as well, I know there was at least one Russian person asking a few years ago.

Balgair · Post by **Balgair** » Wed Jan 22, 2014 4:16 am

Just censused Pozzo and Festung, lua file looks fine to me. My upload at 02:13:17 contains those realms, it'll be interesting to see how many updates I'm credited with; unfortunately Festung is connected, but there were only 91 found total, so I'll guess that very few updates will be found if Pozzo isn't processing properly (it had 200+ each faction).

Edit: well that was inconclusive, although I have suspicions that it's low enough to have been only the connected realms. Wiped the file and tried again with only Pozzo this time, upload time 03:02:54. Whether it records any updates or not may give some clues as to where things are going wrong.

Edit 2: Jan 22, 14 03:02:54 0 0 Y

So, it gives credit for submissions on the servers (proven by none being on 0 submissions), but doesn't process the data. The lua has the same spellings, accents and all, in both TimesPlus and the actual data, so it must be somewhere in the database processing?

Post by **bringoutyourdead** » Wed Jan 22, 2014 11:30 am

Re: Cyrillic
We still have the same problem that caused Rollie to decline making the required changes. When the mySQL database structure was initially set up he had to specify the coding of the character set allowed in the database. The only way to update that to include the expanded character set that recognizes Cyrillic is to:

backup the databases just in case.

unload all the data into temp holding files
blow the existing databases away

initialize all the tables again with the new coding. Oh by the way.. I expect that Rollie did this on the fly manually when he first set up the system so it isn't scripted... Metalbeast would have to recreate all this process by hand (again).

and then reload all the data again from the temp files.

And while the current hardware is much faster then what Rollie had... This whole process would be a multiple day procedure during which the site is down.

re: bad processing on site for realms with accents... I'm turning on the Bat light to get the Beast's attention.

Balgair · Post by **Balgair** » Wed Jan 22, 2014 12:04 pm

Ahh, I see, thanks for the explanation, makes sense then, sounds like way too much hassle under current circumstances.

Balgair · Post by **Balgair** » Tue Mar 18, 2014 9:46 am

Just thought I'd bump this thread since the site's being worked on - still no data visible on accented servers, even when they have a good number of submissions.

Post by **bringoutyourdead** » Wed May 14, 2014 3:20 pm

re-verified the problem.
the code that processes the finished census run section of the CensusPlus data
[TimesPlus] section of the data file is being handled correctly.
This means we have server counts and Activity numbers that are valid.

The code that process the actual character information is failing on the accents and listing them as unknown servers.
I should have the specific problem found shortly and will submit my suggested update to Metalbeast to get this fixed.. It may take me a day or two.. and ? for Metalbeast to implement.

Post by **bringoutyourdead** » Wed May 14, 2014 8:16 pm

I think I found the bug...
I made corrections and submitted to Metalbeast for 2nd check and approval.

Have to admit I'm not fully sure why the change is needed but I expect that oops there is a class involved.. [never mind .. programming joke]

Post by **bringoutyourdead** » Mon May 19, 2014 12:42 pm

After further study the fix I suggested failed to work..going deeper I discovered that the accent character problem is a larger problem then first recognized.

Class names which have accents are not being processed correctly
Newer language options were not coded even when the character set wasn't the problem (mxES, ptBR, itIT).

I've asked Metalbeast to modify the logging of uploaded file processing to allow me to get a better grasp of the problems.
I've gotten a pretty good idea of the process flow here on the site.

Most processes are triggered by the search queries that users issue.
The two upload processes work through the validation the file and subsequent relocation to a holding directory.
A system timer process kicks off every xxx number of seconds and if new upload(s) exist and one isn't already in process the oldest upload gets picked for processing.
Also (though I haven't confirmed this yet) other system timer processes get triggered at regular intervals to create in advance the graphs and other larger data tables that can't be created in reasonable time due to user requests.

Post by **bringoutyourdead** » Thu Nov 13, 2014 3:09 pm

Sigh... this problem is back again:
EU-Pozzo dell'Eternità is getting corrupted in processing character data and failing as eu-pozzodell\'eternità, the \' is valid for escaping the '.. the problem is the space going missing between Pozzo & dell.
EU-Aggra (Português) is also showing issues again.. it is suppose to get truncated to EU-Aggra instead the failures are showing as EU-Aggr.

I thought both of these had been fixed.. sigh.

also a couple of feminine forms are not being converted.. that may be my problem.. I'll need to check.

edit: and I found the problem.. it is the users....
some people are not completely deleting the old CensusPlus and replacing all the files, or they have real old data in their character data and need to purge their local database.

I am seeing good processing and bad processing of all the problem realms and feminine to masculine conversions.

I will be sending more PM to the users that I can identify having data problems.