We've all tried it...
Playing Toon A, while Census is running, but you want to play toon B. However, your Census have been running for 30 minutes, and another 30 minutes is left before it finishes.
"Should I stay or should I go" plays in my head..
If I switch now, the first 30ish minutes on toon B will be wasted, in regards to Census. If I stay on toon A, I don't play the toon I want to.. Or maybe I just need a few things from my bank alt.
How about, if Census is running, when you log out of a toon, Census will resume where you were on the last toon.
Ofc, you need to be on same realm/faction for this to work, and maybe a 5 min limit when switching (so the realms haven't changed too much).
This will make Census much more smooth for the player..
I hope this makes sense..
Suggestion: Continue Census on another toon
Suggestion: Continue Census on another toon
Doing census mainly on Draenor; Raluf - Nimsay - Lusmo - Quixx - Sosyan - Garthog - Trubin - Zalistra - Zesmi and Djaang
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
This is an interesting suggestion.
Unfortunately besides the obvious user side issues you have pointed out, there are some technical problems that make this very unlikely to happen.
The biggest issues are:
First that when you log out a whole bunch of working state data for the addon goes away.
All of this would have to be restructured as tables to be saved on exit and I am unsure that would be practicable in the event that a player randomly logs out.
These are issues Blizzard controls not addon programmers.
Second it would require a complete rewrite of the census run queuing system to be able to handle starting randomly somewhere in mid progress.
I will put this into the possible TODO list... but to be honest it is far more work then I am currently willing to handle at this time.
Unfortunately besides the obvious user side issues you have pointed out, there are some technical problems that make this very unlikely to happen.
The biggest issues are:
First that when you log out a whole bunch of working state data for the addon goes away.
All of this would have to be restructured as tables to be saved on exit and I am unsure that would be practicable in the event that a player randomly logs out.
These are issues Blizzard controls not addon programmers.
Second it would require a complete rewrite of the census run queuing system to be able to handle starting randomly somewhere in mid progress.
I will put this into the possible TODO list... but to be honest it is far more work then I am currently willing to handle at this time.
That is a really awesome idea, which I'd love to happen, however I can see the problems with it, so appreciate it likely won't happen, it sounds a nightmare to code. I also have a feeling it doesn't affect that many of us; I think Fuxie, you're on Draenor Horde like me - there's probably only a handful of realms which take so very long to census (I saw 4600 in one census last night...), so of the regular people here, probably we're the only ones who'd actually benefit! 
The basic concept isn't that complex
Example:
CP does a /WHO command and writes to data file (as normal)
CP dumps the queue to a secondary data file
- data file contains:
- - Time stamp
- - Realm
- - Faction
- - Queue data
Rinse and repeat until "/census me" is run, then delete secondary data file.
When you start a census run, addon starts by checking if secondary data file exists, if it does, read it:
- If realm, faction and time is WRONG, delete the file and start over.
- Else resume census from data file.
Implementation, on the other hand.....
Example:
CP does a /WHO command and writes to data file (as normal)
CP dumps the queue to a secondary data file
- data file contains:
- - Time stamp
- - Realm
- - Faction
- - Queue data
Rinse and repeat until "/census me" is run, then delete secondary data file.
When you start a census run, addon starts by checking if secondary data file exists, if it does, read it:
- If realm, faction and time is WRONG, delete the file and start over.
- Else resume census from data file.
Implementation, on the other hand.....
Doing census mainly on Draenor; Raluf - Nimsay - Lusmo - Quixx - Sosyan - Garthog - Trubin - Zalistra - Zesmi and Djaang
4600 is nothingBalgair wrote:(I saw 4600 in one census last night...)
Today's census yielded this
Code: Select all
[20:30:07] Census+ Finished Taking data. Found 1693 new characters and saw 5260. Took 1 Hr 4 Min.Doing census mainly on Draenor; Raluf - Nimsay - Lusmo - Quixx - Sosyan - Garthog - Trubin - Zalistra - Zesmi and Djaang
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
I am probably going to be sorry about this...
But I am going to start the investigation on restarting census in progress on disconnect.
this would also cover logging out on one character and logging in on another in the same realm/faction.
No promise on timeline or even if this will ever go live.
Log out and disconnects under Blizzard control should be handled.
Client crashes or disconnects that Blizzard doesn't handle wouldn't save the needed data.
Also there will be a maximum time limit between stop and restart, if the duration is too long the restart will be treated as a fresh start.
Processing order will remain the same for now... as that would require a complete rewrite of that code to handle the new process flow.
I had/have a number of other changes that I want to implement in the code.
The base of this code is so old and programming practices in the World of Warcraft environment have changed over the years.
I want to get all of the event drive code into the same file so we don't have the odd ball fresh start race conditions.
I want to get rid of all or at least most of the existing .XML code since that code isn't flexible enough to handle quick changes.
While the new options interface has proven to be very flexible and extensible, since there are multiple interactions between various elements the coding was done as straight procedural code.
This isn't very code efficient. I used way to many lines of code, modern code reuse practices would shrink the file size considerably.
I want to tweak the code to get faster census runs... without triggering any limits by Blizzard... which will be interesting, considering Blizzard hasn't identified the actual limits as far as I know.
But I am going to start the investigation on restarting census in progress on disconnect.
this would also cover logging out on one character and logging in on another in the same realm/faction.
No promise on timeline or even if this will ever go live.
Log out and disconnects under Blizzard control should be handled.
Client crashes or disconnects that Blizzard doesn't handle wouldn't save the needed data.
Also there will be a maximum time limit between stop and restart, if the duration is too long the restart will be treated as a fresh start.
Processing order will remain the same for now... as that would require a complete rewrite of that code to handle the new process flow.
I had/have a number of other changes that I want to implement in the code.
The base of this code is so old and programming practices in the World of Warcraft environment have changed over the years.
I want to get all of the event drive code into the same file so we don't have the odd ball fresh start race conditions.
I want to get rid of all or at least most of the existing .XML code since that code isn't flexible enough to handle quick changes.
While the new options interface has proven to be very flexible and extensible, since there are multiple interactions between various elements the coding was done as straight procedural code.
This isn't very code efficient. I used way to many lines of code, modern code reuse practices would shrink the file size considerably.
I want to tweak the code to get faster census runs... without triggering any limits by Blizzard... which will be interesting, considering Blizzard hasn't identified the actual limits as far as I know.
Fantastic, it'll be a big help for us on the highest pop realms if it is possible to implement that one - I've just given up on letting my censuses complete now and if I want to switch character, I do, since 90% of the time it's in the neverending lvl 100s, would be great if I could jump to an alt without losing my progress. Totally understand if it doesn't happen in the end, but glad you're going to look into it
^^ QFTBalgair wrote:Fantastic, it'll be a big help for us on the highest pop realms if it is possible to implement that one - I've just given up on letting my censuses complete now and if I want to switch character, I do, since 90% of the time it's in the neverending lvl 100s, would be great if I could jump to an alt without losing my progress. Totally understand if it doesn't happen in the end, but glad you're going to look into it
Fantastic suggestion? It must have been suggested by some sort of genious..
Doing census mainly on Draenor; Raluf - Nimsay - Lusmo - Quixx - Sosyan - Garthog - Trubin - Zalistra - Zesmi and Djaang
I skimmed through the posts being the internet badboy I am, also did a ctrl+f to see if anyone mentioned it above and didn't seem to. In its curent state, does CensusPlus save the results it has seen when I press stop? The status text i get says something like Found x characters and 0 new.
Gah - Not only did I read any of the posts above but I can't even properly quote the text. My brain is not working properly today. Sorry guys.
Gah - Not only did I read any of the posts above but I can't even properly quote the text. My brain is not working properly today. Sorry guys.
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
On a census run two things are being recorded.
The character activity on the realm across all levels we can reach.
If you look at your data file this is the [TimesPlus] section of the table.
Since this is a full run snapshot summary based upon class it doesn't get recorded until the census run finishes the full queue of queries.
The [Servers] section of the table records each character that was captured by either the census run,
or by target focus,
or dragging your pointer over a character on your world screen with your mouse.
These are individual data points and do not need a census run completion or even on the later two cases a start of census.
When the client is issued a character left world event on log out,
the client per instructions in the .TOC file saves the CensusPlus table as it currently exists in memory out to the appropriate account sub-directory.
(And on login onto a character reads this data back into memory for the game.)
If the client crashes (doesn't happen much or at all these days) or you blue screen or lose power then nothing for any addon gets saved.
If you get disconnected from the server, the client gets the left world event and does its clean up saves.
If you press STOP on CensusPlus, data stops being added to the table.. but nothing is written to disk until the left world event happens.
The character activity on the realm across all levels we can reach.
If you look at your data file this is the [TimesPlus] section of the table.
Since this is a full run snapshot summary based upon class it doesn't get recorded until the census run finishes the full queue of queries.
The [Servers] section of the table records each character that was captured by either the census run,
or by target focus,
or dragging your pointer over a character on your world screen with your mouse.
These are individual data points and do not need a census run completion or even on the later two cases a start of census.
When the client is issued a character left world event on log out,
the client per instructions in the .TOC file saves the CensusPlus table as it currently exists in memory out to the appropriate account sub-directory.
(And on login onto a character reads this data back into memory for the game.)
If the client crashes (doesn't happen much or at all these days) or you blue screen or lose power then nothing for any addon gets saved.
If you get disconnected from the server, the client gets the left world event and does its clean up saves.
If you press STOP on CensusPlus, data stops being added to the table.. but nothing is written to disk until the left world event happens.
bringoutyourdead, not trying to stress you or anything, but almost half a year since last word...
Are you still working on this, and if so, are we getting close or a long way to go still?
Any highlights you can share?
Are you still working on this, and if so, are we getting close or a long way to go still?
Any highlights you can share?
Doing census mainly on Draenor; Raluf - Nimsay - Lusmo - Quixx - Sosyan - Garthog - Trubin - Zalistra - Zesmi and Djaang
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
It is in a galaxy far far away.
I am still trying to get the website to handle the guild/guild realm issue..
and also the current hour most needed...
and possible update on the very old version of phpBB
and cleaning up some security issues that have killed some member accounts,
since those are sharing the same code environments.
I have also started the process to check if WoW 6.2 will impact CensusPlus.
Due to issues, that took multiple days to get the 39GB of PTR code..
and then 2 days later with the next patch.. I got locked out.
Not that addon's are enabled for the PTR yet.. I should have the lock removed before then.
Oh and nearly every census run I have done lately has been to get the EU most wanted out of the single digit level.
I am still trying to get the website to handle the guild/guild realm issue..
and also the current hour most needed...
and possible update on the very old version of phpBB
and cleaning up some security issues that have killed some member accounts,
since those are sharing the same code environments.
I have also started the process to check if WoW 6.2 will impact CensusPlus.
Due to issues, that took multiple days to get the 39GB of PTR code..
and then 2 days later with the next patch.. I got locked out.
Not that addon's are enabled for the PTR yet.. I should have the lock removed before then.
Oh and nearly every census run I have done lately has been to get the EU most wanted out of the single digit level.
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
As you might have guessed, if you have been following my question posts lately, I have returned to working on the CensusPlus code.
I looked to see if we could return to collecting data outside of census runs.. and Balgair is correct.. unless Blizzard makes some API changes.. that is a no go zone.
I have done some work on resume from connection drop for CensusPlus and I think I can make that work.. but in does involve major work on the run time processing.
The Plan is if the following happens.. it will be possible to resume the census run on next startup if:
Run stops due to disconnect or logout and client drops back to character or WoW client login screen.
The following will remove any chance of resume.. Stopping Census Run, or crash of WoW client or computer.
Resume should happen if next login is on SAME Realm/ Same Faction and within a time window from last recorded /who action.
This will allow you to logout of one character on your realm list and login on another .. IF SAME faction.
The time window hasn't been determined.
If it has been too long since last census action.. characters will not be seen that could have been seen during the census run... and activity tracking will be compromised.
At this time I am toying with either a 5 or 10 minute delay window.. unless you cross to the next hour zone.
I your drop out to resume window happens at the top of the hour xx:00,
I am not yet sure how CensusPlus will record the activity data.. hour before or hour after.
I need to do more research.. but am inclined to just force new start instead of resume.
To make this happen a number of data items need to be changed from local in addon only status to Global status (in another datatable), as only Global data can be written/read to/from storage by the client.
This may have an impact on upload size if your last use of CensusPlus is not run to completion.
On completion the temporary data needed to recover and resume is no longer needed and is deleted.
As of last night:
I have most of the 'Can we resume' coding working.. and then
I trashed my modified CensusPlus.. error attempting to do addition on nul field.
If it was a typo error.. I may have it fixed soon.. or if not, I may have to rewrite more sections of the code.
One of the reasons I am not giving an estimate when these changes will be available to you.
I looked to see if we could return to collecting data outside of census runs.. and Balgair is correct.. unless Blizzard makes some API changes.. that is a no go zone.
I have done some work on resume from connection drop for CensusPlus and I think I can make that work.. but in does involve major work on the run time processing.
The Plan is if the following happens.. it will be possible to resume the census run on next startup if:
Run stops due to disconnect or logout and client drops back to character or WoW client login screen.
The following will remove any chance of resume.. Stopping Census Run, or crash of WoW client or computer.
Resume should happen if next login is on SAME Realm/ Same Faction and within a time window from last recorded /who action.
This will allow you to logout of one character on your realm list and login on another .. IF SAME faction.
The time window hasn't been determined.
If it has been too long since last census action.. characters will not be seen that could have been seen during the census run... and activity tracking will be compromised.
At this time I am toying with either a 5 or 10 minute delay window.. unless you cross to the next hour zone.
I your drop out to resume window happens at the top of the hour xx:00,
I am not yet sure how CensusPlus will record the activity data.. hour before or hour after.
I need to do more research.. but am inclined to just force new start instead of resume.
To make this happen a number of data items need to be changed from local in addon only status to Global status (in another datatable), as only Global data can be written/read to/from storage by the client.
This may have an impact on upload size if your last use of CensusPlus is not run to completion.
On completion the temporary data needed to recover and resume is no longer needed and is deleted.
As of last night:
I have most of the 'Can we resume' coding working.. and then
I trashed my modified CensusPlus.. error attempting to do addition on nul field.
If it was a typo error.. I may have it fixed soon.. or if not, I may have to rewrite more sections of the code.
One of the reasons I am not giving an estimate when these changes will be available to you.
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
I may have made the final changes to CensuPlus for the next release... or maybe not.
Since the US realms are done, I thought I would see if I could corrupt the EU data... no wait.. just kidding.
Not kidding
On my EU account, I have been able to start Census runs and then log out in the middle of the run where the data needed to do a resume of the run on login was correctly saved.
It took a little more coding today as the in game final tallies of new and seen again characters didn't quite match reality.
A purely in game summation problem, not a data collection issue.
It doesn't do much good to save the data if you can't correctly use it after Blizzard reloads the tables from the saved files.
That required much thought and work to make sure the correct actions and data were used if the actual start of census processing was a New Start or a Continuing action.
Also the Census finish processing had to be modified to make sure we didn't get caught in a continuing loop that would add more and more incorrect data.
At this time.. things look good, but I need to force all of the possible edge conditions.. so I can fix any problems before you users hit them and break the addon.
If things go well and no edge issues are found, I will do a push and make an announcement when the next version is available.
For those of you who don't see a need for this new resume on disconnect feature, you don't need to upgrade.
Both the current version and the next version will be acceptable for uploading to this site.
Since the US realms are done, I thought I would see if I could corrupt the EU data... no wait.. just kidding.
Not kidding
On my EU account, I have been able to start Census runs and then log out in the middle of the run where the data needed to do a resume of the run on login was correctly saved.
It took a little more coding today as the in game final tallies of new and seen again characters didn't quite match reality.
A purely in game summation problem, not a data collection issue.
It doesn't do much good to save the data if you can't correctly use it after Blizzard reloads the tables from the saved files.
That required much thought and work to make sure the correct actions and data were used if the actual start of census processing was a New Start or a Continuing action.
Also the Census finish processing had to be modified to make sure we didn't get caught in a continuing loop that would add more and more incorrect data.
At this time.. things look good, but I need to force all of the possible edge conditions.. so I can fix any problems before you users hit them and break the addon.
If things go well and no edge issues are found, I will do a push and make an announcement when the next version is available.
For those of you who don't see a need for this new resume on disconnect feature, you don't need to upgrade.
Both the current version and the next version will be acceptable for uploading to this site.
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
Owie! Owie! who took the Betadine!
After giving up on Blizzard and their Bnet issues.. I purged my Censusplus data.
And got cut on a lot of edges.
I am having to back fill and create the missing table elements for fresh installs and purged databases.
If Blizzard gets their issues resolved so I can do a number of census runs on larger population realms, I may be able to maintain my internal release date...
Otherwise it is going to slip into April.
After giving up on Blizzard and their Bnet issues.. I purged my Censusplus data.
And got cut on a lot of edges.
I am having to back fill and create the missing table elements for fresh installs and purged databases.
If Blizzard gets their issues resolved so I can do a number of census runs on larger population realms, I may be able to maintain my internal release date...
Otherwise it is going to slip into April.
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
Only found one edge that needed sanding and smoothing..
Start followed by Stop where two data fields hadn't yet initialized.. triggered infinite loop which needed full exit to kill off.
but still haven't been able to do multiple logout/continue testing on large pop servers yet... maybe this weekend if Blizzard actually fixes the problems they created.
Start followed by Stop where two data fields hadn't yet initialized.. triggered infinite loop which needed full exit to kill off.
but still haven't been able to do multiple logout/continue testing on large pop servers yet... maybe this weekend if Blizzard actually fixes the problems they created.
-
bringoutyourdead
- Forums Admin & general flunky
- Posts: 1432
- Joined: Fri Nov 07, 2008 1:11 pm
- Location: Texas, USA
Found a potential update killer... or maybe not...
I think I will release the update and have the user decide when to take advantage of the new feature.
The problem is, since we submit jobs from CensusPlus to Libwho to server and then wait for response from server, we can and probably will lose any data requested from the current active census queue job.
Once CensusPlus passes the active job to Libwho, the addon removes that job from the queue as it is Libwho that is responsible for insuring a response returns from the server or even if it has to resubmit the job multiple times if there isn't a timely response.
It is due to this process flow that you can see three apparently different job entries if you have the CensusPlus window open, have the CensusPlus button visible and have verbose logging. In order you see next job, last job and current job at the three locations.
Depending on where we are in the drill down of /who requests determines how much potential data loss we suffer.
Having the census run break at queue top LEVEL queries has the greatest potential for missing data.
As we go down the sub-query chain the potential for missing data gets less and less until we hit the lowest sub-query by letter in character name.
At the lowest level we have so much redundancy that we probably won't lose any data as the other 14 letter selectors will probably provide at least one match on the missed game characters.
Since the drill down only really happens at the Max character level (currently 100) and depending on how long since the last Expansion pack was released the previous Max level.
Only the most populated servers do the full drill down... and only these servers are most desired to have the break for character switch or disconnect.
The realms that need this feature the most will have the least data loss due to the break/resume process.
I think I will release the update and have the user decide when to take advantage of the new feature.
The problem is, since we submit jobs from CensusPlus to Libwho to server and then wait for response from server, we can and probably will lose any data requested from the current active census queue job.
Once CensusPlus passes the active job to Libwho, the addon removes that job from the queue as it is Libwho that is responsible for insuring a response returns from the server or even if it has to resubmit the job multiple times if there isn't a timely response.
It is due to this process flow that you can see three apparently different job entries if you have the CensusPlus window open, have the CensusPlus button visible and have verbose logging. In order you see next job, last job and current job at the three locations.
Depending on where we are in the drill down of /who requests determines how much potential data loss we suffer.
Having the census run break at queue top LEVEL queries has the greatest potential for missing data.
As we go down the sub-query chain the potential for missing data gets less and less until we hit the lowest sub-query by letter in character name.
At the lowest level we have so much redundancy that we probably won't lose any data as the other 14 letter selectors will probably provide at least one match on the missed game characters.
Since the drill down only really happens at the Max character level (currently 100) and depending on how long since the last Expansion pack was released the previous Max level.
Only the most populated servers do the full drill down... and only these servers are most desired to have the break for character switch or disconnect.
The realms that need this feature the most will have the least data loss due to the break/resume process.