user-tailored "most wanted list"

[uigrad]

When people use the most wanted list, it's good for the site (because it fills in the holes), and it's good for the user's stats (since they'll find a lot of users that need updates). But, to use the most wanted list, I generally have to delete characters I've made on other realms, create a new character for the realm, and then run the census.

I have low level characters on a bunch of realms, and don't want to make any more, so I prefer to check how many updates (in the last 30 days) exist for my realms, and use that to decide which ones need a census.

I built a simple perl script that will query the realm data pages at WR, and show me the number of updates over the last 30 days for each of my realms:

Code: Select all

#!/usr/bin/perl
@serverlist = qw(741 783 80 42 85 654 32 77 21 26 46 67 43 535 57 49 559);

foreach $id (@serverlist) {
    $document = `lynx -nolist -pseudo_inlines -dump "http://www.warcraftrealms.com/census.php?serverid=$id"`;

    @doc = split("\n", $document);
    $server = IndexOf("Server: ", \@doc);
    $ratio = IndexOf("Ratio: ", \@doc);

    printf ("%4sA,%4sH %s\n", $doc[$ratio+2] =~ /\s+(\d+) : (\d+)\*/, $doc[$server] =~ /(.*) GO/);
}

sub IndexOf {     # pass in value, array reference
   my ( $value, $arrayref ) = ( shift, shift );
   foreach my $i ( 0 .. @$arrayref-1 )  {
      return $i if ($$arrayref[$i] =~ /$value/);
      }
}

Output looks like:

Code: Select all

  14A,   7H    Server:      [Antonidas................]
  92A,   5H    Server:      [Arathor..................]
  31A,  61H    Server:      [Burning Blade............]
   2A, 133H    Server:      [Crushridge...............]
  17A,   6H    Server:      [Durotan..................]
  23A,   5H    Server:      [Executus.................]
  47A,  68H    Server:      [Feathermoon..............]
  20A,  45H    Server:      [Hellscream...............]
   6A,  24H    Server:      [Illidan..................]
  40A, 111H    Server:      [Mal'Ganis................]
 258A, 355H    Server:      [Medivh...................]
  59A,  36H    Server:      [Shattered Hand...........]
 160A, 312H    Server:      [Silvermoon...............]
  16A,  17H    Server:      [Smolderthorn.............]
  89A,  45H    Server:      [Stormrage................]
  42A,  62H    Server:      [Stormscale...............]
 166A,  65H    Server:      [Twisting Nether..........]

Based on that output, I can see that Crushridge has had plenty of horde updates (133) recently, but only 2 alliance updates. It would be a prime place for me to run a census.

Since the warcraftrealms.com site knows the realms that I have characters for, it would be nice if it could print such a customized list for me.

Rollie, I know you're unlikely to implement this because of all the other things on your wishlist, and that's fine. I'm hoping someone else finds my script useful, though. It should run fine from any command line in unix. Windows users might be able to run it using cygwin.

[SannaSK]

you are brilliant. i was just wondering if there were some way i could do this. no way could i have built this code on my own. now i just have to find some way to run this. can i put it in a ... webpage somehow? i have WinXP64, i'd like to avoid installing an entire program just for this.

When I started making my armyhorde of alts (all named Jaecen, as my main is Jae and I'm taking census : p), I realized real quick I was going to lose track of who-what-when-where-why-omfg, so I bookmarked each server in a folder in Firefox, and just thought, "well, I can open them all in tabs." And Firefox went "lolnoob you're opening 26 tabs, are you sure you want to do this?" and i kind of went .

so you get massive internet cookies for this. yay! now i just have to do something with it.... *ponders*

[Rollie]

Slick script =)

[SannaSK]

Noobcoder's Misadventures with Perl.

So first I went to my website and made a wr_cen.pl file in the base directory, into which I C+Ped the entirety of the script there, no alterations. When I attempted to load the page, I got a 500 Internal Server error. Asked a couple of friends, moved the file to the /cgi-bin/ folder, and got a blank white page.

Headkeyboarded for a bit, then broke down and installed Cygwin, as well as another program called Strawberry Perl. Opened them, attempted to copy-paste the script directly into the black-background box, and all that came up was a ^V. My friend said 'roflnoob, you have to put the code into a .pl file on your computer.' I said, 'oh, well that makes sense at least.'

So I've got this wr_cen.pl and durned if I can figure out how to make the little command line box thinger access it. I am such a perl noob. how i learned html, php, mysql is beyond me --owait, i am epixphail at php/mysql, that probably answers that-- but yeah, to anyone who doesn't already know what to do with a perl script, don't try this at home yet. When I figure out what to do (which really means 'when my more knowledgeable friends tell me what i'm doin rong), I'll edit this post with Direcshuns For Perl Dummies.

Until then, I guess I just, uh... open my 26 tabs... : <.


edit 1, friday morning. my friend says I need to install Lynx, a standalone text-only browser. Do you suppose if I alter that to be 'firefox,' would it still work? Maybe that's why my attempts at running it on my webhost didn't work, it was looking for this lynx thing that didn't exist : p

edit 2, friday midnight. Ok, my friend figured out that when I installed Cygwin, all the alternate packages that I thought were set to install were specifically set to /not/ install. Fixed that by rerunning the setup and installing perl (4 reelz this time) and lynx. currently stuck at the 2 "bad interpreter" errors. still working.

edit 3, sat noonish. the internet makes me think there's something with Windows CFLRs in my notepad file that's screwing up the command line's ability to process mah file. Just to be clear, my friend can run the script fine on her linux box; i'm not trying to say my confusion is coming because there's something wrong with the script. it's totally not, the script is fine. everyone else can run it... just not me, because i'm an effing noob : p.

edit 4, sat noonish but about an hour later than the above. progress! typing 'perl c:/tempy/wr_cen.pl' stopped giving me error messages and just did nothing. (that's pretty sad when that qualifies as progress.) i searched for 'lynx' in the cygwin folder and got nothing that remotely looks like an EXE file, which i find strange but don't know what to do about. in the $document line of the .pl file, i replaced 'lynx' with 'firefox' and ran the command again. got repeating lines of
"sh: firefox: command not found
A, H"
so it's halfway working. I don't know what I did that it was giving me the error messages and then it wasn't.

Have primarily come to the conclusion that this is not a recommendable way for your average Warcraft Realms census taker to create their own customizable most wanted list. : p. However, still working on this.

edit 5, about 2 minutes later than above. First I'm going to multiclass to rogue just long enough to stab Cygwin in the faise, then truss it up by its toenails and stab it some more in other soft parts of its body. It specifically /did not install/ the Lynx.exe and related folders, because i'm missing some other .dll somewhere. ARGH. still working.

EDIT 6, about 15 minutes after above: SUCCESS, so far as I can tell. I can't open the lynx.exe directly, it QQs about can't find a 'cygcrypto-0.9.8.dll' file (which exists just fine in the cygwin/bin directory). i reinstalled the packages i had chosen before, chose a different mirror, and all of a sudden it WORKS now. the Cygwin command line gives me the nice happy text display like it does above. Mwahahaha!

Damned if I know what I did tho, esp in regards to the suddenly-disappearing 'bad interpreter' returns. But it works now! Yay!!

Next on the list: figure out why UniUploader hates me. but that's for another thread : [.


edit 7, about 30 minutes later. I lied. the command line box queries two servers and locks up. *sigh* i give up. in this amount of time, i could probably have coded a php script to do everything server-side and just handed it to rollie. well maybe i could have. i would have been far less clueless about this whole mess. *tears some hair out*

[Rollie]

So I thought I'd give it a shot myself.

Copy/Pasted the above code into test.pl on my server. From the command line (I use putty), ran

perl test.pl

And it worked like a charm!

Do you have shell access to your web server?

[SannaSK]

Shell... my webhost runs CPanel, and while there is an option that says "SSH/Shell Access," clicking it only gives me an option to set up an SSH key, which I did in the hopes it would then give me /another/ option which it did not. Which is odd, because the video that it shows clearly displays the test-user having the ability to open a shell window from that page. Maybe I have to go enable something somewhere else.

TLDR i do 'have' shell if only i can figure it out.



Edit.

ok. What ended up happening is this.

After myself and my wonderful friend J spent 2 days bashing our heads against the Cygwin install on my machine, my other friend M shows up and tells me to put the line

print "Content-type:text/html\n\n";

on the line before

foreach $id (@serverlist) {

and SOMEHOW. SUDDENLY. the /cgi-bin/wr_cen.pl file on my webserver works. Not the one on my machine that I tried to use with Cygwin, but the one on my webserver randomly works now. I'm one enterbreak short, because I get the results as a long single line of text, but that works for my personal preference anyway. I C+P all of it into a notepad anyway so I can delete the server's line when I've taken census on that server today.


This is not a viable solution for personalized most wanted lists for individuals who aren't already very familiar with things like 'perl, scripts, cgi-bin.' If you have to install anything on your computer that you're feeling iffy about, this is not for you (unless you have a really good friend willing to hold your hand through like every fraggin step, and then some). On the other hand, once it works, it works like a charm (assuming your Cygwin doesn't freeze up every single time).

Thanks very much, OP. b^_^b Army of census alts, release the hounds!

[uigrad]

I'll put out a shameless plug for freeshell.org. They let you open a shell account for a dollar (used to be free, but they had problem with scammers).

Step 1) download putty (best freeware terminal for windows)
Step 2) use PuTTY to telnet to freeshell.org, use 'new' as account name.
Step 3) send a dollar bill to the address it gives you to finish account creation
Step 4) run any perl scripts or CGI scripts that you want.

Alternatively, you could use silenceisdefeat.org. I think it is similar to freeshell.org, and it lets you start an account with a dollar, and allows you to send it through paypal. I haven't used it as much.


edit: by the way, your friend is correct about the Content-type. If you serve static documents, then the webserver uses a combination of things (file extension, local .htaccess file, global settings) to figure out how to serve it. If the document is generated by a CGI script of any kind, then you need to provide the content header yourself.

Alternatively, you could put this at the top of your document:

Code: Select all

use CGI qw/:standard/; 
print header();

[SannaSK]

Pardon my necro on this thread X D

when editing a .pl, does the encoding type matter? After much stabbiting and a great deal of assistance from more knowledgeable friends, I finally got this script to run on my website. After a long absence away from CensusPlus, I picked it back up again and went into my website to try to update the serverlist with the servers I had chars on.

And now the .pl won't load at all. It gives me a giant string of
A, H
A, H
A, H
and no actual data. I'm rather distressed, as my only other option is the unwieldy method of bookmarking each server in warcraftrealms.com and opening them all in tabs, which seems like an unkind thing to do to Rollie's bandwidth.

/all/ I did was open the .pl, paste in the new string of numbers, and save. The only thing I can think is that the encoding type matters. It's currently set to us-ascii. Is this good? Bad? Did it break for some completely unrelated reason?

[Hybuir]

If someone has a working site with this script, this may just bring me back to the board lol

[Kosh]

@SannaSK, since there aren't any non-ASCII characters in the script, the character encoding shouldn't matter, as long as it's some form of "Western" encoding. It's more likely that you somehow broke the script with your alteration, or broke the line-endings by using a bad editor (I believe Notepad does this). UNIX uses line-feed (^J), and MS-DOS's carriage-return / line-feed combination tends to break a lot of UNIX tools.

If you still have your original version of the script, try running it, instead. If you still get the blank "A, H" lines, that would indicate that lynx isn't working for you anymore (or can't reach the website).


Now, thanks to Hybur for bumping this, I decided to try this script out for myself. I like what it does, but realized that I am also interested in the "Update dates". Here's my modified script to add the two dates at the end of each line:

Code: Select all

#!/usr/bin/perl
@serverlist = qw(741 783 80 42 85 654 32 77 21 26 46 67 43 535 57 49 559);

foreach $id (@serverlist) {
	$document = `lynx -nolist -pseudo_inlines -dump "http://www.warcraftrealms.com/census.php?serverid=$id"`;
#	$document = `cat tmpcen.txt`;

	@doc = split("\n", $document);
	$server = IndexOf("Server: ", \@doc);
	$ratio = IndexOf("Ratio: ", \@doc);
	$update = IndexOf("Update ~ ", \@doc);

	chop( $serverline = `echo "$doc[$server]" | sed -e 's/  */ /g;s/\] GO.*/]/;y/./ /'` );

	printf( "%4sA,%4sH %s (%s,%s)\n", $doc[$ratio+2] =~ /\s+(\d+) : (\d+)\*/, $serverline, $doc[$update] =~ /.* ~ (.*)/, $doc[$update+1] =~ /.* ~ (.*)/ );
}

sub IndexOf {		# pass in value, array reference
	my ( $value, $arrayref ) = ( shift, shift );
	foreach my $i ( 0 .. @$arrayref-1 )  {
		return $i if ($$arrayref[$i] =~ /$value/);
		}
}

And here's the output from that serverlist:

Code: Select all

 328A, 375H  Server: [Antonidas                ] (2010-5-22,2010-5-22)
 141A,   4H  Server: [Arathor                  ] (2010-5-23,2010-5-18)
  13A,  10H  Server: [Burning Blade            ] (2010-5-23,2010-5-23)
   3A, 217H  Server: [Crushridge               ] (2010-5-21,2010-5-21)
  27A, 177H  Server: [Durotan                  ] (2010-5-22,2010-5-21)
 147A,   4H  Server: [Executus                 ] (2010-5-19,2010-5-20)
  29A, 190H  Server: [Feathermoon              ] (2010-5-23,2010-5-23)
 170A, 489H  Server: [Hellscream               ] (2010-5-22,2010-5-22)
 127A, 118H  Server: [Illidan                  ] (2010-5-20,2010-5-20)
  52A, 184H  Server: [Mal'Ganis                ] (2010-5-22,2010-5-22)
  28A,   7H  Server: [Medivh                   ] (2010-5-17,2010-5-15)
 325A, 103H  Server: [Shattered Hand           ] (2010-5-23,2010-5-23)
  29A, 225H  Server: [Silvermoon               ] (2010-5-23,2010-5-23)
   5A, 317H  Server: [Smolderthorn             ] (2010-5-10,2010-5-18)
 138A,  48H  Server: [Stormrage                ] (2010-5-22,2010-5-22)
   2A,   3H  Server: [Stormscale               ] (2010-5-19,2010-5-18)
  44A, 161H  Server: [Twisting Nether          ] (2010-5-22,2010-5-22)

I also didn't really like all of the extra space around the Server (and the dots after the name), so I took the opportunity to "fix" those, as well. As a bonus, this keeps each line under 80 characters, so it will still fit on even a narrow (80-column) terminal. Note that I used "sed" to accomplish this, so if your installation somehow has Perl and lynx, but not sed (!?), this version won't work for you.

If you liked the dots, but otherwise like this version, you can just remove the ";y/./ /" from near the end of like 13.

[Hybuir]

moar dots! (I kid)

[Kosh]

moar dots! (I kid)

Oh don't worry, I can whip up an extra-special, "moar dots" edition, just for you

[Kosh]

I was thinking about all of those Windows people who can't run scripts, and since it will probably be even longer than Blizzard's definition of Soon? before Rollie implements something like this on the site, I decided to whip up my own web-service.

Now, I'm not a GUI person, so it isn't "pretty", and it's on a personal machine, so if it starts getting hammered, I'll just pull the script, but it can be found at http://mackosh.dyndns.org/~kosh/censerv/censerv.php.

How it works is, you pass the list of serverIDs you are interested in to the script by specifying "?servList=" and your list of IDs in the URL. You can use commas or spaces between the numbers, but since spaces are illegal in a URL, your browser should replace them with "%20", making the list longer and harder to read.

I make no effort to manage cookies, so you will always need to specify the "servList", but you can bookmark it once you have the list the way you want.

To pick on the OP's server list, here's what the URL would look like:
(big long url with long list of numbers -- hover or edit post to see it (or click the link))

I implemented a cache, so if multiple people ask for the same ID (or you keep hitting "refresh"), Rollie's server won't keep getting hit. I've currently got the cache time set at an hour; how often does the data behind those pages get updated, Rollie?

I haven't gotten around to implementing "dots" / "moardots" mode, but I'll mention one of the options that is there now: "verbose". The page defaults to verbose, but you can make it slightly less so by appending "&verbose=false" to the URL.


[Edit]prettify second link so it doens't force the page width out[/Edit]

[Kosh]

Well, I've already got a hit from GoogleBot, which encouraged me to get robots.txt set up for that domain (I have metadata in the page not to cache, etc. it, but of course the page is already loaded by then).

To make sure that other spiders (whether well-behaved or not) won't kick off queries when following the second link above, I've added another option to my script, and edited that link to add "&cache=only". This will return the cached information, even if it has expired.

You (probably) don't want to enable that option for your normal use.

[1974ER]

Thank you Kosh, I bookmarked the page with my little personal list.

[Kosh]

You're welcome Being an "Epic Censi", is it really a "little list", though?

I'm glad it's useful to some people. As of when I write this, 29 different server IDs have been viewed (including 1, 2, 3, 4, the example list I give you with no list ), so we'll see if more people notice it / start using it.

Oh, and I masked the second URL in my prior post, so I didn't have to side-scroll anymore.

[1974ER]

Hehehe :D:D , at least for the time being, yes. I just wanted to test it and put in just four servers on which I actually play on, not to overload your computer or WCR.

Edit: I will probably increase it to about 20 later on, to cover additionally those realms where I have "permanent" census alts. As for being "Epic Censi", well, Rollie deemed it appropriate to surprise me with the title a while back. As for reaching the figures I have... well, I have gone through quite a large number of deleted level 1 alts. Having only one account and an old computer to work with places its limits on censusing efforts.

All and all, I am pretty happy to note that I might break the 10M updates barrier near the end of June / early July, depending on how big a hit trying to do lots of Midsummer Fire Festival quests will take on my submission figures during that time period.

[Kosh]

Don't worry. I query each page from WR sequentially, so more numbers just take longer. The bigger load would be if multiple people are all querying different (uncached) numbers at the same time.

I don't currently impose any limit to the number of serverIDs you can request; the only limits are system limits on URL / variable length. I know that 17 IDs work just fine, since that's what is in the example URL.

It's looking like Rollie's process to update the server-level stats runs sequentially across the servers, since I have noticed that different servers get refreshed at different times. That probably throws out any kind of "smart" caching (when the cache lasts longer when no updates are expected), but I might bump up the cache life to two or three hours.

Good luck with your "army".

Oh, and we're up to 34 unique IDs now (and someone discovering that "All" isn't a valid ID :p ).

[FuxieDK]

and someone discovering that "All" isn't a valid ID :p

That would be me

[Kosh]

and someone discovering that "All" isn't a valid ID :p

That would be me

Seriously, even if I were nutty enough to implement "All", would you really want such a page?

Don't worry, it's a good test of my input validation.

Speaking of validation, the generated pages do validate, amazingly enough.

Oh, and Hybuir, just for you, I've added a "very special" feature... Just add "&moardots" to a query.