Wednesday, July 29, 2009

Automated Testing

Testing is a very important part of software. Automated tests let you know that your code is working. I was reading Wagic’s blog and he mentions how constructs tests. (Wagic lets you play Magic against the computer on PSP or Windows.)

The way Wagic tests Magic cards is very simple and easy to understand. The tests are in plain text which makes the tests probably shorter than if they were coded in C++ which is the language that Wagic uses.

The tests consist of an init section, init is short for initialization or beginning, a do section which plays a card, and the assert section which checks to make sure that the card did what it was supposed to do.

I don’t usually cut and paste anything but the blog does a good job describing the parts of the test.

Wagic Blog
Wagic Card Testing

INIT
When it loads a test file, the Testing Suite initializes the game with the data from the initial state. So for example:

[INIT]
FIRSTMAIN
[PLAYER1]
hand:Ancestor’s Chosen
graveyard:swamp,grizzly bears,dragon engine
manapool:{5}{W}{W}
[PLAYER2]
graveyard:black knight

The above init state is loaded by the testing suite. The Game engine is “forced” to go to the “first main phase” of player 1. Player 1 has {5}{W}{W} in his manapool, Ancestor’s Chosen in his hand, and 3 cards in his graveyard. Player 2 has 1 card in his graveyard. Obviously what we want to test here is Ancestor’s Chosen ability: When Ancestor’s Chosen comes into play, you gain 1 life for each card in your graveyard. The obvious advantage is that I don’t have to create a specific deck with Ancestor’s Chosen to playtest it in order to test its ability.

DO
As I said in my other article, the testing suite is like a CPU player, except it is very dumb: Instead of computing the best moves for its cards, it just takes orders from the test file. This is the second part of the test file for Ancestor’s Chosen:

[DO]
Ancestor’s Chosen

It looks simple? Well it is. What I tell the “dumb AI player” here is: just click on the Ancestor’s Chosen card. And that’s it. Since player 1 has the mana in its mana pool, and it’s his first main phase, clicking on the ancestor’s chosen card will put it into play.

ASSERT
Now that we’ve done the actions needed in the test, what we want is to compare the new state of the Game, with what we expect to happen. We write what we expect to happen in the third part of the test file:

[ASSERT]
FIRSTMAIN
[PLAYER1]
inplay:Ancestor’s Chosen
life:23
graveyard:swamp,grizzly bears,dragon engine
manapool:{0}
[PLAYER2]
graveyard:black knight
[END]

So, if you compare this to the initial state we described above, the differences are the following: Ancestor’s Chosen is in play, player 1 has no more left mana in his manapool, and he has 23 life (initial life of player 1 wasn’t described in the initial state. In that case the game uses the default value of 20).

Now, this is what the test expects to happen. The testing suite will of course compare this to the actual state of the game. If the expected and the actual states are not equivalent, then the test fails and the program lets me know about the error.

Here is the full test file for Ancestor’s Chosen:

#Testing Ancestors chosen
[INIT]
FIRSTMAIN
[PLAYER1]
hand:Ancestor’s Chosen
graveyard:swamp,grizzly bears,dragon engine
manapool:{5}{W}{W}
[PLAYER2]
graveyard:black knight
[DO]
Ancestor’s Chosen
[ASSERT]
FIRSTMAIN
[PLAYER1]
inplay:Ancestor’s Chosen
life:23
graveyard:swamp,grizzly bears,dragon engine
manapool:{0}
[PLAYER2]
graveyard:black knight
[END]

6 comments:

Xuelynom said...

I've looked at your code and wagic code, why do you put all your classes in the same directory ? wouldn't it be easier to regroup GUI classes, engine classes, AI classes in their own package ?

willow said...

@Xuelynom: I started Wagic with very little idea of "what" it would become. Initially there where maybe 10 files tops, and I didn't see the point of putting them in several directories and/or packages.
Actually I still wouldn't see the point now, except I should have gone with better naming conventions. Every class AI related should start with "AI", etc...
So, short answer: no specific reason, really, but I'm not sure separate into different packages would solve anything really...
Is your nickname a reference to a famous game developer, by the way?

@forge:thanks for the article :)

Radek said...

Excellent method of testing! I work on CCG engine too (not for MTG) and have many problems with card testing, cause adding any new ability to engine can ruins previously working things. My testing methodic is following: I use two AI players, set the initials conditions for game (fill game zones – hand, game etc., set other game params), artificially set testing cards costs to high level for AI choosing algorithm, run the game and watch detailed logs. So this is not the fully autotest… (
I think I can try to use the same method for my engine, thanks for idea! )
@willow:
Just one question: for me the main problem is not the one card abilities works, but the abilities interactions. So, how you solve this type of problem? (I mean complex game scenarios, when one card interacts with another and may take effect at next turn etc.) Do you write a long test macros or something like that?

p.s. Sorry for “broken” English )

willow said...

@Radek: yes.
Basically, in the [DO] section, I just ask the engine to "click" on some elements of the GUI. so if I write "ancestor's chosen", it means "click on ancestor's chosen". But if I want to test end of turn triggered abilities, I have another keyword "next" to go to next phase, "choice n" whenever there is a dialog box with a choice to make, "yes" if I want to interrupt, etc...
You can have a look at more complex tests on Wagic's SVN.

Radek said...

@willow
Thnx – already looked some tests.
persist2.txt is awesome )) – is that I mean, thanks again, I’ll take it into account.

Xuelynom said...

Well the point is organisation... it's easier for someone new in the project to make changes when he knows that all and only the card panel classes are in mtgforge.gui.components. It doesnt solve anything in the same way that putting the plumbing in the bathroom doesnt help the plumber, but it helps him to know where to plumb :D

Xuelynom is Molyneux with an error, a child coming from a Gotlib book ( http://www.marcelgotlib.com/ )