Start of HT 2010

2010-01-25T15:24:00.000-08:00

Moved back to Oxford on Saturday of -1st week. Had a lot of fun moving the usual handful of boxes out of the car, over a few inches of ice and into my house trying to avoid the foot long icicles hanging from the guttering (welcome to student housing). Managed to arrive at exactly the same time as about 5 other people on my street which was great. As usual got Mum to buy me a steak in Cafe Rouge to help me survive the rest of term without raw meat.

The rest of 0th week was spent in the library (from Sunday onwards) doing vac work and revising for collections. My wake up call was cycling to the river in the snow to cox the first boat. Training was going well although we only were able to get one VIII out and that had two novices in. Although the morning outings got me out of bed they also subjected everyone I happened to see to my complaints about not being able to feel my toes until about 9pm, when they were thoroughly warmed up after watching four episodes of House curled up in a blanket.

Unfortunately Hilary Term didn't start so well. Had two collections on Friday which I did pretty poorly on. I'm entirely blaming that on feeling ill for most of Thursday (who knew snow was bad for you). By Sunday I felt much better, cured by running between the star wars marathon (all 6 films) happening in my basement and the library to get the last of the vac work finished.

Monday morning started with a 9am probability lecture, fun times! My diary looked horrific with about 16 hours worth of lectures scheduled. Considering I'm supposed to be taking 72 hours of lectures this term (which equates to 9 hours a week) I realised I needed to cut down on some of the courses. On Monday night I stole Mason (tutor) away from his dinner and we chatted about courses (unfortunately we were sat right at the bottom of the stairs leading to the dining hall. Subsequently most of the conversation was spent with me acknowledging half the college as they walked past, wondering why people were taking photos of the cat, and then trying to avoid the cat scratching my legs out). Was able to narrow it down to 80 hours which is far more sensible...

Tuesday night was formal hall - classic lamb shank (which I'm really starting to get fed up of), with a dessert of football in the bar followed by a cheeseboard of probability served in the library, going on until I can't remember what hour of the morning (probably not that late considering my track record of this term so far).

Wednesday turned out to be a pretty good evening. There was a surprise Captains' meeting (only a surprise since I apparently no longer process information in emails), so had to grab Dave (men's rowing captain) from the union and run down to a lecture theatre in Christ Church and listen to Pembroke and Christ Church whinge at each other for about an hour and a half. For me the meeting consisted of a lot of face-palming and wishing I had brought some work with me to make the time a bit more productive. On the way out, after locking all the captains inside the college gates, Dave asked if I'd eaten dinner yet. This evolved into us eating at Quod (which turns out is a really expensive restaurant on high street) and him paying for it. I wish being Boat Club captain was usually this decadent. On my walk home I bumped into all the freshers queuing outside NOPS (North Oxford Property Services) for the Jericho student housing release and did my Mother Theresa bit part by bringing out tea and hot water bottles. I have unfortunately not had either the mugs or the hot water bottles returned so I refuse to do anything charitable for the freshers again.

The main event of Friday was getting soaked THROUGH semi-waterproof tracksuit bottoms and my sexy Somerville rowing splash jacket whilst cycling the 15 minute ride from my house to kickboxing - then attempting to box in soaking wet trackies. On arriving back home I was so exhausted that even climbing up the stairs to the warm shower was far too much effort. Naturally I solved the problem by curling up in the basement and watching House (this term's addiction, almost half way through season 2 and we only started watching it a week and a half ago).

On Saturday the college had their first bop of Hilary term. For some reason it couldn't be held in college so I had to walk down to the OTHER END OF TOWN dressed as Evita. It was a good fun night, the venue was packed and everyone was in a good mood. Plenty of bad dancing was had.

Highlight of the week was on Sunday (yesterday) when all of Somerville seemed to have a hangover. As the only one exempt from this rule I was able to observe and laugh, although had an equally unproductive day until the evening when I was finally able to start writing the PHP program for facebook (with half of the time working on it spent figuring out why eclipse was being so temperamental).

Lessons Learned in PHPT Writing

2008-05-25T10:59:00.000-07:00

This is pretty much the same as the page from the PHP wiki (which can be found here), less some stuff which has been added by other people.

I decided to write this mainly because I won’t be writing tests as a full time job any more so am using this to pass the responsibility over to the new testers! But also with the impending TestFests and general focus on testing in the community thought it would be worthwhile to share my knowledge. Some of the points below are quite specific to the extensions that I’ve written tests for (array, ctype, dir, ereg, imap, mbstring, mysql and pcre), but hopefully will be useful for testing in general and should highlight the kind of things that should be focused on to produce a good test suite. If anyone wishes to add to, change or edit this list then please feel free, I’d be interested to see what other people have to say!

I'd like to thank Zoe Slattery (zoe), Steve Seear (stevseea) and Robin Fernandes (robinf) for their help with producing this.

General:

Keep basic tests basic: test a function with the minimum number of arguments, maximum number of arguments, and, if any flags or options can be set, each one of those (include any checks to ensure that they have been set).
Include files, although annoying for debugging reasons, save massive amounts of code reuse and keep tests clean and consistent.

If using a function from an include file, write a comment explaining what the function is doing to save a reader’s time.

Use include files for variables that will change, for example passwords and names of databases and email servers. If using these, remember to generalise expected output.

Read the documentation, write tests that reflect what the documentation says the function should do.

For example, in the directory functions the resource parameter in some functions is explained to be “opened using opendir”, try passing a resource opened with fopen(), this according to the documentation shouldn’t work (alas, it did).

Use titles (in the –TEST– section) that explain what the test is actually doing! These should be one line only
Tests should be intuitive, if something very complicated is happening separate a test out.

If being non-intuitive is unavoidable, COMMENTS ARE USEFUL.
Lay tests out in an easy to read fashion, remember that other people might actually read them at some point.

If the documentation says that behaviour is unpredictable, don’t write a test for that scenario.

Argument Types:

When testing different data types for arguments that require integers, be careful of supplying resources. If no error message is returned, there is a chance that the resource has been converted to an integer which will be unpredictable, thus making the expected output impossible to create.
When testing arguments that relate to the length or offset of something, pass negative numbers, as well as numbers less and greater than the length.
If a function takes a resource as an argument, test different types of resources (file system, directory, database connection etc.) in case there is no type check.

Expected Output:

This list is quite specific to using EXPECTF, the same ideas apply to EXPECTREGEX though!

Run a test multiple times to check if the expected output needs to be generalised to accommodate for variable output.
Don’t put comments in the test explaining what the expected output should be, that’s what the expected output section is for :-).
Use echo statements to separate parts of the expected output (e.g. a counter if using iterations etc.)
Remember that the expected output has to reflect what will come out of a test if everything is working. For example a test with a statement conditionally echoed out that has not been included in the expected output saying that something is not available, with a comment next to it saying “don’t fail if this happens” is nonsensical, the test will fail if this happens as the string is not in the expected output.
If var_dumping a string where some of the string is generalised (i.e. replaced with %d/%s etc), check the string length is not also variable.

Example:

string(%d) "The next bit might vary in length: %s"

Change numbers generated in resources and objects to %d.

Example:

object(C)#%d (1) {
["p"]=>
bool(true)
}

resource(%d) of type (stream)

casting a resource to a string: you don’t know how many digits the number will be, so character count has to be %d too!

string(%d) "Resource id #%d"

Try to write the expected output yourself, instead of copying and pasting from a console. This ensures that you don’t miss any odd/unexpected behaviour.
Watch out for line endings on different platforms and how that effects the character length of a string
Avoid using %a unless it really is unavoidable. I mean REALLY unavoidable.
Watch out for error messages in the expected output:

Change the file location to %s
Line numbers should be changed to %d, though this can mean bugs don’t get noticed (for example all errors saying “on line 0”). If leaving the line number in, ensure it’s the right one!
Some errors are platform specific, if they’re being generated using strerror(errno) in the source code then change the error to %s.

Specific Extensions:

These are specific to extensions I have tested, but could probably be used in other extensions too.

MBString:

‘unexpected’ behaviour can come from not remembering byte length vs character length in strings.

e.g. in mb_ereg the byte length of a matched part of a string is returned instead of the character length, so for example when 15 multibyte characters are matched, mb_ereg returned int(35). (see mb_ereg_basic.phpt).

base64 encode any multibyte strings that appear in the expected output.

Anything requiring an external server (e.g. MySQL, IMAP):

Use an include file where users can set their own username, password and server name as variables, then ensure that you use the variables throughout the tests.
Whenever any of these appear in the expected output, remember to replace them with %s (if using EXPECTF).
If something doesn’t appear to be working as expected, it could be that the server you’re using doesn’t support this functionality (to save bogus bugs).
If possible, test tests on more than one server before committing.

Test Case Generator:

This script was written by Zoe Slattery (thank you!) and can be found in the PHP 5_3 source code in scripts/dev/generate_phpt.php. Although it’s in the 5_3 source code, I think I’m right in saying that it can be used to create tests for all versions of PHP. I use it for all the tests I have written and have found it to be incredibly useful! The basic format of it is that you pass a function name and a type of test (either basic, error or variation) and it will generate a template for the test. Below is how I have it set up on my machine (windows), and how I use it.

Use on command Line:

php.exe generate_phpt.php –h

will give a good overview of how to use the test case generator, there are things that I set every time I use it so have that in a batch file.

gen.bat file:

cd <place to store generated tests>
php.exe <location of generate_phpt.php script> -s <location of source code> -f %*

I then type the function name and the type of test I want, e.g.

gen.bat array_key_exists –b

This will generate a basic test for array_key_exists. All I need to do now is fill out the EXPECTF section of the file!

Variation tests:

gen.bat array_key_exists –v

will generate variation tests. There will be one for each of the functions argument and an empty test. The numbered variations will iterate over an array of different data types and pass them as an argument, these tests are very useful! Again all that needs doing with these is changing the title (in the --TEST-- section), add a comment explaining what the test is doing and filling out the EXPECTF section. With the blank variation test is a template for any other variation tests the function needs.

Josie's Blog

Start of HT 2010

Lessons Learned in PHPT Writing