Tuesday, 9 February 2016

What are you smoking? What is the real meaning of 'Smoke Testing?'

As a mathematician (sorry!) I have a keen interest in defining terms before they are used - it's kind of become a habit.

That's why things like 'I was so shocked I literally died' really grind my gears (see also 'bricked', 'decimated' . .)

One term I hear multiple meanings of often is 'Smoke Testing' - I was discussing this with Dave Hay (highly recommend his blog at http://portal2portal.blogspot.co.uk) and we came up with three that we'd heard on our projects:

'Initial power on/run one transaction' test

In electronics, apply power to the system and turn it on - the 'smoke test' here is that if smoke starts appearing, then something is wrong. 

In software, if you can't even log on, or the disk starts spinning like crazy, or all the RAM is used quickly it's similar.

'Increase load until it breaks to determine the maximum safe load'

This has a car analogy: Yay! Keep increasing the revs/speed until smoke starts coming out of the engine. At that point, back off a little and you have your max safe speed. OK, you should be more scientific but you get the point.

In software, crank up the load-testers until you run out of CPU, RAM, I/O, connections or whatever. Hopefully it won't actually crash, just run low on resources.

It might crash though but not immediately; For example it might queue transactions and eventually the queue will fill.

'Test end-to-end' (rarer)

This one I hadn't heard of until the other day. The idea is to blow smoke into the system from one end and see if it comes out where you expect it to i.e. is the flow correct and are there any leaks? Also, is the smoke that comes out the same type as you put in?

In software, this is analagous to watching for side-effects as well as output. Is the Java heap constantly growing? What about the database connection pool? Memory leak?

Anyone got any more?

“When I use a word,” Humpty Dumpty said, in rather a scornful tone, “it means just what I choose it to mean—neither more nor less.” “The question is,” said Alice, “whether you can make words mean so many different things.”

LEWIS CARROLL (Charles L. Dodgson, a mathematician!), Through the Looking-Glass