Wednesday, 16 December 2015

The Magical Progress Fairy - are you using one on your project?

Some time ago IBM had an advert featuring a 'Universal Business Adapter' (YouTube here) which would magically connect anything to anything.

Not surprisingly, everybody wanted one. However not everybody got the joke that it didn't actually exist - you could do everything it did, but you would actually need to build it using SOA/ESB/Integration tooling. (If you do actually want to build one, start here).

Another thing that everybody seems to want on their project is the Magical Progress Fairy. A lot of people think they already have one . . .

What is a Magical Progress Fairy?

A Magical Progress Fairy (MPF or latin: hopeus overexperiencus) is a mythical creature that you invoke when your project is not going as fast as you want it to, you're behind and you're likely to miss deadlines. It magically improves your productivity, vitalises your team, eliminates any problems and gets you right back on track - without you making any changes to your approach at all!

I don't do that . .

Perhaps you don't - but do any of these things sound familiar?

"We only made 20 story points in our last sprint - to get back on track we need to make 30 in the next sprint"

So how is that going to happen? What are you actually going to do to improve productivity by 50%? A lot of the time we hear phrases like 'Everybody buckle down', 'Let's all really concentrate for this one' or 'we all need to work harder'.

On the other hand, everyone is already working hard and concentrating - and we need to keep in mind the Agile principle that progress should be sustainable indefinitely.

"We had a bad sprint last time - a lot of things came up to disrupt us"

And these won't happen on the next sprint? The internet won't drop out? A team member won't get sick? There won't be an unexpected bug in the tooling / database etc?

"We weren't experienced enough on the technology before - we're better now"

Maybe - but can you prove it? Are you sure you now know everything you need to know? What else is around the corner?

All of these approaches (some might call them excuses - I would never do that) rely on the Magical Progress Fairy to get you out of trouble. Magically progress will improve, mistakes will not be made, people will become more productive and disruptions will not happen . . .

OK, maybe I might need a Magical Progress Fairy - but what else can I do?

There are a number of things you might try but the first is to be realistic about what you've already managed to do and what you will be able to do in the future:

Accept reality and calculate your project velocity

Project velocity is a measure of how productive your team is, in your environment, working on your user stories, using their skills in the technology and method chosen for your project. It's not a single one of any of these, a measure of how 'good' the team is - or an individual. All of the project factors have a part to play.

To calculate your project velocity, simply divide up how many story points you actually managed to get done in the previous sprint by the number that you planned to deliver in that sprint.


We plan a sprint of 2 weeks (10 days). When we did planning poker we decided that one story point was going to be one 'ideal day'. We have a team of 5 people, so that gives us 50 ideal days, so 50 story points to plan with.

At the end of the sprint we found that we 'only' achieved 32 story points.

So, our velocity is 32 / 50=0.64. Note that it is not '1.0 - except for Bill had a dentist appointment and Sarah's PC needed a new hard drive, we spent time planning the sprint - and we used 2.5 person days for 1 morning of playback to the business'

Of course, you can have a velocity greater than 1.0. 

If you'd finished all the user stories in 9 days - people do get ahead , then  you'd have had a velocity of 50 / 45 or 1.11 (I know it's recurring but hey . .)

Once you know your velocity, use it to plan further sprints realistically

So, if your velocity was 0.64 and you have another 10 day sprint - then plan for 50*.64=32 story points

If it was 1.11 then plan for 50 *1.11=55.5 story points (if your planning poker has a 0.5 card - otherwise it's 55 - no quarts in pint pots).

Don't invoke the Magical Progress Fairy

If you planned 50 story points and you made 32, then that is what you made. No ifs, but buts. That's all you proved your team can make. Why will next time be any different?

Don't extend the sprint

'We'll be finished if we just have another day / 2 days / week / month . . '. One of the advantages of the sprint system is that it shows you 'point in time' progress with nowhere to hide. Either the software works or it doesn't. Software is often 95% finished for 95% of the time you ask the developer 'how it's going'

Finish early? Shut down the sprint, calculate velocity and start the next one

Congratulations - you made a velocity greater that 1 - but that just shows your estimating was on the low side vs reality. Now you know that, go ahead with that knowledge.

My velocity has changed after the next sprint - what do I do?

So, your velocity after sprint 1 was 32. Your velocity after spring 2 was 44. Great! This isn't the magical progress fairy. Something was different - maybe your team did get better, or the dev server didn't crash or there wasn't a big queue at the coffee shop.

So, use your new velocity to plan the next sprint- that's now our up to date measure of reality.

If you have put real interventions in e.g. given people faster computers, given them a copy of production data to test with etc, then this will allow you to measure if this made any difference.

A quick quiz - which of the following might actually work? 

No prizes except maybe getting your project back on track . .
  • Work evenings and weekends
  • Increase the project duration by adding more sprints
  • Adding more people to your project - see The Mythical Man Month
  • Cut the scope by removing user stories
  • Yelling at the team more - motivation is the key
  • Re-prioritising user stories and revisiting what might be acceptable as a minimum viable product
  • Re-plan the sprint every day: hey, why not replan every hour?
  • Concentrate on the happy path and make some exception paths into a manual activity e.g. 'If paying by credit card, click 'next'. If paying by cash or check/cheque, please visit one of our high street stores'
  • Add more RAM, CPU or disk to the development system - maybe - but is this the real issue?
  • Cut/eliminate testing (especially NFRs) - let the users do it in the beta
  • Try different sprint lengths - overheads like planning and playback can eat into the time. Beware long sprints as this reduces agility and means that progress can be obscured.
  • Re-estimate everything over and over until we find an estimate we like that fits the plan.
  • Ignore technical debt - we've gotta hit that deadline - hardcode everything!
  • Calculate your velocity, use it to plan further sprints, bite the bullet and accept reality.

Finally, Beware the 'hidden hours' of evenings and weekends working - especially when you don't know about it!

Remember we planned for a 10 day, 50 person/day sprint? If 4 people worked the weekend, it's actually now a 58 person/day sprint. This may give you more development output but it's skewing  your velocity!

It means that in the next sprint, you're now assuming that 4 people will work the weekend - and the same going forward.

If you plan for an 8 hour day and 2 of the team actually work 12 hour days all the way through 'to get it done and show dedication' - they're making a mockery of your velocity calculations.

Watch out for people staying late, doing work at home in the evenings of weekends - if they do it, you need to take account of it otherwise they're becoming your Magical Progress Fairy  . .

Thursday, 10 December 2015

El Capitan Time Machine on NAS running very slowly?

So I upgraded my Macs to El Capitan and everything was rosy - with the exception of Time Machine.

Time machine backups to local USB drives (Seagate 2TB if it makes any difference . .) seemed to work fine, but network backups to a Synology DS415Play NAS were getting 'stuck' on the 1st few MB - and staying stuck, seemingly forever.

The symptoms were visible in the 'progress' of Time Machine in the status bar:

The 'backing up' would stay in single megabytes.

After checking the network etc, I then tried Time Machine preferences:
Originally I had my NAS time machine connected as the 'auto discovered name' i.e 'AndyNAS'.

I tried the 'add or remove backup disk' option to remove the connection and re-add it. It didn't seem to help - I got the same problem.

I tried connecting directly to the Time Machine drive using Finder and for no reason that I could think of, used the numeric IP address to connect to the NAS rather than the name. This seemed to work fine and the Time Machine share was visible and seemed to be working normally.

Finder->Go->Connect To Server

In a last ditch attempt, as I have my NAS on a static IP on my LAN (, I tried removing the original Time Machine NAS connection using 'Add or Remove Backup Disk' and then tried connecting using the 'dotted quad' 'numeric' IP address as shown above - and everything started working again!

Not sure exactly what fixed it - could be any of the following:
  • Adding and Removing the Backup Disk / NAS connection in Time Machine Preferences
  • Doing a direct afp connect to the NAS Time Machine share in Finder
  • Using the 'dotted quad' / Numeric IP address rather than the name
  • Pure luck  and the 'turn it off then on again' effect - but it did fix 3 out of 3 of the macs I tried it on

To be fair, I'm not that bothered - it's working again, and for all three Macs (work, personal and wife's) backing up via Time Machine to the NAS after the El Capitan upgrade. It's also encrypting fine.

As a side note, I didn't have to re-create my Time Machine backups - as long as I re-entered the encryption password then it 'found' the old backups and carried on as before.

As usual, this worked for me - your mileage may vary but it might be worth a try.


After an interesting dialog box of 'Time machine has checked your backup - it needs to backup from scratch' (effectively) I found that having to back up 150GB or so was taking a long time.

If you check 'console' and search for 'mdworker', you might see a number of entries, especially starting with the word 'deny'. Apparently, this isn't helping your time machine speed!

Believe it or not, the fix for this is to start your Mac in 'safe mode' - then restart again normally.

To start in safe mode, shutdown your Mac. Then start it up. Immediately after the first 'bong', hold down the shift key. When you see the startup screen, you can release the shift key. If you're successful, you'll see 'safe mode' written on the screen somewhere.

(Apple guidance here

After you've started up in safe mode, give it a minute or two to finish loading everything then shut down again.

Now check your time-machine backup speed - assuming you have a decent network speed (I highly recommend gigabit) you'll see it fly along . . .

Resetting Mac SMC and NVRAM / PRAM

Having just applied Apple's latest update to El Capitan and temporarily lost my bluetooth connection, I thought I'd post this as a reminder to myself if nothing else . .

(My bluetooth was fixed by resetting the SMC)

Remove any devices e.g. USB/Thunderbolt and try a reboot first: I tried recently to reset as my bluetooth had died again. Resetting with two displays plugged into the thunderbolt ports didn't work

See also here

Also, for bluetooth, try deleting the bluetooth plist in /Library/Preferences

Resetting the NVRAM / PRAM

  • Shut down the Mac completely (not just standby)
  • Power on the Mac
  • Immediately after the startup 'boing', press and hold Command, Option, P and R until the Mac restarts and you hear the second 'Boing' (yep, this is a finger stretcher!)
  • Apple's link here

Resetting the SMC

(Note that this is for Macs with non-removable battery - check Apple's notes first)
  • Shut down the Mac completely (not just standby)
  • Make sure you have a power supply plugged in
  • Press Shift, Control, Option (on the left side of the keyboard) and the Power key all at the same time
  • Release them all at the same time
  • Apple's link here
(BTW 'Option' is the key that is marked with  'alt' on the top and a symbol on the bottom, at least on UK keyboards - see also

Wednesday, 9 December 2015

BPM Good Practice, Leading Practice, 'Best Practice'

Notwithstanding the 'Lord Wessex Effect' and whether good practice is good for your particular implementation, there is a lot of good practices that are worth considering.

The following might be useful - one of the authors you might recognise :-)

Tuesday, 8 December 2015

SQL JOINs - a handy visual representation with examples

Found this today on, written by C.L. Moffatt: A really useful and visual guide to all the different types of SQL JOINs

If you need to know the difference between INNER JOIN, LEFT JOIN, OUTER JOIN, RIGHT JOIN etc etc then try here:

It even has a handy 'print out and put on the wall' graphic! Great stuff!

Monday, 7 December 2015

Running an IBM BPM saved search from the REST API

This may seem obvious to many people, but it wasn't to me, so:

To run a saved search in IBM BPM from the REST API the URL is:


To restrict the size of results add ?size=<size> at the end