We continue our tour around the clock today with a look at the next source of Problems - "New Changes", writes Robin Yearsley.
Please note - I have deliberately separated out "upgrades and Patches" as the next Source of problems which we will cover tomorrow.
Strictly speaknig they are - and should be - classed as "New Changes" but for reasons we'll go into tomorrow we have created a separate 'source' for them. Please bear in mind that these criteria for "New Changes" also apply to "upgrades and patches"
So how do Changes cause Problems?
Well, strictly speaking they don't!
Failed changes cause incidents, incidents are resolved - with service restoration taking place - then the underlying root cause is assessed / analysed and then eliminated through Problem Management.
Great Changes don't cause problems. They enable improvement.
Now, we won't get bogged down with the strict ITIL workflow here but we're going to focus on what you can do to prevent (or minimise) the ultimate source of problems: "New Changes".
There are several key reasons often cited, we discuss each one here...
Lack of approval:-
The change did not receive the appropriate level, duration, depth or quality of review prior to approval. The result - the change is (often unwhittingly) mis-approved. Think of it as a time-bomb. It's now been approved (armed) and it's heading for your production environment. It should have had some aspect of it validated or raised as an objection during a CAB meeting or senior level approval meeting - but this didn't happen. So, some impact is going to happen as a result.
Tackling this challenge is best done through your CAB meetings, but you can also create special "advisory" groups who pre-approve forthcoming change packages prior to it going into the full CAB for final approval. Such advisory froups will generally focus on specific elements of change such as the "Technical Design Group" or the "High Availability Group". You should aim to create several of these groups but tag the responsibility onto a group that already exisits to avoid internal warfare/politics about the (alledged) resource utilisation of your Change Process.
Remember - all change involves some degree of risk. You will NEVER be able to consider all possibilities - you just need to ensure that your Change Process and the people who facilitate and approve changes are provided with the best information and the best opportunity to mitigate any risks prior to implementation.
Poor testing:-
It amazes me how often poorly (or un-tested) changes are approved to enter production. One classic reason for this is because the CEO shouts loudly and wants some new functionality in place ASAP! Everyone beneath the CEO suddenly cuts corners, moves mountains and drives the change through in double quick time - missing all the appropriate check-points. There's not much science here: poor testing = higher risk change. It's that simple. Plus - CEO's shouting loudly will not alter this risk factor. Ask yourself the question - How loud will they shout if all your systems fail and your organization takes a massive 'hit'? You need strong IT leaders to stand up, be counted and present the risks in the correct way. Reasoned discussion leads to reasoned decisions.
Another factor lies within the testing methodology itself. Are the right skilled resources carrying out the testing, are the testers allowed enough quality time to implement a fix to any given bug, are testers influenced by teh business to much? The list goes on... but the consideration is simple - the quality of your testing function is directly proportionate to the quality of the change package that you are expected to implement.
Poor implementation planning:-
Do your changes come attached with a thought out and walked-through quality implementation plan? If the change is of any reasonable size - have the implementation team walked through it with the Change team to review the key stages and ensure that all appropriate support activities are in place? There's no substitute for great planning. It prevents the pain of production issues.
The best Change teams ask difficult questions, especially "What if this happens...", type questions. If it's badly planned and thought through - don't let it in!
Poor execution of implementation plan:-
The ironic thing is - the best laid plans, poorly executed, are a waste of time. The best laid plans also need to be flawlessly executed. If things go wrong, as they often do, small implenmentation workarounds and fall back positions should have already been identified and documented read-to-go.
The best advice I can give about plan execution is this - have the best person for the job in charge of ensuring timely and accurate progress against the plan - but if things fall away too fast - that person knows when to effectively 'pull the plug' on the change and restore normal service. The best person will not carry on regardless.
Impact of another (concurrent) change:-
Mmmm, do you really know everthing that's going on?
For example will the facilities folks be testing the UPS systems over the weekend and therefore when your technical guys try to implement their new system - there's no power available!?
For Change Management to be really effective - you need to have what we call, "One complete version of the truth".
All change programmes should ultimately 'come together' into this one version - preferably onto one high level change plan. This activity can actually drive up the quality of Changes too - since it means IT and Technology folks have to talk regularly with their change business counterparts - about change!
I've lost count of the number of times Change teams have identified 'risky' change going on elsewhere when the IT folks were also planning a change.
So, what else can we do to prevent "New Change" being the sources of Problems.
Well, there are two final prevention items that can be employed:-
1. Your acceptance into production criteria.
- Did the change pass all pre-production criteria?
- Did the change get approved by the right folks?
- Did the change implementation plan get walked through (or did it sail through?)
2. Playing it tough.
- Learn to say 'no'.
- If you can't (or are un-empowered too) - get someone on your side who can.
The bottom line with this particular source of Problem is this:-
If the change is in bad shape - it enters production at your peril!
Tomorrow, we'll move onto look at a very special sub-set of Change: "upgrades and patches", which we believe deserve it's own special Source number (#3).
In the meantime, if you have any comments about this article, please feel free to share by posting a comment below.
Posts
