ZDS Toolkit - Deliver Metrics

ZDS

ZDS Toolkit

Using tools to measure and maintain service levels is a sure sign that your change management operation has moved up to CMM level 4. The Deliver metrics tool provides this capability. To view an on-line demonstration, please call Dave Goodall at 714 993 1267 or email me at dave.goodall@gmail.com

Installation and Operation Guide   Microsoft Word Format   Adobe PDF Format

A line-step migration that shows a final status of 'Succeeded' may have failed requiring the user to reset and re-run it many times before getting a successful end result.

Consider a 25 line package being migrated from RICE -> DEV -> TEST ->: PROD that all end up successfully migrated to PROD. From one perspective that's a zero percent failure rate.

If every line migrated through every step successfully the first time the user would have done 100 migrations. But suppose the user had problems with one or more lines and had to re-run some step migrations 19 times. The failure rate, and your Deliver service level (or lack of it), is 19/119 or 16%.

To your end users these failure rates, and not the end result, are the true measure of the service level you provide them.

The tool identifies the true failure rates by considering the result of EVERY line-step migration that users ran in the selected period. It also identifies, and classifies the errors that contribute to the overall failure rates for Custom and Standard object migrations. This gives you the information you need to create an action plan to improve your service levels and monitor them thereafter.

The Deliver metrics tool obtains this information by recursing over your Kintana logs directory tree and analyzing each log. It can be directed to process for any date range - typically for a year. Once tuned the tool is capable of processing up to 30,000 files per hour.

Four reports are generated:

The first level report is a simple summary of your current migration service levels:

2007 Service Levels

Custom Succeeded                       65830   84.09%

Custom Failed                          12456   15.91%

Standard Succeeded                     40302   87.05%

Standard Failed                         5997   12.95%

The second breaks the failures down to the broad categories contributing to the failure rate:

Kintana Deliver Errors By Type 2007

Custom
  Transport
    KB Kintana Bug                      1034   28.23%
    IP Insufficient Privileges           343    8.22%
    FT File Transfer                     301    6.34%
    ..
  Process
    Logical Error                       1562   38.12%
    ..
Standard
    ..

The third breaks each broad category down in turn to the individual reasons:

Kintana Deliver Errors By Reason 2007

Custom
  ..
  FT File Transfer       63     .85% Could not find file
  FT File Transfer       28     .43% no space left on deviceCould not find file
  FT File Transfer       13     .22% ftp login to destination environment failed
  FT File Transfer        3     .04% read-only file system
  ..

Lastly the tool puts out an html formatted log that can be viewed in any browser.

The log lists every individual package-line-step migration, it's status code, and reason.

Each line is a link to the underlying kintana execution log. Clicking on a line drills down into the specific log.

Data mining is an iterative process. Working through the log allows results to be verified, existing categories to be revised, and new ones added.

An important part of this is setting up the configuration directives needed to compensate for bugs in the Kintana execution logs and for the under and over reporting that is built-in to the system.

Several discovery and refinement runs are required to generate the final database.

The next step is to examine each broad error category and determine what can be done to eliminate or minimize errors in that category.

The end result is new target service levels for Custom and Standard object migrations and an action plan for achieving it.

The process starts with setting up an initial configuration file to control the operation of the tool.

There are four configuration groups:

1. Tuning: This section directs the tool to ignore specified test packages, and contains initial settings for tuning the tool to run at it's maximum speed consistent with accurate results.

[Tuning]
Test Packages=12345,12346,.... 99998,99999
Custom Percentage=60
Standard Percentage=40
Max File Size=170000

2. Objects: The names of the Standard and Custom objects migrated on all your workflows:

[Custom Error Objects]
Cnt=62
1=SQL Script
2=C++ Library
3=Java Class
..

3. Error Types: Broad categories of migration errors for Standard and Custom objects. Most of these are generic and common to all installations. Some will be specific to your own.

[Custom Error Types]
Cnt=14
CS=Command Script
FT=File transfer
IP=Insufficient Privileges
KB=Kintana Bug
LE=Logical Error
..
XX=Not Classified

4. Error Codes: Unique error strings that characterize a specific migration failure. Each string is associated with an error type (category) and optionally with a correction directive.

[Custom Error Codes]
Cnt=182
2=LE| |ORA-00942: table or view does not exist
3=LE|f|ORA-00955: name is already used by an existing object
16=LE|f|PL/SQL ERROR 201
..
182=LE|s|ORA-02261: such unique or primary key already exists in the table

We supply the base configuration. You must supply the names of the objects you are using.

The tuning, error type, and error code sections are refined as part of the discovery process on successive iterations of the tool.

You can purchase The Deliver metrics analysis tool under a non-transferable, single site license for you to run under your control.

If you prefer it, we will run the analysis on our own equipment, and return the results to you. Because of the large amount of data involved we consider it preferable for you to run the tool on the back shift, and send us the output for refinement.

The refinement process requires that we be able to drill into the execution logs. We would need either read-only access to the logs directory tree, or a copy on DVD.

In either case you will need to supply us with your non-disclosure agreement.

To view an on-line demonstration please call or email Dave Goodall at 714 993 1267 dave.goodall@gmail.com  

Demand Metrics

You may also want to look at our companion Demand Metrics analysis tool.