As a young consultant at Coopers, I had the privilege of being included as the technology person on a number of engagements with other specialties. One such was the Victoria’s Secret engagement, where I was able to work with the firm’s top retail experts. I am going to make a point here, about knowing your customer, but not without telling the story.
Our customers in the Detroit office were mostly from the manufacturing practice, and the guys teased me about shipping out to the Victoria’s Secret facility. “Wear a hardhat,” one wag said, “in case a box of panties falls on you.” We did, in fact, keep hardhats in the office.
I did not know a corset from a camisole, so I resolved to study the catalog until I knew the names of all the items.
The retail people were different. My tech counterpart arrived from Chicago with just a rollaboard, same as me. He was chafed because he had had to wait for Charles, the retail expert, with his train of checked baggage. Bemberg lining, doctor’s sleeves, Aston-Martin cufflinks. They were a different species.
My side of the engagement was to evaluate the client’s competence in software management, capacity utilization, contingency planning, staffing, budgeting, and so forth – routine work for me.
I also ran the day-to-day activities of collecting data and conducting interviews. Victoria’s middle managers were, unbelievably, all attractive women. I would have to tell my guys to stop hyperventilating. “Yes, she’s hot. She’s also a VP. We’re interviewing her tomorrow.”
The men who worked there seemed inured to Victoria’s charms. The head of store ops banged through the statistics from memory. He knew which item, color, and style sold best in each market.
“The black satin tap,” he said, on this topic, “that one.” He pointed, without looking up, at a promotional poster. I confronted a life-size photo of a dark-haired woman modelling this item, to good effect. I did not know, initially, a corset from a camisole, and so I resolved to study the catalog – no, not the illustrated one – until I knew the names of all the items.
The firm’s seniormost retail expert, Marge Meek, took me under her wing. She was a retail god. Like, personal friends with Marhsall Field, or something. Marge took me to visit some stores, which turns out to be pretty important in retail.
“Okay Mark, who is the Victoria’s Secret customer?” Well, to start with, she is young, fit, well-educated, and upwardly mobile. I rattled off what I had read in the annual report.
“Now look around. Is that who you see here?” I am a tech guy. It would never have occurred to me to visit a store and study the customers. Marge offered her own characterization, which was a little less flattering, but undeniably accurate.
Back at the job site, we reprised our field trip for the team. Our engagement partner had his own opinion. “Women that date Mexicans,” was Dean’s pronouncement. He was not well-liked by the retail people.
Every so often, I will write up a standard quantitative procedure, usually because someone has asked me about it. For instance, see Pay Plan Math, What Is Accuracy, and Know Your Time Series. Today, it’s weighted-factor analysis for product selection. At a high level, this procedure is:
Gather your requirements and selection criteria
Quantify how important each criterion is
Grade the vendor responses
Compute numerical scores
Gather Requirements and Criteria
First, through interviews and maybe some direct observation, discover why we need the product. In my business, this is generally a software product, but it could be anything. Next, determine what are the requirements and selection criteria.
Selection criteria are the features we will evaluate to decide which product is the best fit, whereas requirements are features the product must have to even be considered. Don’t make the mistake of thinking requirements are just extra-special criteria.
If you’re looking to buy a car, and gas mileage is on the list, then a hybrid will score well on that criterion. If you’re only looking to buy a hybrid, then that’s the category, and you’re not looking at gas cars at all.
The purpose of requirements is to define the category of product we’re looking for. If you’re writing an RFP, the criteria are what the vendors respond to, and the requirements determine which vendors get the RFP. When in doubt, send them the RFP anyway and let the vendor figure it out.
For example, if I am selecting cybersecurity software, I might want endpoint protection (EPP), endpoint detection and response (EDR), managed detection and response (MDR), or even a security operations center (SOC). These all address the same problem, but they’re not the same product.
Quantify Importance of the Criteria
In the chart, I show criteria scored on a scale of 1 to 5, which is typical. Then, for the sake of example, I norm these to a total score of 100. This is probably overkill, but it’s fun to have 100 as a baseline. Later, we’ll do the same with the final score. Clients love simple numbers.
One way to explore the criteria is do a forced ranking from most important to least important. This is not amenable to quantitative methods, but it’s a good way to get started. Spend an hour in front of the whiteboard while the client staff fight it out over the ranking, then let them each do the 1 to 5, and average their responses.
Yet another way is to give each participant 100 points that he can allocate as desired across the criteria. This is the most accurate, in terms of understanding tradeoffs, and it makes the math easy.
I like to keep the cost analysis separate from the features. It is possible to turn the price proposal into another row among the criteria, but no one really thinks this way. What you’re shooting for is, “this one scored 84 out of a hundred, and it’s $100,000 more than the one that scored 74,” with traceability back to the features that account for the difference.
Grade the Vendor Responses
Maybe you’ve sent an RFP and are now grading the proposals, or maybe you’re doing your own research. Using an RFP is handy because you can include the criteria and let the vendors tell you how they propose to meet them. In either case, you (and the committee) are responsible for assigning a number to indicate how well the product meets each criterion.
Here again, the 1 to 5 scale is popular and easy to use. Obviously, grades supported by numbers are best. For gas mileage, you can assign 1, 2, 3, 4, and 5 to specific ranges of MPG. Something like “vendor support” can be tied to a service-level agreement in hours or minutes.
Compute the Final Score
This is called weighted-factor analysis because each product is scored according to its criteria grades, and the criteria have different weights. It’s just like computing a weighted average. Since we’ve normed the weights to 100 and we’re using a 5-point grading scale, we divide the totals by five to produce a score out of 100. You can present this as a percentage if you want.
In our carefully contrived example, vendor #3 comes out on top even though they had the lowest raw score, because they scored well on the criteria that mattered most.
When data scientists say that “our precision exceeds our accuracy,” this is what they mean. Do not take this fundamentally subjective numerical score out to two decimal places. The point of this procedure is not so much to generate a number, but to make the variables explicit.
The idea is that sum of many small decisions will be more accurate than one big one, particularly if there is consensus among the participants. Everyone on the committee should be able to say why the chosen product scored ten points better than the runner-up.
Also, to be a little bit pragmatic: now everyone has their fingerprints on the decision. No one can complain that they weren’t consulted, or question how the decision was made.
Funny aside: One of my first consulting projects was the selection of a networking vendor for Ford Credit. We did the full procedure: interviews, requirements, criteria, an RFP, a selection committee, bidder conferences, sealed bids, etc. Digital Equipment (DEC) won. Remember them? And then some big shot from the Glass House swooped in and gave the contract to IBM. What about our fancy RFP project? Well, it was “defective” because it failed to produce IBM as the winner. There was a saying in those days, “no one gets fired for buying IBM.” It was seen as the safe choice – and the only choice for risk-averse executives.
My first Big Six consulting engagement, right out of MBA school, was solving a catastrophic failure in the Accounts Payable system of American Motors Corp. You may recall AMC, they produced the Gremlin and the original Jeep. This was right around the time of their acquisition by Chrysler, a sensitive time for the company. The building still wore the red, white, and blue AMC logo, but the Chrysler employee newspaper was on the stand in the cafeteria.
It was on me to figure out what in hell had caused this popular and bulletproof software to fail.
They were also just about to launch two new assembly plants in Canada, at Brampton and Bramalea, Ontario. The launch, and maybe even the acquisition, was jeopardized because AMC had suddenly lost the ability to pay its suppliers’ invoices. They had devolved to a purely manual process, paying months late, and their suppliers were threatening to cut them off. Without a functioning A/P system, there would not be many parts shipping to the new plants in Canada.
The classical A/P function revolves around the “three-way match.” Starting with the invoice, you must locate the purchase order for the goods and the slip from the receiving department showing that the correct goods had arrived. As you can imagine, a giant manufacturing company cannot possibly perform this task on paper. American Motors had been running the McCormack & Dodge suite of accounting software, and that was the proximate cause of the failure. My assignment was to diagnose and fix the failure.
The Director of the A/P department had collected all of the invoices, receivers, and purchase orders into file boxes on tables in a huge room. This had been a big conference room, maybe, or a gymnasium, and he had hired a platoon of “account temps” to run around the room looking for three-way matches. Once someone found a match, they would run down the hall to the cashier and authorize payment of the invoice. It was like a demented Chuck Barris TV game show.
The mad rush to pay months-old invoices was destroying any organization that might once have existed.
For me, as a programmer, this provided a stunning visualization of what this work must have looked like in the dark days before computers. Of course, in those days, they would have been prepared for it. Here, the mad rush to pay months-old invoices was destroying any organization that might once have existed in the file boxes. The A/P director’s job was on the line and, over the weeks of my engagement, he aged ten years. This poor devil was my client. I could see the dark circles and the grey hair progressing as I greeted him each morning.
I should note that a new consultant doesn’t get a big job like this on his own but, as “senior schmuck onsite,” I was running the engagement. Occasionally, higher-ranking consultants would show up to offer an opinion, not do any actual work, and bill four hours to the job. Also, as the only one with any computer skills, it was on me to figure out what in hell had caused this popular and bulletproof software to fail.
Our method had two prongs of attack. First, we brought in several junior, not yet CPA, staffers from our audit practice, and put them to work matching invoices. This was basically the same process as in the gym downstairs, only our people were going to be smarter and look for patterns that might provide some clues. Plus, we could bill for them at 100% realization.
Meanwhile, I would learn everything I could about the failing A/P system and its friends, the Purchasing system and the General Ledger system. I read all three APRMs (Application Programmer’s Reference Manual, pronounced “A-Parm”) from cover to cover. I read all the Job Control Language, the job streams, and much of the COBOL source code.
The only people dumber than the A/P department are these consultants!
I also got invited to defend our work at an executive meeting on the top floor of the AMC building, where I met the Vice President of Purchasing. This was a big bull of a man, obviously some kind of ex-jock with a lot of red meat in his diet. He pounded my guy mercilessly, and the preliminary stats from our auditors were no defense. “The only people dumber than the A/P department,” he roared, “are the consultants hired by the A/P department!”
Eventually, I traced the failure to one specific job running one specific program, P1X030, the “matching module” itself. All data flowing into, out of, or around this module were good, except that something like 90% of invoices went unmatched. I called my manager up from Detroit and we went over the results.
I enjoyed working with Ken. Back in those days, computer skills were considered déclassé. I was the only consultant who could write a lick of code, and Ken was our only “technical” manager. Eventually, the firm would get rid of Ken, and then me, in favor of a more golf-oriented practice.
“What about the exception report?” Ken asked, “is it dummied out?” I checked the JCL. Systems programmers would often streamline an implementation by suppressing some of its printouts but, no, P1X030 was faithfully printing a list of its reasons for rejecting 90% of the invoices. “Let’s go for a walk,” Ken said.
We walked about half a mile, the length of the big, mainframe computer facility. There, lying on the output table, was P1X030’s exception report. Ken rapped on the window of the control room and spoke with the operator. The report spooled off his printer every night, and then lay unclaimed on the table. The operator had been collecting the old reports, and he was relieved to the be rid of them. This was line-printer paper, boxes of it. I waited while Ken went to find a hand truck.
The problem, printed mechanically line after line, was that the Purchasing department had been neglecting the important task of generating proper purchase orders. They had been ordering the suppliers, probably in the same tones I had heard in the boardroom, simply to ship now and worry about the numbers later.
Purchasing had evidently instructed the suppliers to invent random P.O. numbers. Our auditors had found a few clinkers, like 12345678 and 00000000, but mostly we had no clue. If anyone had thought to ask a supplier, they would have been afraid to admit it and, anyway, it would have been the Purchasing department doing the asking.
I wrote up our findings and Ken presented them to AMC management. He wheeled his hand truck into the boardroom and, for dramatic effect, read off the first few variants of “missing or invalid purchase order number.” We included a report from P1X030, tabulating the various ways in which its safety features had been defeated.
There was no system failure for me to fix, so that concluded our engagement. As to the failure we did find, management seemed eager to fix that one on their own.
Clients are often surprised when I say, “my work here is finished.” The consultant’s handbook says you should hang around until they’re sick and tired of you. I feel it is better, when retained for a specific task – like a startup – to do the job in good style and then move on.
So it is with Provider Exchange Network. The business model is established, the software is up and patented, and the staff are fluent in their new roles. It has been my pleasure to work with the talented people at PEN, and I wish them all the best.
