The Power of Experience

I have been rereading Gary Klein’s landmark book on decision-making, Sources of Power.  Klein’s genius was something other sciences take for granted: field work.  Klein and his team spent years studying how experts make high-stakes decisions in real life.  This is truly “what they don’t teach you in business school.”

The short version is that formal methods for decision making are rarely used in real-life conditions.  Indeed, the people studied by Klein were not even conscious of making decisions.  They just knew what to do.  When a surgeon must make a snap decision, with someone’s life on the line, there’s no time for a weighted-factor analysis.

Most research on decision-making bleaches out the importance of prior experience

Klein points out that most psychology research, in an effort to produce controlled conditions, bleaches out the importance of prior experience.  If you do all your research in a laboratory, then you will only learn how people make decisions in a laboratory – not in combat, say, or a forest fire.

Like his better-known colleagues Kahneman and Tversky, much of Klein’s research was funded by military organizations.  They would like their gunners and squadron leaders not to make fatal blunders under fire.  Also included are doctors, firefighters, and nuclear power plant operators.

The power of experience seems obvious enough, but Klein figured out exactly how it works, in a framework called the Recognition-Primed Decision Model.  This consists of using imagination plus experience to generate possible courses of action, and then conducting mental simulations to predict the likely results.

Various “sources of power” follow from the model:

  • Intuition/Experience
  • Mental Simulation
  • Seeing Leverage Points
  • Seeing Patterns and Anomalies
  • Reasoning by Analogy
  • Anticipating Intentions

What we think of as intuition is really expert recognition.  One firefighter recounted a narrow escape because he’d had a “premonition” the building he was working in was about to collapse.  This might have been a warning from God – or it might have been the million subtle cues he was unconsciously observing.

This may seem like a different realm from business, where we have ample time to make decision trees, compute expected values, perform cost-benefit analyses, and – there’s always time for one more Big Four consulting study.  This is an illusion, however.  Whether they know it or not, managers are under constant pressure to make decisions and take action faster than their competitors.

My mentor at AutoNation, Kevin Westfall, had a plaque in his office with this quote from General George S. Patton, “a good plan, executed right now, is far better than a perfect plan executed next week.”  Kevin and I had both arrived from our previous employer with some impatience over their decision protocols.

In an area that could easily devolve into pop psychology, I was impressed by Klein’s scientific rigor.  Every study is cross-checked, blind, double-blind, sanitized, etc.  Every result is turned into a training program, and then the trainees are tested.  In one project, his team redesigned the user interface for a computerized weapons system, making its operators 20% more effective.

Since experience is so powerful, Klein takes up the question of how best to gain it.  That is, what are the key lessons from the old-timers in various domains?  In the infantry, this might mean knowing how fast your squad can move over terrain, what their best range is for engagement, and being able to gauge those distances by eye.

The cornerstone of the book is the RPD framework, and then Klein spends a chapter on each “source of power,” plus his research methods and training programs.  If that sounds like too much psychology for you, skip the text and just read the case studies.  They’re amazing.

Paying Bills for American Motors

My first Big Six consulting engagement, right out of MBA school, was solving a catastrophic failure in the Accounts Payable system of American Motors Corp.  You may recall AMC, they produced the Gremlin and the original Jeep.  This was right around the time of their acquisition by Chrysler, a sensitive time for the company.  The building still wore the red, white, and blue AMC logo, but the Chrysler employee newspaper was on the stand in the cafeteria.

It was on me to figure out what in hell had caused this popular and bulletproof software to fail. 

They were also just about to launch two new assembly plants in Canada, at Brampton and Bramalea, Ontario.  The launch, and maybe even the acquisition, was jeopardized because AMC had suddenly lost the ability to pay its suppliers’ invoices.  They had devolved to a purely manual process, paying months late, and their suppliers were threatening to cut them off.  Without a functioning A/P system, there would not be many parts shipping to the new plants in Canada.

The classical A/P function revolves around the “three-way match.”  Starting with the invoice, you must locate the purchase order for the goods and the slip from the receiving department showing that the correct goods had arrived.  As you can imagine, a giant manufacturing company cannot possibly perform this task on paper.  American Motors had been running the McCormack & Dodge suite of accounting software, and that was the proximate cause of the failure.  My assignment was to diagnose and fix the failure.

The Director of the A/P department had collected all of the invoices, receivers, and purchase orders into file boxes on tables in a huge room.  This had been a big conference room, maybe, or a gymnasium, and he had hired a platoon of “account temps” to run around the room looking for three-way matches.  Once someone found a match, they would run down the hall to the cashier and authorize payment of the invoice.  It was like a demented Chuck Barris TV game show.

The mad rush to pay months-old invoices was destroying any organization that might once have existed.

For me, as a programmer, this provided a stunning visualization of what this work must have looked like in the dark days before computers.  Of course, in those days, they would have been prepared for it.  Here, the mad rush to pay months-old invoices was destroying any organization that might once have existed in the file boxes.  The A/P director’s job was on the line and, over the weeks of my engagement, he aged ten years.  This poor devil was my client.  I could see the dark circles and the grey hair progressing as I greeted him each morning.

I should note that a new consultant doesn’t get a big job like this on his own but, as “senior schmuck onsite,” I was running the engagement.  Occasionally, higher-ranking consultants would show up to offer an opinion, not do any actual work, and bill four hours to the job.  Also, as the only one with any computer skills, it was on me to figure out what in hell had caused this popular and bulletproof software to fail.

Our method had two prongs of attack.  First, we brought in several junior, not yet CPA, staffers from our audit practice, and put them to work matching invoices.  This was basically the same process as in the gym downstairs, only our people were going to be smarter and look for patterns that might provide some clues.  Plus, we could bill for them at 100% realization.

Meanwhile, I would learn everything I could about the failing A/P system and its friends, the Purchasing system and the General Ledger system.  I read all three APRMs (Application Programmer’s Reference Manual, pronounced “A-Parm”) from cover to cover.  I read all the Job Control Language, the job streams, and much of the COBOL source code.

The only people dumber than the A/P department are these consultants!

I also got invited to defend our work at an executive meeting on the top floor of the AMC building, where I met the Vice President of Purchasing.  This was a big bull of a man, obviously some kind of ex-jock with a lot of red meat in his diet.  He pounded my guy mercilessly, and the preliminary stats from our auditors were no defense.  “The only people dumber than the A/P department,” he roared, “are the consultants hired by the A/P department!”

Eventually, I traced the failure to one specific job running one specific program, P1X030, the “matching module” itself.  All data flowing into, out of, or around this module were good, except that something like 90% of invoices went unmatched.  I called my manager up from Detroit and we went over the results.

I enjoyed working with Ken.  Back in those days, computer skills were considered déclassé.  I was the only consultant who could write a lick of code, and Ken was our only “technical” manager.  Eventually, the firm would get rid of Ken, and then me, in favor of a more golf-oriented practice.

“What about the exception report?” Ken asked, “is it dummied out?”  I checked the JCL.  Systems programmers would often streamline an implementation by suppressing some of its printouts but, no, P1X030 was faithfully printing a list of its reasons for rejecting 90% of the invoices.  “Let’s go for a walk,” Ken said.

We walked about half a mile, the length of the big, mainframe computer facility.  There, lying on the output table, was P1X030’s exception report.  Ken rapped on the window of the control room and spoke with the operator.  The report spooled off his printer every night, and then lay unclaimed on the table.  The operator had been collecting the old reports, and he was relieved to the be rid of them.  This was line-printer paper, boxes of it.  I waited while Ken went to find a hand truck.

The problem, printed mechanically line after line, was that the Purchasing department had been neglecting the important task of generating proper purchase orders.  They had been ordering the suppliers, probably in the same tones I had heard in the boardroom, simply to ship now and worry about the numbers later.

Purchasing had evidently instructed the suppliers to invent random P.O. numbers.  Our auditors had found a few clinkers, like 12345678 and 00000000, but mostly we had no clue.  If anyone had thought to ask a supplier, they would have been afraid to admit it and, anyway, it would have been the Purchasing department doing the asking.

I wrote up our findings and Ken presented them to AMC management.  He wheeled his hand truck into the boardroom and, for dramatic effect, read off the first few variants of “missing or invalid purchase order number.”  We included a report from P1X030, tabulating the various ways in which its safety features had been defeated.

There was no system failure for me to fix, so that concluded our engagement.  As to the failure we did find, management seemed eager to fix that one on their own.

Thoughts on Creator Mode

I have been experimenting with LinkedIn’s new “Creator Mode.”  It offers some features that should help to promote my consulting practice.  This is a for-profit enterprise, not merely “building my personal brand.”  I hate that expression.  Personally, I am a human being (and a nice guy) and not actually a product.

I am not a number! I am a free man! – Number Six

Among other things, Creator Mode allows people to “follow” me.  This emulates the Instagram concept, which is fundamentally hierarchical, having followers and influencers instead of the peer relationship we know as “connection.”

This makes it a little awkward when I want to connect with someone.  No one on Twitter or Instagram sends a personal request to “please follow me.”  On the flip side, if I think we have a relationship, I want it to be mutual.  I don’t want to become your follower.

I guess if you’re a superstar influencer with thousands of followers, that’s more valuable to you than connections.  Not to me, though.  I have built up my reputation through, as Tom Peters says, “having accomplishments and people who know about them.”

As I wrote here, I try to limit connections to people I have worked with, or met, or could at least identify in a police lineup.  Followers are people whom I don’t know, but who enjoy my content.  I try to post relevant material from HBR, McKinsey, and my own blog.

There should be a way to have both connections and followers, so here’s the workaround: when I am sending out connection requests, as I did this weekend for Autovate Austin, I turn off Creator Mode.  When I’m back to posting content, I turn it on.  This is a partial solution, and maybe LinkedIn will figure out a better one.

Edtech Unicorns and JIT Training

Udemy went IPO last week, and PitchBook just published a note on the category, so I thought to write about my positive experiences with Coursera.  Online learning is segmented by subject, level, and quality of instruction.  See the research note for a complete rundown.

The edtech boom has not waned now that most schools and universities are again meeting in person. 

Coursera is oriented toward college credit and professional certification.  My instructor for neural nets, Coursera co-founder Andrew Ng, is a professor at Stanford.  They offer online degree programs in conjunction with major universities.  For example, you can earn a Master’s in Data Science through CU Boulder.

I was intrigued by that, but … I have a specific business problem to solve, and I already have grad-level coursework in statistics.  It doesn’t make sense for me to sit through STAT 561 again.  For me, the “all you can eat” plan is a better value at $50 per month.

What I need, today, is to move this code off my laptop and into the cloud.  For that, I can take the cloud deployment class.  If I run into problems with data wrangling, there’s a class for that, too.  This reminds me of that scene in The Matrix, where Trinity learns to fly a helicopter.

People can gain the skills they need, as and when they need them – not as fast as Trinity, but fast enough to keep up with evolving needs on the job.  I think this is the future of education, and 37 million students agree with me.