The Power of Experience

I have been rereading Gary Klein’s landmark book on decision-making, Sources of Power.  Klein’s genius was something other sciences take for granted: field work.  Klein and his team spent years studying how experts make high-stakes decisions in real life.  This is truly “what they don’t teach you in business school.”

The short version is that formal methods for decision making are rarely used in real-life conditions.  Indeed, the people studied by Klein were not even conscious of making decisions.  They just knew what to do.  When a surgeon must make a snap decision, with someone’s life on the line, there’s no time for a weighted-factor analysis.

Most research on decision-making bleaches out the importance of prior experience

Klein points out that most psychology research, in an effort to produce controlled conditions, bleaches out the importance of prior experience.  If you do all your research in a laboratory, then you will only learn how people make decisions in a laboratory – not in combat, say, or a forest fire.

Like his better-known colleagues Kahneman and Tversky, much of Klein’s research was funded by military organizations.  They would like their gunners and squadron leaders not to make fatal blunders under fire.  Also included are doctors, firefighters, and nuclear power plant operators.

The power of experience seems obvious enough, but Klein figured out exactly how it works, in a framework called the Recognition-Primed Decision Model.  This consists of using imagination plus experience to generate possible courses of action, and then conducting mental simulations to predict the likely results.

Various “sources of power” follow from the model:

  • Intuition/Experience
  • Mental Simulation
  • Seeing Leverage Points
  • Seeing Patterns and Anomalies
  • Reasoning by Analogy
  • Anticipating Intentions

What we think of as intuition is really expert recognition.  One firefighter recounted a narrow escape because he’d had a “premonition” the building he was working in was about to collapse.  This might have been a warning from God – or it might have been the million subtle cues he was unconsciously observing.

This may seem like a different realm from business, where we have ample time to make decision trees, compute expected values, perform cost-benefit analyses, and – there’s always time for one more Big Four consulting study.  This is an illusion, however.  Whether they know it or not, managers are under constant pressure to make decisions and take action faster than their competitors.

My mentor at AutoNation, Kevin Westfall, had a plaque in his office with this quote from General George S. Patton, “a good plan, executed right now, is far better than a perfect plan executed next week.”  Kevin and I had both arrived from our previous employer with some impatience over their decision protocols.

In an area that could easily devolve into pop psychology, I was impressed by Klein’s scientific rigor.  Every study is cross-checked, blind, double-blind, sanitized, etc.  Every result is turned into a training program, and then the trainees are tested.  In one project, his team redesigned the user interface for a computerized weapons system, making its operators 20% more effective.

Since experience is so powerful, Klein takes up the question of how best to gain it.  That is, what are the key lessons from the old-timers in various domains?  In the infantry, this might mean knowing how fast your squad can move over terrain, what their best range is for engagement, and being able to gauge those distances by eye.

The cornerstone of the book is the RPD framework, and then Klein spends a chapter on each “source of power,” plus his research methods and training programs.  If that sounds like too much psychology for you, skip the text and just read the case studies.  They’re amazing.

GPS Trackers and OBD Ports

While I was working at Safe-Guard, in 2018, we adopted and co-branded a GPS tracker from ZAZ.  Shortly thereafter, we learned that our crosstown rival, EasyCare, was backing another such product, called SAVY.

Of the two, SAVY had more consumer-friendly features in their mobile app, which I feel is decisive.  This point of strategic positioning is the focus of today’s post.

Neither hookup did well, demonstrating that providers of “paper” F&I products are ill-equipped to deploy hardware.  I took the installer training, just for grins, with Hector Delgado.  So, at least I have a useful skill to fall back on.

I also consulted briefly for LoJack, in 2012, helping them sort out issues around preloading – issues they solved, ultimately, by selling the brand to Spireon.  Old-timers will recall LoJack used to work on radio.  It’s GPS now, like all the others.  “New LoJack by Spireon” is, in fact, old Spireon plus the stronger brand name.  The field today consists of:

    • Ikon
    • LoJack
    • Recover
    • SAVY
    • ZAZ

The model for all of these is that the dealer installs the tracking devices and uses them for lot management, and then sells them through to customers as theft protection.  They’re often sold as a nonnegotiable “preload,” which makes sense from the dealer’s perspective because it would cost another $50 of technician time to remove the device if the customer doesn’t want it.  You can see how consumer mobile app-appeal figures into our story.

If the device is drawing power from an OBD port, it can report the vehicle’s battery condition along with its location.  There’s a lot more you can do with OBD data, but manufacturers can be prickly about connecting to those other pins.  The typical device consists of the GPS chip, a cell modem, and an accelerometer.  You may have noticed that your iPhone also includes these parts, but not the OBD plug.

Speaking of those other pins, subprime lenders and BHPH dealers can wire the device to do starter interrupt.  That is, the OBD-powered devices.  The Recover device I saw at NADA is battery powered.  The argument for a battery-powered device is that it’s easier to install.  The opposing argument is around battery life, especially if you are selling it through, and the advanced capabilities available to an OBD scanner.

Connected Car Features

This brings me to the consumer features:

    • Service reminders
    • Teen driving
    • Driver performance
    • OBD health scan
    • Dealer inventory
    • Service scheduling
    • Credit application
    • Trip history
    • Recall notification
    • Digital glovebox

The astute reader will note that many of these features also aid the dealer in customer retention.  On the other hand, dealer-friendly features don’t mean a thing if the customer doesn’t use the app.  So, preloading can work against you if F&I fails to upsell the device properly.

Also, as mentioned above, your iPhone can support most of these functions on its own.  I run Life 360, which adds “insurance referral” to the driver performance feature.  The advantage to the dealer-installed device is that it’s physically attached to the vehicle.  By the way, you can buy a home OBD scanner for $30 at Walmart.

The dealer-installed GPS tracker is an amalgam of all these capabilities.  The key to success is exploiting them creatively and packaging them in ways that appeal to the consumer.

What is Accuracy?

Suppose you have tested positive for a rare and fatal disease, and your doctor tells you the test is 90% accurate.  Is it time to put your affairs in order?  Fortunately, no.  “Accuracy” means different things to different people, and it’s surprisingly easy to misinterpret.

What the 90% means to your doctor is that if ten people have the disease, then the test will detect nine of them.  This is the test’s “sensitivity.”  Sensitivity is important because you want to detect as many cases as possible, for early treatment.

On the other hand, like Paul Samuelson’s joke about the stock market having predicted nine of the last five recessions, sensitivity doesn’t tell you anything about the rate of false positives.  

If you’re into machine learning, you probably noticed that sensitivity is the same as “recall.”  Data scientists use several different measures of accuracy.  For starters, we have precision, recall, naïve accuracy, and F1 score.

There are many good posts on how to measure accuracy (here’s one) but few that place it in the Bayesian context of medical testing.  My plan for this article is to briefly review the standard accuracy metrics, introduce some notation, and then connect them to the inference calculations.

Accuracy Metrics for Machine Learning

First, here is the standard “confusion matrix” for binary classification.  It shows how test results fall into four categories: True Positives, True Negatives, False Positives, and False Negatives.  Total actual positives and negatives are P and N, while total predicted are and .

These are not only definitions, they’re numbers that express probabilities like the sensitivity formula, above.  This notation will come in handy later.  The standard definition of accuracy is simply the number of cases which were labeled correctly – true positives and true negatives – divided by the total population.

Unfortunately, this simple formula breaks down when the data is imbalanced.  I care about this because I work with insurance data, which is notoriously imbalanced.  The same goes for rare diseases, like HIV infection – which afflicts roughly 0.4% of people in the U.S.  Doctors use a metric called “specificity.”

The FP term in the denominator penalizes the model for false positives.  You can think of specificity as “recall for negatives.”  Doctors want a test with high sensitivity for screening, and then a more specific test for confirmation.  A good explainer from a medical perspective is here.

In a machine learning context, you want to optimize something called “balanced accuracy.”  This is the average of sensitivity and specificity.  For more on imbalanced data and machine learning, see my earlier post.

Bayes Theorem and Medical Testing

Bayes’ Theorem is a slick way to express a conditional probability in terms of its converse.  It allows us to convert “is this true given the evidence?” into “what would be the evidence if this were true?”

This kind of reasoning is obviously important for interpreting medical test results, and most people are bad at it.  I’m one of them.  I can never apply Bayesian reasoning without first making the diagram:

In this diagram, A is the set of people who have the disease and B is the set of people who have tested positive.  U is the universe of people that we’ve tested.  We have to make this stipulation because, in real life, you can’t test everyone.

We might assume that the base rate of disease in the wide world is A/U, but we only know about the people we’ve tested.  They may be self-selecting to take the test because they have risk factors, and this would lead us to overestimate the base rate.

Even within our tidy, tested universe, we can only estimate A by means of our imperfect test.  This is where some probability math comes in handy.  The true positives, people who tested positive and in fact have the disease, are the intersection of sets A and B.  Here they are, using conditional probability:

That is, the probability of testing positive if you’re sick, P(B|A), times the base probability of being sick, P(A).  Again, though, P(A) can be found only through inference – and medical surveillance.  Take a moment and think about how you would obtain these statistics in real life.

Mostly, you are going to watch the people who tested positive, set B, to see which ones develop symptoms.  The Bayesian framework gives you four variables to play with – five, counting the intersection set itself – so you can solve for P(A) in terms of the other ones:

That is, the probability of being sick if you’ve tested positive, P(A|B), times the probability of testing positive, P(B).  We know P(B) because we know how many people we’ve tested, U, and how many were positive.  Now that we’re in a position to solve for P(A) let’s bring back the other notation.

Accuracy Metrics and Bayes Theorem

Machine learning people use the accuracy metrics from the first section, above, while statistics people use the probability calculations from this second section.  I think it’s useful, especially given imbalanced medical (or insurance) data, to combine the two.

Now, we can rewrite the two conditional probability calculations, above, in terms of accuracy.  Set A = P, set B = P̂ , and the various metrics describe how they overlap.

And:

Giving our sick group as:

Finally, since you’re still worried about your positive test result … let’s assume the disease has a base rate of 1% – twice as virulent as HIV.  Recall that we never said what the test’s specificity was.  Since the test has good sensitivity, 90%, let’s say that specificity is weak, only 50%.

You are among 504 patients who tested positive.  Of these, only nine actually have the disease.  Your probability of being one of the nine is P(A|B).  This is the test’s precision, which works out to 1.8%.

Taxonomy of D2C Providers

PayLink was the finance source we chose for Safe-Guard’s D2C program, and now they have launched their own D2C program, Olive.  This looks like a strong program, and I’m flattered they’ve kept many features from my blueprint.  I especially like name, which is a play on “all of” coverage.

“We estimate the market size of the ‘sweet spot’ for post-OEM warranty VSC sales will continue to grow to 109 million vehicles by 2024”

This is a $260 billion market.  Colonnade estimates that the “sweet spot” of vehicles less than 11 years old but past their OEM warranty is 87 million units, and growing.  That’s not to mention the 40 million sold each year without a service contract, at franchise and independent dealers.  I wrote about the different segments and how to value them in The Case for D2C.

Olive is positioned to address both segments, by partnering with automakers.  They claim two of the global top five.  I have reason to believe these are Volkswagen and Nissan, but I couldn’t find a source.  Like the original, Olive uses digital marketing and online origination through a consumer facing mobile-responsive (but not mobile-first) platform.

Not one to believe in coincidence, I reckoned that maybe there is only one winning formula, and this is it.  That would be bad news for APC, US Direct, Dialog Direct, Endurance, Infinite, Forever, Car Shield, Car Chex, Delta, Omega, and Concord.

If you are a data-science inclined tech strategy consultant, you might think of this as a seven-dimensional feature space

I taxonomized a bit in my first post on the topic, Direct to Consumer VSC Sales, and today we will work out a generalized framework.  This means mixing apples and oranges.  Many of the competitors I studied are “pure play” B2C.

Business to consumer (B2C) is everyday marketing, with branding, advertising, and SEO.  Direct to consumer (D2C) means that you were marketing through a channel, and now you’re going direct.  It also means that you have a channel conflict to manage.  Think of an auto finance company that does both direct and indirect lending.

My research found seven features that characterize a consumer-facing VSC vendor.  You can think of these as design choices for a new entrant:

    1. Administrator
    2. Payment plan
    3. Touch points
    4. Lead source
    5. Marketing
    6. Branding
    7. Eligibility

Contract administrator – At Safe-Guard, obviously, we were the admin, although we branded the contract for various OEM clients.  Endurance administers their own contract, and Forever Car has an exclusive partner.  Most B2C vendors offer multiple contracts, with a variety of coverage choices.  Tec Assured works for the dealer, offering whichever contract is sold in the dealership.

Payment plan – No one pays cash for a service contract.  APC has its own finance arm, and Dialog Direct is part of Budco.  The challenge here is the credit-card security (PCI) standards.  See My Shift in the Call Center.  Admins recognize the need for a subscription-based VSC but none has yet cracked the code (maybe JM&A).  Bundling the contract with a payment plan achieves the same effect.

Touch points – In the earlier post, I described three points in the lifecycle which a vendor could target.  This dovetails with the lead source, below.  If the vendor is working with an OEM or a dealer, they can focus on new customers who didn’t buy in the dealership.  They also have access to service dates.  The pure B2C vendors generally aim for the warranty’s end, or they don’t choose a touchpoint at all.

Lead source – If the vendor is working with an OEM or a dealer (or any affinity group) then that’s a source of leads.  This may also qualify as a “relationship” for spam-law purposes.  Then, there are the usual sources, like vehicle registration lists.  This brings us to marketing.

Marketing – The pure B2C vendors use pure pull marketing, developing a brand through SEO, social, and advertising.  Who could forget the Car Shield ads featuring Ice-T?  Old-school telemarketing and direct mail are still in use – love those postcards!  As in the earlier post, my favorite approach is digital marketing, from a lead list, with call-center backup, driven by a CRM like Salesforce or Nutshell.

Branding – Branding is complicated in this space.  The pure B2C vendors must develop an online brand, even though they sell multiple admins’ contracts.  On the other hand, the affinity vendors may develop the client’s brand, or they may create a separate brand as a way of mitigating channel conflict.

Eligibility – Depending on the touch point, the easiest way to deal with pre-existing conditions is to sell while there’s still OEM warranty remaining, or enforce a thirty-day waiting period.  Servicecontract.com uses an inspection at Pep Boys, and there are also mobile inspection services.  Dealers using Tec Assured, obviously, have their own inspection capability.

Let’s demonstrate the framework with some worked examples:If you are a data-science inclined tech strategy consultant, you might think of this as a “feature space,” with each of the competitors staking out their territory on a seven-dimensional Go board.  You might also want to boil it down to three dimensions so your client can understand the diagram.

In the earlier post, I cited two broad categories: those that work with the dealer as a partner and lead source, and those that are pure B2C consumer-facing.  Here, I have shown a little more of the complexity.  Affinity marketing doesn’t stop with dealers, and some D2C vendors are hybrids.

If you’re a new entrant, this framework can help you structure your go-to market strategy.  If you’re an incumbent, you can play seven-D Go and outflank the competition.