Tag: top

AI-Based VSC Risk Rating

I have been working on a startup that will use artificial intelligence to rate vehicle service contracts.  For a VSC provider, increased accuracy means sharper pricing and, potentially, lower reserves.  Outsourcing this work to a specialist bureau means reduced costs, too.  Our business model is already used for risk rating consumer credit, and the technology is already used for risk rating auto insurance.

In this article, I present an example using auto insurance data.  If you would like to see how our approach works with VSC data, please get in touch.  We are currently seeking VSC providers for our pilot program.

The French MTPL dataset is often cited in the AI literature.  It gives the claims history for roughly 600,000 policies.  Of these, I used 90% for training and set aside 10% for testing.  So, the results shown here are not just “curve fitting,” but predictions against new data.

The Gini Coefficient

The challenge with insurance data is that most policies never have a claim.  This is known as the imbalanced data problem.  If you’re training an AI classifier, it can achieve 95% accuracy simply by predicting “no claim” every time.  You will want to use an objective function that heavily penalizes false negatives, and you may also want to oversample the “with claim” cases. 

The dashed line in the chart above represents cumulative actual claims, sorted in order of increasing severity.  This is called the Lorenz curve.  You can see that it’s zero all the way across and then, at the 95% mark, the claims kick in. 

The blue line is the Lorenz curve for the predictive model.  A good fit here would be a deep concavity that hugs the dotted line.  That would mean the model is estimating low where the actuals are low (zero) and then progressively steeper.

The Gini index is a measure of the Lorenz curve’s concavity.  This 0.30 is pretty good.  The team that won the Allstate Challenge did it with 0.20.  The downside to Gini is that it only tests the model’s ability to rank relative risks, not absolute ones.  I have seen models up above 0.40 that were still way off on actual dollars.

Mean Absolute Error

The key metric, to my way of thinking, is being able to predict the total claims liability.  This automatically gives you the mean, and Gini characterizes the distribution.  I like MAE because it represents actual dollars, and it’s not pulled astray by outliers (like mean squared error). 

Here, you see that the model overestimates by 1.2%.

You may be wondering why MAE is so high, when we are within $1.00 on the average claim.  That’s because all of the no-claim people were estimated at an average of $72.50, and they’re 95% of the test set.  The average estimate for the group that turned out to have claims (remember, this is out-of-sample data) was $130.70. 

Neural Networks

For claim severity, I trained a small neural net, including my own custom layers for scaling and encoding.  I really like TensorFlow for this, because it saves the trained encoders as part of the model.  You want to use a small neural net with a small dataset, because a bigger one can simply memorize the training data, and not be predictive at all.

This dataset has only nine features and, in fairness, a linear model would fit it just as well.  My code repo is now filled with neural nets, random forests, and two-stage combo models.  What this means for our startup is that we don’t have to hire a platoon of actuaries.  We can get by with a few data scientists using AI as a “force multiplier.”

Earlier this century, I played a key role in moving the industry to electronic origination.  At the time, it was clear that the API approach would liberate VSC pricing from the confines of printed rate cards and broad risk classes.  Each rate quote could be tailored to the individual vehicle.

As I said earlier, our approach is current, proved, and working elsewhere.  It’s just not being used in the VSC industry … yet.

Workflow for Online Car Buying

A few years ago, I published a precedence diagram for the key operations of online car buying.  I was arguing against a linear process, and calling attention to some deadlocks.  Since then, I have been following the industry’s experiments with new process models, and coming to realize that these deadlocks are the great, unanalyzed, obstacle to process reform.

Practices that seem unfair, deceptive, or abusive may actually be crude attempts to solve the deadlock problem.

One example of a deadlock is that you can’t quote an accurate payment until you know the buy rate, and for that you need to submit a credit application.  This is usually solved by iteration.  You do a pre-approval or quote the floor rate, and then change it later.

Likewise, you can’t price protection products until you know the vehicle, but the customer wants to shop by payment.  Protection products are also priced by term, and you don’t know the desired term until you finish structuring the deal.

In fact, even the customer’s choice of vehicle depends on the monthly payment, which is downstream of everything else.  Virtually the only operation that’s not blocked by another operation is valuing the trade.

Like an interlocking puzzle, “we don’t know anything until we know everything.”  Choosing any one item to lock first, without iteration, will result in a suboptimal deal – buying too much car, for too long a term, or overlooking the protection products.

Practices that seem unfair, deceptive, or abusive may actually be crude attempts to solve the deadlock problem.  For instance, quoting a payment with some leg in it, or goal-seeking the full approval amount.

Can you see how this ties into current debates about the hybrid sales model?  F&I presents a menu with a six-month term bump, which might not be optimal, just to compensate for too tight a payment from the desk.

Fortunately, in the world of online car buying, the customer is free to resolve deadlocks through iteration.  This means:

  1. Set up the deal one way
  2. Change any feature, like the term
  3. The change “cascades,” undoing other features
  4. Revisit those other features
  5. Repeat until all features look good together

The in-store process does not support iteration well, and probably never will, but an online process can.  All you need is the well-known concept of a “dirty” flag, to keep track of the cascading changes, along with navigation and a completeness gauge to guide the customer through steps #4 and 5.

You could analyze step #3 at the level of a dozen individual features.  I made that chart, too, but I believe it’s more useful to collect them into the canonical five pages shown here.

By the way, I have previously described the products page in some detail, along with the analytics to drive it.  Discussion of the “random survey question” is here.  Today’s diagram contemplates a mobile app, as do my recent posts, but the same approach will work for a web site.

Deconstructing the Dealership

Remember when dealerships had body shops?  Two out of five still do, but they comprise less than 20% of this $35 billion market.  Somewhere along the line, it became clear that collision repair was better done by specialist facilities, unconnected to the dealer.  Scale, capital investment, brand diversification, and (lack of) synergy were factors.

We may now wonder if parts and service belong in the dealership, thanks in some measure to the rise of automotive eCommerce.  Jim Ziegler warns that Valvoline Express is beating dealers in the shop and online.  Ward’s makes the same point, with emphasis on Google search optimization.  In the same vein, Amazon has come up with a way to sell tires online.

There can be much synergy between the two ends of the business, which can be leveraged to manage and sustain customer relationships – Vincent Romans

My approach is to “follow the money” and, sure enough, here is Carl Icahn buying up repair facilities.  Icahn Automotive Group is a classic consolidation play, with 2,000 locations including Precision Auto Care, Pep Boys, Just Brakes, AutoPlus, AAMCO, Cottman, and CAP.  Icahn is vertically integrated through Federal-Mogul Motorparts, which includes ANCO wipers and Champion spark plugs.

So, will eCommerce pick off the dealer’s profit centers one by one?  In this example, we see the convergence of powerful megatrends.  The traditional dealer model is challenged by two new ones, which I like to call the Best Buy model and Amazon model.

History tells us that the Amazon model will prevail in the end, but it doesn’t tell us what the transformation will look like, or how dealers should prepare.  To learn that, we employ an old tool from Business Process Reengineering, and we discover a surprising result.  Here is a breakdown of the traditional dealer operations:

The Seven Profit Centers of a Car Dealer

  1. New Sales
  2. Used Sales
  3. Finance
  4. Insurance
  5. Parts
  6. Service
  7. Collision Repair

We can consider each operation in terms of how it will respond to the new challenges – and whether it belongs with the others.  We have to start somewhere, so let us define new vehicle sales as the nucleus of the dealership.  The test drive is the process most resistant to eCommerce although, as I wrote last week, there are ways around it.

Used vehicle sales will certainly not stay in the dealership.  It is vulnerable to both consolidators and eCommerce.  This is a shame because taking vehicles in trade used to be a great synergy.  The new specialists are true “auto traders,” using high-volume analytics to trade both ways with the public and the auction.

Coming back to fixed operations, there is a clear synergy.  According to Cox research, customers who are properly introduced to the service department are two and a half times more likely to come back for service.  But there are other ways to exploit this synergy, like the “zero deductible at our dealership” service contract – and the Amazon tire store shows that parts can be separated from service.

Lithia Motors has 186 locations including, by my count, fourteen collision centers.  There is not much synergy between body shops and vehicle sales, or even service, but they run fine as standalone operations connected to the brand.  Likewise, given a branded service contract, I can see Lithia’s mass market franchises supporting shared service facilities.

F&I is the subject of fierce debate, too much to cover here.  Can it be merged into the sales function? Can protection products be sold successfully online?  What is the future of indirect finance?  Do “F” and “I” even belong together anymore?  For our purpose today, we need only observe that while F&I has a workflow linkage to sales, it does not need a physical one.  F&I could just as easily skype in from a call center.

As Carl Icahn would tell you, these are distinct businesses without much synergy, if synergy is defined as “positive return from shared personnel and facilities.”  Dealers organized along these lines will, indeed, be picked apart by eCommerce and consolidation.

On the other hand, if synergy means “positive return from shared customer contact and branding,” then these businesses will hang together.  Dealers organized along this principle will have diverse and independent operations, making them resilient to disruption.  They will have “optionality,” to use Nassim Taleb’s term.

You may be taken aback by this assault on the venerable “rooftop,” and I admitted earlier to being surprised.  However, decoupling and diversification (and divestiture) are textbook responses to an industry in flux.  Just look at how many departments are no longer in department stores.

Car Dealer Megatrends – Conclusion

This is the conclusion of my series on car dealer megatrends.  The first three articles covered the long running trend toward consolidation, steadily improving process maturity, and disruption from new technology.  Like all good megatrends, these three flow together, reinforcing each other to produce a sea change in the industry.  Consolidation means bigger groups with more money to spend on technology, and the scale to exploit improved procedures.

Big dealer groups crave stability, and repeatable successes.  In my trade, software development, we have a formal process maturity model.  The bottom rung is where your success depends on “heroes and luck.”  When you own 20 stores, you are less interested in one superstar killing the pay plan, and much more interested in a hundred guys making base hits.  If you are not clear on this, I recommend the movie version of Moneyball, featuring Brad Pitt as Billy Beane.

We’re making less per transaction, but we’re doing more transactions.

I work mainly in F&I, but you can see the same general idea in the velocity method for new and used car sales.  That idea is margin compression.  The quote above is from Paragon Honda’s Brian Benstock and, last I checked, he was still hard at it.

The locus of high gross shifted from new cars to F&I, and then from finance to products.  Smart people tell me the 100% markup on products will soon be ended, either by competition or by the CFPB.  Today, when you read about the latest PVR record from Group 1 (or whomever) you will also read management downplaying expectations of further such records.

The executive, however, said the group’s F&I operations may have reached the peak in terms of PVR.

Dealership ROI is above 20% but, as you know, highly cyclical.  The stock market has been around 14% lately and, arguably, less volatile.  AutoNation has been chugging along at a steady 10%.  Investors will accept a lower return, in exchange for stability.

AutoNation was founded in the era of big box retail.  My colleague there, Scott Barrett, came from Blockbuster.  It was always our intention to remake auto retail in the image of Circuit City, which, by the way, was the parent of CarMax.

I spoke with an AutoNation executive recently who told me that learning to live with margin compression is an explicit part of their strategy.  It is an iron law of economics that, in a free market, competition will drive margins toward zero.

Have a look at this NADA chart.  In five years, gross has been cut almost in half.  This is a breathtaking diminution, and then you go on the industry forums and find people bitching that vAuto has cut used car gross, and TrueCar has cut new car gross, and now some idiot proposes to cut F&I gross by putting VSC prices online.

Marv Eleazer has called this a race to the bottom, and he’s right, but this is not a race you can opt out of.  That’s not how competition works.  Think of it as a race run in Mexico City.  The smart dealers and big groups are already training to compete in the thin air of lower gross.