Predicting Loan Defaults with AI

I have some time on my hands, so I decided to experiment with some of the new AI assisted code generators.  I wanted something relevant to F&I, and I found this exercise on Coursera.  The “F” side of F&I gets all the attention, but there is plenty of opportunity for AI to rate insurance risk and mechanical breakdown risk.

Note that we are using AI to generate an AI model.  For the Coursera exercise, linear regression is sufficient, but I chose to use neural networks here because they are undeniably machine learning.  See my earlier post on this, “What Is Real AI?”

Today, we’ll look at three popular AI assistants: ChatGPT, GPT-Engineer, and GitHub Copilot.  These are all based on the famous OpenAI large language model, just packaged a little differently.

To start, I worked the problem myself, running several different models.  The goal is to predict the probability of a given loan going bad, based on seventeen variables including credit score, term, and debt to income.  Once I was satisfied with my solution, I turned the problem over to my robot friend, ChatGPT.

ChatGPT

Using the chat window requires you to cut and paste code over to your IDE, so it doesn’t really feel like a code generator.  On the other hand, it’s conversational, so it can tell you its assumptions and you can give clarifying instructions.  Here is the prompt I used:

We need to write a Python script to predict loan defaults using a neural network model, and based on some input data. To start, read the input data from a CSV file and create a data frame. Some of the columns have numeric data and some have categorical data. The last column is the dependent variable. It has values of either zero or one. Next, prepare the data for use in a neural network by running it through an appropriate pipeline. Split off twenty percent of the rows, randomly, for use as a test set. Finally, train the neural network using the remaining eighty percent of the rows. We want to know the probability of a loan default. Test the neural network by comparing its predictions for the test set, and report your results by plotting the ROC curve.

Ordinarily, this would be more interactive, but I wanted to write all the requirements into a single prompt for demonstration purposes, and because that’s the approach preferred by GPT-Engineer.

ChatGPT solved the problem nicely, matching my best score of 0.76.  It plotted its own ROC chart, below, instead of using the Scikit method, even though it did use Scikit for the pipelines.

ChatGPT did several things differently from me, mostly simplifications, and it made some mistakes.  For example, it forgot that its “create neural network” function didn’t include compilation.  The call to compile() can be in or out of the function, so it’s easy to misplace.

The creepiest thing ever?  ChatGPT chose random_state = 42, which is completely arbitrary – and the same number I use!  Like, how did it know?

GPT-Engineer

GPT-Engineer uses the OpenAI API, for which you have to buy tokens.  I think of it as a wrapper for the ChatGPT dialogue above.  GPT-Engineer also runs outside of the IDE.  You invoke it from the command line and pass a file containing the prompt.

The special charm of GPT-Engineer is that it can ask clarifying questions.  Here are the questions it had about my prompt:

1. The specific format and structure of the input CSV file.
2. The exact columns that contain numeric and categorical data.
3. The type of neural network model to be used.
4. The specific pipeline to be used for preparing the data.
5. The method for splitting the data into training and test sets.
6. The method for training the neural network.
7. The method for testing the neural network and comparing its predictions.
8. The method for reporting results and plotting the ROC curve.

Some of these questions are reasonable and some are not.  ChatGPT, on its own, figured out how to handle the file, the pipeline, and the structure of the neural net.  In fact, it was kind of fun comparing notes with ChatGPT like, “oh yeah, well, I’m using batch norms.”

I suspect that “code and train a neural net” is asking too much.  GPT-Engineer crushes routine tasks, as my man Arjan demonstrates here.

GPT Engineer is another nail in the coffin of software developers. In this video, I’ll show you how it works. The tool is crazy powerful. Just specify what you want it to build, and then, well, it just builds it.

GitHub Copilot

Microsoft did a nice job of integrating Copilot into Visual Studio and several other IDEs.  You install an extension, and subscribe to the service for $10 a month.  Microsoft has a big ($13 billion) investment in OpenAI, and they own GitHub.  This means an LLM trained not only on human language, but on a giant repository of source code.

Microsoft advertises Copilot as “pair programming,” like having a buddy look over your shoulder, and it works the same way autocomplete works for text.  It can also define a function based on an inline comment like, “read a file from disk and create a dataframe.”

Copilot didn’t really suit me.  I wanted to see how an AI would code differently from me, as ChatGPT had, but Copilot kept serving up my own code from GitHub.  Also, it kept wanting to define functions where it should have just written the line, like pd.read_csv(“test.csv”).

Conclusion

As I said at the top, part of the fun is having an AI program write an AI program – although, in this case, any decent predictive model would suffice.  OpenAI is, itself, driven by a large language model (LLM).  So, here we have a large, general, neural network helping me to produce a small, tailored one.

What does all this mean for the industry?  Well, for one thing, it is starting to look bad for software developers.  Arjan suggests it will take out the junior engineers first, but I’m not so sure.

Researchers have long feared that the resources required to build and train foundation models would mean a Big Tech oligopoly.  Technologically, there have been good results in the other direction, with small open-source models.  Commercially, however, this is a race between Microsoft and Google.

Microsoft is also introducing other Copilots, and researchers are hard at work on natural language prompting for all computer tasks.  So, the same way I can prompt GPT-Engineer to write some code, you’ll be able to have an AI do whatever you were planning to do on Excel or Tableau.