reference link

1. What is SHAP for?

Say we train a prediction model as such:

• Input: age, gender, occupation
• Output: If the person likes computer games

There are two different ways to interpret the models.

• On a global level, we can check which features were the most predictive. There are numerous methods to predict this, such as the coefficients in the linear regression model and the number of splits in the tree-based models.

• On a local level, we can check the HETEROGENEITY of different features. For example, age was the most important feature across all the individuals, but say for Frank, a 50-year-old who works as a video game tester, his occupation is going to be much more significant than his age in determining whether he like computer games. Identifying which features were most important for Frank specifically involves finding feature importances on a ‘local’ – individual – level.

  Mathematical denotation of this locality is shown below:

  $$ g_{Frank} = \phi_{FrankAge} + \phi_{FrankGender} + \phi_{FrankOccupation} $$

  Note that, I don’t multiply the ϕ values by the corresponding x. Instead, I multiply it by 1 if the feature is present, and 0 if it is not.

2. Key assumptions of SHAP

SHAP is based on the game theory. Consider the following scenario: a group of people are playing a game. As a result of playing this game, they receive a certain reward; how can they divide this reward between themselves in a way which reflects each of their contributions?

There are a few things which everyone can agree on; meeting the following conditions will mean the game is ‘fair’ according to SHAP values:

• The sum of what everyone receives should equal the total reward.
• If two people contributed the same value, then they should receive the same amount from the reward.
• Someone who contributed no value should receive nothing.
• If the group plays two games, then an individual’s reward from both games should equal their reward from their first game plus their reward from the second game.

Translate this into the previous notation. Now we get:

• g_Frank should be equal to p_Frank, the probability the complex model assigned to Frank of liking computer games.
• If two features x contributed the same value to the final prediction, then their coefficients ϕ should have the same value
• If a feature contributed nothing to the final prediction (or if it is missing), then its contribution to g should be 0