Metric Divergence Measures and Information Value in Credit Scoring

Abstract

Recently, a series of divergence measures have emerged from information theory and statistics and numerous inequalities have been established among them. However, none of them are a metric in topology. In this paper, we propose a class of metric divergence measures, namely, 𝐿_𝑝(𝑃‖𝑄), 𝑃≥1, and study their mathematical properties. We then study an important divergence measure widely used in credit scoring, called information value. In particular, we explore the mathematical reasoning of weight of evidence and suggest a better alternative to weight of evidence. Finally, we propose using 𝐿_𝑝(𝑃‖𝑄) as alternatives to information value to overcome its disadvantages.

1. Introduction

The information measure is an important concept in Information theory and statistics. It is related to the system of measurement of information or the amount of information based on the probabilities of the events that convey information. Divergence measures are an important type of information measures. They are commonly used to find appropriate distance or difference between two probability distributions.

Let Δ_𝑛=𝑃=𝑝₁,𝑝₂,…,𝑝_𝑛∣𝑝_𝑖≥0,^𝑛_𝑖=1𝑝_𝑖=1,𝑛≥2, (1) be the set of finite discrete probability distributions as in [1]. For all 𝑃, 𝑄∈Δ_𝑛, the following divergence measures are well known in the literature of information theory and statistics.

Hellinger Discrimination [2].ℎ()=1𝑃‖𝑄2^𝑛_𝑖=1𝑝_𝑖−√𝑞_𝑖². (2)

Shannon’s Entropy [3].𝐻(𝑃)=−^𝑛_𝑖=1𝑝_𝑖log₂𝑝_𝑖, (3) which is sometimes referred to as measure of uncertainty. The entropy 𝐻(𝑃) of a discrete random variable is defined in terms of its probability distribution 𝑃 and is a good measure of randomness or uncertainty.

Note that in the original definition of Shannon’s Entropy the log is to the base 2 and entropy is expressed in bits in information theory. The log can be any other bases and the entropy will be a constant factor of the one in base 2 by the change-base formula of the logarithm function. Hence, without loss of generality, we will assume all the logs are natural logarithms.

Kullback and Leibler’s Relative Information [4].𝐷(𝑃‖𝑄)=^𝑛_𝑖=1𝑝_𝑖𝑝ln^𝑖𝑞_𝑖. (4) Its symmetric form is the well-known 𝐽-divergence.

J-Divergence (Jeffreys [5], Kullback, and Leibler [4]).𝐽(𝑃‖𝑄)=𝐷(𝑃‖𝑄)+𝐷(𝑄‖𝑃)=^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖𝑝ln^𝑖𝑞_𝑖. (5)

Triangular Discrimination [6].Δ(𝑃‖𝑄)=^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖²𝑝_𝑖+𝑞_𝑖. (6)

Symmetric Chi-Square Divergence (Dragomir et al. [7]). One has 𝜓(𝑃‖𝑄)=𝜒²(𝑃‖𝑄)+𝜒²()=𝑄‖𝑃_𝑛𝑖=1𝑝_𝑖−𝑞_𝑖²𝑝_𝑖+𝑞_𝑖𝑝_𝑖𝑞_𝑖, (7) where 𝜒²(𝑃‖𝑄)=∑^𝑛_𝑖=1((𝑝_𝑖−𝑞_𝑖)²/𝑝_𝑖) is the well-known 𝜒²-divergence (Pearson [8]).

Jensen-Shannon Divergence (Sibson [9], Burbea, and Rao [10, 11]).𝐼()=1𝑃‖𝑄2^𝑛_𝑖=1𝑝_𝑖ln2𝑝_𝑖𝑝_𝑖+𝑞_𝑖+^𝑛_𝑖=1𝑞_𝑖ln2𝑞_𝑖𝑝_𝑖+𝑞_𝑖. (8)

Arithmetic-Geometric Divergence (Taneja [12]). Moreover 𝑇(𝑃‖𝑄)=^𝑛_𝑖=1𝑝_𝑖+𝑞_𝑖2𝑝ln^𝑖+𝑞_𝑖2𝑝_𝑖𝑞_𝑖. (9)

Taneja’s Divergence (Taneja [12]). One has 𝑑(𝑃‖𝑄)=1−^𝑛_𝑖=1𝑝_𝑖+√𝑞_𝑖2𝑝_𝑖+𝑞_𝑖2. (10) The information measures 𝐽(𝑃‖𝑄), 𝐼(𝑃‖𝑄) and 𝑇(𝑃‖𝑄) can be written as 𝐽()[𝐼()(,𝐼()=1𝑃‖𝑄=4𝑃‖𝑄+𝑇𝑃‖𝑄)]𝑃‖𝑄2𝐷𝑃‖𝑃+𝑄2+𝐷𝑄‖𝑃+𝑄2,𝑇()=1𝑃‖𝑄2𝐷𝑃+𝑄2‖𝑃+𝐷𝑃+𝑄2‖𝑄. (11)

Relative Information of Type s. Cressie and Read [13] considered the one-parametric generalization of information measure 𝐷(𝑃‖𝑄), called the relative information of type 𝑠 given by 𝐷_𝑠(𝑃‖𝑄)=[𝑠(𝑠−1)]^−1𝑛_𝑖=1𝑝^𝑠_𝑖𝑞_𝑖^1−𝑠−1,𝑠≠0,1. (12) It also has some special cases:(i)lim_𝑠→0𝐷_𝑠(𝑃‖𝑄)=𝐷(𝑄‖𝑃), (ii)lim_𝑠→1𝐷_𝑠(𝑃‖𝑄)=𝐷(𝑃‖𝑄), (iii)𝐷₋₁(𝑃‖𝑄)=(1/2)𝜓²(𝑄‖𝑃), (iv)𝐷_1/2(𝑃‖𝑄)=4ℎ(𝑃‖𝑄), (v)𝐷₂(𝑃‖𝑄)=(1/2)𝜓²(𝑃‖𝑄), (vi)𝐷₂(𝑃‖𝑄)=𝐷₋₁(𝑄‖𝑃), (vii)𝐷₁(𝑃‖𝑄)=𝐷₀(𝑄‖𝑃).

It is shown that 𝐷_𝑠(𝑃‖𝑄) is nonnegative and convex in 𝑃 and 𝑄 in [1].

J-Divergence of Type s [14].𝑉_𝑠(𝑃‖𝑄)=𝐷_𝑠(𝑃‖𝑄)+𝐷_𝑠()=[𝑠(𝑄‖𝑃𝑠−1)]^−1𝑛_𝑖=1𝑝^𝑠_𝑖𝑞_𝑖^1−𝑠+𝑞^𝑠_𝑖𝑝_𝑖^1−𝑠−2,𝑠≠0,1. (13)

It admits the following particular cases:(i)lim_𝑠→0𝑉_𝑠(𝑃‖𝑄)=lim_𝑠→1𝑉_𝑠(𝑃‖𝑄)=𝐽(𝑃‖𝑄), (ii)lim_𝑠→1𝑉_𝑠(𝑃‖𝑄)=𝐽(𝑃‖𝑄), (iii)𝑉₋₁(𝑃‖𝑄)=𝑉₂(𝑃‖𝑄)=(1/2)Ψ(𝑃‖𝑄), (iv)𝑉₀(𝑃‖𝑄)=𝑉₁(𝑃‖𝑄)=𝐽(𝑃‖𝑄), (v)𝑉_1/2(𝑃‖𝑄)=8ℎ(𝑃‖𝑄).

Unified Generalization of Jensen-Shannon Divergence and Arithmetic-Geometry Mean Divergence [14].𝑊_𝑠()=1𝑃‖𝑄2𝐷_𝑠𝑃+𝑄2‖𝑃+𝐷_𝑠𝑃+𝑄2‖𝑄. (14)

It admits the following particular cases:(i)𝑊₋₁(𝑃‖𝑄)=(1/4)Δ(𝑃‖𝑄), (ii)𝑊₀(𝑃‖𝑄)=𝐼(𝑃‖𝑄), (iii)𝑊_1/2(𝑃‖𝑄)=4𝑑(𝑃‖𝑄), (iv)𝑊₁(𝑃‖𝑄)=𝑇(𝑃‖𝑄), (v)𝑊₂(𝑃‖𝑄)=(1/16)Ψ(𝑃‖𝑄).

Taneja proved [14] that all the 3 𝑠-type information measures 𝐷_𝑠(𝑃‖𝑄), 𝑉_𝑠(𝑃‖𝑄), and 𝑊_𝑠(𝑃‖𝑄) are nonnegative and convex in the pair (𝑃,𝑄). He also obtained inequalities regarding the various divergence measures: 14Δ()()()()≤1𝑃‖𝑄≤𝐼𝑃‖𝑄≤ℎ𝑃‖𝑄≤4𝑑𝑃‖𝑄8𝐽()()≤1𝑃‖𝑄≤𝑇𝑃‖𝑄𝜓().16𝑃‖𝑄 (15) Here, we observe thatΔ(𝑃‖𝑄)>0. Hence, from the above inequalities we see that 𝐼(𝑃‖𝑄), ℎ(𝑃‖𝑄), 𝑑(𝑃‖𝑄), 𝐽(𝑃‖𝑄), 𝑇(𝑃‖𝑄), and 𝜓(𝑃‖𝑄) are all positive.

We also note that all 𝑝_𝑖 in the original definition of Δ_𝑛 in [1] are required to be positive. Yet, in realty some 𝑝_𝑖 may be 0. In this case 𝐼(𝑃‖𝑄), 𝑇(𝑃‖𝑄), 𝜓(𝑃‖𝑄), and Δ(𝑃‖𝑄) will be undefined. We have extended the definition of Δ_𝑛 to include the cases when 𝑝_𝑖=0. We assume that 0ln(0)=0, which is easily justified by continuity since 𝑥ln(𝑥)→0 as 𝑥→0. For convenience, we also assume 0ln(0/0)=0 and 𝑡ln(𝑡/0)=0 for 𝑡>0.

A problem with the above divergence measures is that none of them are a real distance, that is, a metric, in topology. In this paper, we will study a class of metric divergence measures 𝐿_𝑝(𝑃‖𝑄). We then study the underlying mathematics of a special divergence measure called information value, which is widely used in credit scoring. We propose using 𝐿_𝑝(𝑃‖𝑄) as alternatives to IV in order to overcome the disadvantages of information value.

The rest of this paper is organized as follows. In Section 2, after reviewing the metric space, we disprove that the above divergence measures are metrics. We then study a class of metric divergence measures 𝐿_𝑝(𝑃‖𝑄). Section 3 is concerned with information value. We examine a rule of thumb and weight of evidence and suggest a better alternative to weight of evidence. We then propose using 𝐿_𝑝(𝑃‖𝑄) as alternatives of IV to overcome the disadvantages of information value. Section 4 presents some numerical results. Finally, the paper is concluded in Section 5.

2. Metric Divergence Measures

2.1. Review of Metric Space

Definition 1. Suppose a real valued function 𝑑∶𝑀×𝑀→𝑅 and that for all 𝑥, 𝑦, 𝑧 of set 𝑀(1)𝑑(𝑥,𝑦)≥0 (nonnegative),(2)𝑑(𝑥,𝑥)=0 if and only if 𝑥=𝑦 (identity),(3)𝑑(𝑥,𝑦)=𝑑(𝑦,𝑥) (symmetry),(4)𝑑(𝑥,𝑧)≤𝑑(𝑥,𝑦)+𝑑(𝑦,𝑧) (triangle inequality).
Such a “distance function” 𝑑 is called a metric on 𝑀, and the pair (𝐷,𝑀) is called a metric space. If 𝑑 satisfies (1)–(3) but not necessarily (4), it is called a semimetric.
A metric space is a topological space in a natural manner, and therefore all definitions and theorems about general topological spaces also apply to a metric space. For instance, in a metric space one can define open and closed sets, convergence of sequences of points, compact space, and connected space.

Definition 2. A metric 𝑑₁ is said to be upper bounded by another metric 𝑑₂ if there exists a positive constant𝑐such that 𝑑₁(𝑥,𝑦)≤𝑐×𝑑₂(𝑥,𝑦) for all 𝑥, 𝑦∈𝑀. In this case, 𝑑₂ is said to be lower bounded by𝑑₁.
If 𝑑₁ is upper bounded by 𝑑₂, then the convergence in the metric space (𝑀,𝑑₂) implies the convergence in the metric space (𝑀,𝑑₁).

Definition 3. Two metrics 𝑑₁ and 𝑑₂ are equivalent if there exist 2 positive constants 𝛼 and 𝛽 such that 𝛼×𝑑₂(𝑥,𝑦)≤𝑑₁(𝑥,𝑦)≤𝛽×𝑑₂(𝑥,𝑦).
If two metrics 𝑑₁ and 𝑑₂ are equivalent, they will have the same convergence.

2.2. Nonmetric Divergence Measures

Proposition 4. None of the divergence measuresΔ(𝑃‖𝑄), 𝐼(𝑃‖𝑄), ℎ(𝑃‖𝑄), 𝑑(𝑃‖𝑄), 𝐽(𝑃‖𝑄), 𝑇(𝑃‖𝑄), and 𝜓(𝑃‖𝑄) are a metric in topology. Indeed, none of them satisfy the triangle inequality.

Proof. We disprove them either numerically or analytically by constructing counter examples in Δ₂. (a)Let𝑝=(0,1), 𝑞=(1,0), and 𝑟=(0.5,0.5). ThenΔ(𝑃‖𝑄)=2, Δ(𝑃‖𝑅)=Δ(𝑅‖𝑄)=2/3, andΔ(𝑃‖𝑅)+Δ(𝑅‖𝑄)=4/3<2=Δ(𝑃‖𝑄). (b)Let𝑝=(0.5,0.5), 𝑞=(0.2,0.8), and 𝑟=(0.3,0.7). Then 𝐼(𝑃‖𝑄)=0.02357767, 𝐼(𝑅‖𝑄)=0.00350105, 𝐼(𝑃‖𝑅)=0.01020550, and 𝐼(𝑃‖𝑅)+𝐼(𝑅‖𝑄)<𝐼(𝑃‖𝑄).(c)Let𝑝=(0,1), 𝑞=(1,0), and 𝑟=(0.5,0.5). Thenℎ(𝑃,𝑄)=1, ℎ(𝑃‖𝑅)=ℎ(𝑅‖𝑄)=(1/2)(2−√2), and ℎ(𝑃‖𝑅)+ℎ(𝑅‖𝑄)=2−√2<1=ℎ(𝑃‖𝑄).(d)Let 𝑝=(0.2,0.8), 𝑞=(0.4,0.6), and 𝑟=(0.3,0.7). Then 𝐷()()()2𝑃‖𝑅+𝐷𝑅‖𝑄−𝐷𝑃‖𝑄=0.2×ln38+0.8×ln73+0.3×ln47+0.7×ln61−0.2×ln24−0.8×ln32=0.2ln31−ln28+0.8ln74−ln33+0.3×ln47+0.7×ln64=0.2×ln36+0.8×ln73+0.3×ln47+0.7×ln64=0.2×ln36+0.8×ln74−0.3×ln36−0.7×ln74=−0.1×ln36+0.1×ln76=0.1ln74−ln39=0.1×ln14<0.1×ln1=0. (16) Hence,𝐷(𝑃‖𝑅)+𝐷(𝑅‖𝑄)<𝐷(𝑃‖𝑄). Indeed, 𝐷(𝑃‖𝑄) is not symmetric either (see [15]).(e)Let𝑝=(0.2,0.8), 𝑞=(0.4,0.6), 𝑟=(0.3,0.7). Then 𝐽()()=3𝑃‖𝑅+𝐽𝑅‖𝑄0.1×ln28+0.1×ln7+40.1×ln37+0.1×ln68=0.1×ln3,𝐽()4𝑃‖𝑄=0.2×ln2+0.2×ln38=0.2×ln3. (17) Hence,𝐽(𝑃‖𝑅)+𝐽(𝑅‖𝑄)<𝐽(𝑃‖𝑄).(f)Let𝑝=(0.5,0.5),𝑞=(0.2,0.8), and 𝑟=(0.3,0.7). Then 𝑇(𝑃‖𝑄)=0.05330024, 𝑇(𝑃‖𝑅)=0.02135897, 𝑇(𝑅‖𝑄)=0.00677313, and𝑇(𝑃‖𝑅)+𝑇(𝑅‖𝑄)<𝑇(𝑃‖𝑄).(g)Let 𝑝=(0.5,0.5), 𝑞=(0.2,0.8), 𝑟=(0.3,0.7). Then, 𝜓(𝑃‖𝑄)=369/400, 𝜓(𝑄‖𝑅)=185/1680, 𝜓(𝑃‖𝑅)=368/1050, and 𝜓(𝑃‖𝑅)+𝜓(𝑅‖𝑄)<𝜓(𝑃‖𝑄).

2.3. A Natural Metric Divergence Measure

If we pick up the common part of 𝐷(𝑃‖𝑄) and 𝐽(𝑃‖𝑄), we will obtain a metric divergence measure 𝑍(𝑃‖𝑄)=^𝑛_𝑖=1ln𝑝_𝑖−ln𝑞_𝑖=^𝑛_𝑖=1𝑝ln^𝑖𝑞_𝑖. (18) Since𝑝_𝑖≤1and|𝑝_𝑖−𝑞_𝑖|≤1 for all1≤𝑖≤𝑛, both𝐷(𝑃‖𝑄)and𝐽(𝑃‖𝑄)are upper bounded by𝑍(𝑃‖𝑄); that is, 𝐷(𝑃‖𝑄)≤𝑍(𝑃‖𝑄), 𝐽(𝑃‖𝑄)≤𝑍(𝑃‖𝑄).

2.4. 𝐿_𝑝-Divergence

Recall that for a real number 𝑝≥1, the𝑙_𝑝-norm of vector𝑥=(𝑥₁,𝑥₂,…,𝑥_𝑛)∈𝐑^𝑛is defined by ‖𝑥‖_𝑝=^𝑛_𝑖=1𝑥_𝑖^𝑝1/𝑝. (19) We will apply the𝑙_𝑝-metric from the𝑙_𝑝-norm to divergence measures to obtain𝑙_𝑝-divergence. For convenience, we will use the upper case notation.

Definition 5. For two probability distributions𝑃and𝑄, one defines their 𝐿_𝑝-divergence as 𝐿_𝑝(𝑃‖𝑄)=^𝑝𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖^𝑝. (20) Here,𝑝≥1is used for superscript, subscript, and radial root. It should not be confused with the vector𝑃.
In particular, when𝑝=1, we have the 𝐿₁ distance: 𝐿₁(𝑃‖𝑄)=^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖. (21) When𝑝=2, we obtain the Euclidean Distance: 𝐿₂(𝑃‖𝑄)=^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖². (22) When𝑝→∞, we have 𝐿_∞(𝑃‖𝑄)=max_{1≤𝑖≤𝑛}𝑝_𝑖−𝑞_𝑖. (23) It is known that𝑙_𝑝-norms are decreasing in𝑝. Moreover, all𝑙_𝑝metrics are equivalent.

Lemma 6. If𝑟>𝑝≥1, then the𝑙_𝑝-norms in𝐑^𝑛satisfy ‖𝑥‖_𝑟≤‖𝑥‖_𝑝≤𝑛^{(1/𝑝−1/𝑟)}‖𝑥‖_𝑟. (24)

Corollary 7. If𝑟>𝑝≥1, then the𝐿_𝑝-divergences satisfy 𝐿_𝑟(𝑃‖𝑄)≤𝐿_𝑝(𝑃‖𝑄)≤𝑛^{(1/𝑝−1/𝑟)}𝐿_𝑟(𝑃‖𝑄). (25)

Theorem 8. 𝐿_𝑝-divergences are all bounded by constant 2 for𝑝≥1; that is,𝐿_𝑝(𝑃‖𝑄)≤2.In particular,𝐿_∞(𝑃‖𝑄)≤1and𝐿₂(𝑃‖𝑄)≤√2.

Proof. We first prove the general case. From Corollary 7, it is sufficient to prove that the𝐿₁-divergence is bounded by 2. Let𝑃=(𝑝₁,𝑝₂,…,𝑝_𝑛)and𝑄=(𝑞₁,𝑞₂,…,𝑞_𝑛)be two probability distributions. Without loss of generality, let us assume that𝑝₁≥𝑞₁, 𝑝₂≥𝑞₂,…,𝑝_𝑘≥𝑞_𝑘, and 𝑝_𝑘+1<𝑞_𝐾+1,…,𝑝_𝑛<𝑞_𝑛. Then, we have 𝐿₁(𝑃‖𝑄)=^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖=^𝑘_𝑖=1𝑝_𝑖−𝑞_𝑖+^𝑛_𝑖=𝑘+1𝑞_𝑖−𝑝_𝑖≤^𝑘_𝑖=1𝑝_𝑖+^𝑛_𝑖=𝑘+1𝑞_𝑖≤^𝑛_𝑖=1𝑝_𝑖+^𝑛_𝑖=1𝑞_𝑖=1+1=2. (26) Noting that |𝑝_𝑖−𝑞_𝑖|≤1for1≤𝑖≤𝑛, we have(𝑝_𝑖−𝑞_𝑖)²≤|𝑝_𝑖−𝑞_𝑖|≤1.Hence, 𝐿_∞(𝑃‖𝑄)=max_{1≤𝑖≤𝑛}𝑝_𝑖−𝑞_𝑖𝐿≤1,₂(𝑃‖𝑄)=^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖²≤^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖=𝐿₁(𝑃‖𝑄)≤√2. (27) Therefore, we have proved the 2 particular cases.

The following result shows that the relative entropy𝐷(𝑃‖𝑄)is lower bounded by the square of the𝐿₁(𝑃‖𝑄). Its proof can be found at [15].

Lemma 9. 𝐷()≥1𝑃‖𝑄2𝐿₁(𝑃‖𝑄)². (28)

Theorem 10. The square root of 𝐽(𝑃‖𝑄) is lower bounded by𝐿₁(𝑃‖𝑄); that is, 𝐽(𝑃‖𝑄)≥𝐿₁(𝑃‖𝑄). (29)

Proof. Applying lemma to𝐷(𝑃‖𝑄)and𝐷(𝑄‖𝑃), we obtain 𝐷()()≥1𝑃‖𝑄+𝐷𝑄‖𝑃2𝐿₁(𝑃‖𝑄)²+12𝐿₁(𝑄‖𝑃)²=𝐿₁(𝑃‖𝑄)². (30) Note that the left hand side is nothing but𝐽(𝑃‖𝑄).The proof is completed by taking the square root on both sides.

Remark 11. 𝐽(𝑃‖𝑄)and hence𝐽(𝑃‖𝑄)are unbounded by any constants. This can be seen by taking𝑃=(𝑠,1−𝑠)and𝑄=(1−𝑠,𝑠), 0<𝑠<1, and taking limit 𝑠→0.Since𝐿_𝑝-divergences are all bounded by constant 2 for𝑝≥1, 𝐽(𝑃‖𝑄)and hence𝐽(𝑃‖𝑄)are not equivalent to𝐿_𝑝-divergences.
We now establish the convexity property for 𝐿_𝑝-divergence, which is useful in optimization.

Theorem 12. 𝐿_𝑝(𝑃‖𝑄)is convex in the pair(𝑝,𝑞), that is, if(𝑃₁,𝑄₁), and(𝑃₂,𝑄₂)are two pairs of probability distributions, then 𝐿_𝑝𝜆𝑃₁+(1−𝜆)𝑃₂‖𝜆𝑄₁+(1−𝜆)𝑄₂≤𝜆𝐿_𝑝𝑃₁‖𝑄₁+(1−𝜆)𝐿_𝑝𝑃₂‖𝑄₂ (31) for all0≤𝜆≤1.

Proof. Let 𝑃₁=(𝑝₁₁,𝑝₁₂,…,𝑝_1𝑛), 𝑃₂=(𝑝₂₁,𝑝₂₂,…,𝑝_2𝑛),𝑄₁=(𝑞₁₁,𝑞₁₂,…,𝑞_1𝑛), and 𝑄₂=(𝑞₂₁,𝑞₂₂,…,𝑞_2𝑛). Then 𝐿_𝑝𝜆𝑃₁+(1−𝜆)𝑃₂‖𝜆𝑄₁+(1−𝜆)𝑄₂=^𝑛_𝑖=1𝜆𝑝_1𝑖+(1−𝜆)𝑝_2𝑖−𝜆𝑞_1𝑖−(1−𝜆)𝑞_2𝑖^𝑝1/𝑝=^𝑛_𝑖=1𝜆𝑝_1𝑖−𝑞_1𝑖+(1−𝜆)𝑝_2𝑖−𝑞_2𝑖^𝑝1/𝑝≤^𝑛_𝑖=1𝜆𝑝_1𝑖−𝑞_1𝑖^𝑝1/𝑝+^𝑛_𝑖=1(1−𝜆)𝑝_2𝑖−𝑞_2𝑖^𝑝1/𝑝=𝜆^𝑛_𝑖=1𝑝_1𝑖−𝑞_1𝑖^𝑝1/𝑝+(1−𝜆)^𝑛_𝑖=1𝑝_2𝑖−𝑞_2𝑖^𝑝1/𝑝=𝜆𝐿_𝑝𝑃₁‖𝑄₁+(1−𝜆)𝐿_𝑝𝑃₂‖𝑄₂. (32) Here, the first inequality is from the well-known Minkowski’s Inequality.

It follows from the following results that we can generate infinitely many metric divergence measures using the existing ones.

Proposition 13. If 𝑑₁(𝑃‖𝑄) and 𝑑₂(𝑃‖𝑄) are two metric divergence measures, so are the following 3 measures:(1)𝛼×𝑑₁(𝑃‖𝑄)+𝛽×𝑑₂(𝑃‖𝑄) for all 𝛼≥0, 𝛽≥0 and 𝛼+𝛽>0,(2)max(𝑑₁(𝑃‖𝑄),𝑑₂(𝑃‖𝑄)),(3)(𝑑₁(𝑃‖𝑄))²+(𝑑₂(𝑃‖𝑄))².

Proof. The proof of (1) and (2) is trivial and hence will be omitted. As for (3), it is sufficient to verify the triangle inequality since nonnegative, identity, and symmetry are all easy to verify.
To begin with, let us first prove an inequality: for any nonnegative𝑎,𝑏,𝑐,𝑑, 𝑎²+𝑏²+𝑐²+𝑑²≤+2𝑎𝑐+2𝑏𝑑𝑎²+𝑏²+𝑐²+𝑑². (33) It is easy to see that inequality (33) is equivalent to the following inequality: 𝑎𝑐+𝑏𝑑≤𝑎²+𝑏²𝑐²+𝑑². (34) Inequality (34) is equivalent to the following inequality: 2𝑎𝑏𝑐𝑑≤𝑎²𝑑²+𝑏²𝑐². (35) Inequality (35) is equivalent to the following inequality: (𝑎𝑑−𝑏𝑐)²≥0. (36) Since inequality (36) is always true, inequality (34) is true.
Now, let us assume𝑃, 𝑄, 𝑅 are 3 arbitrary probability distributions. Since𝑑₁(𝑃‖𝑄)and𝑑₂(𝑃‖𝑄)satisfy the triangle inequality, we have 𝑑₁(𝑃‖𝑄)²+𝑑₂(𝑃‖𝑄)²≤𝑑₁(𝑃‖𝑅)+𝑑₁(𝑅‖𝑄)²+𝑑₂(𝑃‖𝑅)+𝑑₂(𝑅‖𝑄)^21/2=𝑑²₁(𝑃‖𝑅)+2𝑑₁(𝑃‖𝑅)𝑑₁(𝑅‖𝑄)+𝑑²₁(𝑅‖𝑄)+𝑑²₂(𝑃‖𝑅)+2𝑑₂(𝑃‖𝑅)×𝑑₂(𝑅‖𝑄)+𝑑²₂(𝑅‖𝑄)^1/2=𝑑²₁(𝑃‖𝑅)+𝑑²₂(+𝑑𝑃‖𝑅)₂₁(𝑅‖𝑄)+𝑑²₂(𝑅‖𝑄)+2𝑑₁(𝑃‖𝑅)𝑑₁(𝑅‖𝑄)+2𝑑₂(𝑃‖𝑅)𝑑₂(𝑅‖𝑄)^1/2≤𝑑²₁(𝑃‖𝑅)+𝑑²₂()+𝑃‖𝑅𝑑²₁(𝑅‖𝑄)+𝑑²₂(𝑅‖𝑄). (37) The last inequality results from inequality (33).

Remark 14. da Costa and Taneja [16] show that 𝑊_𝑠(𝑃‖𝑄) and 𝑉_𝑠(𝑃‖𝑄) are metrics divergence measures for all 𝑠. SinceΔ(𝑃‖𝑄), 𝐼(𝑃‖𝑄), ℎ(𝑃‖𝑄), 𝑑(𝑃‖𝑄), 𝐽(𝑃‖𝑄), 𝑇(𝑃‖𝑄), and 𝜓(𝑃‖𝑄) are all constant factors of special cases of 𝑊_𝑠(𝑃‖𝑄) or 𝑉_𝑠(𝑃‖𝑄), they are all metric divergence measures by Proposition 13. Yet, da Costa and Taneja did not disprove or discuss any applications of these divergence measures.

3. Information Value in Credit Scoring

Information value, or IV in short, is a widely used measure in credit scoring in the financial industry. It is a numerical value to quantify the predictive power of an independent continuous variable𝑥in capturing the binary dependent variable𝑦. Mathematically, it is defined as [17] IV=^𝑛_𝑖=1𝑔_𝑖𝑔−𝑏_𝑖𝑏𝑔×ln^𝑖/𝑔𝑏_𝑖,/𝑏 (38) where 𝑛 is the number of bins or groups of variable 𝑥, 𝑔_𝑖 and 𝑏_𝑖 are the numbers of good and bad accounts with bin𝑖, and 𝑔 and 𝑏 are the total number of good accounts and bad accounts in the population. Hence,𝑔_𝑖/𝑔and𝑏_𝑖/𝑏are distributions of good accounts and bad accounts. Therefore, ^𝑛_𝑖=1𝑔_𝑖𝑔=^𝑛_𝑖=1𝑏_𝑖𝑏=1. (39) Usually, “good” means 𝑦=0 and “bad” means 𝑦=1. It could be the other way, since IV is symmetric about good and bad. If 𝑔_𝑖/𝑔=𝑏_𝑖/𝑏 for all𝑖=1,…,𝑛, then IV = 0; that is, 𝑥 has no information on 𝑦.

IV is mainly used to reduce the number of variables as the initial step in the logistic regression, especially in big data with many variables. IV is based on an analysis of each individual predictor in turn without taking into account the other predictors.

3.1. IV and WOE

One advantage of IV is its close tie with weight of evidence (WOE), defined by ln((𝑔_𝑖/𝑔)/(𝑏_𝑖/𝑏)).WOE measures the strength of each grouped attribute in separating good and bad accounts. According to [17], WOE is the log of odds ratio, which measures odds of being good. Moreover, WOE is monotonic and linear.

Yet, WOE is not an accurate measure in that it is not the log of odds ratio and hence its linearity is not guaranteed. Indeed, 𝑔_𝑖/𝑔 and 𝑏_𝑖/𝑏 are from two different probability distributions. They represent the number of good accounts in bin𝑖divided by the total number of good accounts in the population and the number of bad accounts in bin𝑖divided by the total number of bad accounts in the population, respectively. In general, 𝑔_𝑖/𝑔+𝑏_𝑖/𝑏≠1 as can be seen from Exhibit  6.2 in [17].

To make WOE a log of odds, let us change its definition to 𝑔WOE1=ln^𝑖/𝑛_𝑖𝑏_𝑖/𝑛_𝑖𝑔=ln^𝑖𝑏_𝑖𝑏=−ln^𝑖𝑔_𝑖 (40) and denote it by WOE1. The cancelled 𝑛_𝑖=𝑔_𝑖+𝑏_𝑖 is the number of accounts in bin𝑖, and so 𝑔_𝑖/𝑛_𝑖+𝑏_𝑖/𝑛_𝑖=1.

As is well known, the logistic regression models the log odds, expressed in conditional probabilities, as a linear function of the independent variable; that is, 𝑃()ln𝑌=1∣𝑥𝑃(𝑌=0∣𝑥)=𝛽₀+𝛽₁𝑥. (41) When 𝑥 falls into bin 𝑖, ln(𝑃(𝑌=1∣𝑥)/𝑃(𝑌=0∣𝑥)) becomes ln(𝑏_𝑖/𝑔_𝑖)=−ln(𝑔_𝑖/𝑏_𝑖). Hence, the WOE1 values are either continuously increasing or continuously decreasing in a linear fashion.

IV and WOE1 can be used together to select independent variables for logistic regression. When a continuous variable 𝑥 has a large IV, we make it a candidate variable for logistic regression if WOE1 values are linear. It is common to plot the WOE1 values versus the mean values of 𝑥 at bin 𝑖.

3.2. A Rule of Thumb of IV

Intuitively, the larger the IV, the more predictive the independent variable. However, if IV is too large, it should be checked for over predicting. For instance, 𝑥 may be a postknowledge variable.

To quantify IV, a rule of thumb is proposed in [17, 18]: (i)less than 0.02: unpredictive,(ii)0.02 to 0.1: weak,(iii)0.1 to 0.3: medium,(iv)0.3+: strong.

In addition, mathematical reasoning of the rule of thumb is given in [18]. In more detail, IV can be expressed as the average of 2 likelihood ratio test statistics 𝐺(𝑃,𝑄) and 𝐺(𝑄,𝑃) of Chi-square distributions with (𝑛−1) degrees of freedom: 2×IV=2^𝑛_𝑖=1𝑝_𝑖𝑝ln^𝑖𝑞_𝑖+2^𝑛_𝑖=1𝑞_𝑖𝑞ln^𝑖𝑝_𝑖=𝐺(𝑃,𝑄)+𝐺(𝑄,𝑃). (42) The close relationship between IV and the likelihood ratio test allows using the Chi-square distribution to assign a significance level.

However, this is doubtful. On the one hand, 𝐺(𝑃,𝑄) and 𝐺(𝑄,𝑃) are not necessarily independent. On the other hand, even if they are independent, it is not enough. Let us assume that 2 × IV follows a Chi-square distribution with 2(𝑛−1) degrees of freedom. Yet, the critical values of the Chi-square distribution are too large compared with the values in the rule of thumb, as can be seen from the Chi-square table in many books about Probability, say [19].

We only list the first several rows of Table 1. When Table 1 grows as DF increases, the values in each column will increase. One may use the Excel function CHIINV(𝑝,𝑑𝑓) or its newer and more accurate version CHISQ.INV.RT(𝑝,𝑑𝑓) to build Table 1, which returns the inverse of the right-tailed probability 1−𝑝 of the Chi-square distribution with 𝑑𝑓 degrees of freedom.

Table 1 

Chi-square table.

The critical values are as small as the values in the rule of thumb only when the degrees of freedom are as small as 6. For instance, there is a probability of 1−0.005=0.995 that a Chi-square distribution with 6 degrees of freedom will be larger than or equal to 0.68, that is, CHIINV(0.995,6)=0.68. (43) Yet, there is a probability of 0.995 that a Chi-square random variable with 10 degrees of freedom will be larger than or equal to 2.16. There is a probability of 0.995 that a Chi-square random variable with 18 degrees of freedom will be larger than or equal to 6.26. On the basis of the above, the rule of thumb is more or less empirical.

3.3. Calculation of IV

The calculation of IV is simple once binning is done. In this sense, IV is a subjective measure. It depends on how the binning is done and how many bins are used. Different binning methods may result in different IV values, whereas the logistic regression in the later stages will not use the information of these bins.

In practice, 10 or 20 bins are used. The more the bins, the better the chance the good accounts will be separated from the bad accounts. Yet, we cannot divide the values of 𝑥 indefinitely since we may not avoid 0 good account or 0 bad account in some bins. To overcome the limitation of the logarithm function in the 𝐽-divergence, the binning should avoid 0 good account or 0 bad account in any bins.

The idea of binning is to assign values of𝑥with similar behaviors to the same group or bin. In particular, the same values of𝑥must fall into the same bin. A natural way of binning is to sort the data first and then divide them into 𝑛 bins with an equal number of observations (the last bin may have less number of observations). This works well if 𝑥 has no repeating values at all. In reality, 𝑥 often has repeating values (called the tied values in statistics), which may cause problems when the tied values of 𝑥 fall into different bins.

Proc Rank in SAS serves, a good candidate for binning (as opposed to function cut in 𝑅). When there are no tied values in 𝑥, it simply divides the values of𝑥into 𝑛 bins. When there are tied values in𝑥, it treats the tied values by its option TIES.

Proc Rank begins with sorting the values of 𝑥 within a BY group. It then assigns each nonmissing value an ordinal number that indicates its rank or position in the sequence. In case of ties, option TIES will be used. Depending on whether TIES = LOW, HIGH, or MEAN (default one), the lowest rank, highest rank, or the average rank will be assigned to all the tied values. Next, the following formula is used to calculate the binning value of each nonmissing value of𝑥: rank×𝑛𝑚+1, (44) where⌊⌋is the floor function, rank the value’s rank, 𝑛the number of bins, and 𝑚 the number of nonmissing observations. Note that the range of the binning values is from 0 to 𝑛−1. Finally, all the values of 𝑥 are binned according to their binning values. In case one bin has less than 5% of the population, we may combine this bin with its neighboring bin.

To illustrate the use of Proc Rank with𝑛=10and TIES = MEAN, let us look at an imaginary dataset with one variable age and 100 observations. Assume this dataset has been sorted and has fifty observations with a value of 10, thirty observations with a value of 20, ten observations with a value of 30, nine observations with a value of 40, and one observation with a value of 50.

The first 50 observations have a tied value of 10. Each of them will be assigned an average rank of 𝑘=25.5 and hence a binning value of ⌊(25.5×10)/101⌋=2. The next 30 observations have a tied value of 20. Each of them will be assigned an average rank of (51+52+⋯+80)/30=65.5 and hence a binning value of ⌊(65.5×10)/101⌋=6. The next 10 observations have a tied value of 30. Each of them will be assigned an average rank of (81+82+⋯+90)/10=85.5 and hence a binning value of ⌊(85.5×10)/101⌋=8. The next 9 observations have a tied value of 40. Each of them will be assigned an average rank of (91+92+⋯+99)/9=95 and hence a binning value of ⌊(95×10)/101⌋=9. The last observation has a rank of 100 and hence will be assigned a binning value of ⌊(100×10)/101⌋=9. In summary, the 100 observations are divided into 4 bins: the first 50 observations, the next 30 observations, the next 10 observations, and the last 10 observations.

Remark 15. Missing values are not ranked and are left missing in Proc Rank. Yet, they may be kept in a separate bin by means of Proc Summary or Proc Means in the calculation of IV.

Remark 16. If 𝑥 has less than𝑘different values, the number of bins by Proc Rank will be less that 𝑘.

After binning is done for 𝑥, a simple SAS program can be written to calculate IV. Meanwhile, WOE1 are calculated per bin as for WOE in [17]. If IV is less than 0.02, we will throw this independent variable. If IV is large than 0.3, over predicting will be checked. If IV is between 0.02 and 0.3 and WOE1 are linear, we will include this independent variable as a candidate variable in logistic regression. If IV is between 0.02 and 0.3 but WOE1 are not linear, we may make transformations of the independent variable to make WOE1 more linear. If a transformation can preserve the rank of the original independent variable, the binning by Proc Rank will be preserved. Therefore, we have obtained the following result.

Proposition 17. IV, when binning by Proc Rank, is invariant under any strictly monotonic transformations.

3.4. Mathematical Properties of IV

IV is the information statistic for the difference between the information in the good accounts and the information in the bad accounts. Indeed, IV is the 𝐽-divergence with distributions of good accounts and bad accounts. Thus, IV is lower bounded by the square of𝐿₁(𝑃‖𝑄)by Theorem 10.

Property 1. IV≥(𝐿₁(𝑃‖𝑄))².

Property 2. IV satisfies the inequalities (15); that is, 14Δ()()()()≤𝑃‖𝑄≤𝐼𝑃‖𝑄≤ℎ𝑃‖𝑄≤4𝑑𝑃‖𝑄IV8()≤1≤𝑇𝑃‖𝑄𝜓().16𝑃‖𝑄 (45) In particular, IV is upper bounded by𝜓(𝑃‖𝑄): 1IV≤2^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖²𝑝_𝑖+𝑞_𝑖𝑝_𝑖𝑞_𝑖. (46) Note that there is a direct proof of the above inequality, which is much easier than that in [14]. Let us assume without loss of generality that 𝑝₁≥𝑞₁, 𝑝₂≥𝑞₂,…,𝑝_𝑘≥𝑞_𝑘, and 𝑝_𝑘+1<𝑞_𝐾+1,…,𝑝_𝑛<𝑞_𝑛.

Proof. Using the identity 𝑝_𝑖/𝑞_𝑖=1+((𝑝_𝑖−𝑞_𝑖)/𝑞₁) and making Taylor’s expansion of function ln(𝑝_𝑖/𝑞_𝑖) around 1 for 𝑖=1,2,…,𝑘, we obtain 𝑝ln^𝑖𝑞_𝑖(1)+1=ln2𝑝_𝑖−𝑞_𝑖𝑞_𝑖−12(1+𝜉)²𝑝_𝑖−𝑞_𝑖𝑞_𝑖²≤12𝑝_𝑖−𝑞_𝑖𝑞_𝑖. (47) Multiplying 𝑝_𝑖−𝑞_𝑖≥0 and summing up from1 to 𝑘, we obtain ^𝑘_𝑖=1𝑝_𝑖−𝑞_𝑖𝑝ln^𝑖𝑞_𝑖≤12^𝑘_𝑖=1𝑝_𝑖−𝑞_𝑖²𝑞_𝑖≤12^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖²𝑞_𝑖. (48) Similarly,ln(𝑞_𝑖/𝑝_𝑖)≤(1/2)((𝑞_𝑖−𝑝_𝑖)/𝑝_𝑖).Multiplying𝑞_𝑖−𝑝_𝑖≥0and summing up from 𝑘 to 𝑛, we obtain ^𝑛_𝑖=𝑘+1𝑞_𝑖−𝑝_𝑖𝑞ln^𝑖𝑝_𝑖≤12^𝑛_𝑖=𝑘+1𝑞_𝑖−𝑝_𝑖²𝑝_𝑖≤12^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖²𝑝_𝑖. (49) The proof is completed by noting that ^𝑛_𝑖=1𝑝_𝑖−𝑞_𝑖𝑝ln^𝑖𝑞_𝑖=^𝑘_𝑖=1𝑝_𝑖−𝑞_𝑖𝑝ln^𝑖𝑞_𝑖+^𝑛_𝑖=𝑘+1𝑝_𝑖−𝑞_𝑖𝑝ln^𝑖𝑞_𝑖=^𝑘_𝑖=1𝑝_𝑖−𝑞_𝑖𝑝ln^𝑖𝑞_𝑖+^𝑛_𝑖=𝑘+1𝑞_𝑖−𝑝_𝑖𝑞ln^𝑖𝑝_𝑖. (50)

Theorem 18. IV is convex in the pair(𝑝,𝑞).

Proof. Applying Theorem 2.7.2 from [15] to both 𝐷(𝑃‖𝑄) and 𝐷(𝑄‖𝑃), we obtain 𝐽𝜆𝑃₁+(1−𝜆)𝑃₂‖𝜆𝑄₁+(1−𝜆)𝑄₂=𝐷𝜆𝑃₁+(1−𝜆)𝑃₂‖𝜆𝑄₁+(1−𝜆)𝑄₂+𝐷𝜆𝑄₁+(1−𝜆)𝑄₂‖𝜆𝑃₁+(1−𝜆)𝑃₂≤𝜆𝐷𝑃₁‖𝑄₁+(1−𝜆)𝐷𝑃₂‖𝑄₂+𝜆𝐷𝑄₁‖𝑃₁+(1−𝜆)𝐷𝑄₂‖𝑃₂=𝜆𝐷𝑃₁‖𝑄₁+𝐷𝑄₁‖𝑃₁+(1−𝜆)𝐷𝑃₂‖𝑄₂+𝐷𝑄₂‖𝑃₂=𝜆𝐽𝑃₁‖𝑄₁+(1−𝜆)𝐽𝑃₂‖𝑄₂. (51)

Remark 19. 𝐽(𝑃‖𝑄) is not convex albeit a metric.

Property 3. If more than 95% of population of𝑥have the same value, then IV = 0. In particular, if𝑥has just one value, then IV = 0.

Proof. Assume more than 95% of population of 𝑥 have the same value𝑥₀. Then, all the population with value𝑥₀will fall into the same bin, called the majority bin. The rest of population whose values are different from𝑥₀will be combined into the majority bin. Thus, there will be only one bin for all the values of𝑥.Therefore 𝑔₁=𝑔, 𝑏₁=𝑏, and hence IV = 0.

Remark 20. If the population whose values are not𝑥₀are not combined into the majority bin, then IV could be larger than 0.02. As shown in Table 2,𝑥has 10000 observations, where 95.8% or 9580 observations have the same value, say 2, and the rest of 420 observations have another value, say 4. Both bins contribute a value larger than 0.02 to IV. Statistically, 4.2% of the population are outliers and can be neglected. Hence, it is more meaningful to say𝑥has no information to𝑦.

Table 2 

Outlier domination to IV.

3.5. Alternatives to IV

As we have seen above, IV has 3 shortcomings: (1) it is not a metric; (2) no groups are allowed to have 0 bad accounts or 0 good accounts; and (3) its range is too broad, from 0 to∞.

Theoretically, any divergence measures of the difference or distance between good distributions and bad distributions can be alternatives to IV. In particular,𝐿_𝑝(𝑃‖𝑄)(𝑝≥1) are good alternatives to IV. They overcome all the 3 shortcomings of IV: (1)𝐿_𝑝(𝑃‖𝑄)are all metrics; (2) they allow bins to have 0 bad accounts or 0 good accounts; and (3) They all have a much narrow range, from 0 to 2.

While𝐿_𝑃(𝑃‖𝑄)do not seem to have a tie with weight of evidence, they can be as quantifiable as IV. For instance, we may adopt a rule of thumb for𝐿_𝑃(𝑃‖𝑄):(i)weak: 6% of its upper bound,(ii)medium: 6% to 30% of its upper bound,(iii)strong: Larger than 30% of its upper bound.

In particular, (i)weak: 0.12 for 𝐿₁(𝑃‖𝑄), 0.085 for 𝐿₂(𝑃‖𝑄), and 0.06 for 𝐿_∞(𝑃‖𝑄),(ii)medium: 0.12 to 0.60 for 𝐿₁(𝑃‖𝑄), 0.085 to 0.424 for 𝐿₂(𝑃‖𝑄), and 0.06 to 0.30 for𝐿_∞(𝑃‖𝑄),(iii)strong: 0.60+ for𝐿₁(𝑃‖𝑄), 0.424+ for𝐿₂(𝑃‖𝑄), 0.3+ for𝐿_∞(𝑃‖𝑄).

Remark 21. If 𝐿₁(𝑃‖𝑄)>0.12, then IV > 0.0144.

The lower bound 0.12 of 𝐿₁(𝑃‖𝑄) can be adjusted as needed. It can also be combined with IV to enhance the accuracy. For instance, for the number of independent variables is large enough, we may select only those which satisfy both lower bounds of 𝐿₁(𝑃‖𝑄) and IV.

4. Numerical Results

To illustrate our results, we use Exhibit  6.2 in [17] but add one column for WOE1. We use the real WOE, not its “more user-friendly” form—100 times WOE.

From Table 3, we see that 𝐿_∞<𝐿₂<𝐿₁<√IV. From Figure 1, we also see that the WOE1 for nonmissing values has a linear trend for variable age.

Table 3 

Calculation of IV and WOE.

Figure 1 

Logical Trend of WOE1 for Variable Age.

5. Conclusions

In this paper, we have proposed a class of metric divergence measures, namely, 𝐿_𝑝(𝑃‖𝑄), 𝑝≥1, and studied their mathematical properties. We studied information value, an important divergence measure widely used in credit scoring. After exploring the mathematical reasoning of a rule of thumb and weight of evidence, we suggested an alternative to weight of evidence. Finally, we proposed using 𝐿_𝑝(𝑃‖𝑄) as alternatives to information value to overcome its disadvantages.