Leo Breiman was the first person to notice the link between random forest and kernel methods. He pointed out that random forests which are grown using i.i.d random vectors in the tree construction are equivalent to a kernel acting on the true margin. Lin and Jeon established the connection between random forests and adaptive nearest neighbor, implying that random forests can be seen as adaptive kernel estimates. Davies and Ghahramani proposed Random Forest Kernel and show that it can empirically outperform state-of-art kernel methods. Scornet first defined KeRF estimates and gave the explicit link between KeRF estimates and random forest. He also gave explicit expressions for kernels based on centred random forest and uniform random forest, two simplified models of random forest. He named these two KeRFs by Centred KeRF and Uniform KeRF,and proved upper bounds on their rates of consistency.
Centred forest is a simplified model for Breiman's original random forest, which uniformly selects an attribute among all attributes and performs splits at the center of the cell along the pre-chosen attribute. The algorithm stops when a fully binary tree of level
k
is built, where
k
∈
N
is a parameter of the algorithm.
Uniform forest is another simplified model for Breiman's original random forest, which uniformly selects an attribute among all attributes and performs splits at a point uniformly drawn on the side of the cell, along the preselected attribute.
From random forest to KeRF
Given a training sample
D
n
=
{
(
X
i
,
Y
i
)
}
i
=
1
n
of
[
0
,
1
]
p
×
R
-valued independent random variables distributed as the independent prototype pair
(
X
,
Y
)
, where
E
[
Y
2
]
<
∞
. We aim at predicting the response
Y
,associated with the random variable
X
, by estimating the regression function
m
(
x
)
=
E
[
Y
|
X
=
x
]
. A random regression forest is an ensemble of
M
randomized regression trees. Denote
m
n
(
x
,
Θ
j
)
the predicted value at point
x
by the
j
-th tree, where
Θ
1
,
⋯
,
Θ
M
are independent random variables, distributed as a generic random variable
Θ
, independent of the sample
D
n
. This random variable can be used to describe the randomness induced by node splitting and the sampling procedure for tree construction. The trees are combined to form the finite forest estimate
m
M
,
n
(
x
,
Θ
1
,
⋯
,
Θ
M
)
=
1
M
∑
j
=
1
M
m
n
(
x
,
Θ
j
)
. For regression trees, we have
m
n
=
∑
i
=
1
n
Y
i
1
X
i
∈
A
n
(
x
,
Θ
j
)
N
n
(
x
,
Θ
j
)
, where
A
n
(
x
,
Θ
j
)
is the cell containing
x
, designed with randomness
Θ
j
and dataset
D
n
, and
N
n
(
x
,
Θ
j
)
=
∑
i
=
1
n
1
X
i
∈
A
n
(
x
,
Θ
j
)
.
Thus random forest estimates satisfy, for all
x
∈
[
0
,
1
]
d
,
m
M
,
n
(
x
,
Θ
1
,
…
,
Θ
M
)
=
1
M
∑
j
=
1
M
(
∑
i
=
1
n
Y
i
1
X
i
∈
A
n
(
x
,
Θ
j
)
N
n
(
x
,
Θ
j
)
)
. Random regression forest has two level of averaging, first over the samples in the target cell of a tree, then over all trees. Thus the contributions of observations that are in cells with a high density of data points are smaller than that of observations which belong to less populated cells. In order to improve the random forest methods and compensate the misestimation, Scornet defined KeRF by
m
~
M
,
n
(
x
,
Θ
1
,
…
,
Θ
M
)
=
1
∑
j
=
1
M
N
n
(
x
,
Θ
j
)
∑
j
=
1
M
∑
i
=
1
n
Y
i
1
X
i
∈
A
n
(
x
,
Θ
j
)
,
which is equal to the mean of the
Y
i
's falling in the cells containing
x
in the forest. If we define the connection function of the
M
finite forest as
K
M
,
n
(
x
,
z
)
=
1
M
∑
j
=
1
M
1
z
∈
A
n
(
x
,
Θ
j
)
, i.e. the proportion of cells shared between
x
and
z
, then almost surely we have
m
~
M
,
n
(
x
,
Θ
1
,
…
,
Θ
M
)
=
∑
i
=
1
n
Y
i
K
M
,
n
(
x
,
x
i
)
∑
ℓ
=
1
n
K
M
,
n
(
x
,
x
ℓ
)
, which defines the KeRF.
The construction of Centred KeRF of level
k
is the same as for centred forest, except that predictions are made by
m
~
M
,
n
(
x
,
Θ
1
,
…
,
Θ
M
)
, the corresponding kernel function, or connection function is
K
k
c
c
(
x
,
z
)
=
∑
k
1
,
…
,
k
d
,
∑
j
=
1
d
k
j
=
k
k
!
k
1
!
…
k
d
!
(
1
d
)
k
∏
j
=
1
d
1
⌈
2
k
j
x
j
⌉
=
⌈
2
k
j
z
j
⌉
, for all
x
,
z
∈
[
0
,
1
]
d
.
Uniform KeRF is built in the same way as uniform forest, except that predictions are made by
m
~
M
,
n
(
x
,
Θ
1
,
…
,
Θ
M
)
, the corresponding kernel function, or connection function is
K
k
u
f
(
0
,
x
)
=
∑
k
1
,
…
,
k
d
,
∑
j
=
1
d
k
j
=
k
k
!
k
1
!
…
k
d
!
(
1
d
)
k
∏
m
=
1
d
(
1
−
|
x
m
|
∑
j
=
0
k
m
−
1
(
−
ln
|
x
m
|
)
j
j
!
)
, for all
x
∈
[
0
,
1
]
d
.
Relation between KeRF and random forest
Predictions given by KeRF and random forests are close if the number of points in each cell is controlled:
Assume that there exist sequences
(
a
n
)
,
(
b
n
)
such that, a.s.
,
a
n
≤
N
n
(
x
,
Θ
)
≤
b
n
and
a
n
≤
1
M
∑
m
=
1
M
N
n
x
,
Θ
m
≤
b
n
,
then almost surely
,
|
m
M
,
n
(
x
)
−
m
~
M
,
n
(
x
)
|
≤
b
n
−
a
n
a
n
m
~
M
,
n
(
x
)
.
Relation between infinite KeRF and infinite random forest
When the number of trees
M
goes to infinity, then we have infinite random forest and infinite KeRF. Their estimates are close if the number of observations in each cell is bounded:
Assume that there exist sequences
(
ε
n
)
,
(
a
n
)
,
(
b
n
)
such that, a.s.
E
[
N
n
(
x
,
Θ
)
]
≥
1
,
P
[
a
n
≤
N
n
(
x
,
Θ
)
≤
b
n
∣
D
n
]
≥
1
−
ε
n
/
2
,
P
[
a
n
≤
E
Θ
[
N
n
(
x
,
Θ
)
]
≤
b
n
∣
D
n
]
≥
1
−
ε
n
/
2
,
Then almost surely,
|
m
∞
,
n
(
x
−
m
~
∞
,
n
(
x
)
|
≤
b
n
−
a
n
a
n
m
~
∞
,
n
(
x
)
+
n
ε
n
(
max
1
≤
i
≤
n
Y
i
)
.
Assume that
Y
=
m
(
X
)
+
ε
, where
ε
is a centred Gaussian noise, independent of
X
, with finite variance
σ
2
<
∞
. Moreover,
X
is uniformly distributed on
[
0
,
1
]
d
and
m
is Lipschitz. Scornet proved upper bounds on the rates of consistency for centred KeRF and uniform KeRF.
Providing
k
→
∞
and
n
/
2
k
→
∞
, there exists a constant
C
1
>
0
such that, for all
n
,
E
[
m
~
n
c
c
(
X
)
−
m
(
X
)
]
2
≤
C
1
n
−
1
/
(
3
+
d
log
2
)
(
log
n
)
2
.
Providing
k
→
∞
and
n
/
2
k
→
∞
, there exists a constant
C
>
0
such that,
E
[
m
~
n
u
f
(
X
)
−
m
(
X
)
]
2
≤
C
n
−
2
/
(
6
+
3
d
log
2
)
(
log
n
)
2
.
