Geometric complexity theory (GCT), is a research program in computational complexity theory proposed by Ketan Mulmuley and Milind Sohoni. The goal of the program is to answer the most famous open problem in computer science – whether P = NP – by showing that the complexity class P is not equal to the complexity class NP.
The idea behind the approach is to adopt and develop advanced tools in algebraic geometry and representation theory (i.e., geometric invariant theory) to prove lower bounds for problems. Currently the main focus of the program is on algebraic complexity classes. Proving that computing the permanent cannot be efficiently reduced to computing determinants is considered to be a major milestone for the program. These computational problems can be characterized by their symmetries. The program aims at utilizing these symmetries for proving lower bounds.
The approach is considered by some to be the only viable currently active program to separate P from NP. However, Ketan Mulmuley believes the program, if viable, is likely to take about 100 years before it can settle the P vs. NP problem.
The program is pursued by several researchers in mathematics and theoretical computer science. Part of the reason for the interest in the program is the existence of arguments for the program avoiding known barriers such as relativization and natural proofs for proving general lower bounds.