The term mathematical theory is used to refer to a mathematical subfield. A theory can be a body of knowledge, and so in this sense a "mathematical theory" refers to an area of mathematical research. This is distinct from the idea of mathematical models. Branches of mathematics like group theory and number theory are examples of this.
References
Mathematical theory Wikipedia(Text) CC BY-SA