The term transformation theory refers to a procedure and a "picture" used by P. A. M. Dirac in his early formulation of quantum theory, from around 1927.
This "transformation" idea refers to the changes a quantum state undergoes in the course of time, whereby its vector "moves" between "positions" or "orientations" in its Hilbert space. Time evolution, quantum transitions, and symmetry transformations in Quantum mechanics may thus be viewed as the systematic theory of abstract, generalized rotations in this space of quantum state vectors.
Remaining in full use today, it would be regarded as a topic in the mathematics of Hilbert space, although, technically speaking, it is somewhat more general in scope. While the terminology is reminiscent of rotations of vectors in ordinary space, the Hilbert space of a quantum object is more general, and holds its entire quantum state.
(The term further sometimes evokes the wave–particle duality, according to which a particle (a "small" physical object) may display either particle or wave aspects, depending on the observational situation. Or, indeed, a variety of intermediate aspects, as the situation demands.)