Higher-order modulation is a type of digital modulation usually with an order of 4 or higher. Examples: quadrature phase-shift keying (QPSK), and m-ary quadrature amplitude modulation (m-QAM).
References
Higher-order modulation Wikipedia(Text) CC BY-SA