The term complex representation has slightly different meanings in mathematics and physics.
In mathematics, a complex representation is a group representation of a group (or Lie algebra) on a complex vector space.
In physics, a complex representation is a group representation of a group (or Lie algebra) on a complex vector space that is neither real nor pseudoreal. In other words, the group elements are expressed as complex matrices, and the complex conjugate of a complex representation is a different, non-equivalent representation. For compact groups, the Frobenius-Schur indicator can be used to tell whether a representation is real, complex, or pseudo-real.
For example, the N-dimensional fundamental representation of SU(N) for N greater than two is a complex representation whose complex conjugate is often called the antifundamental representation.