Trace vector decoder is a system that uses a microprocessor's trace mode to decode encrypted code just-in-time before it is executed and possibly re-encrypt it after the execution. It can be used to enforce copy protections for some computer systems.
Trace Vector in Motorola 68000
As an example, Motorola 68000 has a trace mode where a trace exception vector is executed before each instruction in the main program. The processor automatically changes execution to the trace exception vector before executing any instruction from the main program. The trace exception vector decodes the instruction that will be executed after the exception vector. The next time trace exception happens the old decoded location may possibly be re-encrypted.
Following code snippet is an example of a program initializing a trace exception routine.
MOVEM.L Stack,D0-D7/A0-A6 ; Initialize registers Stack MOVE.L #$4E730000,-(SP) ; Start loading trace exception MOVE.L #$00000010,-(SP) ; vector into stack MOVE.L #$0004DDB9,-(SP) MOVE.L #$BD96BDAE,-(SP) MOVE.L #$B386B586,-(SP) MOVE.L #$D046D246,-(SP) MOVE.L #$0246A71F,-(SP) MOVE.L #$00023C17,-(SP) MOVE.W #$2C6F,-(SP) MOVE.L SP,($24).W ; Set trace exception vector ORI.W #$A71F,SR ; Enter trace mode NOP ; Trace vector happens now for the first time. ; Code after this line is encrypted.A disassembly of the trace exception vector that is loaded on the stack:
TraceCode: MOVE.L (2,SP),A6 ; Load return address from ; supervisor stack. MOVE.W (SP),D6 ; Load condition codes of the main ; program. AND.W #$A71F,D6 ADD.W D6,D0 ADD.W D6,D1 EOR.L D1,D6 EOR.L D2,D6 EOR.L D6,(A6) ; Decrypt 8 bytes ahead in main EOR.L D6,(4,A6) RTE ; Return from exceptionNote that registers altered in the trace vector affect the main program that is being traced. Usually registers are pushed onto stack in any exception vector, because altering them would break the main program. However, purpose of this vector is to obfuscate the code against reverse engineering.
It should also be noted that condition code register (CCR) affects the decryption process. For example, an arithmetic operation in the main program having the 0 number as a result, will cause zero flag bit to be set in CCR. This will cause the value in (SP) to be changed in the trace vector. This is also done to obfuscate against reverse engineering.
Rob Northen copylock has a trace vector decoder that was used on the Amiga and Atari ST platforms.