← All writeups

Reverse engineering

ghost_protocol (reverse engineering / MIPS VM keygen)

Flag: P2P{n0_m0r3_sh4d0ws_1n_th3_m4ch1n3} Activation code (protocol key): n0_m0r3_sh4d0ws_1n_th3_m4ch1n3

Category: Reverse engineering (MIPS, anti-analysis, custom bytecode VM, keygen) Provided file: ghost_protocol (1775995121_.ghost_protocol) Solved: 2026-05-31

Challenge description:

A mysterious binary was found on a compromised server during an incident response engagement. Analysts believe it contains the activation code for a dormant threat actor's command infrastructure. The binary appears to implement some kind of custom protocol verification system.

1. Recon

$ file ghost_protocol
ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, stripped

Little-endian MIPS32 (LSB → use qemu-mipsel), static, stripped, ~68 KB. Strings reveal a heavy anti-analysis arsenal: /proc/self/status, /proc/self/maps, /proc/self/environ, TracerPid:, LD_PRELOAD, LD_LIBRARY_PATH, docker, containerd, systemd, xinetd, socat, screen, tmux: server, /dev/shm/. Notably no plaintext P2P{ flag anywhere in the file — every output string is built at runtime.

First run under emulation prints a decoy:

$ qemu-mipsel ./ghost_protocol
P2P{d3bugg3r_d3t3ct3d_n1c3_try}

A qemu -strace shows the giveaway: a forked child does ptrace(PTRACE_ATTACH, parent) (request 16) which returns ENOSYS under qemu → the program concludes it's being debugged and emits the taunt. The real flag is on the path where all checks pass.

2. Static analysis (Ghidra headless + radare2)

main is at 0x22280 (loaded by entry), inside a giant function FUN_00021d70. Decompiling (Ghidra headless with a Java decompiler-dump script, since this build lacks PyGhidra) reveals three layers.

Layer 1 — environment tripwires (decoy gate)

A boolean analysis_detected is set by:

if (analysis_detected || timing > 150_000_000ns) { print_decoy(); return; }

The decoy is emitted char-by-char (P,2,P,{,...) via a putchar helper.

Layer 2 — keyed self-integrity + encrypted bytecode

The clean path:

  1. Computes a key byte K = rolling hash over the first 0x40 bytes of its own file (/proc/self/exe, with an argv[0] fallback — which is what qemu hits). K = 0x65.
  2. malloc(0x8b2) and copies an encrypted blob from 0x104a2 (file offset 0x4a2).
  3. Decodes it in two passes:
    • Pass 1 – position-dependent operand transform: for each instruction, operand bytes ^= (pos+n) ^ 0x3c. Instruction lengths come from a table at 0x10d54.
    • Pass 2 – XOR each opcode byte with K.
  4. Initializes a VM state and runs it.

Key insight: the VM fetch is op = (K ^ blob[pc]) ^ 0xA7. Pass-2 XORed the opcode with K, and the fetch XORs it back — so K cancels out entirely. The self-integrity hash is decorative for execution; the only environment-dependent variable that matters is the permutation flag below.

Layer 3 — a custom register VM

A register-based VM (8 regs, PC, ZF, a byte stack, a call/loop stack). 20 opcodes, decoded as pseudo = blob[pc] ^ 0xA7:

op mnemonic op mnemonic op mnemonic op mnemonic
1 MOV r,imm 6 AND 0xB JZ 0x10 RET
2 MOV r,r 7 SHL 0xC JNZ 0x11 READ
3 ADD 8 SHR 0xD PUSHB 0x12 WRITE
4 SUB 9 CMP 0xE POPB 0x13 ROL
5 XOR 0xA JMP 0xF LOOP/CALL 0x14 HALT

The opcode-fetch contains the permutation gate at 0x22810:

0x22810  lw   at, 0x1bc(s0)     ; perm flag (from the parent-process comm check)
0x22814  beqz at, 0x22838       ; if 0 -> no permutation
0x22818..0x22834               ; else swap bits 0/3 of every opcode  => total scramble

The flag at state+0x1bc comes from checking the parent process comm against the container/multiplexer blocklist. Under qemu it ends up non-zero, every opcode is permuted, the VM spins, and nothing prints — which is exactly why the naive "force the clean path" patch produced no output.

3. Offline simulation

Rather than fight the environment, I re-implemented the decoder + VM in Python (vm_sim.py), which faithfully reproduced the program's real strings:

$ python3 vm_sim.py
=== perm=False ... ===
OUT: Enter protocol key: ACCESS DENIED.      # matches the binary exactly
=== perm=True ... ===
OUT:                                          # scrambled -> spins, no output

So perm=False is the intended state, and the program is a keygen-checker: print Enter protocol key:, read 30 bytes, validate, then branch to ACCESS GRANTED (0x7c2) or ACCESS DENIED (0x83e).

The per-character check

Disassembling the validator (0xDC):

r6 = 1 (valid)        ; r6 stays 1 only if every char passes
r5 = 0x5a (rolling state)
for each of 30 chars:
    READ r0
    <per-char ops: XOR const, XOR r5, optional ROL, ADD const, AND 0xff>
    if (transformed != TARGET_i) r6 = 0
    r5 = TARGET_i ^ rol8(r5, 1)          ; deterministic, independent of input
if (r6 != 0) -> ACCESS GRANTED else ACCESS DENIED

Because r5 evolves only from the (fixed) per-char targets, the checks are solvable left-to-right.

Greedy solver

solve_key.py runs the real decoded VM and, for each of the 30 positions, brute-forces the byte (0–255) that keeps the validity flag r6 intact after that char's check:

$ python3 solve_key.py
KEY ascii: n0_m0r3_sh4d0ws_1n_th3_m4ch1n3
VM final r6: 1
VM output: Enter protocol key: ACCESS GRANTED.

4. Dynamic confirmation on the real binary

Two surgical patches (both far past the 0x40-byte header the integrity hash covers, so the key stays valid):

vaddr original patched effect
0x223cc jal putchar j 0x22060 + nop force the clean path (skip decoy)
0x22814 beqz at,0x22838 b 0x22838 force permutation flag = 0 
$ printf 'n0_m0r3_sh4d0ws_1n_th3_m4ch1n3' | qemu-mipsel ./ghost_patched2
Enter protocol key: ACCESS GRANTED
$ printf 'wrongkeywrongkeywrongkeywrongk'   | qemu-mipsel ./ghost_patched2
Enter protocol key: ACCESS DENIED

Confirmed both ways.

5. Result

The recovered activation code is the protocol key:

n0_m0r3_sh4d0ws_1n_th3_m4ch1n3

Wrapped in the CTF's flag format (the decoy established the P2P{...} convention):

P2P{n0_m0r3_sh4d0ws_1n_th3_m4ch1n3}

6. Takeaways

Artifacts (~/Downloads/)