When I did magnetometer calibrations on an IMU project some years back, I remember I found

this paper very useful. It's a Honeywell paper, but applies to any magnetometer measurements. The rotational equations for tilt compensation may seem tricky but is simply a

matrix rotation of the measured mag vector, opposite to the rotation of the measured gravity vector (pitch/roll). Final azimuth is the corrected mag vector loosing it's Z component (projected onto X,Y plane), the paper uses atan to express angle vs north. In the rotations, remember each rotation is a matrix multiplication which is non-commutative so order of rotations matters.

EDIT: On commutativity, to illustrate order of rotations; if you represent pitch and roll by angles (not by a gravity vector), the angles need be relative to some axis system. If that system has a in a pitch-roll rotation order, you need to correct the mag vector in backwards order; first reverse roll - then reverse pitch.