Good news everyone!
mfoxdogg managed to put me in contact with Morten Mikkelsen, which has written a thesis on bump mapping, with weight on sampled normal maps. He assured me that my current method will be fine for "plain" normal maps (like the ones Moure has used here), but will fail for sampled normal maps (as in the torus I posted).
To render the sampled normal maps correctly it's absolutely essential that the tangent space used during reconstruction matches the tangent space used during sampling ("baking"). In order to make this easier, Morten has created some open source code to consistently create "proper" tangents for meshes, even if vertex order is changed etc. Blender and xNormal now uses his code, and thus I think we should do as well.
In order to support his code, we'll also need a vertex welder (he has one available for that too), as his tangent space code returns an unindexed mesh back.
In addition we'll need to add two new vectors to the RayDifferential class: "tangent" and "bitangent". I propose we add another constructor which accepts these, the old ones just normalize dpdu and dpdv. These new fields will then be used when transforming the normalmap normal back into object space, instead of the current dpdu, dpdv.
edit: Hmm can we just reuse dpdu,dpdv directly for this in GetShadingGeometry? I forgot that the shading geometry's dpdu,dpdv doesn't actually follow u or v.
Since this will require the storage of an additional vector per vertex (the bitangent can be recovered on the fly), I think we should make a parameter to control this behavior. It should then be enabled for "baked" meshes only.
May contain traces of nuts.