A query for the fully rigid...

slippy

Likes Bikes and Dirt
Sounds like you found the problem. 1 or 2 Nm either way shouldn't (in theory) have been making all that much difference. I run all my stems with a 5mm spacer on top similar reasons. Annoying how some standard stem caps have an internal lip which looks good without compression but when you tighten it hits the top of the steerer and prevents full compression (slightly different problem I know, but same solution).
 
Top