I would say that what stops the hub from rotating on the splines even minutely is the friction between the end of the hub and the crank bearing journal
That is created by the nut being tightened to the correct torque of course
it seems to me that the plain washer is critical in transmitting that force through from the nut to the hub end as it is a bigger diameter and much thicker than the locking washer alone.
If that were the case you wouldn't need splines. A secured splined coupling that is required to transmit loads in both directions is not ideal. A tapered or keyed coupling should have been used.
If it were the tension created by the nut that prevented rotational movement a spline wouldn't be necessary and a plain shaft with the coupling loosely slipped over would suffice. Clearly this wouldn't work.
It's likely that however tight the nut is there will be relative movement of the coupling on the splines. The only practical solution is to lock the nut in place. There are numerous means of doing this and proprietary locking devices available.
