Just for completeness (and for anyone else with the issue). I had installed the brackets upside down. Once corrected wobble is almost completely gone.

Here is a photo of how it is supposed to look. https://photos.app.goo.gl/KDGUrMKjzvzNBHZN7