So it turns out that the clanking sounds was the belt rubbing up against the mudguard when downshifting. Not sure why depressing the clutch made it go away. Anyway, I considered tightening the belt but everything seemed fine so I ended up removing the mudguard, bending it a bit and bolting back on. No more issues.
I would double check the belt tightness and make sure things are straight. I bought an awesome laser off of eBay that magnetically sits on the rear pulley. My belt was moving around and hitting the tire sidewall and it was chewing it up a bit so be careful.
By the way the noise, having to wrench on it and upgrading parts is all part of the charm.
