So it turns out that the clanking sounds was the belt rubbing up against the mudguard when downshifting. Not sure why depressing the clutch made it go away. Anyway, I considered tightening the belt but everything seemed fine so I ended up removing the mudguard, bending it a bit and bolting back on. No more issues.
