Yes....Sounds very much like you have got something sharp or abrasive on your clothing which will be rapidly speeding up the wear on the leather.
Also try to avoid leaving the seat belt buckle over the surface of the seat or seat back. That can cause damage as you get in and out. Move the buckle to the side of the seat before entry or exit.
The secret is to get out of the car as little as possible...just keep driving !!
