Bit of advice?

[clivvy]

Had a funny one today, will cut a long story short. Neighbours and I share a drive that goes front to back. We both have right of way. A few years ago he tarmacced over the concrete drive on HIS "half" and we sort of fell out. Every now and then I move one of my cars down the drive towards the back so I can swap overvthe cars we have. Today I came home and did just that. Neighbours mum who lives at the house came home from a long holiday and her son, the one we have trouble with, came banging on the door. Basically he threatened me, said the land was his and I didn't ask to park the car there and he said don't park there any more. He got very intense because I started to explain that I meant no harm and thought it was ok as it was temporary. He then threatened me, and said if I did it again, he would pull it off the drive himself. At thus point I said don't threaten me, he said I am threatening you, and I said don't, and don't you touch my car, he then continued with his threat and said he would pull it off the drive. I told him to speak to his solicitor to get things clear because perhaps I have mis understood the agreement for the drive. I understand that yes, the land is probably his and I have right of way, but at the front, half is mine if that is how he wants to be.
My issue is, the threats. I don't mind being asked nit to use it as a parking spot, which I don't, but I need advice on the threat. Is there anything I can do at all? He's made myself and my girlfriend a but scared, and put us on edge, now I am worried that the next time I use the drive for access, and leave the car for a short time, he's going to damage it. Should I seek legal advice, report to police or leave it alone? Problem is we are now on edge over it, and very upset...