Not sure of the legalities, but your word against another would probably not hold, i suppose the best way would be to have the cat chipped and have any documentation that goes with it. I'm no cat expert but some cats are known for having 2 homes, a friend of mine who had their cat from a kitten had to accept that when their cat went missing for long periods of time it was at its other home, proof of this was it always came back in tip top condition and quite often had a different colour flea collar on. It just didn't disappear for a few days, it was more like months.
As far as i know, it still does it to this day.
Get yourself a goldfish, they find it very difficult to get to a neighbours house.