Forgive me, but I am going to play devils advocate. Are you sure he knew you were pulling over to talk, and didn't assume you were pulling over to let him pass. I've been on both ends of this many times, letting people pass who were clearly in a bigger hurry than I, and passed people who pulled to the side to let me get by. It seems to be a pretty common courtesy most do. Maybe he incorrectly assumed this is why you pulled over?
Since you asked, If I were in your situation as describe, there is no chance I would have slowed (I actually would have sped up) or done anything to let someone get past me until I was safely at the spot. At that point you can correctly make the claim that you were the first there, and not just a guy driving down the road.