That's my feeling too. Unless someone walks by it and pulls it, the last person to putt pulls the pin, and the first person to hole out is responsible for replacing it.