Ok i cant believe that this has not been suggested but here goes. You set the button to be a lockable button. The button will not do anything if the person is not in the button allow list. The button could be lockpicked using a hoe or some tool.
