Hi there
Strange behavior for sure, first I would Increase the delay to 250, or longer and test again, are you hardware debouncing the switch or no then I would move the attach and detach interrupts functions outside of the ISR’s (I feel that’s where something is breaking)
set a flag and do it in a if , else logic in the main code block.
That will fix it.
HTH
GL 
PJ
ps. just be sure your button push get’s just one interrupt by slowing things down.
check out how I do the sleep button interrupt and ISR’s in this code example demo