What you described is a Latch switch, of the various types you would have:
a) a Momentary, where the circuit is only open or closed while the button is pushed.
b) a Flip-Flop, where the circuit can be either open or closed, but state is changed by the same switch.
c) a Latch where the circuit is either open or closed, but stays that way when triggered by the main switch, another switch or power loss to the circuit allows it to change state.
What type of voltages and type (AC or DC) are you controlling? Would help with determining the right relay specs to recommend.
To a small child, the perfect granddad is unafraid of big dogs and fierce storms but absolutely terrified of the word “boo.” – Robert Brault