I.m sorry I got distracted,what did u say?
Got it.You might let him know you have a good business going and you will not rock that boat with any major changes. "sorry, it is just not happening," and" you are being .direct with him as you can." Polite .. so as not to cause resentment but very firm. Then reassess as you go along. Be prepared if he decides he cant handle things your way and wants to buy out.
Tell him that is not part of your business arrangement and You don.t plan on any changes that could potentially damage that.