Boat stability- my guests are afraid of uncomfortable trip and/or sleeping on the boat.

There are 2 main considerations- the yacht itself and the area.
There are some areas that are rougher than others – the Cyclades, Antigua, Malta.
When selecting your charter itinerary, ask about the different areas that are available for your destination.
Greece- opt for Saronic Gulf, Ionian Sea or the Sporades
Antigua – get a bigger boat or look at Grenadines direct (some charters start at Union). There is always the BVI that is quite comfy.
Malta- no alternative areas, Malta is pretty rough, not ideal for first time charterers.
Selecting your yacht- there are things to look at. The primary concern is the length, the longer the boat, the more wavelengths it covers and the more comfortable. When considering multiple yachts, ask for advice. The shape of the hull and the weight of the boat might point to this or that boats. Are brokers will be able to help.
As for a family vacation, consider a big catamaran, ideally a motor yacht; or a motor sailer if in the Med (gulet and gulet-like boats). There are some wonderful high end motor sailors like Iraklis L. In the Caribbean, there are some wonderful big yachts, both motor and sail.
Zero speed stabilizers- some boats offer zero speed stabilization. If you want to be sure that the nights are comfortable, choose a boat with zero speed stabilization. Check out the video if the concept of zero speed stabilization is new.