"Krom" can be used to express an exception as well as an addition. The difference is sometimes difficult to tell without context.
In the case of
Krom Petro la tuta familio vojaĝis.
Peter did not travel (if the context does not tell, that did). One can assume that Peter is member of the family, so it wouldn't make sense to mention Peter as an addition, so he must be the exception.
In the case of
Krom Petro ankaŭ ĉiuj aliaj familianoj vojaĝis.
Peter did travel. In this case "ankaŭ" implies that it is an addition an not an exception.
So to answer your question: You can remove the ambiguity by adding words like "ankaŭ". To make sure that we talk about an exception, you can put it at the end.
La tuta familio vojaĝis, krom Petro.
Other words that you can use instead of "krom":
More on that once more in PMEG.