If you say gefratoj, you are talking of brothers and sisters (siblings).
You could use it, in the same way somebody asking if you have siblings would use gepatroj. If you know there are just brothers, then you should use fratoj, in the same way a person who knows you have brothers when asking you how many brothers you have would.