If you say gefratoj, you are talking of brothers and sisters (siblings).
You can use it, in the same way somebody asking if you have siblings would use gepatroj. If you know there are just brothers, then you should use fratoj, in the same way a person who knows you have brothers would do when asking you how many brothers you have.