se - under the condition (that)

cxu - whether (or not)

In your case it is a question of whether or not a person is friendly, and therefore the word to use is cxu

The other phrase may be used, I suppose, in a very unusual context meaning something like "If he is friendly, then I don't know*.

"Kiu estas la malamiko en la libro?"

"Mi ne scias se li estas amika"

se - under the condition (that)

ĉu - whether (or not)

In your case it is a question of whether or not a person is friendly, and therefore the word to use is ĉu

The other phrase may be used, I suppose, in a very unusual context meaning something like "If he is friendly, then I don't know*.

"Kiu estas la malamiko en la libro?"

"Mi ne scias se li estas amika"