In saying something like "The group is big" ("La grupo estas granda"), I'm referring to the noun "group" singularly (i.e. the group itself is big). How would I refer to it collectively as in English, as in the sentence "The group are scared" (i.e. the members of the group are scared)? "La grupo estas timaj" doesn't sound like it would make sense. I could just keep it simple and say "La membroj de la grupo estas timaj", but I wonder if there is any Esperanto equivalent of using collective nouns in the above manner.
I think you're falling victim to L1 interference here. While it is possible in English to use both singular and plural with certain nouns (team, group, but also data), this is not necessarily the same in other languages, including Esperanto.
As per the grammatical rules (rule 3 of the fundamental 16), there is agreement between nouns and adjectives, so La grupo estis tima is correct usage. This would sound perfectly fine to a native speaker of, say, German.