For starters, Umisho was different from most harem shows. The male characters played a non-trivial role, and the girls weren't all automatically bonkers for the main guy. In short, it was a show about what a bunch of highschool boys and girls in a swimming club would be like.
And all the characters had their quirks. And the plot, though thin, was interesting. How they all deal with the meets and tournaments and such was pretty cute.
So yeah, nobody would probably watch the series without the fanservice, but it was still good enough a show otherwise to keep people interested.