Worser is not a word, the most correct statement is, "I have never seen a worse attempt at trolling". However, the original statement is not grammatically incorrect. In the statement, "I have never seen worse trolling", trolling is a noun, not an 'action' (never mind how you categorize like THAT).
That said, Elfen Lied is a forgettable anime, IMO, precisely because the plot really isn't that DEEP (tragic romance, gee, where have I seen that before?) and the 'themes' of racism and 'evils of humanity' are explored in a rather plain black and white way.