Clovis is not a racist genocidal bastard. His hate of Elevens was more of them "killing" Lelouch and Nunnally. He was very lenient towards Elevens, especially trying to restore their economy faster than normal instead of just enslaving them.
Probably because of hearing about Lelouch making friends with Elevens before he became governor.
In the end, Clovis only ordered the destruction of the Shinjuku Ghetto to achieve his goals efficiently. Remember that Lelouch also killed in order to achieve his goals. He sacrificed what were supposed to be his allies as well.
The real racist in the siblings was only Cornelia.