Cowardly would have been dodging the fight. Guilford knew that Zero had no chance in a 1-on-1 fight and simply used the duel call as a way to trap him. If he came, he died, if he stayed away, his men would die and he would be exposed as a fake Zero (since the real one would never let his men die, or so the public believed).
Guilford never expected Zero to actually accept his challenge. Thats why he was confused when Zero asked him what the rules would be.