Easy, Shirou. If Shiki's not in Nanaya or the even more GAR Satsujinki mode, then Shiki's actually very frail and sickly. Shirou, on the other hand, trains his body constantly and has cannonly beat up Shinji for hurting Sakura.
Not to mention, Shirou is close to Fuji-nee.
Living with that woman would turn anyone bad ass for just self-defense.
So, if it's just a fist fight, I'd give it to Shirou.
Now, UBW VS Shiki's Mystic Eyes . . .?
Sorry Shirou, but Shiki would pown your ass before you'd even utter, "As I pray, Unlimited Blade Works . . ."
Now . . . as for ARCHER VS Shiki . . .
Sorry Shiki, you wouldn't even get a chance to take off your glasses before Hurnting stabbed you through the head.