The only reason those Unity games you played ran poorly were because they were made poorly. I do see lots of poorly optimized Unity games though. When I've used it, it's been very easy to do, but then again I'm aware of all that performance stuff.
It just seems to me that those developers simply were not knowledgeable and mindful of performance optimizations. For example, if a developer didn't know what occlusion culling was, they wouldn't ever consider baking it in Unity, even though it takes a few clicks anyway. On the other hand, if they were using UE4, this inexperienced developer wouldn't need to consider it because UE4 does it automatically.
It would run better or the same as UE4 in Unity if you do consider those things, unless you have a crazy graphically-intense game, which is what UE4 is suited for. You, however, are aiming for making a game with very bare graphics. From a just-graphics point of view, it's much easier to get that in Unity (not that it's hard in UE4, it's just that you actually have to change lots of default settings).
But you are right that UE4 is more accessible for someone without programming experience, because of stuff like their Blueprints, and their visual shader creation, and lots of nice art and level design oriented features.
So yeah, definitely try them both.