Plug gap will be optimised to provide a meaty enough spark to give reliable fuel ignition and burn, and also to ensure the spark jumps the gap reliably and with minimal jitter. At normal atmospheric pressure air breaks down between two smooth metallic hemispheres at around 30kV/cm. Shaping the electrodes to say a fine point will increase local field strength and enhance spark initiation. Reducing the gap will enhance spark initiation. Higher engine compression (gas density) will hold off sparking for a given voltage. When I rebuilt a Ford Xflow to race spec’ I had to reduced the gaps to obtain good ignition performance at the higher compression ratio. If the gap is too large for the spark to jump it will find another route. If it tracks in the ignition coil it will soon produce a preferred path there and hence damage the coil. So, the biggest possible gap is not necessarily the best choice. Conversely too small a spark may not produce a full burn.
Think I actually understand that. So maybe it's a bit of both, engine requirement, but also depending on the type of plug?