When running only some tests, sometimes extra, unnecessarily work was
done, like bringing up the cluster without alpha gates.