Undo part of #6495 (#6508)
Undo part of #6495 (#6508)

Since we don't actually run the upgrade tests for anything but Istio, this change effectively reverts the change and removes the logic that isn't exercised today to (hopefully) unbreak things.
knative/serving,knative/serving