[base-builder-swift] Reduce size 1.6GB and misc improvements (#7387)
[base-builder-swift] Reduce size 1.6GB and misc improvements (#7387)

1. Reduce image size by 1.6GB by deleting unneeded directory containing
swift tools that were already copied to /usr.
2. Use apt-get instead of apt to avoid warnings.
google/oss-fuzz,skia-dev/oss-fuzz,skia-dev/oss-fuzz,skia-dev/oss-fuzz,skia-dev/oss-fuzz,skia-dev/oss-fuzz,skia-dev/oss-fuzz,robertswiecki/oss-fuzz,robertswiecki/oss-fuzz,skia-dev/oss-fuzz,google/oss-fuzz,robertswiecki/oss-fuzz,google/oss-fuzz,google/oss-fuzz,google/oss-fuzz,skia-dev/oss-fuzz,google/oss-fuzz,robertswiecki/oss-fuzz,google/oss-fuzz,robertswiecki/oss-fuzz,skia-dev/oss-fuzz,google/oss-fuzz,google/oss-fuzz,robertswiecki/oss-fuzz,skia-dev/oss-fuzz,robertswiecki/oss-fuzz,robertswiecki/oss-fuzz,google/oss-fuzz,google/oss-fuzz,robertswiecki/oss-fuzz,skia-dev/oss-fuzz,robertswiecki/oss-fuzz,robertswiecki/oss-fuzz