Woof. To save everyone the trouble I had:

If you get a the manifest is missing a Swift tools version specification error when adding a new package that has a 0.x.x version format, check that it’s actually pulling the latest version.