Isn't this just allowing anyone to access your API regardless of
whether they have the secret key or not?
Yes, The implicit grant lacks the ability to authenticate the client, which the other grants can do — further introducing attack vectors that the authorization grants, which require a client secret, do not experience.
The use case for implicit grant is authentication of end users and access by client to a resource (possibly owned by the end user) and should be used by SPA Apps (and any other Javascript/User-agent-based app).
If client authentication is important then a different grant must be implemented.
Implicit grant is more secure in the sense that it wouldn't expose the client secret, which can be shared across your internal applications.
Here's a good read on when to use which (OAuth2) grants and OIDC flows https://community.apigee.com/articles/41719/when-to-use-which-oauth2-grants-and-oidc-flows.html