Refresh tokens are generally not bearer tokens. At least they're not based on the RFC 6749 spec. However, it really depends on the authorization server's implementation and policies. And, that's the crux of Oauth in general. Not everyone adheres to the spec.
Anyways, access tokens are generally bearer tokens and considered stateless. You send them to the resource server (the server hosting whatever API), and they will not care who you are.
However, a refresh token is NOT sent to a resource server. It's sent to an authorization server, and the refresh token generally has more security implementations to prevent abuse. As per RFC 6749 spec:
(G) The client requests a new access token by authenticating with
the authorization server and presenting the refresh token. The
client authentication requirements are based on the client type
and on the authorization server policies.
(H) The authorization server authenticates the client and validates
the refresh token, and if valid, issues a new access token (and,
optionally, a new refresh token).
Generally, this requires a client ID and client secret to be sent for authentication to enforce refresh token / client binding. But, it goes even one step further:
(C) Assuming the resource owner grants access, the authorization
server redirects the user-agent back to the client using the
redirection URI provided earlier (in the request or during
client registration). The redirection URI includes an
authorization code and any local state provided by the client
earlier.
(D) The client requests an access token from the authorization
server's token endpoint by including the authorization code
received in the previous step. When making the request, the
client authenticates with the authorization server. The client
includes the redirection URI used to obtain the authorization
code for verification.
(E) The authorization server authenticates the client, validates the
authorization code, and ensures that the redirection URI
received matches the URI used to redirect the client in
step (C). If valid, the authorization server responds back with
an access token and, optionally, a refresh token.
Basically, what this tells you is that any refresh token request must come from the same URI/URL as in the original redirection URI from the authorization code grant portion of a typical OAuth flow. So, what does this mean? Even if someone stole your refresh token, if the authorization server is implemented to the RFC spec, then it will throw an invalid_grant error if you try to make the request from another domain / URI. Of course, this is all with the caveat that the authorization provider implements the spec and policies to do this. Google definitely does this.