In fact a "fat Cantor set" gives a counterexample. The construction is slightly complicated. Lucky for you it's much simpler now than it was an hour ago; it turned out I could throw away a lot of detailed estimates.
Fat Cantor Sets
Given a sequence $(\lambda_n)\subset(0,1)$ we construct the corresponding "fat Cantor set" exactly as in the construction of the well known middle-thirds Cantor set, except that instead of removing the middle third of each interval in going to the next stage, for each interval $I$ at stage $n$ we remove the middle interval of length $\lambda_n|I|$. (Here and below we write $|I|$ for the length of an interval $I$.
So we have $$K=\bigcap_{n=0}^\infty K_n,$$where $K_0=[0,1]$ and $$K_n=\bigcup_{j=1}^{2^n}I_{n,j},$$where for each $n$, $(I_{n,j})$ is a collection of disjoint closed intervals with $|I_{n,j}|=\delta_n$.
(Just to make sure we're on the same page we note that if we take $\lambda_n=1/3$ we get the middle-thirds Cantor set.)
So far we have just a "variable dissection ratio" Cantor set, or some such term. To be "fat" a Cantor set is supposed to have positive measure. It's evident that $$m(K_{n+1})=(1-\lambda_n)m(K_n),$$so $m(K)=\prod_n(1-\lambda_n)$; now the basic result about infinite products says
$m(K)>0$ if and only if $\sum\lambda_n<\infty$.
The Counterexample
We will construct a fat Cantor set as above, so that if $d(y)=d(y,K)$ then $$\limsup_{y\to x}\frac{d(y)}{|x-y|^\alpha}=\infty\quad(x\in K, \alpha>0).$$
Note The condition $m(\Bbb R\setminus F)<\infty$ is really irrelevant; if you want that then construct $K\subset[0,1]$ as below and let $F=(-\infty,0]\cup K\cup[1,\infty)$. Note that $\{0,1\}\subset K$, so that if $y\in[0,1]$ then $d(y,F)=d(y,K)$.
Note we will use notation as in the previous section.
Suppose $x\in K$. Fix $n$ for a while. There exists $j$ so that $$x\in I_n=I_{n,j}.$$ Let $y_n$ be the center of $I_n$.
Now, recalling that $|I_n|=\delta_n$, the middle subinterval of $I_n$ that gets excluded from $K_{n+1}$ is $$J=\left(y_n-\frac{\lambda_n}{2}\delta_n,y_n+\frac{\lambda_n}{2}\delta_n\right).$$Since the endpoints of $J$ turn out to be points of $K$ it follows that $$d(y_n)=\frac{\lambda_n\delta_n}2.$$(In fact all we need below is $d(y_n)\ge\lambda_n\delta_n/2$, which is clear just because $J\cap K=\emptyset$.)
Since $x\in I_n$ and $y_n$ is the center of $I_n$ we have $$|x-y_n|\le\delta_n/2.$$So for $\alpha>1$ we have $$\frac{|x-y_n|^\alpha}{d(y_n)}\le c\frac{\delta_n^\alpha}{\lambda_n\delta_n}=c\frac{\delta_n^{\alpha-1}}{\lambda_n}.$$
It's clear that $\delta_n\le 2^{-n}$, so we're done if we can find $\lambda_n$ with $\sum\lambda_n<\infty$ and $$\lim_n\frac{2^{-n(\alpha-1)}}{\lambda_n}=0\quad(\alpha>1).$$For example, $$\delta_n=\frac1{(n+2)^2}.$$
($n+2$ instead of $n$ just to make sure $\lambda_n<1$ for all $n\in\Bbb N$, even if someone takes $0\in\Bbb N$.)
Moral
Variable-dissection-ratio Cantor sets are useful for constructing all sorts of examples. So what's above was an obvious thing to try (once I actually tried it it came out simpler than I expected).
Unimportant Comment
I keep saying I was surprised how simple this was. For the benefit of anyone who feels it's not all that simple, comments on how it could have been (and was at first) much worse:
I was surprised that the trivial estimate $|x-y_n|\le\delta_n/2$ was sufficient. Of course in general there's not much more to be said about $|x-y_n|$. But there is more you can say if $x$ lies in the right half of the left half of $I_n$, or the left half of the right half; in the original I needed an argument showing that for almost every $x$ that happens for infinitely many $n$.
And I was surprised that the trivial estimate $\delta_n\le2^{-n}$ was enough. In fact $\delta_n=2^{-n}\prod_{j=1}^n(1-\lambda_j)$; I assumed at first I was going to have to deal with that...
