In a recent post (Possible bug in hypergeometric function AppellF1) I reported a buggy behaviour of the function AppellF1[].
Now I would like to aks about the implementation which possibly can explain this behaviour, and specifically the appearance of the octogonal region in (x,y) where the implementation of the function is well-behaved. (I'm asking also because I have not received an answer to that post).
There are two definitions for AppellF1[] which could be used for an implementation ( http://mathworld.wolfram.com/AppellHypergeometricFunction.html )
Series representation
$$F1(a,b1,b2,c,x,y) = \sum _{m=0}^{\infty } \sum _{n=0}^{\infty }\frac{ (a)_{m+n} }{(c)_{m+n}} (\text{b1})_m (\text{b2})_n \frac {x^m}{m!} \frac {y^n}{n!} $$
Sum[
Pochhammer[a, m + n]/
Pochhammer[c, n + m] Pochhammer[b1, m] Pochhammer[b2, n] x^m/m! y^n/
n!, {m, 0, \[Infinity]}, {n, 0, \[Infinity]}]
(* Out[2843]= AppellF1[a, b1, b2, c, x, y] *)
Integral representation
$$F1(a,b1,b2,c,x,y) = \frac{\Gamma (c)}{\Gamma (a) \Gamma (c-a)} \int_0^1 t^{a-1} (1-t)^{-a+c-1} (1-t x)^{-\text{b1}} (1-t y)^{-\text{b2}} \, dt$$
fctF1[a_, b1_, b2_, c_, x_, y_] :=
Gamma[c]/(Gamma[a] Gamma[c - a])
Integrate[
t^(a - 1) (1 - t)^(c - a - 1) (1 - x t)^-b1 (1 - t y)^-b2, {t, 0,
1}]