Without loss of generality, we can assume $x\ge y$.
- If $x\ge y \ge 1$, then $$(x^y+y^x)\left(\frac{1}{x}+\frac{1}{y}\right)\ge (x+y)\left(\frac{1}{x}+\frac{1}{y}\right)\ge4.$$
- If $1\ge x\ge y>0$, then $$x^y=\frac{1}{\left(\frac{1}{x}\right)^{y}}=\frac{1}{\left(1+\frac{1-x}{x}\right)^y}\ge \frac{1}{1+y\left(\frac{1-x}{x}\right)}=\frac{x}{x+y-xy},$$ and similarily $$y^x\ge\frac{y}{x+y-xy}.$$ Hence $$(x^y+y^x)\left(\frac{1}{x}+\frac{1}{y}\right)\ge \frac{\left(\frac{1}{x}+\frac{1}{y}\right)^2}{\frac{1}{x}+\frac{1}{y}+1}\ge 4.$$
- If $x\ge 1 \ge y>0$, how to go on?