While an individual photon has no rest frame, when two photons move apart, it makes sense to ask where their centre of mass is, as described here.
If I proposed that while a photon has no mass, a system of two parting photons does have mass, how would I quantify that mass, in a manner consistent with the prevailing theories of physics?
In particle physics, it is common to define the invariant mass $M$ of a system of $n$ particles with four-momenta $$p_i =(E_i, \vec{p}_i), \quad (i=1, \ldots n, \; p_i^2=E_i^2-\vec{p}_i^2=m_i^2),$$ by the Lorentz invariant quantity $$M^2:= \left(p_1+\ldots +p_n\right)^2=(E_1+\ldots +E_n)^2-(\vec{p}_1+\ldots+\vec{p}_n)^2.$$ In the case of two photons (where $E_1= |\vec{p}_1|$, $E_2=|\vec{p}_2|$), the invariant mass of this two-particle system is given by $$M^2=(p_1+p_2)^2=(|\vec{p}_1|+|\vec{p}_2|)^2-(\vec{p}_1+\vec{p}_2)^2.$$ In the reference frame where $\vec{p}_1 + \vec{p}_2=\vec{0}$ (the so-called center-of-mass frame), the invariant mass (squared) of the two-photon system becomes $$ M^2=4E_{\rm CM}^2$$ with $E_{\rm CM}:= |\vec{p}_1|=|\vec{p}_2|$.
Mass is rest energy divided by c$^2$, so you could define the mass as hf$_1$+hf$_2$. However, for it to be useful as a concept in physical analysis, mass should be constant and localized. In the case of two free photons parting in opposite directions, I don't see a use for the concept of total mass. In the case of an opaque box containing light and with infinite Q, it might be useful.
