That the math font for \mathcal changes, when package txgreeks is loaded, is not an incompatibility, but intended behavior, the abstract of package txgreeks:
The TX Fonts 1 of Young Ryu provide a very complete replacement for
the default math fonts of TeX and LaTeX, containing all CM symbols and
even all symbols from the AMS fonts, and more. In particular upright
shapes for the Greek letters are available (they are necessary in
French mathematical typography). The txgreeks package allows LaTeX
users who use the TX fonts to easily select the shapes (italic or
upright) for the Greek lowercase and uppercase letters. This is
compatible with using arbitrary text fonts in the document.
Thus it is not a surprise, that txgreek loads package txfonts, which enables the TX fonts for math, including the font for \mathcal.
The CM/AMS fonts are using italic for lower case Greek letters by default. The remaining task to get italic upper case Greek letters is
discussed and answered in question "Greek letters in italic in math equation".
This is a modification of egreg's answer. It changes the macro definitions of \var<GreekLetterName> with \GreekLetterName. The upright symbols are not lost, but available with prefix var:
\documentclass[a5paper]{article}
\usepackage{amsmath}
\makeatletter
\@for\@tmp:=Gamma,Delta,Theta,Lambda,Xi,Pi,Sigma,Upsilon,Phi,Psi,Omega\do{%
\expandafter\let\expandafter\@@tmp\csname\@tmp\endcsname
\expandafter\let\csname\@tmp\expandafter\endcsname\csname var\@tmp\endcsname
\expandafter\let\csname var\@tmp\endcsname\@@tmp
}
\makeatother
\begin{document}
\subsection*{Lowercase Greek letters in italics}
\[\alpha\beta\gamma\delta\epsilon\varepsilon\dots
\]
\subsection*{Uppercase Greek letters}
\subsubsection*{Italics}
\[\Gamma \Delta \Theta \Lambda \Xi \Pi
\Sigma \Upsilon \Phi \Psi \Omega\]
\subsubsection*{Upright}
\[\varGamma \varDelta \varTheta \varLambda \varXi \varPi
\varSigma \varUpsilon \varPhi \varPsi \varOmega\]
\end{document}

txgreeksis an addendum for packagetxfonts, which indeed changes many math fonts. If you want to have uppercase Greek math letters in italics, then see question Greek letters in italic in math equation. – Heiko Oberdiek Jun 30 '15 at 14:04txgreek, with or withoutISO, uses a different font for\mathcal. – egreg Jun 30 '15 at 14:04\mathcalis not out of work here. And if I do not useISO, the lower case Greeks are still italic. You should minimize your code as much as possible, show some picture of what you see and explain, what you want to have. "Out of work" does not give good information on your problem especially as you are changing the fonts here. Firstamsfonts, thentxgreeks(txfonts). What do you want to have? – LaRiFaRi Jun 30 '15 at 14:34\mathcal. Math Greek upper case letters are made italic by the code in the answer. And the lower case letters are italic by default. – Heiko Oberdiek Jun 30 '15 at 14:37