This is quite easy with xparse and \NewDocumentCommand, having two optional arguments, however, you need to check for their existence.
Please note that oo could be replaced by gg to allow for optional {}{} arguments, but that's not recommended, but see below in this answer as an alternative.
\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand{\com}{oo}{%
\lambda%
%Check if the first arg is given
\IfValueT{#1}{%
% Now check if has a 2nd argument as well.
\IfValueTF{#2}{% Yes, the 2nd one is present, use {f}(x) style
\{#1\}(#2)%
}{% No, use [f] style
[#1]%
}%
}%
}
\begin{document}
$\com$
$\com[f]$
$\com[f][x]$
\end{document}

Update with gg type (use it with care!) and another optional argument to use that instead of \lambda
\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand{\com}{oo}{%
\lambda%
%Check if the first arg is given
\IfValueT{#1}{%
% Now check if has a 2nd argument as well.
\IfValueTF{#2}{% Yes, the 2nd one is present, use {f}(x) style
\{#1\}(#2)%
}{% No, use [f] style
[#1]%
}%
}%
}
\NewDocumentCommand{\comother}{O{\lambda}gg}{%
#1%
%Check if the first arg is given
\IfValueT{#2}{%
% Now check if has a 2nd argument as well.
\IfValueTF{#3}{% Yes, the 2nd one is present, use {f}(x) style
\{#2\}(#3)%
}{% No, use [f] style
[#2]%
}%
}%
}
\begin{document}
$\com$
$\com[f]$
$\com[f][x]$
$\comother$
$\comother{f}$
$\comother{f}{x}$
% Now with \beta instead of \lambda
$\comother[\beta]$
$\comother[\beta]{h}$
$\comother[\beta]{h}{y}$
\end{document}

$\lambda \{f\}(x)$? – David Richerby Mar 18 '17 at 13:44xparse.\com=\lambda;\com[f]=\lambda[f];\com{f}(x)=\lambda\{f\}(x). That way your code resembles the output and is easier to understand. – Manuel Mar 18 '17 at 14:46