You need to define the command in the preamble and also load graphicx package.
\documentclass[11pt]{report}
\usepackage[utf8]{inputenc}
\usepackage[english,ukrainian]{babel}
\usepackage{graphicx}
\DeclareRobustCommand{\capitalhyphen}{\raisebox{0.24ex}{\resizebox{0.4em}{\height}{-}}\kern-0.07em}
\begin{document}
\chapter{ФІЗИКО\capitalhyphen МАТЕМАТИЧНИЙ}
\end{document}

*File List*
report.cls 2007/10/19 v1.4h Standard LaTeX document class
size11.clo 2007/10/19 v1.4h Standard LaTeX file (size option)
inputenc.sty 2008/03/30 v1.1d Input encoding file
utf8.def 2008/04/05 v1.1m UTF-8 support for inputenc
t1enc.dfu 2008/04/05 v1.1m UTF-8 support for inputenc
ot1enc.dfu 2008/04/05 v1.1m UTF-8 support for inputenc
omsenc.dfu 2008/04/05 v1.1m UTF-8 support for inputenc
babel.sty 2008/07/08 v3.8m The Babel package
english.ldf 2005/03/30 v3.3o English support from the babel system
ukraineb.ldf 2008/03/21 v1.1l Ukrainian support from the babel system
t2aenc.dfu 2008/04/05 v1.1m UTF-8 support for inputenc
graphicx.sty 1999/02/16 v1.0f Enhanced LaTeX Graphics (DPC,SPQR)
keyval.sty 1999/03/16 v1.13 key=value parser (DPC)
graphics.sty 2009/02/05 v1.0o Standard LaTeX Graphics (DPC,SPQR)
trig.sty 1999/03/16 v1.09 sin cos tan (DPC)
graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live
pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX
infwarerr.sty 2010/04/08 v1.3 Providing info/warning/error messages (HO)
ltxcmds.sty 2011/11/09 v1.22 LaTeX kernel commands for general use (HO)
t2acmr.fd 2001/08/11 v1.0a Computer Modern Cyrillic font definitions
supp-pdf.mkii
pdftexcmds.sty 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO)
ifluatex.sty 2010/03/01 v1.3 Provides the ifluatex switch (HO)
ifpdf.sty 2011/01/30 v2.3 Provides the ifpdf switch (HO)
epstopdf-base.sty 2010/02/09 v2.5 Base part for package epstopdf
grfext.sty 2010/08/19 v1.1 Manage graphics extensions (HO)
kvdefinekeys.sty 2011/04/07 v1.3 Define keys (HO)
kvoptions.sty 2011/06/30 v3.11 Key value format for package options (HO)
kvsetkeys.sty 2012/04/25 v1.16 Key value parser (HO)
etexcmds.sty 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO)
epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Live
***********
If you add
\usepackage[unicode]{hyperref,xcolor}
\hypersetup{citebordercolor=green,linkbordercolor=white}
You get an error that \height is undefined. hyperref needs to do a lot of normalising of characters to meet the restricted encoding allowed in pdf bookmarks. It apparently disables the latex protection mechanism to do that.
You can add \let\height\relax then you get no error but the bookmark list isn't right. however
\usepackage{hyperref,xcolor}
\hypersetup{citebordercolor=green,linkbordercolor=white}
seems to work
\DeclareRobustCommandinstead of\newcommandor your definition but\protect\capitalhyphenwhen used in a moving argument. See this question for a technical description of the problem and this question for a similar problem. – egreg Jan 31 '13 at 16:55\DeclareRobustCommand{\capitalhyphen}{...}is the correct way; sorry if my comment has been misleading. Then you don't have to use\protectin front of\capitalhyphen– egreg Jan 31 '13 at 17:20graphicxpackage to also be included, since you're using\resizebox. – Werner Jan 31 '13 at 17:22\protect(that doesn't belong there), I get no error. It's probably a problem of encoding; retype the chapter title, but before runningpdflatexfor the first time, remove the.auxand.tocfiles. – egreg Jan 31 '13 at 17:43