If you insert \showtokens{#3} at the beginning of your redefined \chapterlinesformat, you'll see this on the terminal and in the log file:
> \interlinepenalty \@M Digital Services of the University\@@par .
\chapterlinesformat #1#2#3->\showtokens {#3}
\myExtractFirstLetter {#3}\makeb...l.32 \chapter*{Digital Services of the University}
So, your #3 does not start with the first letter of the title because KOMA-Script inserted a very high \interlinepenalty (10000) to make sure the title can't be broken between two pages without your explicit consent.
Robust method
Simplest setup without hyperref nor nameref:
\documentclass{scrbook}
\usepackage[T1]{fontenc} % for the non-ASCII demo title
\usepackage{lmodern} % can be arbitrarily scaled
\usepackage{xcolor}
\makeatletter
\long\def@my@extract@first@char#1\MyFirstChar#2#3\my@endTitle{#2}
\newcommand{\myExtractFirstChar}[2]{%
\edef#2{\unexpanded\expandafter{@my@extract@first@char #1\my@endTitle}}%
}
\let\MyFirstChar@firstofone
\makeatother
\renewcommand{\chapterlinesformat}[3]{%
\myExtractFirstChar{#3}{\MyExtractedFirstChar}%
\makebox(0,20)[l]{%
\hspace{-20pt}%
\fontsize{60}{6}\selectfont
\color{black!15!white}%
\MyExtractedFirstChar
}%
#3% <--------- don't forget!
}
\begin{document}
\chapter{\MyFirstChar{Œ}uvres complètes}
Here is some text.
\end{document}

If you use hyperref or nameref, it is not strictly necessary but might be nicer, for some applications, to explicitly tell gettitlestring that the \MyFirstChar command in titles does not matter:
\documentclass{scrbook}
\usepackage[T1]{fontenc} % for the non-ASCII demo title
\usepackage{lmodern} % can be arbitrarily scaled
\usepackage{xcolor}
\usepackage[expand]{gettitlestring}
\usepackage{nameref}
\usepackage{hyperref}
\makeatletter
\long\def@my@extract@first@char#1\MyFirstChar#2#3\my@endTitle{#2}
\newcommand{\myExtractFirstChar}[2]{%
\edef#2{\unexpanded\expandafter{@my@extract@first@char #1\my@endTitle}}%
}
\let\MyFirstChar@firstofone
\GetTitleStringDisableCommands{\let\MyFirstChar@firstofone}
\makeatother
\renewcommand{\chapterlinesformat}[3]{%
\myExtractFirstChar{#3}{\MyExtractedFirstChar}%
\makebox(0,20)[l]{%
\hspace{-20pt}%
\fontsize{60}{6}\selectfont
\color{black!15!white}%
\MyExtractedFirstChar
}%
#3% <--------- don't forget!
}
\begin{document}
\chapter{\MyFirstChar{Œ}uvres complètes\label{chap}}
Here is some text. The chapter title is ``\nameref{chap}.''
% You need \usepackage{refcount} in order to test this.
%
% \edef\zzz{\unexpanded\expandafter\expandafter\expandafter{%
% \getrefbykeydefault{chap}{name}{dflt}}}%
% \show\zzz % > \zzz=macro:->Œuvres complètes.
\end{document}
The left side shows the PDF bookmark:

More automatic but less robust solution
Another, however less robust way to solve the problem, consists in using a simple regular expression in order to find the first letter in \chapterlinesformat's third argument. The command \myExtractFirstLetter, defined below, assigns its second argument locally, but this could easily be made global if you prefer.
\documentclass{scrbook}
\usepackage{lmodern} % can be arbitrarily scaled
\usepackage{xcolor}
\usepackage{xparse}
\ExplSyntaxOn
\seq_new:N \l__jendrik_match_seq
% The second argument must be a macro or expl3 token list variable
\NewDocumentCommand \myExtractFirstLetter { m m }
{
% The (?i) makes the [a-z] class case-insensitive
\regex_extract_once:nnNTF { (?i)[a-z] } {#1} \l__jendrik_match_seq
{ \seq_get_left:NN \l__jendrik_match_seq #2 }
{ \tl_clear:N #2 } % no match -> clear the #2 macro / tl var
}
\ExplSyntaxOff
\renewcommand{\chapterlinesformat}[3]{%
\myExtractFirstLetter{#3}{\myFirstLetter}%
\makebox(0,20)[l]{%
\hspace{-20pt}%
\fontsize{60}{6}\selectfont
\color{black!15!white}%
\myFirstLetter
}%
#3% <--------- don't forget!
}
\begin{document}
\chapter*{Digital Services of the University}
Here is some text.
\end{document}
