You can load the package mathtools and write:
\[
a(i,k) \leftarrow
\min\Big\{0, r(k,k) + \sum_{\mathclap{i'\mid i'\notin \{i,k\}}}
\max\{0, r(i',k)\}\Big\}
\]
However, I always prefer some space left around such sum:

\[
a(i,k) \leftarrow
\min\Big\{0, r(k,k) + \;\sum_{\mathclap{i'\mid i'\notin \{i,k\}}}\;
\max\{0, r(i',k)\}\Big\}
\]
As well:
Notice that the syntax for the prime is neither i^{'} nor i{'}. It is i'.
It might be better to use \mid instead of |. Here, the result is the same, but the symbol | has four different meanings (relation, operator, open delim, close delim), and it is a good practice to always use one of \mid, \vert, \lvert, \rvert.
In my opinion, \sum_{i|i\notin X} is redundant, you can write just \sum_{i\notin X}. It is a common practice that summation is over the very first variable in the sum's index.
Typing \Bigl\{ ... \Bigr\} instead of \Big\{...\Big\} costs two keystrokes more, and you keep the sematics correctly, as well in some cases it is necessary (absolute value |.|, reverse-bracket open intervals ].[ etc.)
You can make the braces after \max bigger, it doesn't cost any space and it increases readibility and visual appearance.

\[
a(i,k) \leftarrow
\min\Bigl\{0, r(k,k) + \;\sum_{\mathclap{i'\notin \{i,k\}}}\;
\max\bigl\{0, r(i{'},k)\bigr\}\Bigr\}
\]
\[and\], rather than$\displaystyleand$: if set inline it will spoil the appearance of the page. – egreg Jan 12 '13 at 22:29mathtools, you can use a few dozen\!negative spacers in a pinch. – Brian61354270 Sep 22 '23 at 00:52