Using \autoref with a starred * part, chapter, section in a report

Question

I'm writing my thesis in LaTex and I'm having issues with \autoref. The issues are illustrated in the MWE below. I included the whole preamble in case there are conflicting packages in it (hyperref and titlesec don't mix well).

The issue seems to be the * in \chapter*{} and \section*{}. It messes up the numbering in \autoref{} even though I added \addtocounter{}.

\documentclass[11pt,a4paper,twoside]{report}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage[main=english,german]{babel}
\usepackage{libertine}      %font
\usepackage{lettrine}
\usepackage{geometry}       % page margins
\geometry{
    left=2.54cm,
    right=2.54cm,
    top=2.5cm,
    bottom=2.5cm,
}
\headsep = 5mm
\newcommand{\headspace}{\vspace*{-1.5cm}}       % remove headspace on select pages
\makeatletter                           %% Define a HUGE 
\newcommand\HUGE{@setfontsize\Huge{35}{35}}
\makeatother
\usepackage[doublespacing]{setspace}
\usepackage{fancyhdr}       % headers & footers
\usepackage{booktabs}
\usepackage{tabularx}
\newcolumntype{Y}{>{\centering\arraybackslash}X}
\usepackage{longtable}
\usepackage{pgfplotstable}
\newcommand{\sbx}[2][c]{%
    \begin{tabular}[#1]{@{}c@{}}#2\end{tabular}}
\usepackage{pdflscape}
\usepackage{afterpage}
\usepackage{rotating}
\usepackage{ragged2e}
\usepackage{pdfpages}
\usepackage{tikz}
\usepackage{subcaption}
\usepackage{enumerate}
\usepackage{array}
\newcolumntype{x}[1]{>{\centering\arraybackslash\hspace{0pt}}p{#1}}
\usepackage{apacite}            %bibliography
\usepackage{natbib}             %bibliography
\usepackage{bibentry}           %bibliography
\renewcommand{\bibpreamble}{\vskip2cm}          % for spacing in between references and list of reference
\usepackage[titletoc]{appendix}     % package to get the word appendix in the TOC
\usepackage{titlesec}
\makeatletter                                                   % for TOC "Part" and spacing
\renewcommand*\l@part[2]{%
    \ifnum \c@tocdepth >-2\relax
    \addpenalty{-@highpenalty}%
    \addvspace{1.5em @plus\p@}%
    \setlength@tempdima{3em}%
    \begingroup
    \parindent \z@ \rightskip @pnumwidth
    \parfillskip -@pnumwidth
    {\leavevmode
        \large \bfseries Part #1\hfill\null}\par
    \nobreak
    \global@nobreaktrue
    \everypar{\global@nobreakfalse\everypar{}}%
    \endgroup
    \fi}
\makeatother
\addto\captionsenglish{\renewcommand{\contentsname}{Table of Contents}} % change name of TOC (doesn't work with titlesec package
\titleclass{\part}{top}                 % formatting part pages (https://tex.stackexchange.com/questions/431868/part-page-styling?rq=1)
\titleformat{\part}
[display]
{\thispagestyle{empty}\raggedleft\fontfamily{LinuxBiolinumT-OsF}\selectfont\HUGE\bfseries}              %removed page style to get rid of page number on parts, font
{\vspace{8cm}\MakeUppercase{\partname} \thepart}
{0pt}
{\titlerule[.7pt]\vspace{1cm}\Huge}
\titleclass{\chapter}{top}              % formatting chapter pages (https://tex.stackexchange.com/questions/431868/part-page-styling?rq=1)
\titleformat{\chapter}
[display]
{\raggedleft\fontfamily{LinuxBiolinumT-OsF}\selectfont\HUGE\bfseries}
{\vspace{3pt}{\chaptername} \thechapter}
{0pt}
{\vspace{1pc}\Huge}
%
\titlespacing*{\chapter}{0pt}{1.8cm}{1cm}
%
\usepackage[english,linktocpage=true]{hyperref}
\hypersetup{
    colorlinks,
    linkcolor={blue!50!black},
    citecolor={blue!30!black},
    urlcolor={blue!80!black},
}
%\newcommand{\link}[1]{{\color{blue!80!black}\href{#1}{#1}}}
%
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathabx}
\usepackage{csquotes}
%\biboptions{sort&compress}
\usepackage{listings}
\lstset{language=R,
    basicstyle=\small\ttfamily,
    stringstyle=\color{darkgray},
    otherkeywords={0,1,2,3,4,5,6,7,8,9},
    morekeywords={TRUE,FALSE},
    deletekeywords={data,frame,length,as,character},
    keywordstyle=\color{blue},
    commentstyle=\color{darkgray},
}
\usepackage{xparse}                 % to make a signature line
\makeatletter
\newcommand\signaturelinewidth{5cm}
\newcommand\signaturelineheight{.4pt}
\newcommand\signaturedashleaderatom{\kern .1pt.\kern .1pt}
\newcommand\signaturelineraise{.4ex}
\newcommand\signaturelabelindent{.5cm}
\newcommand\signaturetextindent{}
\NewDocumentCommand\signatureline{
    O{\signaturelinewidth}      % line width
    O{\signaturelabelindent}    % label margin
    O{%                           text indentation
        \ifx@empty\signaturetextindent
        .5\dimexpr #2\relax
        \else
        \signaturetextindent
        \fi
    }
    m                           % label
    O{\signaturelineheight}     % line height
    O{\signaturelineraise}      % line raise
    D||{}                       % text
}{%
    \parbox[t]{#1}{%
        \leftskip #3%
        \mbox{\strut #7}%
        \vskip -#6%
        \hrule height #5%
        \vskip #6%
        \normalsize
        \leftskip #2%
        \rightskip #2%
        \strut #4%
    }%
}
\NewDocumentCommand\signaturedash{
    O{\signaturelinewidth}      % line width
    O{\signaturelabelindent}    % label margin
    O{%                           text indentation
        \ifx@empty\signaturetextindent
        .5\dimexpr #2\relax
        \else
        \signaturetextindent
        \fi
    }
    m                           % label
    O{\signaturedashleaderatom} % leader atom
    O{\signaturelineraise}      % line raise
    D||{}                       % text
}{%
    \parbox[t]{#1}{%
        \leftskip #3%
        \mbox{\strut #7}%
        \vskip -#6%
        \leftskip 0pt%
        \hrule height 0pt%
        \leavevmode\cleaders\hbox{#5}\hfill\kern 0pt%
        \hrule height 0pt%
        \vskip #6%
        \normalsize
        \leftskip #2%
        \rightskip #2%
        \strut #4%
    }%
}
\makeatother
\usepackage{afterpage}                                  % command for blank pages
\newcommand\blankpage{
    \null
    \thispagestyle{empty}
    \addtocounter{page}{-1}
    \newpage
}
\begin{document}
\setcounter{tocdepth}{2}
\tableofcontents
\part{Part 1 title} \label{part:1}
\chapter{Chapter 1 title}   \label{chap:1}
Reference to \autoref{chap:1}. Correct. 
\chapter{Chapter 2 title}   \label{chap:2}
\addtocounter{chapter}{1}
Reference to \autoref{chap:2}. Correct. 
\section{Section 1 title}   \label{sec:1}
Reference to \autoref{sec:1}. Correct.
\part{Part 2 title}    \label{part:2}
\addtocounter{part}{1}
\addtocontents{toc}{\protect\contentsline{chapter}{\large Part 2 \vspace{.5em}}{}{}}
\chapter{Chapter 3 title \ Why does this repeat?} \label{chap:3}
\addtocounter{chapter}{1}
\addtocontents{toc}{Chapter 3}
This is the wrong reference: \autoref{part:2}. It should be part 2.
\chapter{My super awesome chapter 4}   \label{chap:4}
\addtocounter{chapter}{1}
This is the wrong reference: \autoref{chap:3}. It should be chapter 3.
\section{Section 2 title}  \label{sec:2}
\addtocounter{section}{1}
This is the wrong reference: \autoref{sec:2}. It should be section 4.1.
\end{document}

---

EDIT I realized the problem is from using the * in \chapter*{}. I don't want the chapters to have a number in the table of contents (which \chapter*{} does), but at the same time, be able to be referenced to by \autoref{}. So I how do I remove the number from in front of (e.g.) a chapter name when using \chapter{}? The TeX documentation says to use * ...