The job of ~/.config/xfce4/xinitrc is to start the whole XFCE session. Since you aren't starting any program, the session ends as soon as the script exits.
You could mkae that script include the system script. (The man page says to copy the system script, but don't do that: the system script is likely to change from version to version.) Find out where the system script is; for example it's /etc/xdg/xfce4/xinitrc on Ubuntu. In your ~/.config/xfce4/xinitrc, run your initial customizations, then the system script.
#!/bin/sh
xset b off
. /etc/xdg/xfce4/xinitrc
Alternatively, add xset b off as an application in the autostart list (Settings → Session and Startup → Application Autostart → Add).