Blender's physics aren't real world physics, and there tends to be things that they're good at and things that they're bad at. In the case of rigid body physics, they're good at low energy collisions between primitive objects, and they're bad at dealing with actual meshes. This is a solvable problem: we just have to make our physical systems in a way that they work well with rigid body physics. Rather than trying to rely on one system to both render well and physics well, we'll abstract the physics from the render.
We'll start by disabling rigid body physics on the conveyor belt mesh. Then we'll make some simple, stretched-out cubes (but with scale applied) to make our conveyor belt out of objects. Here, I'm using a clamp to constraint to acquire the curve as a transformation (rather than a deformation):

And we can see that I have physics set up for this cube as well, rigid body, animated. We'll duplicate those a bunch and parent them to an empty:

Now when we move that empty, its children will inherit its location, then clamp to the curve-- so, rotating about the curve. We can just copy your X location keyframes from your rendering conveyor belt onto this empty.

We can disable rendering on these cubes, probably put them on their own collection somewhere out of the way, maybe set them to a wireframe display. They exist only to do our physics.