The key differences are mostly to do with physical form with some contribution from impurities
Coal is largely carbon but, depending on type, contains a range of impurities. Some anthracites are low in impurities, smokeless coals have been treated to drive off many impurities.
Charcoal is usually made from heating wood in an oxygen depleted atmosphere. It is, like smokeless coal, mostly carbon.
Graphite is mostly mined from mineral deposits and is almost pure carbon.
So why do they all burn differently?
The biggest factor is their physical form. But some coals also burn because they have some impurities that burn more easily than pure carbon. Purer coals are harder to ignite than "raw" coals, partly because of this.
But the major factor is the physical form of the substance. Charcoal is inherently porous partly because of how it is made. That porosity makes it much easier to ignite than many coals which are usually less porous and more dense. Porosity makes it easier for oxygen to penetrate the bulk solid, making it easier for the burning to happen and speeding the rate of combustion once burning has started. Coals are also often somewhat porous, though less so than charcoal. And the impurities in raw coals often lead to higher porosity when burning starts. Coked and smokeless coals tend to be porous as the coking process also enhances porosity by driving out impurities and leaving voids or because coal powder is treated and reconstituted into not-very-dense lumps with voids between the particles.
Graphite is created geologically when coal is subject to heat and much pressure. This, like the treatments that make smokeless coal, drives out impurities and compacts the result leaving dense, pure carbon (or because graphite crystallises out of a liquid rock as fairly pure graphite). This leaves very few voids in the solid so the only way it can burn is at the surface which limits the speed of combustion. It will still burn but can only do so slowly. By the way, the other major allotrope of pure carbon, diamond, will also burn but is also poor fuel for much the same reason.
If you crushed graphite into a powder and blew it into a furnace, it would be as fine a fuel as coal or charcoal. As would coal or charcoal if powdered.
So the primary differences are the physical form of the material: porous materials make it easier for oxygen to get "inside" the material to propagate the burning reaction.