There are a few drawbacks to fully automated flight systems. High on this list are component and sensor failures, and eroded flight skills on the part of the pilots. Sensors do fail on occasion, which can either put the aircraft into a dangerous attitude, while the automated systems may lull the pilots into a sense of complacency and leave them unprepared to handle an unexpected emergency.
Consider these incidents in which automated flight systems played a significant role:
Air New Zealand acceptance flight, 2008. On an acceptance flight, when ANZ was reclaiming an A320 they had leased to XL Airways Germany, a frozen angle of attack sensor led to the aircraft stalling and crashing into the ocean after the auto flight system behaved unpredictably... ironically, while the pilots were testing the auto stall recovery.
AF 440... in this case, the auto flight system wasn't directly responsible. However, the combination of the auto flight system dropping out due to frozen pitot tubes, and an inexperienced flight crew, quite possibly with eroded basic piloting skills due to over reliance on auto flight systems, led to a high altitude stall and the loss of the aircraft and everyone on board. Contributing factors are the independent sidestick design (as contrasted to the directly connected center flight stick that might have alerted the pilot that the copilot had the stick all the way back), and auto-throttle not giving any visual cues that it had changed power input (as in moving the throttles).
Qantas 72 angle of attack sensor failure, sending conflicting signals to the flight control computer. The A330 didn't crash, but the pilots had to wrestle with the plane while it went through several extreme climb/dive cycles, until they could stabilize it. One crew member and 11 passengers suffered serious injuries, while 8 crew and 99 passengers suffered minor injuries. This could easily have been a fatal accident.
AF 296. This is the one that we've all seen the videos... while making a low flyby at a regional airshow, it appears that the A300's flight computer mistook this for an approach and went into a landing cycle and put the plane into a forest. In all fairness, this was an early effort at automated flight systems, and that particular issue has been resolved.
Air Inter 148 This was a combination of pilot error and the complexities of a fully automated flight system. On approaching Strasbourg, the pilots entered 3.3 into the flight system for descent, appearing to want a down angle on the nose of 3.3 degrees, when they had actually entered a descent rate of 3300 feet per minute... the flight computer was in the wrong mode. Contributing factors: the auto flight system actually increased the descent rate due to some turbulence encountered, and Air Inter had not installed a ground proximity warning system on their aircraft. Consequently, the aircraft flew into the ground. In this case, rather than eliminating human error, the automated flight control system appears to have encouraged it with a potentially confusing input method. The article notes that at least three fatal crashes of the A320 are traceable to confusion by pilots with the automatic flight control system.
These are a recounting of a few incidents in which the auto flight system played a role in a crash or near crash. It should also be noted that human error also played a role in the majority of them.
The flip side is - how many crashes have been prevented by auto flight systems? That's harder to predict.
I did find this article on airliner crashes per million miles flow, although some interpretation is required, as Boeing airliners have been flying for a lot longer than Airbus, and clearly the later versions of both have much better safety records than their earlier predecessors.
If we take two modern contemporaries that are heavily used today, such as Boeing 777 with some automation but a more traditional approach to cockpit and autopilot design, and Airbus A330 which uses a much more automated flight control system, we find that they have almost identical crash rates.
Right now, it would appear that fully automated flight systems versus semi automated is a draw... neither has a clear advantage.