For the benefit of this thread I undertook a little test session. Having both an aluminium and carbon example of the same series frame is quite handy, because it gives a high level of consistency to the handling characteristics.
The Equipment:
Frames:
Frame 1 is a 2011/12 Giant Anthem aluminium with Fox Float RP2 rear shock and Rock Shox Reba RL dual-air fork with quick release dropouts, aluminium seatpost and stem, carbon bar.
Frame 2 is a 2011/12 Anthem Advanced SL carbon frame, including rear swingarm & upper rockers, with Fox Float RP23 shock and Fox F100 RLC fork with 15mm through-axle, carbon/aluminium composite stem, carbon bar and seatpost.
Both frames are medium size and roll on 26" wheels
In terms of drivetrain, both run a bastardised mix of SLX & XT, both 3x10. The aluminium bike has a non-clutched derailleur, while the carbon bike has a clutched model; the clutch was switched OFF for consistency & comparison of noise transfer.
Wheels:
Wheelset 1 is an aluminium Shimano M785 XT set, quick release at both ends, fitted with Maxxis Ignitor LUST tyres, tubeless.
Wheelset 2 is an aluminium Shimano M775 XT set, quick release rear, through axle front, fitted with standard foldable Maxxis Crossmark tyres, tubed. Both XT sets have 24 spokes front & rear, the M785 is a few grams lighter, but not enough difference to impact the test. I could have got away with one set, but decided to use both to minimise fork swapping.
Wheelset 3 is a Dirty Mongrel Racing custom build, Carbonal M623 rims, Shimano M9000 XTR hubs with 32 DT Aerolite spokes per end, quick release rear, through axle front, with Maxxis Crossmark LUST tyres, tubeless.
All hubs have 10-degree ratchet profiling, so pedal pick-up is consistent across all wheels.
All tyres were run at 22psi front, 24psi rear, although the tubed tyres could probably have been run a little harder to compensate for the softer sidewalls.
The Test Course:
I used a shortened version of the Commonwealth Games course at Lysterfield; mostly gravel-topped but in places worn through to natural surface including tree roots and exposed rocks, with a wide range of corners, climbs, descents and higher-speed sections and some rock garden technical features. The course is 4.2km long, and I tried to pick the same lines as much as possible.
The Test:
A single lap was ridden in each configuration; first all-aluminium, second carbon frame/alu wheels, third all carbon, finally swapping the forks to fit the carbon wheels to the aluminium frame.
The meaty bits:
Not too surprisingly, the all-aluminium rig transmits a lot of trail chatter, and although the tight & steep geometry & small wheels produce agile handling, there is some floatiness when pointing at corners. Switching to the aluminium-shod carbon frame, it's immediately apparent the gravelly trail buzz is reduced. The lighter frame is easier to chuck around, but the slightly floaty feel in corners remains. Side note: this second lap was ridden with the through-axle version of the XT wheels. For me at least there is no significant change in handling attributable to the axle/fork configuration. More aggressive riders may notice more difference.
Lap Three & onto the full carbon package. The gravelly trail buzz is further reduced, but the bigger bumps become slightly more apparent. It's a weird paradox, you feel the size of the bumps more due to the greater stiffness of the wheels, but the vibration damping of the resin component of the composite layup takes the sting out of the initial hit. The handling is noticeably sharper; there is less delay in information getting from the handlebar & fork and through the wheel to the ground and it darts into corners, but as noted earlier in the thread, the stiffness does introduce a little bit of skittishness on mid-corner bumps, although that can be dialled out with an adjustment of tyre pressure. The other factor to the agility is the lack of weight, in this instance there's a 300g difference between the carbon wheels and the XT sets. The reduction in rotating mass cuts the inertia, so it's easier to change direction, and faster to wind up out of corners.
Finally the aluminium frame/carbon wheels. The trail buzz is about on par with the opposite alu/carbon mix - more than all-carbon, but less than all-aluminium, but the handling gains are all there. You do feel the bumps a bit more because you don't have as much carbon (well, resin really) to soak up the vibrations, but they're still softer than aluminium.
The Wrap
Clearly all-carbon is the optimal package, but that's not always possible. If it has to come down to a choice, aluminium frame on carbon wheels is a better package than carbon frame on aluminium wheels.