The question can be answered in a more than one way.
They are all equally important in that if one is poor quality (or mismatched) then the whole chain collapses. But that is being pedantic. I think one way to think about it is to look at the size of the market and the variety of technologies available that will give an indication of how manytheories there are ondesign.
There are zillions of speaker designs which reflects the variety of musical styles you can create by changing the speaker. They are mechanical tranducers with many variables - materials used for the cones etc, crossover specifications, quality of cabinet construction, degree of internal bracing etc etc etc so there is more to play with in their design. It is also the bit that interacts with the room - get that wrong and no amount of money spent on amplification will correct it.
Next I think would come the pre-amp - it is taking a small signal and has to amplify it with minimal distortion and avoiding the introduction of noise. And you can have the choice of passive or active pre-amps.
The the power-amp is, IMO, a more mature market than either of the other two - it is taking a signal of reasonable strength and should be easier to get right because noise (from the mains, RFI, etc) is proportionately less than in the pre-amp section.
I have met only a few people with separate pre- and power amps but they seem to be of the view that they buy a good power amp and 'fine tune' the amplification by trying different pre-amps.
Overall I think it is easier to get the speakers wrong than any other component.