The wind turbine gearbox is a critical equipment transforming the speed of the rotor hub to the generator, the condition of which is the reflection of operational efficiency and reliability of wind turbines. As the initial stage of the wind turbine gearbox, the fault feature extraction of the planetary gear set is challenging since it is prone to be affected by complicated structure, vibration from other high-speed stages and background noise. In this paper, a double Q factor wavelet-based sparse decomposition is applied to the fault feature extraction of the wind turbine planetary gearbox. Considering the sparsest wavelet coefficients, the vibration signal is iteratively decomposed into high Q and low Q components. The fault feature is generally hidden in the low Q component. With further demodulation, the fault information of planetary gears can be easily detected.