Spectral or Spatial? Leveraging Both for Speaker Extraction in Challenging Data Conditions