Interpretable machine learning (IML) is crucial in biomedical research, where understanding how models make decisions is as important as their accuracy. This study examines the stability of Permutation Feature Importance (PFI), a model-agnostic interpretability method, when applied to neural networks trained on high-dimensional single-cell RNA sequencing (scRNA-seq) data. Using data from healthy and hydrosalpinx-affected human fallopian tube cells, five neural networks were trained with varying hyperparameters and input feature sets. All models achieved similar classification accuracy, allowing a fair comparison of feature importance rankings. PFI was applied to assess the contribution of each gene to model predictions, and stability was evaluated across three levels: different models, repeated runs with varying random seeds, and datasets with different numbers of input genes. Kendall’s coefficient of concordance and the Jaccard index were used to measure consistency in feature rankings. Results showed high agreement in repeated PFI runs on the same model and moderate consistency across models with different hyperparameters, indicating that PFI produces stable and interpretable outputs when the input data and model architecture remain unchanged. However, substantial variation appeared when input dimensionality was reduced, showing that PFI rankings are sensitive to the number of input features. These findings emphasize that while PFI is reliable for comparing similar models trained on the same data, its interpretability can degrade in low-dimensional or filtered feature spaces. This insight supports the development of more dependable explainable AI tools for high-dimensional biological data, particularly in genomics and precision medicine.