Document resource
Background Precise cell type annotation in H&E-stained tumor slides remains a major challenge, hindering reproducible identification of the tumor immune microenvironment and clinical biomarkers for immunotherapy. While spatial single-cell transcriptomics enables high-resolution cell mapping, its practical application is limited by cost and complexity. Vision transformer-based pathology foundation models offer promise for digital pathology, yet direct single-cell type prediction remains largely unexplored.Methods We developed Hist2Cell, a vision transformer-based deep learning model that leverages pretrained pathology foundation models for hierarchical cell type prediction directly from H&E whole slide images. The model first assigns cells to five broad categories (Tier 1: neoplastic, immune, stromal, epithelial, dead) and then further classifies them into 17 subtypes (Tier 2), including nine immune and four stromal subtypes. For breast cancer, we further distinguish CD4+, CD8+ T cells, and M1/M2 macrophages (Tier 2+). Training utilized 100K image patches (40x magnification) from 161 FFPE samples, with cell type labels derived from spatial single-cell transcriptomics (Xenium).Results Among the foundation models used for encoder of vision transformer model, Hist2Cell adopting H-optimus-0 1 achieved the best performance, with AUROC of 0.71 (Tier 2) and 0.72 (Tier 2+) in cell type prediction. In breast cancer model, AUROC’s were 0.84 for malignant cells, 0.79 for endothelial cells, and 0.71 for CD4+ T cells. In the test of clinical utilities, Hist2Cell accurately mapped spatial immune cell distributions and identified clinically relevant features from routine pathology images, including tumor-infiltrating lymphocyte (TIL) regions, lymphatic vessel invasion, and tertiary lymphoid structures (TLS).Importantly, we demonstrate the clinical utility of Hist2Cell for predicting treatment response to:Targeted therapy prediction: Using the Yale trastuzumab-treated breast cancer cohort,2 Hist2Cell-derived spatial features enabled prediction of trastuzumab response.Immunotherapy (IO) response prediction: In the Histo-Miner CPI dataset,3 Hist2Cell features stratified immune checkpoint inhibitor (ICI) responders and non-responders.Conclusions Hist2Cell enables high-resolution mapping of tumor and immune cell types from standard histology slides and supports robust extraction of spatial biomarkers for cancer immunotherapy, including the identification and prediction of TIL-enriched regions, TLS regions, and lymphatic vessel invasion. It also demonstrated accurate prediction of patient response to both targeted and immunotherapy treatments. Hist2Cell has strong potential to advance precision oncology and guide patient selection for immunotherapies.References Saillard C, Jenatton R, Llinares-López F, Mariet Z, Cahané D, Durand E, Vert J-P. H-optimus-0. 2024. Available at: https://github.com/bioptimus/releases/tree/main/models/h-optimus/v0. Accessed Aug 16, 2024.Farahmand S, Fernandez AI, Ahmed FS, Rimm DL, Chuang JH, Reisenbichler E, Zarringhalam K. HER2 and trastuzumab treatment response H&E slides with tumor ROI annotations (Version 3) [Data set]. The Cancer Imaging Archive. 2022. https://doi.org/10.7937/E65C-AM96Lucas S, Lorenz C, Brägelmann J, Bozek K, Helbig D, Persa OD, Dengler S, Kreuter A, Laimer M. Histo-Miner: Use case CPI dataset [Data set]. Zenodo. 2025. https://doi.org/10.5281/zenodo.13986860