DIALOGUE - multi cellular programs#

DIALOGUE [JAR22] discovers multicellular programs (MCPs): coordinated gene-expression patterns that vary together across samples / niches in multiple cell types at once.

The pipeline has three phases:

  1. Sample-pseudobulk each cell type and run penalized multiple-CCA on the cell-type-specific feature spaces to obtain MCP weights and per-cell scores.

  2. For every ordered pair of cell types and every shared MCP, fit a hierarchical linear model of one cell type’s MCP score against the partner cell type’s pseudobulk expression of candidate genes; record signed z-scores.

  3. Combine per-gene evidence across pairs (BH-then-Fisher), then refit the per-cell scores with iterative non-negative least squares on the retained gene set.

Best results require at least ~10 samples per condition. The data here is a subset of the ulcerative colitis cohort from the original paper.

Setup#

import warnings

warnings.filterwarnings("ignore", category=DeprecationWarning, module="pertpy")

import pandas as pd
import pertpy as pt
import scanpy as sc

Dataset#

adata = pt.ds.dialogue_example()
adata
AnnData object with n_obs × n_vars = 5374 × 6329
    obs: 'nCount_RNA', 'nFeature_RNA', 'cellQ', 'gender', 'location', 'clinical.status', 'cell.subtypes', 'pathology', 'origin', 'subset', 'sample', 'path_str'
    var: 'name'
    layers: None (.X)

DIALOGUE expects three adata.obs columns: the cell type, the sample / niche label, and a per-cell quality covariate (here cellQ, typically derived from log-counts).

We also encode pathology status as a string column for downstream comparisons.

sc.pp.pca(adata)
sc.pp.neighbors(adata)
sc.tl.umap(adata)
sc.pl.umap(
    adata,
    color=["clinical.status"],
)
../../_images/e57a0d3bd8ce374a42689654324d2d9c7f9650bd324c60ba71c293b09104f087.png
sc.pl.umap(
    adata,
    color=["sample"],
)
../../_images/6669db7dedc93a9f2bc0c72734b88442810026cf626613dcd93bb393ce677e18.png
# ensure that every cell type is represented in every sample
isecs = pd.crosstab(adata.obs["cell.subtypes"], adata.obs["sample"])

isecs
(isecs > 3).sum(axis=1)
cell.subtypes
CD8+ IELs      29
CD8+ IL17+      9
CD8+ LP        30
Macrophages    30
TA2            30
dtype: int64
# based on what we see above, remove CD8+ IL17+ because it's poorly represented across samples
adata = adata[adata.obs["cell.subtypes"] != "CD8+ IL17+"]
isecs = pd.crosstab(adata.obs["cell.subtypes"], adata.obs["sample"])

# then remove the any sample which now has an unrepresented cell type
keep_pts = list(isecs.loc[:, (isecs > 3).sum(axis=0) == isecs.shape[0]].columns.values)
adata = adata[adata.obs["sample"].isin(keep_pts), :].copy()
adata
AnnData object with n_obs × n_vars = 5156 × 6329
    obs: 'nCount_RNA', 'nFeature_RNA', 'cellQ', 'gender', 'location', 'clinical.status', 'cell.subtypes', 'pathology', 'origin', 'subset', 'sample', 'path_str'
    var: 'name'
    uns: 'pca', 'neighbors', 'umap', 'clinical.status_colors', 'sample_colors'
    obsm: 'X_pca', 'X_umap'
    varm: 'PCs'
    obsp: 'distances', 'connectivities'
    layers: None (.X)

Running DIALOGUE#

The pipeline has three phases, called in order on the same AnnData:

  1. fit_programs(adata) — pseudobulk per sample, run penalized multiple-CCA, score every cell.

  2. test_celltype_pairs(adata) — fit a per-pair HLM of one cell type’s program score against the partner’s pseudobulk expression of candidate genes.

  3. refine_scores(adata) — Fisher-combine the per-gene evidence across pairs and refit per-cell scores with iterative non-negative least squares.

Results live on adata.obsm["X_dialogue"], adata.obs["mcp_0"/"mcp_1"/...], and adata.uns["dialogue"].

dl = pt.tl.Dialogue(
    celltype_key="cell.subtypes",
    sample_key="sample",
    cell_quality_key="cellQ",
    n_programs=3,
)
dl.fit_programs(adata)
dl.test_celltype_pairs(adata)
dl.refine_scores(adata)
AnnData object with n_obs × n_vars = 5156 × 6329
    obs: 'nCount_RNA', 'nFeature_RNA', 'cellQ', 'gender', 'location', 'clinical.status', 'cell.subtypes', 'pathology', 'origin', 'subset', 'sample', 'path_str', 'mcp_0', 'mcp_1', 'mcp_2'
    var: 'name'
    uns: 'pca', 'neighbors', 'umap', 'clinical.status_colors', 'sample_colors', 'dialogue'
    obsm: 'X_pca', 'X_umap', 'X_dialogue_cca', 'X_dialogue'
    varm: 'PCs'
    obsp: 'distances', 'connectivities'
    layers: None (.X)
sc.pl.umap(
    adata,
    color=["mcp_0", "mcp_1", "clinical.status"],
    ncols=1,
    cmap="coolwarm",
    vcenter=0,
)
../../_images/8a75c46df324437045da22fdce25c064d045f7d2f65a8f850c2123e539c09b56.png

test_phenotype_association fits one HLM per (cell type, program) of the MCP score against a phenotype, controlling for cellQ and the sample random effect, and returns a signed z-score table like R’s DIALOGUE.pheno. The indicator is the boolean column itself (or the first level of a categorical column), so positive values mean higher scores in pathological samples here. The All row pools the cells of every cell type.

dl.test_phenotype_association(adata, condition_key="pathology")
MCP1 MCP2 MCP3
CD8+ IELs 2.845029 -2.131332 -2.567863
CD8+ LP 2.638867 -2.039325 -2.495653
Macrophages 2.043727 -2.333838 -1.357344
TA2 3.903422 -3.782445 -2.211240
All 2.790751 -3.134785 -2.068562

mcp_0 looks significantly associated with pathology status. Two helper plots zoom in:

dl.plot_split_violins(adata, condition_key="path_str", program="MCP1")
../../_images/ddd89b3595e34436abfc7753754407ad9ce8bdebb23bd33045178edff0378207.png
dl.plot_pairplot(adata, color="clinical.status", program="MCP1")
../../_images/2dce39ec00dade04baf5380737e7c7cd91863c38ec6c26ff875e531fa32b02c9.png

There are two complementary ways to inspect program-associated genes.

find_extreme_score_genes runs rank_genes_groups between the cells with the highest and lowest MCP scores per cell type. It does not enforce any cross-cell-type consistency, so it surfaces correlates within each cell type.

extrema_genes = dl.find_extreme_score_genes(adata, program="MCP1")
# top 10% of TA2 MCP1 scores compared to the bottom 10%
extrema_genes["TA2"].head(10)
names scores logfoldchanges pvals pvals_adj
0 RPL39 26.214300 6.928117 1.715737e-70 3.619634e-67
1 MT-ND3 21.704226 6.214807 1.859983e-65 2.354366e-62
2 OLFM4 21.569912 8.761182 8.527645e-67 1.349287e-63
3 SLC12A2 19.000534 6.624269 1.312781e-51 9.231766e-49
4 SET 15.800262 4.905701 1.341284e-42 4.467887e-40
5 RPL37A 15.444179 1.514294 6.411654e-42 2.028968e-39
6 SPINK1 14.968813 5.184348 5.683817e-40 1.712994e-37
7 RPS27 14.915593 1.606700 1.577860e-39 4.539216e-37
8 RPL34 14.799312 2.091359 1.867438e-38 4.924591e-36
9 RPS29 12.529554 1.578393 3.067488e-30 3.733486e-28

get_program_genes exposes the refined signatures from the pair-level / iterative-NNLS path that matches the original DIALOGUE paper. The candidate genes are those with consistent evidence across pairs after Fisher-combining z-scores.

# The multilevel-modeling step was already run as part of test_celltype_pairs(adata).
# The aggregated per-gene results live on adata.uns['dialogue']:
adata.uns["dialogue"]["gene_pvalues"]["TA2"].head()
gene program up programF CD8+ IELs CD8+ LP Macrophages p_up p_down n_up nf_up n_down nf_down N Nf coef
0 ADD3 MCP1 True MCP1.up 3.819921 3.331202 2.532002 1.929602e-07 1.0 3 1.0 0 0.0 3 1.0 0.464133
1 AGR3 MCP1 True MCP1.up 8.018788 6.976151 5.932591 3.220437e-16 1.0 3 1.0 0 0.0 3 1.0 0.478667
2 ARGLU1 MCP1 True MCP1.up 4.124109 5.701559 4.158928 6.356856e-11 1.0 3 1.0 0 0.0 3 1.0 0.356597
3 ATP1B1 MCP1 True MCP1.up 5.339227 6.143988 7.224583 1.897656e-14 1.0 3 1.0 0 0.0 3 1.0 0.000000
4 ATRX MCP1 True MCP1.up 3.873063 5.145607 5.238130 3.379364e-11 1.0 3 1.0 0 0.0 3 1.0 0.366751

The per-gene Fisher-combined table for each cell type is also exposed:

ta2_genes = dl.get_program_genes(adata, program="MCP1", celltype="TA2")

How much do the two sets overlap on TA2?

sig_genes = extrema_genes["TA2"][extrema_genes["TA2"]["pvals_adj"] < 0.05]
up_genes_extrema = set(sig_genes[sig_genes["logfoldchanges"] > 0]["names"])
down_genes_extrema = set(sig_genes[sig_genes["logfoldchanges"] < 0]["names"])
ta2_up = set(ta2_genes["up"])
overlap = ta2_up & up_genes_extrema
print(f"extrema_up={len(up_genes_extrema)}  dialogue_up={len(ta2_up)}  overlap={len(overlap)}")
extrema_up=757  dialogue_up=87  overlap=87
ta2_down = set(ta2_genes["down"])
overlap_down = ta2_down & down_genes_extrema
print(f"extrema_down={len(down_genes_extrema)}  dialogue_down={len(ta2_down)}  overlap={len(overlap_down)}")
extrema_down=1502  dialogue_down=95  overlap=95