DIALOGUE - multi cellular programs#
DIALOGUE [JAR22] discovers multicellular programs (MCPs): coordinated gene-expression patterns that vary together across samples / niches in multiple cell types at once.
The pipeline has three phases:
Sample-pseudobulk each cell type and run penalized multiple-CCA on the cell-type-specific feature spaces to obtain MCP weights and per-cell scores.
For every ordered pair of cell types and every shared MCP, fit a hierarchical linear model of one cell type’s MCP score against the partner cell type’s pseudobulk expression of candidate genes; record signed z-scores.
Combine per-gene evidence across pairs (BH-then-Fisher), then refit the per-cell scores with iterative non-negative least squares on the retained gene set.
Best results require at least ~10 samples per condition. The data here is a subset of the ulcerative colitis cohort from the original paper.
Setup#
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning, module="pertpy")
import pandas as pd
import pertpy as pt
import scanpy as sc
Dataset#
adata = pt.ds.dialogue_example()
adata
AnnData object with n_obs × n_vars = 5374 × 6329
obs: 'nCount_RNA', 'nFeature_RNA', 'cellQ', 'gender', 'location', 'clinical.status', 'cell.subtypes', 'pathology', 'origin', 'subset', 'sample', 'path_str'
var: 'name'
layers: None (.X)
DIALOGUE expects three adata.obs columns: the cell type, the sample / niche label, and a per-cell quality covariate (here cellQ, typically derived from log-counts).
We also encode pathology status as a string column for downstream comparisons.
sc.pp.pca(adata)
sc.pp.neighbors(adata)
sc.tl.umap(adata)
sc.pl.umap(
adata,
color=["clinical.status"],
)
sc.pl.umap(
adata,
color=["sample"],
)
# ensure that every cell type is represented in every sample
isecs = pd.crosstab(adata.obs["cell.subtypes"], adata.obs["sample"])
isecs
(isecs > 3).sum(axis=1)
cell.subtypes
CD8+ IELs 29
CD8+ IL17+ 9
CD8+ LP 30
Macrophages 30
TA2 30
dtype: int64
# based on what we see above, remove CD8+ IL17+ because it's poorly represented across samples
adata = adata[adata.obs["cell.subtypes"] != "CD8+ IL17+"]
isecs = pd.crosstab(adata.obs["cell.subtypes"], adata.obs["sample"])
# then remove the any sample which now has an unrepresented cell type
keep_pts = list(isecs.loc[:, (isecs > 3).sum(axis=0) == isecs.shape[0]].columns.values)
adata = adata[adata.obs["sample"].isin(keep_pts), :].copy()
adata
AnnData object with n_obs × n_vars = 5156 × 6329
obs: 'nCount_RNA', 'nFeature_RNA', 'cellQ', 'gender', 'location', 'clinical.status', 'cell.subtypes', 'pathology', 'origin', 'subset', 'sample', 'path_str'
var: 'name'
uns: 'pca', 'neighbors', 'umap', 'clinical.status_colors', 'sample_colors'
obsm: 'X_pca', 'X_umap'
varm: 'PCs'
obsp: 'distances', 'connectivities'
layers: None (.X)
Running DIALOGUE#
The pipeline has three phases, called in order on the same AnnData:
fit_programs(adata)— pseudobulk per sample, run penalized multiple-CCA, score every cell.test_celltype_pairs(adata)— fit a per-pair HLM of one cell type’s program score against the partner’s pseudobulk expression of candidate genes.refine_scores(adata)— Fisher-combine the per-gene evidence across pairs and refit per-cell scores with iterative non-negative least squares.
Results live on adata.obsm["X_dialogue"], adata.obs["mcp_0"/"mcp_1"/...], and adata.uns["dialogue"].
dl = pt.tl.Dialogue(
celltype_key="cell.subtypes",
sample_key="sample",
cell_quality_key="cellQ",
n_programs=3,
)
dl.fit_programs(adata)
dl.test_celltype_pairs(adata)
dl.refine_scores(adata)
AnnData object with n_obs × n_vars = 5156 × 6329
obs: 'nCount_RNA', 'nFeature_RNA', 'cellQ', 'gender', 'location', 'clinical.status', 'cell.subtypes', 'pathology', 'origin', 'subset', 'sample', 'path_str', 'mcp_0', 'mcp_1', 'mcp_2'
var: 'name'
uns: 'pca', 'neighbors', 'umap', 'clinical.status_colors', 'sample_colors', 'dialogue'
obsm: 'X_pca', 'X_umap', 'X_dialogue_cca', 'X_dialogue'
varm: 'PCs'
obsp: 'distances', 'connectivities'
layers: None (.X)
sc.pl.umap(
adata,
color=["mcp_0", "mcp_1", "clinical.status"],
ncols=1,
cmap="coolwarm",
vcenter=0,
)
test_phenotype_association fits one HLM per (cell type, program) of the MCP score against a phenotype, controlling for cellQ and the sample random effect, and returns a signed z-score table like R’s DIALOGUE.pheno.
The indicator is the boolean column itself (or the first level of a categorical column), so positive values mean higher scores in pathological samples here.
The All row pools the cells of every cell type.
dl.test_phenotype_association(adata, condition_key="pathology")
| MCP1 | MCP2 | MCP3 | |
|---|---|---|---|
| CD8+ IELs | 2.845029 | -2.131332 | -2.567863 |
| CD8+ LP | 2.638867 | -2.039325 | -2.495653 |
| Macrophages | 2.043727 | -2.333838 | -1.357344 |
| TA2 | 3.903422 | -3.782445 | -2.211240 |
| All | 2.790751 | -3.134785 | -2.068562 |
mcp_0 looks significantly associated with pathology status. Two helper plots zoom in:
dl.plot_split_violins(adata, condition_key="path_str", program="MCP1")
dl.plot_pairplot(adata, color="clinical.status", program="MCP1")
There are two complementary ways to inspect program-associated genes.
find_extreme_score_genes runs rank_genes_groups between the cells with the highest and lowest MCP scores per cell type. It does not enforce any cross-cell-type consistency, so it surfaces correlates within each cell type.
extrema_genes = dl.find_extreme_score_genes(adata, program="MCP1")
# top 10% of TA2 MCP1 scores compared to the bottom 10%
extrema_genes["TA2"].head(10)
| names | scores | logfoldchanges | pvals | pvals_adj | |
|---|---|---|---|---|---|
| 0 | RPL39 | 26.214300 | 6.928117 | 1.715737e-70 | 3.619634e-67 |
| 1 | MT-ND3 | 21.704226 | 6.214807 | 1.859983e-65 | 2.354366e-62 |
| 2 | OLFM4 | 21.569912 | 8.761182 | 8.527645e-67 | 1.349287e-63 |
| 3 | SLC12A2 | 19.000534 | 6.624269 | 1.312781e-51 | 9.231766e-49 |
| 4 | SET | 15.800262 | 4.905701 | 1.341284e-42 | 4.467887e-40 |
| 5 | RPL37A | 15.444179 | 1.514294 | 6.411654e-42 | 2.028968e-39 |
| 6 | SPINK1 | 14.968813 | 5.184348 | 5.683817e-40 | 1.712994e-37 |
| 7 | RPS27 | 14.915593 | 1.606700 | 1.577860e-39 | 4.539216e-37 |
| 8 | RPL34 | 14.799312 | 2.091359 | 1.867438e-38 | 4.924591e-36 |
| 9 | RPS29 | 12.529554 | 1.578393 | 3.067488e-30 | 3.733486e-28 |
get_program_genes exposes the refined signatures from the pair-level / iterative-NNLS path that matches the original DIALOGUE paper. The candidate genes are those with consistent evidence across pairs after Fisher-combining z-scores.
# The multilevel-modeling step was already run as part of test_celltype_pairs(adata).
# The aggregated per-gene results live on adata.uns['dialogue']:
adata.uns["dialogue"]["gene_pvalues"]["TA2"].head()
| gene | program | up | programF | CD8+ IELs | CD8+ LP | Macrophages | p_up | p_down | n_up | nf_up | n_down | nf_down | N | Nf | coef | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | ADD3 | MCP1 | True | MCP1.up | 3.819921 | 3.331202 | 2.532002 | 1.929602e-07 | 1.0 | 3 | 1.0 | 0 | 0.0 | 3 | 1.0 | 0.464133 |
| 1 | AGR3 | MCP1 | True | MCP1.up | 8.018788 | 6.976151 | 5.932591 | 3.220437e-16 | 1.0 | 3 | 1.0 | 0 | 0.0 | 3 | 1.0 | 0.478667 |
| 2 | ARGLU1 | MCP1 | True | MCP1.up | 4.124109 | 5.701559 | 4.158928 | 6.356856e-11 | 1.0 | 3 | 1.0 | 0 | 0.0 | 3 | 1.0 | 0.356597 |
| 3 | ATP1B1 | MCP1 | True | MCP1.up | 5.339227 | 6.143988 | 7.224583 | 1.897656e-14 | 1.0 | 3 | 1.0 | 0 | 0.0 | 3 | 1.0 | 0.000000 |
| 4 | ATRX | MCP1 | True | MCP1.up | 3.873063 | 5.145607 | 5.238130 | 3.379364e-11 | 1.0 | 3 | 1.0 | 0 | 0.0 | 3 | 1.0 | 0.366751 |
The per-gene Fisher-combined table for each cell type is also exposed:
ta2_genes = dl.get_program_genes(adata, program="MCP1", celltype="TA2")
How much do the two sets overlap on TA2?
sig_genes = extrema_genes["TA2"][extrema_genes["TA2"]["pvals_adj"] < 0.05]
up_genes_extrema = set(sig_genes[sig_genes["logfoldchanges"] > 0]["names"])
down_genes_extrema = set(sig_genes[sig_genes["logfoldchanges"] < 0]["names"])
ta2_up = set(ta2_genes["up"])
overlap = ta2_up & up_genes_extrema
print(f"extrema_up={len(up_genes_extrema)} dialogue_up={len(ta2_up)} overlap={len(overlap)}")
extrema_up=757 dialogue_up=87 overlap=87
ta2_down = set(ta2_genes["down"])
overlap_down = ta2_down & down_genes_extrema
print(f"extrema_down={len(down_genes_extrema)} dialogue_down={len(ta2_down)} overlap={len(overlap_down)}")
extrema_down=1502 dialogue_down=95 overlap=95