Draft:Nonlinear Multimodal Embeddings

• Comment: Google scholar returns only 1 result for "Nonlinear multimodal embeddings". Please do not resubmit without a second opinion from someone not connected with the subject matter. Caleb Stanford (talk) 23:38, 26 March 2025 (UTC)

• Comment: In accordance with Wikipedia's Conflict of interest policy, I disclose that I have a conflict of interest regarding the subject of this article. YoniNewman (talk) 18:23, 18 March 2025 (UTC)

---

Nonlinear multimodal embeddings are representation learning techniques used to project data from different modalities, such as text, image, audio or video, into a shared latent space. This allows for semantically similar content across modalities to be mapped to nearby points. Unlike linear approaches, nonlinear multimodal embeddings employ non-linear transformation that can capture complex relationships between different data types and enable more effective cross-modal retrieval, fusion and analysis.

Motivation

While traditional multimodal learning methods demonstrate the value of integrating different data types, they often rely on linear projections that struggle to capture the complex, non-linear relationships inherent in cross-modal data. Real world connections between modalities such as images and text rarely follow simple linear patterns. For instance, an image might relate to certain words through abstract concepts, metaphorical relationships, or cultural associations that linear models cannot adequately represent. Nonlinear multimodal embeddings address this limitation by employing more sophisticated mathematical transformations capable of modeling intricate interdependencies. This nonlinear approach is particularly motivated by applications requiring fine-grained semantic understanding across modalities. As datasets expand, preserving local structures and overall semantic coherence becomes crucial, requiring embedding methods that handle the data's nonlinearity.

Approaches and Methods

Canonical-correlation analysis based methods

Canonical-correlation analysis (CCA) was first introduced in 1936 by Harold Hotelling and is a fundamental approach for multimodal learning. CCA aims to find linear relationships between two sets of variables. Given two data matrices $X\in \mathbb {R} ^{n\times p}$ and $Y\in \mathbb {R} ^{n\times q}$ representing different modalities, CCA finds projection vectors $w_{x}\in \mathbb {R} ^{p}$ and $w_{y}\in \mathbb {R} ^{q}$ that maximizes the correlation between the projected variables:

$\rho =\max _{w_{x},w_{y}}{\frac {w_{x}^{\top }\Sigma _{xy}w_{y}}{{\sqrt {w_{x}^{\top }\Sigma _{xx}w_{x}}}{\sqrt {w_{y}^{\top }\Sigma _{yy}w_{y}}}}}$

such that $\Sigma _{xx}$ and $\Sigma _{yy}$ are the within-modality covariance matrices, and $\Sigma _{xy}$ is the between-modality covariance matrix. However, standard CCA is limited by its linearity, which led to the development of nonlinear extensions, such as kernel CCA and deep CCA.

Kernel CCA

Kernel canonical correlation analysis (KCCA) extends traditional CCA to capture nonlinear relationships between modalities by implicitly mapping the data into high dimensional feature spaces using kernel functions. Given kernel functions $K_{x}$ and $K_{y}$ with corresponding Gram matrices $K_{x}\in \mathbb {R} ^{n\times n}$ and $K_{y}\in \mathbb {R} ^{n\times n}$, KCCA seeks coefficients $\alpha$ and $\beta$ that maximize:

$\rho =\max _{\alpha ,\beta }{\frac {\alpha ^{\top }K_{x}Ky\beta }{{\sqrt {\alpha ^{\top }K_{x}^{2}\alpha }}{\sqrt {\beta ^{\top }K_{y}^{2}\beta }}}}$

To prevent overfitting, regularization terms are typically added, resulting in:

$\rho =\max _{\alpha ,\beta }{\frac {\alpha ^{T}K_{x}K_{y}\beta }{{\sqrt {\alpha ^{T}\left(K_{x}^{2}+\lambda _{x}K_{x}\right)\alpha }}{\sqrt {\;\beta ^{T}\left(K_{y}^{2}+\lambda _{y}K_{y}\right)\beta }}}}$

where $\lambda _{x}$ and $\lambda _{y}$ are regularization parameters. KCCA has proven effective for tasks such as cross-modal retrieval and semantic analysis, though it faces computational challenges with large datasets due to its $O(n^{2})$ memory requirement for sorting kernel matrices.

KCCA was proposed independently by several researchers.

Deep CCA

Deep canonical correlation analysis (DCCA), introduced in 2013, employs neural networks to learn nonlinear transformations for maximizing correlation between modalities. DCCA uses separate neural networks $f_{x}$ and $f_{y}$ for each modality to transofrm the original fata before applying CCA:

$\max _{W_{x},W_{y},\theta _{x},\theta _{y}}\operatorname {corr} \left(f_{x}(X;\theta _{x}),f_{y}(Y;\theta _{y})\right)$

where $\theta _{x}$ and $\theta _{y}$ represent the parameters of the neural networks, and $W_{x}$ and $W_{y}$ are the CCA projection matrices. The correlation objective is computer as:

$\operatorname {corr} (H_{x},H_{y})=\operatorname {tr} \left(T^{-1/2}H_{x}^{T}H_{y}S^{-1/2}\right)$

where $H_{x}=f_{x}(X)$ and $H_{y}=f_{y}(Y)$ are the network outputs, $T=H_{x}^{T}H_{x}+r_{x}I$, $S=H_{y}^{T}H_{y}+r_{y}I$ and $r_{x},r_{y}$ are the regularization parameters. DCCA overcomes the limitations of linear CCA and kernel CCA by learning complex nonlinear relationships while maintaining computational efficiency for large datasets through mini-batch optimization.

Graph based methods

Graph based approaches for nonlinear multimodal embeddings leverage graph structure to model relationships between entities across different modalities. These methods typically represent each modality as a graph and then learn embedding that preserve cross-modal similarities, enabling more effective joint representation of heterogeneous data.

One such method is cross modal graph neural networks (CMGNNs) that extend traditional graph neural networks (GNNs) to handle data from multiple modalities by constructing graphs that capture both intra-modal and inter-modal relationships. These networks model interactions across modalities by representing them as nodes and their relationships as edges.

Another graph based method is Deep Graph Matching Networks (DGMNs) focus on establishing correspondences between nodes in graphs from different modalities by learning a matching function that aligns similar entities across heterogeneous data sources. This approach combines graph neural networks with attention mechanisms to compute node-to-node similarity scores across modalities.

Nonlinear manifold alignment

Manifold alignment is a class of machine learning algorithms that produce projections between sets of data, given that the original data sets lie on a common manifold. Nonlinear manifold alignment extends traditional manifold learning techniques to align data manifolds from different modalities or domains. These methods assume that high-dimensional data from each modality lies on a lower-dimensional manifold, and seek transformations that preserve both the geometric structure within each manifold and the correspondence relationships between manifolds. Techniques such as local tangent space alignment (LTSA) represent the local geometry of the manifold using tangent spaces, which are then aligned to provide global coordinates of the data points. This method is particularly effective for heterogeneous multimodal data, where linear alignment techniques may fall short.

See also

• Canonical correlation
• Multimodal learning
• Deep learning
• Nonlinear dimensionality reduction