Abstract: Let $*$ denote the t -product between two third-order tensors proposed by Kilmer and Martin (Linear Algebra Appl 435(3): 641-658, 2011). The purpose of this work is to study fundamental computation over the set $\operatorname{St}(n, p, l):=\left\{\mathcal{X} \in \mathbb{R}^{n \times p \times l} \mid \mathcal{X}^{\top} * \mathcal{X}=\right. \mathcal{I}\}$, where $\mathcal{X}$ is a third-order tensor of size $n \times p \times l(n \geqslant p)$ and $\mathcal{I}$ is the identity tensor. It is first verified that St ( $n, p, l$) endowed with the Euclidean metric forms a Riemannian manifold, which is termed as the (third-order) tensor Stiefel manifold in this work. We then derive the tangent space, Riemannian gradient, and Riemannian Hessian on St ($n, p, l$). In addition, formulas of various retractions based on t-QR, t-polar decomposition, t-Cayley transform, and t-exponential, as well as vector transports, are presented. It is expected that analogous to their matrix counterparts, the derived formulas may serve as building blocks for analyzing optimization problems over the tensor Stiefel manifold and designing Riemannian algorithms.

Key words: Tensor, t-Product, Stiefel manifold, Retraction, Vector transport, Manifold optimization