Hadamard product (matrices) explained

In mathematics, the Hadamard product (also known as the element-wise product, entrywise product or Schur product^[1]) is a binary operation that takes in two matrices of the same dimensions and returns a matrix of the multiplied corresponding elements. This operation can be thought as a "naive matrix multiplication" and is different from the matrix product. It is attributed to, and named after, either French mathematician Jacques Hadamard or German mathematician Issai Schur.

The Hadamard product is associative and distributive. Unlike the matrix product, it is also commutative.^[2]

Definition

For two matrices and of the same dimension, the Hadamard product

(sometimes ^{[3] [4] [5]}) is a matrix of the same dimension as the operands, with elements given by

For matrices of different dimensions (and, where or), the Hadamard product is undefined.

For example, the Hadamard product for two arbitrary 2 × 3 matrices is:

Properties

• The Hadamard product is commutative (when working with a commutative ring), associative and distributive over addition. That is, if A, B, and C are matrices of the same size, and k is a scalar: $$\backslash begin$$

A \odot B &= B \odot A, \\ A \odot (B \odot C) &= (A \odot B) \odot C, \\ A \odot (B + C) &= A \odot B + A \odot C, \\\left(kA\right) \odot B &= A \odot \left(kB\right) = k\left(A \odot B\right), \\ A \odot 0 &= 0 \odot A = 0.\end

• The identity matrix under Hadamard multiplication of two matrices is an matrix where all elements are equal to 1. This is different from the identity matrix under regular matrix multiplication, where only the elements of the main diagonal are equal to 1. Furthermore, a matrix has an inverse under Hadamard multiplication if and only if none of the elements are equal to zero.^[6]
• For vectors and, and corresponding diagonal matrices and with these vectors as their main diagonals, the following identity holds:^[7] $$\backslash mathbf^*\; (A\; \backslash odot\; B)\backslash mathbf\; =\; \backslash operatorname\backslash left(\_\backslash mathbf^*\; A\; \_\backslash mathbf\; ^\backslash mathsf\backslash right),$$ where denotes the conjugate transpose of . In particular, using vectors of ones, this shows that the sum of all elements in the Hadamard product is the trace of where superscript T denotes the matrix transpose, that is,

. A related result for square and, is that the row-sums of their Hadamard product are the diagonal elements of : $$\backslash sum\_i\; (A\; \backslash odot\; B)\_\; =\; \backslash left(B^\backslash mathsf\; A\backslash right)\_\; =\; \backslash left(AB^\backslash mathsf\backslash right)\_.$$ Similarly, $$\backslash left(\backslash mathbf\backslash mathbf^*\backslash right)\; \backslash odot\; A\; =\; \_\backslash mathbf\; A\; \_\backslash mathbf^*$$ Furthermore, a Hadamard matrix-vector product can be expressed as: $$(A\; \backslash odot\; B)\; \backslash mathbf\; =\; \backslash operatorname\backslash left(A\; D\_\backslash mathbf\; B^\backslash mathsf\; \backslash right)$$ where is the vector formed from the diagonals of matrix .

• The Hadamard product is a principal submatrix of the Kronecker product.^{[8] [9] [10]}
• The Hadamard product satisfies the rank inequality $$\backslash operatorname(A\; \backslash odot\; B)\; \backslash leq\; \backslash operatorname(A)\; \backslash operatorname(B)$$
• If and are positive-definite matrices, then the following inequality involving the Hadamard product holds:^[11] $$\backslash prod\_^n\; \backslash lambda\_i(A\; \backslash odot\; B)\; \backslash ge\; \backslash prod\_^n\; \backslash lambda\_i(A\; B),\backslash quad\; k=1,\backslash ldots,n,$$ where is the th largest eigenvalue of .
• If and are diagonal matrices, then^[12] $$\backslash begin$$

D (A \odot B) E &= (D A E) \odot B = (D A) \odot (B E) \\ &= (AE) \odot (D B) = A \odot (D B E).\end

• The Hadamard product of two vectors

and is the same as matrix multiplication of the corresponding diagonal matrix of one vector by the other vector: $$\backslash mathbf\; a\; \backslash odot\; \backslash mathbf\; b\; =\; D\; \_\; \backslash mathbf\; b\; =\; D\; \_\; \backslash mathbf\; a$$

• The vector to diagonal matrix

  \operatorname{diag}

  operator may be expressed using the Hadamard product as: $$\backslash operatorname(\backslash mathbf)\; =\; (\backslash mathbf\; \backslash mathbf^T)\; \backslash odot\; I$$ where

is a constant vector with elements and is the identity matrix.

The mixed-product property

$$(A\; \backslash otimes\; B)\; \backslash odot\; (C\; \backslash otimes\; D)\; =\; (A\; \backslash odot\; C)\; \backslash otimes\; (B\; \backslash odot\; D),$$ where

 is Kronecker product, assuming has the same dimensions of and with .

$$(A\; \backslash bull\; B)\; \backslash odot\; (C\; \backslash bull\; D)\; =\; (A\; \backslash odot\; C)\; \backslash bull\; (B\; \backslash odot\; D),$$ where

 denotes face-splitting product.^[13]

$$(A\; \backslash bull\; B)(C\; \backslash ast\; D)\; =\; (A\; C)\; \backslash odot\; (B\; D),$$ where

 is column-wise Khatri–Rao product.

Schur product theorem

See main article: Schur product theorem. The Hadamard product of two positive-semidefinite matrices is positive-semidefinite. This is known as the Schur product theorem,^[6] after Russian mathematician Issai Schur. For two positive-semidefinite matrices and, it is also known that the determinant of their Hadamard product is greater than or equal to the product of their respective determinants:$$\backslash det(\backslash odot)\; \backslash ge\; \backslash det\; \backslash det.$$

Analogous operations

Other Hadamard operations are also seen in the mathematical literature,^[14] namely the and (which are in effect the same thing because of fractional indices), defined for a matrix such that:

For

and for

The reads:

A is defined as:^{[15] [16]}

In programming languages

Most scientific or numerical programming languages include the Hadamard product, under various names.

In MATLAB, the Hadamard product is expressed as "dot multiply": a .* b, or the function call: times(a, b).^[17] It also has analogous dot operators which include, for example, the operators a .^ b and a ./ b.^[18] Because of this mechanism, it is possible to reserve * and ^ for matrix multiplication and matrix exponentials, respectively.

The programming language Julia has similar syntax as MATLAB, where Hadamard multiplication is called broadcast multiplication and also denoted with a .* b, and other operators are analogously defined element-wise, for example Hadamard powers use a .^ b.^[19] But unlike MATLAB, in Julia this "dot" syntax is generalized with a generic broadcasting operator . which can apply any function element-wise. This includes both binary operators (such as the aforementioned multiplication and exponentiation, as well as any other binary operator such as the Kronecker product), and also unary operators such as ! and √. Thus, any function in prefix notation f can be applied as f.(x).^[20]

Python does not have built-in array support, leading to inconsistent/conflicting notations. The NumPy numerical library interprets a*b or a.multiply(b) as the Hadamard product, and uses a@b or a.matmul(b) for the matrix product. With the SymPy symbolic library, multiplication of objects as either a*b or a@b will produce the matrix product. The Hadamard product can be obtained with the method call a.multiply_elementwise(b).^[21] Some Python packages include support for Hadamard powers using methods like np.power(a, b), or the Pandas method a.pow(b).

In C++, the Eigen library provides a cwiseProduct member function for the class (a.cwiseProduct(b)), while the Armadillo library uses the operator % to make compact expressions (a % b; a * b is a matrix product).

In GAUSS, and HP Prime, the operation is known as array multiplication.

In Fortran, R, APL, J and Wolfram Language (Mathematica), the multiplication operator * or × apply the Hadamard product, whereas the matrix product is written using matmul, %*%, +.×, +/ .* and ., respectively.The R package matrixcalc introduces the function hadamard.prod for Hadamard Product of numeric matrices or vectors.^[22]

Applications

The Hadamard product appears in lossy compression algorithms such as JPEG. The decoding step involves an entry-for-entry product, in other words the Hadamard product.

In image processing, the Hadamard operator can be used for enhancing, suppressing or masking image regions. One matrix represents the original image, the other acts as weight or masking matrix.

It is used in the machine learning literature, for example, to describe the architecture of recurrent neural networks as GRUs or LSTMs.^[23]

It is also used to study the statistical properties of random vectors and matrices.^{[24] [25]}

The penetrating face product

According to the definition of V. Slyusar the penetrating face product of the p×g matrix

and n-dimensional matrix (n > 1) with p×g blocks ( ) is a matrix of size of the form:^[26]

Example

If

then

Main properties

where

denotes the face-splitting product of matrices, where is a vector.

Applications

The penetrating face product is used in the tensor-matrix theory of digital antenna arrays. This operation can also be used in artificial neural network models, specifically convolutional layers.^[27]

See also

• Frobenius inner product
• Pointwise product
• Kronecker product
• Khatri–Rao product

Notes and References

1. Davis. Chandler. 1962. The norm of the Schur product operation. Numerische Mathematik. 4. 343–44. 10.1007/bf01386329. 1. 121027182.
2. Web site: Million. Elizabeth. April 12, 2007. The Hadamard Product. September 6, 2020. buzzard.ups.edu.
3. Web site: Hadamard product - Machine Learning Glossary. machinelearning.wtf.
4. Web site: linear algebra - What does a dot in a circle mean?. Mathematics Stack Exchange.
5. Web site: Element-wise (or pointwise) operations notation?. Mathematics Stack Exchange.
6. Web site: Million. Elizabeth . The Hadamard Product. 2 January 2012.
7. Book: Horn. Roger A.. Charles R.. Johnson. Matrix analysis. Cambridge University Press. 2012.
8. Liu . Shuangzhe . Trenkler . Götz. 2008 . Hadamard, Khatri-Rao, Kronecker and other matrix products . International Journal of Information and Systems Sciences . 4 . 1 . 160–177.
9. Liu. Shuangzhe. Leiva. Víctor. Zhuang. Dan. Ma. Tiefeng . Figueroa-Zúñiga. Jorge I.. 2022. Matrix differential calculus with applications in the multivariate linear model and its diagnostics. Journal of Multivariate Analysis. 188. 104849. 10.1016/j.jmva.2021.104849. 239598156 . free.
10. Liu. Shuangzhe. Trenkler. Götz. Kollo. Tõnu. von Rosen. Dietrich. Baksalary. Oskar Maria. 2023. Professor Heinz Neudecker and matrix differential calculus. Statistical Papers. 65 . 4 . 2605–2639 . en. 10.1007/s00362-023-01499-w.
11. Hiai. Fumio. Lin. Minghua. On an eigenvalue inequality involving the Hadamard product. Linear Algebra and Its Applications. February 2017. 515. 313–320 . 10.1016/j.laa.2016.11.017. free.
12. Web site: Project . buzzard.ups.edu . 2007 . 2019-12-18.
13. Slyusar. V. I. . End products in matrices in radar applications. . Radioelectronics and Communications Systems . 1998 . 41 . 3. 50–53.
14. Reams . Robert . 1999 . Hadamard inverses, square roots and products of almost semidefinite matrices . Linear Algebra and Its Applications . 288 . 35–43 . 10.1016/S0024-3795(98)10162-3 . free.
15. Web site: Wetzstein . Gordon . Lanman . Douglas . Hirsch . Matthew . Raskar . Ramesh . Supplementary Material: Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting . MIT Media Lab.
16. Book: Cyganek, Boguslaw . Object Detection and Recognition in Digital Images: Theory and Practice . John Wiley & Sons . 2013 . 9781118618363 . 109.
17. Web site: MATLAB times function.
18. Web site: Array vs. Matrix Operations.
19. Web site: Vectorized "dot" operators . 31 January 2024.
20. Web site: Dot Syntax for Vectorizing Functions . 31 January 2024.
21. Web site: Common Matrices — SymPy 1.9 documentation .
22. Web site: 16 May 2013 . Matrix multiplication . 24 August 2013 . An Introduction to R . The R Project for Statistical Computing.
23. Sak . Haşim . Senior . Andrew . Beaufays . Françoise . 2014-02-05 . Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition . cs.NE . 1402.1128 .
24. Neudecker. Heinz. Liu. Shuangzhe. Polasek. Wolfgang. 1995. The Hadamard product and some of its applications in statistics. Statistics . 26. 4. 365–373. 10.1080/02331889508802503 .
25. Neudecker. Heinz. Liu. Shuangzhe . 2001. Some statistical properties of Hadamard products of random matrices. Statistical Papers. 42. 4 . 475–487. 10.1007/s003620100074 . 121385730 .
26. Slyusar. V. I.. March 13, 1998. A Family of Face Products of Matrices and its properties . Cybernetics and Systems Analysis C/C of Kibernetika I Sistemnyi Analiz. 1999.. 35. 3. 379–384. 10.1007/BF02733426. 119661450.
27. Ha D., Dai A.M., Le Q.V.. 2017. HyperNetworks.. The International Conference on Learning Representations (ICLR) 2017. – Toulon, 2017.. Page 6. 1609.09106.