---
title: "SIMD in Rust: Optimizing Matrix Multiplication"
description: "Leveraging Rust’s SIMD support for accelerating matrix multiplication with considerations for portability and correctness"
url: "https://mayorana.ch/en/blog/simd-matrix-mult-rust"
language: "en"
date: "2025-10-01"
author: "Mohamed Bennekrouf"
tags:
  - "rust"
  - "simd"
  - "advanced"
fr_version: "https://mayorana.ch/fr/blog/simd-matrix-mult-rust.md"
---

# SIMD in Rust: Optimizing Matrix Multiplication

Rust’s **SIMD (Single Instruction, Multiple Data)** capabilities enable parallel processing of multiple data elements in a single CPU instruction, ideal for computationally intensive tasks like matrix multiplication. I’ll explain how to leverage `std::arch` for maximum throughput, address portability across architectures (e.g., x86_64 with SSE/AVX vs. ARM with NEON), and highlight challenges and solutions for ensuring correctness and performance.

<div class="svg-container" style="margin:2rem 0;">
<svg class="lo3-fig" viewBox="0 0 800 250" width="100%" style="height:auto;max-width:780px;display:block;margin:0 auto;" role="img" aria-label="Comparison of a scalar loop processing one f32 at a time versus an AVX instruction processing 8 f32 lanes at once">
<style>
.lo3-fig{--bg:#f8fafc;--box:#ffffff;--tx:#1e293b;--mut:#64748b;--ln:#cbd5e1;--ac:#FF6B00}
:root.dark .lo3-fig,[data-theme="dark"] .lo3-fig{--bg:#0f172a;--box:#1e293b;--tx:#f8fafc;--mut:#94a3b8;--ln:#475569}
.lo3-fig .box{fill:var(--box);stroke:var(--ln);stroke-width:1.5}
.lo3-fig .fin{fill:var(--box);stroke:var(--ac);stroke-width:2}
.lo3-fig .lane{fill:var(--bg);stroke:var(--ln);stroke-width:1}
.lo3-fig .lane-on{fill:var(--ac);stroke:var(--ac);stroke-width:1}
.lo3-fig .tx{fill:var(--tx);font:600 12px ui-sans-serif,system-ui,sans-serif;text-anchor:middle}
.lo3-fig .mut{fill:var(--mut);font:500 11px ui-sans-serif,system-ui,sans-serif;text-anchor:middle}
.lo3-fig .ac{fill:var(--ac);font:700 12px ui-sans-serif,system-ui,sans-serif;text-anchor:middle}
.lo3-fig line{stroke:var(--ln);stroke-width:1.5}
</style>
<defs>
<marker id="lo3-arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0,0 L10,5 L0,10 z" fill="var(--ln)"/>
</marker>
</defs>
<!-- left: scalar -->
<rect x="40" y="30" width="320" height="110" rx="6" class="box"/>
<text x="200" y="52" class="tx">Scalar loop</text>
<text x="200" y="70" class="mut">c[i][j] += a[i][k] * b[k][j]</text>
<rect x="80" y="82" width="20" height="20" class="lane-on"/>
<rect x="105" y="82" width="20" height="20" class="lane"/>
<rect x="130" y="82" width="20" height="20" class="lane"/>
<rect x="155" y="82" width="20" height="20" class="lane"/>
<rect x="180" y="82" width="20" height="20" class="lane"/>
<rect x="205" y="82" width="20" height="20" class="lane"/>
<rect x="230" y="82" width="20" height="20" class="lane"/>
<rect x="255" y="82" width="20" height="20" class="lane"/>
<text x="200" y="122" class="mut">1 f32 per instruction, 8 iterations</text>
<!-- right: SIMD -->
<rect x="440" y="30" width="320" height="110" rx="6" class="fin"/>
<text x="600" y="52" class="tx">AVX: _mm256_mul_ps / add_ps</text>
<text x="600" y="70" class="mut">8 dot-product terms at once</text>
<rect x="480" y="82" width="20" height="20" class="lane-on"/>
<rect x="505" y="82" width="20" height="20" class="lane-on"/>
<rect x="530" y="82" width="20" height="20" class="lane-on"/>
<rect x="555" y="82" width="20" height="20" class="lane-on"/>
<rect x="580" y="82" width="20" height="20" class="lane-on"/>
<rect x="605" y="82" width="20" height="20" class="lane-on"/>
<rect x="630" y="82" width="20" height="20" class="lane-on"/>
<rect x="655" y="82" width="20" height="20" class="lane-on"/>
<text x="600" y="122" class="ac">8 f32 per instruction, 1 iteration</text>
<!-- merge to result -->
<line x1="200" y1="140" x2="200" y2="168" marker-end="url(#lo3-arrow)"/>
<line x1="600" y1="140" x2="600" y2="168" marker-end="url(#lo3-arrow)"/>
<path d="M200,168 L200,190 L600,190 L600,168" fill="none" stroke="var(--ln)" stroke-width="1.5"/>
<rect x="270" y="195" width="260" height="45" rx="6" class="box"/>
<text x="400" y="222" class="tx">~8x fewer loop iterations</text>
</svg>
</div>

## Vectorizing Matrix Multiplication with SIMD

Matrix multiplication (e.g., \( C = A \times B \), where \( A \) is \( m \times n \), \( B \) is \( n \times p \), and \( C \) is \( m \times p \)) involves computing dot products of rows and columns. A naive scalar implementation for a 4x4 matrix is:

```rust
fn matrix_mult_scalar(a: &[[f32; 4]; 4], b: &[[f32; 4]; 4], c: &mut [[f32; 4]; 4]) {
    for i in 0..4 {
        for j in 0..4 {
            c[i][j] = 0.0;
            for k in 0..4 {
                c[i][j] += a[i][k] * b[k][j];
            }
        }
    }
}
```

This processes one `f32` at a time, which is slow. SIMD can compute multiple elements simultaneously (e.g., 8 `f32` with AVX on x86_64). Here’s how to vectorize it using `std::arch`:

### Selecting SIMD Instructions
On x86_64 with AVX (256-bit registers), use:
- `_mm256_loadu_ps`: Load 8 `f32` into a 256-bit register.
- `_mm256_mul_ps`: Multiply two 256-bit vectors.
- `_mm256_add_ps`: Add two 256-bit vectors.
- `_mm256_storeu_ps`: Store results back to memory.

### Vectorized Implementation
Assuming \( p \) is a multiple of 8 (padding if needed), vectorize the inner loop:

```rust
#[cfg(target_arch = "x86_64")]
use std::arch::x86_64::*;

unsafe fn matrix_mult_simd(a: &[[f32; 8]; 8], b: &[[f32; 8]; 8], c: &mut [[f32; 8]; 8]) {
    for i in 0..8 {
        for j in (0..8).step_by(8) { // Process 8 elements of C[i][j..j+8]
            let mut sum = _mm256_setzero_ps(); // Zeroed 256-bit register
            for k in 0..8 {
                let a_vec = _mm256_set1_ps(a[i][k]); // Broadcast a[i][k]
                let b_ptr = b[k][j..].as_ptr();
                let b_vec = _mm256_loadu_ps(b_ptr);  // Load 8 elements of B
                let prod = _mm256_mul_ps(a_vec, b_vec);
                sum = _mm256_add_ps(sum, prod);      // Accumulate
            }
            _mm256_storeu_ps(c[i][j..].as_mut_ptr(), sum); // Store 8 results
        }
    }
}
```

This computes 8 dot product terms per iteration, reducing loop iterations by 8x. Wrap this in outer loops, optionally unrolling or tiling (e.g., 8x8 blocks) for better cache usage.

## Using Rust’s SIMD Tools

- **`std::arch`**: Provides raw intrinsics, requiring `unsafe` and manual architecture targeting (e.g., `#[cfg(target_arch = "x86_64")]`). Enable AVX with `--features avx2` in `Cargo.toml`.
- **Crates like `packed_simd`**: Offers portable abstractions:
  ```rust
  use packed_simd::f32x8;

  fn matrix_mult_simd_portable(a: &[[f32; 8]; 8], b: &[[f32; 8]; 8], c: &mut [[f32; 8]; 8]) {
      for i in 0..8 {
          for j in (0..8).step_by(8) {
              let mut sum = f32x8::splat(0.0);
              for k in 0..8 {
                  let a_vec = f32x8::splat(a[i][k]);
                  let b_vec = f32x8::from_slice_unaligned(&b[k][j..]);
                  let prod = a_vec * b_vec;
                  sum = sum + prod;
              }
              sum.write_unaligned(&mut c[i][j..]);
          }
      }
  }
  ```
  This hides architecture specifics, falling back to scalar code if SIMD isn’t available.

## Challenges Across Architectures

- **Instruction Set Availability**: AVX is x86_64-specific; ARM uses NEON (128-bit, 4x `f32`). AVX code fails on ARM or older x86 CPUs without AVX.
  - **Solution**: Use `#[cfg]` for conditional compilation or runtime feature detection with `std::is_x86_feature_detected!("avx2")`. Fallback to scalar or narrower SIMD (e.g., SSE2).
- **Alignment**: AVX prefers 32-byte aligned memory. Unaligned loads (`_mm256_loadu_ps`) are slower.
  - **Solution**: Align data with `#[repr(align(32))]` or pad arrays, trading memory for speed.
- **Portability**: Hardcoding AVX locks you to x86_64. `packed_simd` helps, but performance varies (e.g., NEON’s 4-wide vs. AVX’s 8-wide).
  - **Solution**: Abstract with crates or write multiple implementations, selecting at runtime.
- **Correctness**: Floating-point associativity changes with SIMD summation order, risking numerical drift.
  - **Solution**: Test against scalar results with known inputs; use `fsum` or pairwise reduction for precision.

<div class="svg-container" style="margin:2rem 0;">
<svg class="lo3b-fig" viewBox="0 0 800 370" width="100%" style="height:auto;max-width:780px;display:block;margin:0 auto;" role="img" aria-label="Dispatch tree selecting an AVX2, SSE2, NEON or scalar matrix multiply kernel at runtime, all validated against the same scalar baseline">
<style>
.lo3b-fig{--bg:#f8fafc;--box:#ffffff;--tx:#1e293b;--mut:#64748b;--ln:#cbd5e1;--ac:#FF6B00}
:root.dark .lo3b-fig,[data-theme="dark"] .lo3b-fig{--bg:#0f172a;--box:#1e293b;--tx:#f8fafc;--mut:#94a3b8;--ln:#475569}
.lo3b-fig .box{fill:var(--box);stroke:var(--ln);stroke-width:1.5}
.lo3b-fig .fin{fill:var(--box);stroke:var(--ac);stroke-width:2.5}
.lo3b-fig .hd{fill:var(--bg);stroke:var(--mut);stroke-width:1.5}
.lo3b-fig .ti{fill:var(--tx);font:700 13px ui-sans-serif,system-ui,sans-serif;text-anchor:middle}
.lo3b-fig .tx{fill:var(--tx);font:600 12px ui-sans-serif,system-ui,sans-serif;text-anchor:middle}
.lo3b-fig .mut{fill:var(--mut);font:500 11px ui-sans-serif,system-ui,sans-serif;text-anchor:middle}
.lo3b-fig .ac{fill:var(--ac);font:700 13px ui-sans-serif,system-ui,sans-serif;text-anchor:middle}
.lo3b-fig line,.lo3b-fig path.ln{stroke:var(--ln);stroke-width:1.5;fill:none}
</style>
<defs>
<marker id="lo3b-arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0,0 L10,5 L0,10 z" fill="var(--ln)"/>
</marker>
<marker id="lo3b-arrowac" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M0,0 L10,5 L0,10 z" fill="var(--ac)"/>
</marker>
</defs>
<!-- single call site -->
<rect x="280" y="18" width="240" height="46" rx="6" class="box"/>
<text x="400" y="40" class="ti">matrix_mult(a, b, c)</text>
<text x="400" y="56" class="mut">one call site in your code</text>
<line x1="400" y1="64" x2="400" y2="86" marker-end="url(#lo3b-arrow)"/>
<!-- dispatch -->
<rect x="240" y="88" width="320" height="46" rx="6" class="hd"/>
<text x="400" y="110" class="tx">Pick a kernel: cfg + feature detect</text>
<text x="400" y="126" class="mut">first match wins, checked once at startup</text>
<!-- fan-out bus -->
<line x1="400" y1="134" x2="400" y2="158"/>
<line x1="108" y1="158" x2="692" y2="158"/>
<line x1="108" y1="158" x2="108" y2="178" marker-end="url(#lo3b-arrowac)"/>
<line x1="303" y1="158" x2="303" y2="178" marker-end="url(#lo3b-arrow)"/>
<line x1="497" y1="158" x2="497" y2="178" marker-end="url(#lo3b-arrow)"/>
<line x1="692" y1="158" x2="692" y2="178" marker-end="url(#lo3b-arrow)"/>
<!-- four kernels -->
<rect x="20" y="180" width="176" height="70" rx="6" class="fin"/>
<text x="108" y="203" class="ac">AVX2 — 8 × f32</text>
<text x="108" y="222" class="mut">_mm256_mul_ps</text>
<text x="108" y="240" class="mut">is_x86_feature_detected!</text>
<rect x="215" y="180" width="176" height="70" rx="6" class="box"/>
<text x="303" y="203" class="tx">SSE2 — 4 × f32</text>
<text x="303" y="222" class="mut">_mm_mul_ps</text>
<text x="303" y="240" class="mut">older x86_64 CPUs</text>
<rect x="409" y="180" width="176" height="70" rx="6" class="box"/>
<text x="497" y="203" class="tx">NEON — 4 × f32</text>
<text x="497" y="222" class="mut">vmulq_f32</text>
<text x="497" y="240" class="mut">cfg(target_arch aarch64)</text>
<rect x="604" y="180" width="176" height="70" rx="6" class="box"/>
<text x="692" y="203" class="tx">Scalar fallback</text>
<text x="692" y="222" class="mut">c[i][j] += a * b</text>
<text x="692" y="240" class="mut">compiles everywhere</text>
<!-- merge bus back to one baseline check -->
<path class="ln" d="M108,250 L108,274 L400,274"/>
<path class="ln" d="M303,250 L303,274"/>
<path class="ln" d="M497,250 L497,274"/>
<path class="ln" d="M692,250 L692,274 L400,274"/>
<line x1="400" y1="274" x2="400" y2="296" marker-end="url(#lo3b-arrow)"/>
<rect x="170" y="298" width="460" height="58" rx="6" class="box"/>
<text x="400" y="322" class="tx">Every kernel must match the scalar baseline</text>
<text x="400" y="342" class="mut">SIMD reorders the sum — assert results within tolerance</text>
</svg>
</div>

## Benchmarking against the scalar version
- **Benchmarking**: Use `criterion` to compare SIMD vs. scalar:
  ```rust
  use criterion::{black_box, Criterion};
  fn bench(c: &mut Criterion) {
      let a = [[1.0_f32; 8]; 8];
      let b = [[2.0_f32; 8]; 8];
      let mut c = [[0.0_f32; 8]; 8];
      c.bench_function("simd", |b| b.iter(|| unsafe { matrix_mult_simd(black_box(&a), black_box(&b), black_box(&mut c)) }));
      c.bench_function("scalar", |b| b.iter(|| matrix_mult_scalar(black_box(&a), black_box(&b), black_box(&mut c))));
  }
  ```
  Expect SIMD to be 4-8x faster for large matrices.
- **Profiling**: Use `perf` on Linux (`perf stat -e cycles,instructions`) to confirm instruction reduction (e.g., 8x fewer multiplications).
- **Assembly Inspection**: Run `cargo rustc --release -- --emit asm` or use `godbolt.org` to verify tight loops with SIMD instructions (e.g., `vmulps`, `vaddps`).

## Practical Example Outcome

For a 1024x1024 matrix, AVX could reduce runtime from seconds to milliseconds on a modern CPU, assuming good data locality. Profiling should show an 8x instruction reduction in the inner loop, with benchmarks confirming significant speedups.

## Conclusion
For maximum throughput on a known architecture (e.g., x86_64 with AVX), use `std::arch` to vectorize matrix multiplication’s inner loop, tiling for cache efficiency. For portability, switch to `packed_simd`, accepting some overhead. Address challenges like alignment and feature detection with conditional compilation and runtime checks, ensuring both speed and correctness in a production system.
