Skip to main content

swiglu_ffn

Function swiglu_ffn 

Source
pub fn swiglu_ffn(
    x: &Tensor,
    fc1_w: &Tensor,
    fc2_w: &Tensor,
    gate_w: &Tensor,
) -> Result<Tensor>
Expand description

SwiGLU feed-forward: w2(silu(w1(x)) * w3(x)), all projections bias-free.