MultiModal SigLIP Activation

A custom PyTorch activation module that applies the SigLIP function—defined as
𝑥
× 𝜎
( 𝑥
) x×σ(x)—to both image and text inputs simultaneously. Designed for multi-modal neural network architectures, this module supports image tensors with shape
[ 𝑏
𝑎
𝑡
𝑐
ℎ _
𝑠
𝑖
𝑧
𝑒
, 𝑐
ℎ 𝑎
𝑛
𝑛
𝑒
𝑙
𝑠
, ℎ
𝑒
𝑖
𝑔
ℎ 𝑡
, 𝑤
𝑖
𝑑
𝑡
ℎ ]
[batch_size,channels,height,width] and text tensors with shape
[ 𝑏
𝑎
𝑡
𝑐
ℎ _
𝑠
𝑖
𝑧
𝑒
, 𝑠
𝑒
𝑞
𝑢
𝑒
𝑛
𝑐
𝑒
_ 𝑙
𝑒
𝑛
𝑔
𝑡
ℎ ,
𝑒
𝑚
𝑏
𝑒
𝑑
𝑑
𝑖
𝑛
𝑔
_ 𝑑
𝑖
𝑚
] [batch_size,sequence_length,embedding_dim]. It provides a seamless way to integrate non-linear activations across different data modalities in a deep learning pipeline.