xHC: Expanded Hyper-Connections – Scale Residual Streams Wider, Push Model Intelligence Further

A paper published on r/LocalLLaMA proposes expanded hyper-connections (xHC) that widen residual streams in neural networks, aiming to improve model intelligence and scaling properties.

This architectural modification directly addresses the bottleneck in residual stream capacity that limits depth and width scaling in transformers. If validated, xHC could enable training larger models without proportional increases in parameter count or compute, potentially lowering the cost-per-utility for LLM and vision model builders. The approach may also reduce the need for complex normalization or gating mechanisms currently used to stabilize deep networks.

Operationally, builders should evaluate whether xHC can replace or augment existing residual connections in their training pipelines, particularly for models beyond 70B parameters. If the method generalizes, it could simplify scaling recipes by removing the need for auxiliary loss tuning or layer-specific learning rate schedules, making deeper architectures easier to deploy. Current infrastructure for data-parallel training remains unchanged, but forward-pass memory requirements may shift depending on implementation.