PaCA: Partial Connection Adaptation

PaCA (ICLR 2025) is a PEFT method that targets training speed and activation memory. It fine-tunes r randomly selected input columns (“partial connections”) of each targeted pretrained weight. The only trainable parameters are the updates of these columns; unlike LoRA, PaCA adds no extra low-rank matrices.

Methods such as LoRA run a small low-rank branch next to every pretrained layer. The branch needs few FLOPs, but it is executed sequentially with the pretrained layer and it keeps its own copy of the input activation for the backward pass. PaCA avoids both:

• One GEMM per layer. The adapted columns are written into the pretrained weight right before its matmul and restored right after it, so forward and backward each run a single matmul, exactly like the frozen base layer.
• Partial activations. The gradient of the adapted columns only depends on the matching r input channels, so only those are saved for the backward pass instead of the full input.

In terms of the resulting weights, a PaCA adapter is equivalent to a LoRA adapter of rank r whose A matrix is a fixed column selector: W' = W + delta @ S, where S selects the r columns. It can therefore be converted to a LoRA adapter of rank r without loss via convert_to_lora(), e.g. to serve it with a LoRA inference stack.

When to use PaCA:

• You want LoRA-level accuracy but faster fine-tuning steps and lower activation memory, e.g. to fit longer sequences or larger batches on the same GPU.
• You want an adapter that merges exactly into the base weights (the update is a plain column update).

Current constraints:

• Only nn.Linear layers (including quantized linear layers) are supported.
• PaCA runs the matmul itself instead of calling the base layer’s forward, so forward hooks registered on the base layer are not called (a warning is emitted). Register hooks on the PaCA layer instead.
• Mixed adapter batches (adapter_names argument) are not supported.
• Training with sharded weights (FSDP, DeepSpeed ZeRO-3) has not been tested yet.
• torch.compile is supported: under compilation the column swap, matmul and restore run as opaque custom ops (torch.ops.peft.paca_linear_forward/_backward), so the compiler does not copy the base weights. This requires torch.library.custom_op (PyTorch >= 2.4); otherwise the eager implementation is traced.
• paca_alpha / r scales the column update and therefore acts as a multiplier of the effective learning rate.
• For inference, merge the adapter (merge_and_unload()); the unmerged forward still swaps the columns in and out on every call.

Usage

from transformers import AutoModelForCausalLM
from peft import PacaConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B")
config = PacaConfig(r=16, paca_alpha=16, target_modules="all-linear", random_seed=0)
model = get_peft_model(model, config)
model.print_trainable_parameters()

By default, every targeted layer fine-tunes r randomly selected columns; in the paper, this performed on par with selecting the columns by weight norm or by accumulated gradients. To choose the columns yourself, for example to try another selection strategy or to use a different number of columns per layer, pass paca_indices_pattern. Its keys are layer names or regular expressions, as in LoRA’s rank_pattern:

config = PacaConfig(
    r=16,
    paca_alpha=16,
    target_modules=["q_proj", "v_proj"],
    paca_indices_pattern={"layers.0.self_attn.q_proj": [0, 2, 4, 6], "layers.1.self_attn.v_proj": list(range(32))},
)

Layers that match a key fine-tune the given columns, all other targeted layers select r random columns. The scaling is paca_alpha / r with the r of the config in every layer, so paca_alpha means the same for all layers.

The abstract from the paper is:

Prior parameter-efficient fine-tuning (PEFT) algorithms reduce memory usage and computational costs of fine-tuning large neural network models by training only a few additional adapter parameters, rather than the entire model. However, the reduction in computational costs due to PEFT does not necessarily translate to a reduction in training time; although the computational costs of the adapter layers are much smaller than the pretrained layers, it is well known that those two types of layers are processed sequentially on GPUs, resulting in significant latency overhead. LoRA and its variants avoid this latency overhead by merging the low-rank adapter matrices with the pretrained weights during inference. However, those layers cannot be merged during training since the pretrained weights must remain frozen while the low-rank adapter matrices are updated continuously over the course of training. Furthermore, LoRA and its variants do not reduce activation memory, as the first low-rank adapter matrix still requires the input activations to the pretrained weights to compute weight gradients. To mitigate this issue, we propose Partial Connection Adaptation (PaCA), which fine-tunes randomly selected partial connections within the pretrained weights instead of introducing adapter layers in the model. PaCA not only enhances training speed by eliminating the time overhead due to the sequential processing of the adapter and pretrained layers but also reduces activation memory since only partial activations, rather than full activations, need to be stored for gradient computation. Compared to LoRA, PaCA reduces training time by 22% and total memory usage by 16%, while maintaining comparable accuracy across various fine-tuning scenarios, such as fine-tuning on the MMLU dataset and instruction tuning on the Oasst1 dataset. PaCA can also be combined with quantization, enabling the fine-tuning of large models such as LLaMA3.1-70B. In addition, PaCA enables training with 23% longer sequence and improves throughput by 16% on both NVIDIA A100 GPU and INTEL Gaudi2 HPU compared to LoRA.

Benchmark overview

API

PacaConfig

PacaModel

>>> from transformers import AutoModelForCausalLM
>>> from peft import PacaConfig, get_peft_model

>>> base_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m")
>>> config = PacaConfig(r=16, paca_alpha=16, target_modules="all-linear")
>>> model = get_peft_model(base_model, config)