← ML Research Wiki / 2506.17155

Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity

(2025)

Paper Information
arXiv ID
2506.17155
Contents

Abstract

In this paper, we investigate the use of small datasets in the context of offline reinforcement learning (RL).While many common offline RL benchmarks employ datasets with over a million data points, many offline RL applications rely on considerably smaller datasets.We show that offline RL algorithms can overfit on small datasets, resulting in poor performance.To address this challenge, we introduce "Sparse-Reg": a regularization technique based on sparsity to mitigate overfitting in offline reinforcement learning, enabling effective learning in limited data settings and outperforming state-of-the-art baselines in continuous control.

Summary

This paper introduces Sparse-Reg, a regularization technique based on sparsity, aimed at improving sample complexity in offline reinforcement learning (RL) when limited training datasets are available. The authors explore the challenges posed by small datasets in offline RL, showing that existing algorithms often overfit, leading to suboptimal performance. Sparse-Reg mitigates overfitting by inducing sparsity in neural network parameters, balancing essential pattern capture with model generalization. Through experiments in continuous control tasks using the MuJoCo environment within the D4RL benchmark, the authors demonstrate that Sparse-Reg outperforms state-of-the-art methods under various sample sizes, highlighting its effectiveness in enhancing offline RL performance amidst limited data settings. The paper makes a significant contribution to making offline RL more applicable in real-world scenarios where high-quality data collection can be expensive or impractical.

Methods

This paper employs the following methods:

  • Sparse-Reg
  • Regularization
  • Sparsity

Models Used

  • IQL
  • AWAC
  • TD3+BC

Datasets

The following datasets were used in this research:

  • D4RL

Evaluation Metrics

  • Mean-Square-Error (MSE)
  • Normalized score

Results

  • Sparse-Reg significantly improves performance across various offline RL algorithms
  • Sparse-Reg outperforms state-of-the-art algorithms in limited data scenarios

Technical Requirements

  • Number of GPUs: None specified
  • GPU Type: None specified
  • Compute Requirements: 1 million gradient updates (not specifically stated the training duration)

Papers Using Similar Methods

External Resources

References: 49