Skip to content

Datasets for training machine learning algorithms

Important

The data can only be used for non-commercial, scientific or educational purposes.

Please pay close attention to the terms of use and the licensing information included for each dataset. Occasionally, citation to the original work is required.

By downloading or accessing the data available centrally, you agree to the terms and conditions published by the copyright holder of the data. The responsibility is on the users to check the terms of use of the data and to make sure their use case is aligned with what is permitted by the copyright owner. In case explicit terms of use of the data is missing from the provider's side, the NonCommercial-ShareAlike (NC-SA) attribute of the Creative Commons 3.0 license applies.

Datasets are provided through modules and by loading a module environment variables are populated pointing to the dataset. It is important that you load modules to use the datasets and don't use the path directly as:

  1. The path to datasets are subject to change, by using the environment variables from the modules your code will continue to run.
  2. By loading the modules we get an idea of what datasets are in use.
  3. Notice about changes to specific datasets will be in connection to loading the module.

If you think a required, publicly available dataset is missing please contact us through the support form in SUPR. We will evaluate if it is something that should be added.

Dataloading pipeline

The dataloading pipeline is a crucial component for computational efficiency in supervised training. Some recommendations for clusters are (in order of usefulness):

  1. Reading directly from Mimer is usually fast enough, that way you'll save time that would have been used to transfer to $TMPDIR or /dev/shm.
  2. Many small files are slow prefer to bunch the data into only a few larger files (WebDataset, HDF5, zip, Arrow (e.g. datasets package), TFRecord).
  3. Use compression wisely:
    1. Jpeg, png, gz, or other files that don't benefit from (further) compression shouldn't be compressed (zip -0).
    2. Very compressible formats (csv, mmcif, txt, ...) can potentially benefit from compression.
  4. When shuffling data zip, is preferred over tar and if you can, shuffle before processing data points.
  5. Using a profiler is very useful to get an idea what works or not and what part of the pipeline it is that takes time (e.g. see TensorBoard).
  6. Don't spend more time optimising the dataloading pipeline than you'll save from the optimisation.
  7. Batch size, number of preprocessing workers and prefetching strategy can have significant impact as a rule of thumb:
    1. If loading data is very fast, use the 0th (current thread) as worker.
    2. Otherwise you can spend some effort in setting up prefetching and loading multiple workers.

Mounting compressed datasets (ZIP)

Some datasets are distributed as .zip archives for efficient storage and transfer.
On Vera, you can access them directly without extracting, either:

  • on the host using the built-in fuse-archive command, or
  • inside Apptainer containers using the --fusemount option.

See more information in the filesystem documentation.

Available datasets

The following datasets are currently available:

AlphaFold3 data

  • AlphaFold3-data/3.0

These are the datasets needed to run inference with AlphaFold3, note that we are not allowed to distribute the model weights and they thus are not included.

The following databases have been: (1) mirrored by Google DeepMind; and (2) in part, included with the inference code package for testing purposes, and are available with reference to the following:

CO3Dv2 data

  • CO3Dv2-data/20231130-zip: Provided as a number of uncompressed zipfiles.

Common Objects in 3D version 2. Licensed under CC BY-NC 4.0. If you use the dataset please use the following citation:

@inproceedings{reizenstein21co3d,
    Author = {Reizenstein, Jeremy and Shapovalov, Roman and Henzler, Philipp and Sbordone, Luca and Labatut, Patrick and Novotny, David},
    Booktitle = {International Conference on Computer Vision},
    Title = {Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction},
    Year = {2021},
}

ImageNet 1k data

Gated dataset

Access to this dataset is gated and not provided openly to all users. To get access join the related license group in SUPR: https://supr.naiss.se/group/215/member/subscribe/?.

Please cite the main ImageNet paper (Deng et al. CVPR 2009) if you use ImageNet data, in addition to specific follow-up papers: 

@inproceedings{deng2009imagenet,
  title={Imagenet: A large-scale hierarchical image database},
  author={Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li},
  booktitle={2009 IEEE conference on computer vision and pattern recognition},
  pages={248--255},
  year={2009},
  organization={IEEE}
}

ImageNet-1k-data/20250917-hf

Dataset based on ILSVRC/imagenet-1k on HuggingFace hub. To use:

import os
import datasets

dataset = datasets.load_from_disk(os.environ['IMAGENET_1K_DATA_DIR'])

nuScenes data

Gated dataset

Access to this dataset is gated and not provided openly to all users. To get access join the related license group in SUPR: https://supr.naiss.se/group/220/member/subscribe/?.

Access to this dataset is covered by a license group in SUPR.

  • nuScenes-data/1.0-map-1.3-zip: Main data as uncompressed zip files with map extension.
  • nuScenes-lidarseg-data/1.0-zip: Lidarseg extension as uncompressed zip files.
  • nuScenes-panoptic-data/1.0-zip: Panoptic extension as uncompressed zip files.

Non-commercial use only, for detailed terms-of-use see: https://www.nuscenes.org/terms-of-use

  • Overview: https://www.nuscenes.org/overview
  • Trainval (700+150 scenes) is packaged into 10 different archives that each contain 85 scenes.
  • Test (150 scenes) is used for challenges and does not come with object annotations.
  • Mini (10 scenes) is a subset of trainval used to explore the data without having to download the entire dataset.
  • The meta data is provided separately and includes the annotations, ego vehicle poses, calibration, maps and log information.
  • Citation: please use the following citation when referencing nuScenes:
@article{nuscenes2019,
 title={nuScenes: A multimodal dataset for autonomous driving},
  author={Holger Caesar and Varun Bankiti and Alex H. Lang and Sourabh Vora and
          Venice Erin Liong and Qiang Xu and Anush Krishnan and Yu Pan and
          Giancarlo Baldan and Oscar Beijbom},
  journal={arXiv preprint arXiv:1903.11027},
  year={2019}
}

Deprecated datasets

The following datasets are deprecated and are planned to be removed. If you are using any of these and want them to remain, contact support.

  • Argoverse2-data/deprecated
  • BDD100K-data/deprecated
  • BIOSCAN-5M-data/20260109-hf-deprecated
  • BigEarthNet-S2-data/deprecated
  • CryoPPP-data/deprecated
  • E-GMD-data/1.0.0-deprecated
  • KITTI-data/deprecated
  • LSUN-data/deprecated
  • Lyft-Level5-data/deprecated
  • MAN-TruckScenes-data/deprecated
  • MPI-Sintel-data/deprecated
  • MegaDepth-data/deprecated
  • MegaScenes-data/deprecated
  • Microsoft-COCO-data/deprecated
  • Object365-data/deprecated
  • OpenImages-V6-data/deprecated
  • OpenSLR-SLR12-data/deprecated
  • PandaSet-data/deprecated
  • PhysioNet-challenge-2018-data/1.0.0-deprecated
  • Places365-data/deprecated
  • Scene-Flow-data/mix-deprecated
  • Slakh2100-redux-data/deprecated
  • Waymo-Open-Dataset-data/1.2.0-deprecated
  • YouTube-VOS-data/2019-deprecated
  • Zenseact-Open-Dataset-data/deprecated

Available LLM model weights

These models provide the path to the model weights and can be used with the transformers package or vLLM directly. Each of these modules provides environment variables:

  • HF_NAME: Name of the model e.g. openai/gpt-oss-120b
  • HF_MODEL: Path to the modle weights

as well as possibly a few more specific ones related to the model name. To see exactly what a module does use module show <specify module here>, e.g. module show openai-gpt-oss-120b-model/20250826-hf.

Available model modules:

  • openai-gpt-oss-120b-model/20250826-hf

Deprecated LLM model weights

The following models are deprecated and are planned to be removed. If you are using any of these and want them to remain, contact support.

  • HuggingFaceTB-SmolLM2-135M-Instruct-model/12fd25f-deprecated
  • HuggingFaceTB-SmolLM2-135M-model/93efa2f-deprecated
  • HuggingFaceTB-SmolLM3-3B-Base-model/d78a42f-deprecated
  • HuggingFaceTB-SmolLM3-3B-model/a07cc9a-deprecated
  • QuantTrio-GLM-4.5-Air-GPTQ-Int4-Int8Mix-model/1c98fb7-deprecated
  • Qwen-Qwen2-0.5B-model/91d2aff-deprecated
  • Qwen-Qwen2.5-0.5B-Instruct-model/7ae5576-deprecated
  • Qwen-Qwen2.5-VL-3B-Instruct-model/6628554-deprecated
  • Qwen-Qwen3-0.6B-Base-model/da87bfb-deprecated
  • Qwen-Qwen3-Coder-30B-A3B-Instruct-model/573fa39-deprecated
  • Qwen-Qwen3-VL-235B-A22B-Instruct-model/a85c315-deprecated
  • Qwen-Qwen3.5-122B-A10B-GPTQ-Int4-model/5b9f005-deprecated
  • Qwen-Qwen3.5-27B-FP8-model/2e1b213-deprecated
  • Qwen-Qwen3.5-27B-GPTQ-Int4-model/507bda6-deprecated
  • Qwen-Qwen3.5-27B-model/b7ca741-deprecated
  • Qwen-Qwen3.5-397B-A17B-GPTQ-Int4-model/b54fd48-deprecated
  • Qwen-Qwen3.5-9B-model/c202236-deprecated
  • Qwen-Qwen3.6-27B-model/6a9e13b-deprecated
  • Qwen-Qwen3.6-35B-A3B-model/995ad96-deprecated
  • RedHatAI-Qwen3.6-35B-A3B-NVFP4-model/e850c69-deprecated
  • RedHatAI-gemma-4-26B-A4B-it-NVFP4-model/95d1e59-deprecated
  • RedHatAI-gemma-4-31B-it-NVFP4-model/c490598-deprecated
  • Snowflake-snowflake-arctic-embed-l-v2.0-model/ac6544c-deprecated
  • cyankiwi-Qwen3.6-27B-AWQ-INT4-model/c9b937c-deprecated
  • deepseek-ai-DeepSeek-V3.2-Exp-model/194c67e-deprecated
  • llava-hf-llava-v1.6-vicuna-13b-hf-model/2c9a8b5-deprecated
  • llava-hf-llava-v1.6-vicuna-7b-hf-model/c9f1252-deprecated
  • lmsys-vicuna-7b-v1.3-model/236eeea-deprecated
  • mistralai-Devstral-Small-2-24B-Instruct-2512-model/1da725e-deprecated
  • moonshotai-Kimi-K2-Thinking-model/6126819-deprecated
  • neuralmagic-Llama-3.2-11B-Vision-Instruct-quantized.w4a16-model/7f66874-deprecated
  • neuralmagic-Llama-3.3-70B-Instruct-quantized.w8a8-model/dc36722-deprecated
  • neuralmagic-Meta-Llama-3.1-405B-Instruct-quantized.w4a16-model/9db6306-deprecated
  • neuralmagic-Meta-Llama-3.1-70B-Instruct-quantized.w4a16-model/b5b7bd4-deprecated
  • neuralmagic-Meta-Llama-3.1-8B-Instruct-quantized.w4a16-model/a7c0994-deprecated
  • nomic-ai-nomic-embed-text-v1.5-model/e5cf08a-deprecated
  • nomic-ai-nomic-embed-text-v2-moe-model/1066b65-deprecated
  • openai-community-gpt2-model/607a30d-deprecated
  • openai-gpt-oss-120b-model/b5c939d-deprecated
  • openai-gpt-oss-120b-model/bc75b44-deprecated
  • openai-gpt-oss-20b-model/cbf31f6-deprecated
  • openai-gpt-oss-20b-model/6cee5e8-deprecated
  • sentence-transformers-all-MiniLM-L6-v2-model/c9745ed-deprecated
  • unsloth-Llama-3.2-11B-Vision-Instruct-model/677b0c1-deprecated
  • unsloth-Llama-3.2-1B-Instruct-bnb-4bit-model/117a0a1-deprecated
  • unsloth-Llama-3.2-1B-Instruct-model/d2b9e36-deprecated
  • unsloth-Llama-3.2-3B-Instruct-bnb-4bit-model/1372a85-deprecated
  • unsloth-Llama-3.2-3B-Instruct-model/c22b3d3-deprecated
  • unsloth-Llama-4-Maverick-17B-128E-Instruct-model/4d0b9b8-deprecated
  • unsloth-Llama-4-Scout-17B-16E-Instruct-model/afd8e49-deprecated
  • unsloth-Llama-4-Scout-17B-16E-Instruct-model/8de9338-deprecated
  • unsloth-Meta-Llama-3.1-405B-Instruct-bnb-4bit-model/4efb87c-deprecated
  • unsloth-Meta-Llama-3.1-70B-Instruct-bnb-4bit-model/d401984-deprecated
  • unsloth-Meta-Llama-3.1-8B-Instruct-bnb-4bit-model/e78288d-deprecated
  • unsloth-Meta-Llama-3.1-8B-Instruct-bnb-4bit-model/3da6bbe-deprecated
  • unsloth-Meta-Llama-3.1-8B-Instruct-bnb-4bit-model/5b0dd30-deprecated
  • unsloth-Phi-3-medium-4k-instruct-bnb-4bit-model/6f7b2d9-deprecated
  • unsloth-Qwen3.6-27B-NVFP4-model/6db1783-deprecated
  • unsloth-SmolLM2-135M-Instruct-GGUF-model/*-deprecated
  • unsloth-codegemma-7b-model/93a0a77-deprecated
  • unsloth-gemma-7b-it-bnb-4bit-model/5d05931-deprecated
  • unsloth-llama-3-70b-Instruct-bnb-4bit-model/df8fab1-deprecated
  • unsloth-llama-3-8b-Instruct-bnb-4bit-model/e114164-deprecated
  • unsloth-mistral-7b-instruct-v0.3-bnb-4bit-model/12a648d-deprecated
  • zai-org-GLM-4.5-Air-model/a24ceef-deprecated
  • zai-org-GLM-4.6-model/be72194-deprecated