tomaarsen/mpnet-base-gooaq-mnrl-hn-100k

MPNet base trained on GooAQ triplets

This is a sentence-transformers model finetuned from microsoft/mpnet-base on the gooaq-hard-negatives dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: microsoft/mpnet-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • gooaq-hard-negatives
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'MPNetModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/mpnet-base-gooaq-mnrl-hn-100k")
# Run inference
queries = [
    "what is iq for intellectual disability?",
]
documents = [
    'Diagnosing Intellectual Disability A full scale IQ score of around 70 to 75 indicates a significant limitation in intellectual functioning.',
    'An intellectual disability describes below-average IQ and a lack of skills needed for daily living. This condition used to be called “mental retardation.” A learning disability refers to weaknesses in certain academic skills. Reading, writing and math are the main ones.',
    "IQ, short for intelligence quotient, is a measure of a person's reasoning ability. In short, it is supposed to gauge how well someone can use information and logic to answer questions or make predictions.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7053, 0.6088, 0.6604]])

Evaluation

Metrics

Information Retrieval

• Dataset: gooaq-dev
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.802
cosine_accuracy@3	0.931
cosine_accuracy@5	0.959
cosine_accuracy@10	0.97
cosine_precision@1	0.802
cosine_precision@3	0.3103
cosine_precision@5	0.1918
cosine_precision@10	0.097
cosine_recall@1	0.802
cosine_recall@3	0.931
cosine_recall@5	0.959
cosine_recall@10	0.97
cosine_ndcg@10	0.8934
cosine_mrr@10	0.8678
cosine_map@100	0.868

Training Details

Training Dataset

gooaq-hard-negatives

• Dataset: gooaq-hard-negatives at 87594a1
• Size: 99,000 training samples
• Columns: question, answer, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 1000 samples:

  	question	answer	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  details	min: 8 tokens mean: 11.8 tokens max: 24 tokens	min: 13 tokens mean: 59.53 tokens max: 149 tokens	min: 13 tokens mean: 58.38 tokens max: 135 tokens	min: 13 tokens mean: 58.34 tokens max: 128 tokens	min: 13 tokens mean: 59.23 tokens max: 131 tokens	min: 13 tokens mean: 59.02 tokens max: 122 tokens	min: 14 tokens mean: 59.23 tokens max: 183 tokens

• Samples:

  question	answer	negative_1	negative_2	negative_3	negative_4	negative_5
  will acrylic paint stick to polyester?	I recommend acrylic because polyester is a manmade fabric and acrylic is a water-based paint. Oil paints contain solvents. ... You could also use acrylic paint markers with good results. If you are painting on fabric you might want to spray it with some kind of protective coat after it's set for a few day.	Acrylic resin (but not acrylic emulsion, which is the basis of acrylic paint) is a thermoplastic, which means it is one of a group of plastics which can be heated and manipulated repeatedly, whereas polyester resin and epoxy are thermosetting plastics, which use heat or a catalyst to solidify into a solid mass that won ...	Acrylic paint is ideal on foam core because you do not need to use solvents like with oil paints. Acrylic it fast-drying, durable and adheres to most surfaces. It also has an even sheen and does not need a base coat.	The short answer is no, house paint does not generally perform as well as artists' acrylics in terms of durability, lightfastness and appearance. ... Vehicles and binders used in house paints vary from one manufacturer to another, but usually the term “latex” or “acrylic latex” is used to describe the polymer base.	Acrylics are synthetic paints designed to mix and blend just like oils. Acrylic paint dries extremely fast, to the extent it will dry on your palette if you do not use the paint quick enough. It is less forgiving in this manner and you must be decisive when you paint.	You can paint acrylic paint over enamel paint, but only with proper surface preparation. If you just apply a coat of water-based paint, it won't adhere properly on the surface and the colour will fade quickly. That's because the sheen of the enamel paint interferes with the acrylic paint.
  what is the difference between zone 6 and zone 6a?	Subset Zone Temperatures Each gardening zone is divided into two subsets; zone 6 subsets are 6a and 6b. ... Zone 6: This zone has a minimum average of temperatures of -10° to 0°F. Zone 6a: This subzone has a minimum average temperature of -10° to -5° F. Zone 6b: This subzone has a minimum average temperature of -5° to 0°F.	The zone 2 feature allows this receiver to send a second source signal to a separate audio system in another location. For example, you can watch a DVD in the main zone, while others can listen to a CD in zone 2 at the same time. ... An additional audio system (for example, an amplifier) to output the sound in zone 2.	The only difference is the size - Zone 1 measures 43cm, the 2 is 60 - and the further north you live the more likely you are to need the larger.	Zone C may have ponding and local drainage problems that don't warrant a detailed study or designation as base floodplain. Zone X is the area determined to be outside the 500-year flood and protected by levee from 100- year flood.	The USDA Hardiness Zone Map divides North America into 11 separate planting zones; each growing zone is 10°F warmer (or colder) in an average winter than the adjacent zone. If you see a hardiness zone in a gardening catalog or plant description, chances are it refers to this USDA map.	Forward Lookup Zones—Forward Lookup Zones allow the DNS Server to resolve queries where the client sends a name to the DNS Server to request the IP address of the requested host.
  what is the working principle of electroscope?	The principle of an electroscope is based on the atomic structure of elements, the internal structure of metal elements, charge induction, and the idea that like charges repel and opposite charges attract. All elements are composed of electrons, protons, and neutrons - with the electrons surrounding the nucleus.	Detecting EM Waves. To detect the electric fields, use a conducting rod. The fields cause charges (generally electrons) to accelerate back and forth on the rod, creating a potential difference that oscillates at the frequency of the EM wave and with an amplitude proportional to the amplitude of the wave.	['Head/Body houses the optical parts in the upper part of the microscope.', 'Base of the microscope supports the microscope and houses the illuminator.', 'Arm connects to the base and supports the microscope head.']	Handheld and benchtop equipment such as electronic meters are available to test parameters including pH, turbidity (the amount of particulate suspended in a liquid), and electrical conductivity.	Electronystagmography (ENG or electrooculography) is used to evaluate people with vertigo (a false sense of spinning or motion that can cause dizziness) and certain other disorders that affect hearing and vision. Electrodes are placed at locations above and below the eye to record electrical activity.	What is the working principle of magnetic flow meter? Faraday's Law The operation of a electromagnetic flowmeter is based upon Faraday's Law, which states that the voltage induced across any conductor as it moves at right angles through a magnetic field is proportional to the velocity of that conductor.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

gooaq-hard-negatives

• Dataset: gooaq-hard-negatives at 87594a1
• Size: 1,000 evaluation samples
• Columns: question, answer, negative_1, negative_2, negative_3, negative_4, and negative_5
• Approximate statistics based on the first 1000 samples:

  	question	answer	negative_1	negative_2	negative_3	negative_4	negative_5
  type	string	string	string	string	string	string	string
  details	min: 8 tokens mean: 11.56 tokens max: 22 tokens	min: 15 tokens mean: 60.49 tokens max: 183 tokens	min: 15 tokens mean: 58.66 tokens max: 165 tokens	min: 15 tokens mean: 59.37 tokens max: 159 tokens	min: 14 tokens mean: 59.7 tokens max: 159 tokens	min: 14 tokens mean: 58.75 tokens max: 141 tokens	min: 14 tokens mean: 59.91 tokens max: 141 tokens

• Samples:

  question	answer	negative_1	negative_2	negative_3	negative_4	negative_5
  is aiims and jipmer cancelled for 2020?	According to the above-mentioned clause, the AIIMs and JIPMER medical entrances have been scrapped and the admissions will be conducted under NEET. ... The National Testing Agency will be conducting the NEET 2020 examination on May 3, 2020, in the pen and paper mode.	Okeechobee Music and Arts Festival has been canceled for 2019. ... After months of radio silence from the Okeechobee Music and Arts Festival, organizers announced on Thursday afternoon that the 2019 gathering has been canceled, but say it will return in March 2020.	Update: 2020 Minneapolis Summer X Games Cancelled April 24th, 2020: Due to the ongoing COVID-19 concerns, ESPN has announced X Games Minneapolis 2020, originally scheduled for July 17-19, has been cancelled. Please visit the Official X Games website for more information.	New episodes will air at least until September 2020. But that doesn't mean the series is canceled. Chandler Massey, who portrays Will Horton on "Days," took to Twitter following the news to assure fans that the daytime soap opera isn't going anywhere. "We're not being cancelled," he tweeted Tuesday.	New Delhi: Confirming that All India Institute of Medical Sciences (AIIMS) entrance will be merged with National Eligibility cum Entrance Test (NEET) from 2020-21 onwards, the Union Minister of State of Health and Family welfare stated that the move will ensure a common standard of medical education in the country and ...	iZombie was renewed for a third season on March 11, 2016. iZombie was renewed for a fourth season on May 10, 2017, which premiered on February 26, 2018. iZombie was renewed for a fifth and final season on May 2018, which premiered on May 2, 2019.
  vs_installershell.exe has an invalid certificate vs 2019?	Probably due to a pending Windows update not being installed. ... Please go to windows update and make sure all pending windows update are installed and reboot the computer, then go to C:\Program Files (x86)\Microsoft Visual Studio\Installer then run vs_installershell.exe as administrator.	This issue may occur if one or more of the following condition are true: The computer's date and time settings are incorrect. The certificate has expired or is not yet valid.	This issue may occur if one or more of the following condition are true: The computer's date and time settings are incorrect. The certificate has expired or is not yet valid.	['Click Start > Run.', 'Type in MMC and click OK.', 'Go into the File Tab (or Console) > select Add/Remove Snap-in.', 'Click Add > Click Certificates and click Add, then click close.', 'Click OK (in the Add/Remove Snap-in window)', 'Select Computer Account.', 'Select Local Computer.', 'Click the + to expand the Certificates console tree.']	['Click Start > Run.', 'Type in MMC and click OK.', 'Go into the File Tab (or Console) > select Add/Remove Snap-in.', 'Click Add > Click Certificates and click Add, then click close.', 'Click OK (in the Add/Remove Snap-in window)', 'Select Computer Account.', 'Select Local Computer.', 'Click the + to expand the Certificates console tree.']	['Use certmgr. msc command inside Run dialog. Press Win+R keys -> type certmgr. ... ', 'Use Windows 10 to open the certificate. You can also simply double-click your . crt file in order for Windows to open it. ... ', 'Open . crt file inside your favorite browser. Right-click on the .']
  do you get your period on depo?	Many women stop having their menstrual periods after about 6-12 months of getting Depo-Provera® injections. This is no reason to worry. It's not a medical problem and it does not mean you are pregnant. It just means that your ovaries are resting, and the lining of your uterus does not grow.	Many women stop having their menstrual periods after about 6-12 months of getting Depo-Provera® injections. This is no reason to worry. It's not a medical problem and it does not mean you are pregnant. It just means that your ovaries are resting, and the lining of your uterus does not grow.	You will get your menstrual period back within 6-18 months after you stop taking the Depo-Provera® injections.	You will get your menstrual period back within 6-18 months after you stop taking the Depo-Provera® injections.	You will get your menstrual period back within 6-18 months after you stop taking the Depo-Provera® injections.	It's not a medical problem and it does not mean you are pregnant. It just means that your ovaries are resting, and the lining of your uterus does not grow. So there is no menstrual bleeding. You will get your menstrual period back within 6-18 months after you stop taking the Depo-Provera® injections.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	gooaq-dev_cosine_ndcg@10
-1	-1	-	-	0.2701
0.0003	1	4.7646	-	-
0.0501	155	2.4632	-	-
0.1002	310	1.0555	0.9443	0.8279
0.1503	465	0.9589	-	-
0.2004	620	0.8793	0.7198	0.8696
0.2505	775	0.8265	-	-
0.3006	930	0.8259	0.6847	0.8674
0.3507	1085	0.7991	-	-
0.4008	1240	0.7714	0.6501	0.8822
0.4509	1395	0.7613	-	-
0.5010	1550	0.7571	0.6217	0.8866
0.5511	1705	0.712	-	-
0.6012	1860	0.7219	0.6084	0.8847
0.6513	2015	0.7194	-	-
0.7014	2170	0.7229	0.5962	0.8855
0.7515	2325	0.6977	-	-
0.8016	2480	0.7178	0.5888	0.8874
0.8516	2635	0.7046	-	-
0.9017	2790	0.6953	0.5894	0.8908
0.9518	2945	0.6932	-	-
-1	-1	-	-	0.8934

Framework Versions

• Python: 3.11.14
• Sentence Transformers: 5.3.0.dev0
• Transformers: 4.57.3
• PyTorch: 2.9.1+cu130
• Accelerate: 1.12.0
• Datasets: 4.4.2
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}