Model Overview

Description

Nemotron-3-8B-SteerLM is an 8 billion parameter generative language model based on the NVIDIA 8B GPT base model. It has been customized using the SteerLM Method developed by NVIDIA to allow for user control of model outputs during inference
Key capabilities enabled by SteerLM:

• Dynamic steering of responses by specifying desired attributes like quality, helpfulness, and toxicity at inference time.
• Simplified training compared to RLHF techniques like fine-tuning and bootstrapping.

Nemotron-3-8B-SteerLM is part of Nemotron-3, is a family of enterprise ready decoder-only generative text models compatible with NeMo Framework . NVIDIA NeMo is an end-to-end, cloud-native framework to build, customize, and deploy generative AI models anywhere. It includes training and inferencing frameworks, guardrailing toolkits, data curation tools, and pretrained models, offering enterprises an easy, cost-effective, and fast way to adopt generative AI.

Model Architecture

Architecture Type: Transformer Network Architecture: Generative Pre-Trained Transformer (GPT-3) The SteerLM method involves the following key steps:

• Train an attribute prediction model on human annotated data to evaluate response quality.
• Use this model to annotate diverse datasets and enrich training data.
• Perform conditioned fine-tuning to align responses with specified combinations of attributes.
• (Optionally) Bootstrap training through model sampling and further fine-tuning.

SteerLM-8B applies this technique on top of the open-source NVIDIA GPT model architecture. It was pretrained on internet-scale data and then customized using OASST , HH-RLHF , Light, a subset of permissive licensed OpenPlatypus , and some internally collected SFT data.

Input

Input Type	Description
prompts	List[str] - List of input prompts
max_output_token	int - Optional: Maximum number of generated tokens
top_k	int - Optional: Limits model to consider the top K tokens by probability at each output step
top_p	float - Optional: Limits model to consider the top tokens within a certain probability mass p
temperature	float - Optional: Sharpens (when < 1) or flattens (when > 1) the probability distribution of output tokens

Prompt Format:

Single Turn

Multi-Turn or Few-shot/In-context prompting

<extra_id_0>System A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. <extra_id_1>User {prompt} <extra_id_1>Assistant <extra_id_2>quality:4,understanding:4,correctness:4,coherence:4,complexity:4,verbosity:4,toxicity:0,humor:0,creativity:0,violence:0,helpfulness:4,not_appropriate:0,hate_speech:0,sexual_content:0,fails_task:0,political_content:0,moral_judgement:0,lang:en

<extra_id_0>System A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. <extra_id_1>User {prompt 1} <extra_id_1>Assistant <extra_id_2>quality:4,understanding:4,correctness:4,coherence:4,complexity:4,verbosity:4,toxicity:0,humor:0,creativity:0,violence:0,helpfulness:4,not_appropriate:0,hate_speech:0,sexual_content:0,fails_task:0,political_content:0,moral_judgement:0,lang:en {response 1} <extra_id_1>User {prompt 2} <extra_id_1>Assistant <extra_id_2>quality:4,understanding:4,correctness:4,coherence:4,complexity:4,verbosity:4,toxicity:0,humor:0,creativity:0,violence:0,helpfulness:4,not_appropriate:0,hate_speech:0,sexual_content:0,fails_task:0,political_content:0,moral_judgement:0,lang:en

Output

Output	Type	Description

Outputs | List[str] | List of output strings, with one string for each input prompt

Samples

Inference samples

Inference type	Python sample (Notebook)	CLI with YAML
Real time	text-generation-online-endpoint-dolly.ipynb	text-generation-online-endpoint-dolly.sh
Batch	text-generation-batch-endpoint.ipynb	coming soon

Software Integration

Runtime Engine(s): NVIDIA AI Enterprise Toolkit: NeMo Framework Supported Hardware Architecture Compatibility: (Currently being tested)

• H100
• A100 80GB, A100 40GB

Model Version(s)

Nemotron-3-8B-Chat-SteerLM

Dataset

NVIDIA models are trained on a diverse set of public and proprietary datasets. This model was trained on a dataset containing 3.5 Trillion tokens of text. The dataset contains 53 different human languages and 37 programming languages. NVIDIA is committed to the responsible development of large language models and conducts reviews of all datasets included in training. Evaluation MT-Bench

Category	Score
Total	5.47
Writing	7.05
Roleplay	7.02
Extraction	4.9
Stem	7.35
Humanities	9.35
Reasoning	4.15
Math	2.3
Coding	1.65

Intended use

• The 8B-Chat-SteerLM model is for users who want to customize a model’s response during inference.
• Ethical use: Technology can have a profound impact on people and the world, and NVIDIA is committed to enabling trust and transparency in AI development. NVIDIA encourages users to adopt principles of AI ethics and trustworthiness to guide your business decisions by following the guidelines in the NVIDIA NeMo Foundational Models Community License Agreement.

Limitations

• The model was trained on the data that contains toxic language and societal biases originally crawled from the Internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts.
• The Model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.