BitNet b1.58 2B4T - Scaling Native 1-bit LLM

Model Variants

• microsoft/bitnet-b1.58-2B-4T (This repository): Contains the packed 1.58-bit weights optimized for efficient inference. Use this for deployment.
• microsoft/bitnet-b1.58-2B-4T-bf16 : Contains the master weights in BF16 format. Use this only for training or fine-tuning purposes.
• microsoft/bitnet-b1.58-2B-4T-gguf : Contains the model weights in GGUF format, compatible with the bitnet.cpp library for CPU inference.

Model Details

• Architecture: Transformer-based, modified with BitLinear layers (BitNet framework).
  • Uses Rotary Position Embeddings (RoPE).
  • Uses squared ReLU (ReLU²) activation in FFN layers.
  • Employs subln normalization.
  • No bias terms in linear or normalization layers.
• Quantization: Native 1.58-bit weights and 8-bit activations (W1.58A8).
  • Weights are quantized to ternary values {-1, 0, +1} using absmean quantization during the forward pass.
  • Activations are quantized to 8-bit integers using absmax quantization (per-token).
  • Crucially, the model was trained from scratch with this quantization scheme, not post-training quantized.
• Parameters: ~2 Billion
• Training Tokens: 4 Trillion
• Context Length: Maximum sequence length of 4096 tokens.
  • Recommendation: For optimal performance on tasks requiring very long contexts (beyond the pre-training length or for specialized long-reasoning tasks), we recommend performing intermediate long-sequence adaptation/training before the final fine-tuning stage.
• Training Stages:
  1. Pre-training: Large-scale training on public text/code and synthetic math data using a two-stage learning rate and weight decay schedule.
  2. Supervised Fine-tuning (SFT): Fine-tuned on instruction-following and conversational datasets using sum loss aggregation and specific hyperparameter tuning.
  3. Direct Preference Optimization (DPO): Aligned with human preferences using preference pairs.
• Tokenizer: LLaMA 3 Tokenizer (vocab size: 128,256).

How to Use (with transformers)

Please do NOT expect performance efficiency gains (in terms of speed, latency, or energy consumption) when using this model with the standard transformers library, even with the required fork. The current execution paths within transformers do not contain the specialized, highly optimized computational kernels required to leverage the advantages of the BitNet architecture. Running the model via transformers will likely result in inference speeds and energy usage comparable to, or potentially worse than, standard full-precision models within this framework on both CPU and GPU. While you might observe reduced memory usage due to the quantized weights, the primary computational efficiency benefits are not accessible through this standard transformers usage path. For achieving the efficiency benefits demonstrated in the technical paper, you MUST use the dedicated C++ implementation: bitnet.cpp .

Requirements

pip install git+https://github.com/huggingface/transformers.git@096f25ae1f501a084d8ff2dcaf25fbc2bd60eba4

Example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "microsoft/bitnet-b1.58-2B-4T"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16
)

# Apply the chat template
messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "How are you?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
chat_input = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate response
chat_outputs = model.generate(**chat_input, max_new_tokens=50)
response = tokenizer.decode(chat_outputs[0][chat_input['input_ids'].shape[-1]:], skip_special_tokens=True) # Decode only the response part
print("\nAssistant Response:", response)

How to Use (with bitnet.cpp)

Evaluation

License

Bias, Risks, and Limitations

Disclaimer

• Original Model Card

Send Request

curl <AZUREML_ENDPOINT_URL> \
    -X POST \
    -d '{"inputs":"System: You are a helpful assistant that responds like a pirate<|eot_id|>User: What is Deep Learning?<|eot_id|>Assistant: ","parameters":{"max_new_tokens":10,"seed":42}}' \
    -H "Authorization: Bearer <AZUREML_TOKEN>" \
    -H "Content-Type: application/json"

Supported Parameters

• max_length: The maximum length the generated tokens can have. Corresponds to the length of the input prompt + max_new_tokens. Its effect is overridden by max_new_tokens, if also set. Defaults to 20.
• max_new_tokens: The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt.
• min_length: The minimum length of the sequence to be generated. Corresponds to the length of the input prompt + min_new_tokens. Its effect is overridden by min_new_tokens, if also set. Defaults to 0.
• min_new_tokens: The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt.
• early_stopping: Controls the stopping condition for beam-based methods, like beam-search. It accepts the following values: True, where the generation stops as soon as there are num_beams complete candidates; False, where an heuristic is applied and the generation stops when is it very unlikely to find better candidates; "never", where the beam search procedure only stops when there cannot be better candidates (canonical beam search algorithm). Defaults to False.
• max_time: The maximum amount of time you allow the computation to run for in seconds. generation will still finish the current pass after allocated time has been passed.
• stop_strings: A string or a list of strings that should terminate generation if the model outputs them.

• do_sample: Whether or not to use sampling; use greedy decoding otherwise. Defaults to False.
• num_beams: Number of beams for beam search, 1 means no beam search. Defaults to 1.
• num_beam_groups: Number of groups to divide num_beams into in order to ensure diversity among different groups of beams. Defaults to 1.
• penalty_alpha: The values balance the model confidence and the degeneration penalty in contrastive search decoding.

• temperature: The value used to modulate the next token probabilities. Defaults to 1.0.
• top_k: The number of highest probability vocabulary tokens to keep for top-k-filtering. Defaults to 50.
• top_p: If set to a float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. Defaults to 1.0.
• min_p: Minimum token probability, scaled by the probability of the most likely token. Typical values are between 0.01 and 0.2.
• typical_p: Controls local typicality, keeping the smallest set of the most locally typical tokens whose probabilities add up to typical_p or higher. Defaults to 1.0.
• epsilon_cutoff: If set to a float strictly between 0 and 1, only tokens with a conditional probability greater than epsilon_cutoff will be sampled. Defaults to 0.0.
• eta_cutoff: Hybrid of locally typical sampling and epsilon sampling. If set to a float strictly between 0 and 1, a token is only considered if it meets the eta_cutoff threshold. Defaults to 0.0.
• diversity_penalty: Subtracted from a beam’s score if it generates a token already generated by another group at a particular time. Only effective if group beam search is enabled. Defaults to 0.0.
• repetition_penalty: The parameter for repetition penalty. 1.0 means no penalty. Defaults to 1.0.
• encoder_repetition_penalty: Exponential penalty on sequences not in the original input. 1.0 means no penalty. Defaults to 1.0.
• length_penalty: Exponential penalty to the sequence length used with beam-based generation. >0.0 promotes longer sequences, <0.0 encourages shorter sequences. Defaults to 1.0.
• no_repeat_ngram_size: If set to an integer > 0, all ngrams of that size can only occur once. Defaults to 0.
• bad_words_ids: List of lists of token ids that are not allowed to be generated.
• force_words_ids: List of token ids that must be generated. Can be a simple list or a disjunctive constraint for multiple forms.
• renormalize_logits: Whether to renormalize the logits after applying all logits processors. Defaults to False.
• constraints: Custom constraints to ensure the output contains certain tokens as defined by Constraint objects.
• forced_bos_token_id: The id of the token to force as the first generated token after the decoder start token. Useful for multilingual models.
• forced_eos_token_id: The id of the token to force as the last generated token when max_length is reached. Can be a list for multiple end-of-sequence tokens.
• remove_invalid_values: Whether to remove possible NaN and inf outputs to prevent the generation method from crashing. May slow down generation.
• exponential_decay_length_penalty: Tuple (start_index, decay_factor) adding an exponentially increasing length penalty after a certain number of tokens.
• suppress_tokens: List of tokens that will be suppressed during generation.
• begin_suppress_tokens: List of tokens that will be suppressed at the beginning of generation.
• forced_decoder_ids: List of pairs indicating a mapping from generation indices to token indices that will be forced before sampling.
• sequence_bias: Dictionary mapping a sequence of tokens to its bias term. Positive biases increase, negative biases decrease the odds of selection.
• token_healing: Heal tail tokens of prompts by replacing them with their appropriate extensions. Defaults to False.
• guidance_scale: The guidance scale for classifier-free guidance (CFG). CFG is enabled by setting guidance_scale > 1.
• low_memory: Switch to sequential beam search and sequential top-k for contrastive search to reduce peak memory.
• watermarking_config: Arguments used to watermark the model outputs by adding a small bias to a randomly selected set of tokens. If passed as a dict, it will be converted internally.

Example payload

{
  "inputs": "System: You are a helpful assistant that responds like a pirate<|eot_id|>User: What is Deep Learning?<|eot_id|>Assistant: ",
  "parameters": {
    "max_new_tokens": 10,
    "do_sample": true,
    "temperature": 0.8
  }
}