NextCoder-14B

NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits (ICML'2025)

Introduction

• Significantly improvements in code editing, NextCoder-32B has performing on par with GPT-4o on complex benchmarks like Aider-Polyglot with performance increment of 44% from their base model.
• No loss of generalizibility, due to our new finetuning method SeleKT
• Long-context Support up to 32K tokens.

• Type: Causal Language Models
• Training Stage: Post-training with SeleKT
• Architecture: transformers with RoPE, SwiGLU, RMSNorm, and Attention QKV bias
• Number of Parameters: 14.7B
• Number of Paramaters (Non-Embedding): 13.1B
• Number of Layers: 48
• Number of Attention Heads (GQA): 40 for Q and 8 for KV

Requirements

KeyError: 'qwen2'

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "microsoft/NextCoder-14B"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = """
Fix the following function that divides two numbers to handle all the edge cases:

def divide(a, b)
  returm a/b
"""
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=1024
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Evaluation and Performance

Responsible AI Use

Citation

@inproceedings{aggarwal2025nextcoder,
author = {Aggarwal, Tushar and Singh, Swayam and Awasthi, Abhijeet and Kanade, Aditya and Natarajan, Nagarajan},
title = {NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits},
booktitle = {International Conference on Machine Learning},
year = {2025},
url = {https://www.microsoft.com/en-us/research/publication/nextcoder-robust-adaptation-of-code-lms-to-diverse-code-edits/},
}

• Example Notebook
• Original Model Card
• Text Generation Inference Documentation

Send Request

curl <AZUREML_ENDPOINT_URL> \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H "Authorization: Bearer <AZUREML_TOKEN>" \
    -H "Content-Type: application/json"

Supported Parameters

• inputs (string): Input prompt.
• parameters (object):
  • best_of (integer): Generate best_of sequences and return the one if the highest token logprobs.
  • decoder_input_details (boolean): Whether to return decoder input token logprobs and ids.
  • details (boolean): Whether to return generation details.
  • do_sample (boolean): Activate logits sampling.
  • frequency_penalty (float): The parameter for frequency penalty. 1.0 means no penalty Penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim.
  • grammar (object): One of the following
    • #1 (object):
      • type (enum): Possible values: json.
      • value (string): A string that represents a JSON Schema. JSON Schema is a declarative language that allows to annotate JSON documents with types and descriptions.
    • #2 (object):
      • type (enum): Possible values: regex.
      • value (string): The regular expression.
    • #3 (object):
      • type (enum): Possible values: json_schema.
      • value (object):
        • name (string): Optional name identifier for the schema
        • schema (object): The actual JSON schema definition
  • max_new_tokens (integer): Maximum number of tokens to generate.
  • repetition_penalty (float): The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details.
  • return_full_text (boolean): Whether to prepend the prompt to the generated text
  • seed (integer): Random sampling seed.
  • stop (string[]): Stop generating tokens if a member of stop is generated.
  • temperature (float): The value used to module the logits distribution.
  • top_k (integer): The number of highest probability vocabulary tokens to keep for top-k-filtering.
  • top_n_tokens (integer): The number of highest probability vocabulary tokens to keep for top-n-filtering.
  • top_p (float): Top-p value for nucleus sampling.
  • truncate (integer): Truncate inputs tokens to the given size.
  • typical_p (float): Typical Decoding mass See Typical Decoding for Natural Language Generation for more information.
  • watermark (boolean): Watermarking with A Watermark for Large Language Models.
  • stream (boolean): Whether to stream the output tokens or not. Defaults to false.

Example payload

{
  "inputs": "What is Deep Learning?",
  "parameters": {
    "do_sample": true,
    "top_p": 0.95,
    "temperature": 0.2,
    "top_k": 50,
    "max_new_tokens": 256,
    "repetition_penalty": 1.03,
    "stop": ["\nUser:", "<|endoftext|>", "</s>"]
  }
}

OpenAI Chat Completion API compatibility