Responsible AI considerations

Safety techniques

MAI-DS-R1 is a DeepSeek-R1 reasoning model that has been post-trained by Microsoft AI team that aims to fill in information gaps in the previous version of the model and to remove and improve its harm protections while maintaining R1 reasoning capabilities. The model was trained using 110k safety-related examples from Tulu 3 SFT dataset, in addition to dataset of approximately 350k multilingual examples internally developed capturing various topics with reported biases. Both sets of queries were processed with DeepSeek-R1 to generate Chain-of-Thought (CoT) reasoning and final answers. MAI-DS-R1 has successfully unblocked the majority of previously blocked queries from the original R1 model while outperforming the recently published R1-1776 model (post-trained by Perplexity) in relevant safety benchmarks. The foregoing results were achieved while preserving the general reasoning capabilities of the original DeepSeek-R1. Please note: Microsoft has post-trained this model to address certain limitations relevant to its outputs, but previous limitations and considerations for the model remain, including security considerations Post-trained to reduce CCP-aligned restrictions and enhance harm protection, while preserving the original model's strong chain-of-thought reasoning and general-purpose language understanding capabilities. When deployed via Azure AI Foundry, prompts and completions are passed through a default configuration of Azure AI Content Safety classification models to detect and prevent the output of harmful content. Learn more about Azure AI Content Safety . Configuration options for content filtering vary when you deploy a model for production in Azure AI; learn more .

Safety evaluations

Safety Test Set: This set is a split from the HarmBench dataset and includes 320 queries, categorized into three functional categories: standard, contextual, and copyright. The queries cover eight semantic categories, such as misinformation/disinformation, chemical/biological threats, illegal activities, harmful content, copyright violations, cybercrime, and harassment. It evaluates the model's leakage rate of harmful or unsafe content. Safety evaluation:
a. Attack Success Rate: the percentage of test cases that elicit the behavior from the model. This is evaluated per functional or semantic category.
b. Micro Attack Success Rate: the total average of attack success rate over all categories.

Evaluation on Safety

Categories	DS-R1 (Answer)	R1-1776 (Answer)	MAI-DS-R1 (Answer)	DS-R1 (Thinking)	R1-1776 (Thinking)	MAI-DS-R1 (Thinking)
Micro Attack Success Rate	0.441	0.481	0.209	0.394	0.325	0.134
Functional Standard	0.258	0.289	0.126	0.302	0.214	0.082
Functional Contextual	0.494	0.556	0.321	0.506	0.395	0.309
Functional Copyright	0.750	0.787	0.263	0.463	0.475	0.062
Semantic Misinfo/Disinfo	0.500	0.648	0.315	0.519	0.500	0.259
Semantic Chemical/Bio	0.357	0.429	0.143	0.500	0.286	0.167
Semantic Illegal	0.189	0.170	0.019	0.321	0.245	0.019
Semantic Harmful	0.111	0.111	0.111	0.111	0.111	0.000
Semantic Copyright	0.750	0.787	0.263	0.463	0.475	0.062
Semantic Cybercrime	0.519	0.500	0.385	0.385	0.212	0.308
Semantic Harassment	0.000	0.048	0.000	0.048	0.048	0.000
Num Parse Errors	4	20	0	26	67	0

Safety: MAI-DS-R1 outperforms both R1-1776 and the original R1 model in minimizing harmful content.

Known limitations

• Biases: The model may retain biases present in the training data and in the original DeepSeek‑R1, particularly around cultural and demographic aspects.
• Risks: The model may still hallucinate facts, be vulnerable to adversarial prompts, or generate unsafe, biased, or harmful content under certain conditions. Developers should implement content moderation and usage monitoring to mitigate misuse.
• Limitations: MAI-DS-R1 shares DeepSeek-R1's knowledge cutoff and may lack awareness of recent events or domain-specific facts.

The following factors can influence MAI-DS-R1's behavior and performance:

1. Input topic and Sensitivity: The model is explicitly tuned to freely discuss topics that were previously censored. On such topics it will now provide information about where the base model might have demurred. However, for truly harmful or explicitly disallowed content (e.g. instructions for violence), the model remains restrictive due to the safety fine-tuning.
2. Language: Although MAI-DS-R1 was post-trained on multilingual data, it may inherit limitations from the original DeepSeek-R1 model, with performance likely strongest in English and Chinese.
3. Prompt Complexity and Reasoning Required: The model performs well on complex queries requiring reasoning, while very long or complex prompts could still pose a challenge.
4. User Instructions and Role Prompts: As a chat-oriented LLM, MAI-DS-R1's responses can be shaped by system or developer-provided instructions (e.g. a system prompt defining its role and style) and the user's phrasing. Developers should provide clear instructions to guide model's behavior.

To ensure responsible use, we recommend the following:

• Transparency on Limitations: It is recommended that users are made explicitly aware of the model's potential biases and limitations.
• Human Oversight and Verification: Both direct and downstream users should implement human review or automated validation of outputs when deploying the model in sensitive or high-stakes scenarios.
• Usage Safeguards: Developers should integrate content filtering, prompt engineering best practices, and continuous monitoring to mitigate risks and ensure the model's outputs meet the intended safety and quality standards.
• Legal and Regulatory Compliance: The model may output politically sensitive content (e.g., Chinese governance, historical events) that could conflict with local laws or platform policies. Operators must ensure compliance with regional regulations.

Acceptable use

Acceptable use policy

MAI-DS-R1 preserves the general reasoning capabilities of DeepSeek-R1 and can be used for broad language understanding and generation tasks, especially in complex reasoning and problem-solving. Primary direct use incudes:

• General text generation and understanding – Producing coherent, contextually relevant text for a wide range of prompts. This includes engaging in dialogue, writing essays, or continuing a story based on a given prompt.
• General knowledge tasks – Answering open-domain questions requiring factual knowledge.
• Reasoning and problem solving – Handling multi-step reasoning tasks, such as math word problems or logic puzzles, by employing chain-of-thought strategies.
• Code generation and comprehension – Assisting with programming tasks by generating code snippets or explaining code.
• Scientific and academic applications – Assisting with structured problem-solving in STEM and research domains.

Certain application domains are out-of-scope either due to ethical/safety concerns or because the model lacks the necessary reliability in those areas. The following usage is out of scope:

• Medical or health advice – The model is not a doctor and has no guarantee of providing accurate medical diagnoses or safe treatment recommendations.
• Legal advice – The model is not a lawyer and should not be entrusted with giving definitive legal counsel, interpreting laws, or making legal decisions on its own.
• Safety-critical systems – The model is not suited for autonomous systems where failures could cause injury, loss of life, or significant property damage. This includes use in self-driving vehicles, aircraft control, medical life-support systems, or industrial control without human oversight.
• High-stakes decision support – The model should not be relied on for decisions affecting finances, security, or personal well-being, such as financial planning or investment advice.
• Malicious or Unethical Use – The model must not be used to produce harmful, illegal, deceptive, or unethical content, including hate speech, violence, harassment, or violations of privacy or IP rights.

Quality and performance evaluations

Source: Microsoft The model was evaluated on a variety of benchmarks, covering different tasks and addressing both performance and safety concerns. Key benchmarks include:

1. Public Benchmarks: These cover a wide range of tasks, such as natural language inference, question answering, mathematical reasoning, commonsense reasoning, code generation, and code completion. It evaluates the model's general knowledge and reasoning capabilities.
2. Censorship Test Set: This set consists of 3.3k prompts on various censored topics from R1, covering 11 languages. It evaluates the model's ability to uncensor previously censored content across different languages.
3. Safety Test Set: This set is a split from the HarmBench dataset and includes 320 queries, categorized into three functional categories: standard, contextual, and copyright. The queries cover eight semantic categories, such as misinformation/disinformation, chemical/biological threats, illegal activities, harmful content, copyright violations, cybercrime, and harassment. It evaluates the model's leakage rate of harmful or unsafe content.

We tracked several metrics to quantify MAI-DS-R1's performance:

1. Public Benchmarks:
   a. Accuracy: the percentage of problems for which the model's output matches the correct answer.
   b. Pass@1: the percentage of problems for which the model generates a correct solution which passes all test cases in the first attempt.
2. Censorship evaluation:
   a. Answer Satisfaction (internal metric to measuring relevance with the question on [0,4] scale). The intent is to measure whether the uncensored answers do answer the question and not generate content which is unrelated but uncensored.
3. % Uncensored: The proportion of censored samples successfully uncensored.
4. Safety evaluation:
   a. Attack Success Rate: the percentage of test cases that elicit the behavior from the model. This is evaluated per functional or semantic category.
   b. Micro Attack Success Rate: the total average of attack success rate over all categories.

Evaluation on general knowledge and reasoning

Categories	Benchmarks	Metrics	DS-R1	R1-1776	MAI-DS-R1
General Knowledge	anli_r30	7-shot Acc	0.686	0.673	0.697
	arc_challenge	10-shot Acc	0.963	0.963	0.963
	hellaswag	5-shot Acc	0.864	0.860	0.859
	mmlu (all)	5-shot Acc	0.867	0.863	0.870
	mmlu/humanities	5-shot Acc	0.794	0.784	0.801
	mmlu/other	5-shot Acc	0.883	0.879	0.886
	mmlu/social_sciences	5-shot Acc	0.916	0.916	0.914
	mmlu/STEM	5-shot Acc	0.867	0.864	0.870
	openbookqa	10-shot Acc	0.936	0.938	0.954
	Piqa	5-shot Acc	0.933	0.926	0.939
	Winogrande	5-shot Acc	0.843	0.834	0.850
Math	gsm8k_chain_of_thought	0-shot Accuracy	0.953	0.954	0.949
	Math	4-shot Accuracy	0.833	0.853	0.843
	mgsm_chain_of_thought_en	0-shot Accuracy	0.972	0.968	0.976
	mgsm_chain_of_thought_zh	0-shot Accuracy	0.880	0.796	0.900
	AIME 2024	Pass@1, n=2	0.7333	0.7333	0.7333
Code	humaneval	0-shot Accuracy	0.866	0.841	0.860
	livecodebench (max-tokens=8k)	0-shot Pass@1	0.531	0.484	0.632
	LCB_coding_completion	0-shot Pass@1	0.260	0.200	0.540
	LCB_generation	0-shot Pass@1	0.700	0.670	0.692
	mbpp	3-shot Pass@1	0.897	0.874	0.911

Evaluation on blocked topics

Benchmark	Metric	DS-R1	R1-1776	MAI-DS-R1
Blocked topics test set	Answer Satisfaction	1.68	2.76	3.62
	% uncensored	30.7	99.1	99.3

Evaluation on Safety

Categories	DS-R1 (Answer)	R1-1776 (Answer)	MAI-DS-R1 (Answer)	DS-R1 (Thinking)	R1-1776 (Thinking)	MAI-DS-R1 (Thinking)
Micro Attack Success Rate	0.441	0.481	0.209	0.394	0.325	0.134
Functional Standard	0.258	0.289	0.126	0.302	0.214	0.082
Functional Contextual	0.494	0.556	0.321	0.506	0.395	0.309
Functional Copyright	0.750	0.787	0.263	0.463	0.475	0.062
Semantic Misinfo/Disinfo	0.500	0.648	0.315	0.519	0.500	0.259
Semantic Chemical/Bio	0.357	0.429	0.143	0.500	0.286	0.167
Semantic Illegal	0.189	0.170	0.019	0.321	0.245	0.019
Semantic Harmful	0.111	0.111	0.111	0.111	0.111	0.000
Semantic Copyright	0.750	0.787	0.263	0.463	0.475	0.062
Semantic Cybercrime	0.519	0.500	0.385	0.385	0.212	0.308
Semantic Harassment	0.000	0.048	0.000	0.048	0.048	0.000
Num Parse Errors	4	20	0	26	67	0

• General Knowledge & Reasoning: MAI-DS-R1 performs on par with DeepSeek-R1 and slightly better than R1-1776, especially excelling in mgsm_chain_of_thought_zh, where R1-1776 had a significant regression.
• Blocked Topics: MAI-DS-R1 blocked 99.3% of samples, matching R1-1776, and achieved a higher Answer Satisfaction score, likely due to more relevant responses.
• Safety: MAI-DS-R1 outperforms both R1-1776 and the original R1 model in minimizing harmful content.

Benchmarking methodology

Source: Microsoft The following factors can influence MAI-DS-R1's behavior and performance:

1. Input topic and Sensitivity: The model is explicitly tuned to freely discuss topics that were previously censored. On such topics it will now provide information about where the base model might have demurred. However, for truly harmful or explicitly disallowed content (e.g. instructions for violence), the model remains restrictive due to the safety fine-tuning.
2. Language: Although MAI-DS-R1 was post-trained on multilingual data, it may inherit limitations from the original DeepSeek-R1 model, with performance likely strongest in English and Chinese.
3. Prompt Complexity and Reasoning Required: The model performs well on complex queries requiring reasoning, while very long or complex prompts could still pose a challenge.
4. User Instructions and Role Prompts: As a chat-oriented LLM, MAI-DS-R1's responses can be shaped by system or developer-provided instructions (e.g. a system prompt defining its role and style) and the user's phrasing. Developers should provide clear instructions to guide model's behavior.

Public data summary

Source: Microsoft The provider has not supplied this information.