DeepSeek-R1 distillation 으로 Reasoning 모델 만들기¶

• Reasoning 데이터 살펴보기
• SFT 로 distillation
• Math-500 으로 평가

데이터와 아이디어는 SimpileScaling S1 을 참조하였습니다. 비슷한 사례를 실습하는 내용입니다.

In [ ]:

#%%capture
#!pip install unsloth "xformers==0.0.28.post2"
# Also get the latest nightly Unsloth!
#!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install unsloth

#%%capture #!pip install unsloth "xformers==0.0.28.post2" # Also get the latest nightly Unsloth! #!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git" !pip install unsloth

In [ ]:

!nvidia-smi

!nvidia-smi

Model Load & Setup¶

실습은 Unsloth 를 이용하여 LoRA 로 진행합니다.

• Reasoning Capability 가 Full Tuning 이 필요한 일인가에 대해서는 생각해보면, "그렇지 않다" 가 제 의견입니다.
  • 왜 이렇게 생각하는지 실습과 결과를 보고 다시 한번 고찰해보겠습니다.

시작 모델은 Qwen2.5-7B-Instruct 로 진행하겠습니다.

• 위 의문과 연관되는 내용입니다. 7B 정도의 모델로도 충분한가? 이 부분도 실험 결과를 봐야 알 수 있을 것 같습니다.

max_seq_length 를 주의해야합니다!!

In [ ]:

from unsloth import FastLanguageModel
import torch
max_seq_length = 32768 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Qwen/Qwen2.5-7B-Instruct", # or choose "unsloth/Llama-3.2-1B-Instruct"
    max_seq_length = max_seq_length,
    dtype=torch.bfloat16,
    token = "",
)

from unsloth import FastLanguageModel import torch max_seq_length = 32768 # Choose any! We auto support RoPE Scaling internally! dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ model, tokenizer = FastLanguageModel.from_pretrained( model_name = "Qwen/Qwen2.5-7B-Instruct", # or choose "unsloth/Llama-3.2-1B-Instruct" max_seq_length = max_seq_length, dtype=torch.bfloat16, token = "", )

In [ ]:

model = FastLanguageModel.get_peft_model(
    model,
    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 2503,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)

model = FastLanguageModel.get_peft_model( model, r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # Supports any, but = 0 is optimized bias = "none", # Supports any, but = "none" is optimized # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes! use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context random_state = 2503, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ )

In [4]:

Data Prep¶

데이터는 simplescaling/s1K-1.1 에서 다운 받겠습니다.

Reasoning 을 위한 양질의 여러 문제 (수학 & 크로스워드 퍼즐) 이 1k 개 준비 되어있고, 여러 방식으로 풀려진 예시가 있습니다.

그 중에서 DeepSeek-R1 의 풀이를 볼 것입니다.

(참고) 한국어 버전 번역본들도 있습니다, 24.03 기준 아직까지는 좋은 데이터셋을 찾니 못 했습니다.

In [5]:

from datasets import load_dataset

dataset = load_dataset("simplescaling/s1K-1.1", split = "train")

# 한국어 버전 번역본!
#dataset = load_dataset("exp-models/s1K-1.1-Korean", split = "train")

from datasets import load_dataset dataset = load_dataset("simplescaling/s1K-1.1", split = "train") # 한국어 버전 번역본! #dataset = load_dataset("exp-models/s1K-1.1-Korean", split = "train")

In [ ]:

dataset[0]

dataset[0]

In [ ]:

# select "deepseek_grade" == yes
dataset = dataset.filter(lambda x: x["deepseek_grade"] == "Yes")
len(dataset)

# select "deepseek_grade" == yes dataset = dataset.filter(lambda x: x["deepseek_grade"] == "Yes") len(dataset)

대화식으로 변형합니다, Qwen 의 chat template 을 적용하기 전에 <think> </think> 가 나오도록 합니다.

여기서 <think> 는 스페셜 토큰이 아니라 그냥 글자일 뿐입니다 (!!)

In [8]:

def example_to_messages(example):
    """
    단일 example (예: {'solution': '...', 'question': '...', ...})을
    Qwen 스타일 대화 목록(list of dict)으로 변환하는 함수.
    """
    system_content = "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."

    # question을 user 메시지로, solution을 assistant 메시지로 사용
    user_content = example.get("question", "")
    think_content = example.get("deepseek_thinking_trajectory", "")
    attempt_content = example.get("deepseek_attempt", "")

    # 실제 메시지 목록
    return [
        {"role": "user", "content": user_content},
        # 필요하다면 중간에 'assistant' 답변을 넣고 싶을 때만 추가합니다.
        {"role": "assistant", "content": f"<think>\n{think_content}</think>\n{attempt_content}"},
    ]

dataset = dataset.map(lambda x: {"messages": example_to_messages(x)})

def example_to_messages(example): """ 단일 example (예: {'solution': '...', 'question': '...', ...})을 Qwen 스타일 대화 목록(list of dict)으로 변환하는 함수. """ system_content = "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." # question을 user 메시지로, solution을 assistant 메시지로 사용 user_content = example.get("question", "") think_content = example.get("deepseek_thinking_trajectory", "") attempt_content = example.get("deepseek_attempt", "") # 실제 메시지 목록 return [ {"role": "user", "content": user_content}, # 필요하다면 중간에 'assistant' 답변을 넣고 싶을 때만 추가합니다. {"role": "assistant", "content": f"\n{think_content}\n{attempt_content}"}, ] dataset = dataset.map(lambda x: {"messages": example_to_messages(x)})

In [ ]:

dataset["messages"][0]

dataset["messages"][0]

In [10]:

from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(
    tokenizer,
    chat_template = "qwen2.5",
)

def formatting_prompts_func(examples):
    convos = examples["messages"]
    texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos]
    return { "text" : texts, }
pass

from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "qwen2.5", ) def formatting_prompts_func(examples): convos = examples["messages"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } pass

In [ ]:

dataset = dataset.map(formatting_prompts_func, batched = True,)

dataset = dataset.map(formatting_prompts_func, batched = True,)

In [ ]:

dataset[3]["messages"]

dataset[3]["messages"]

In [ ]:

dataset[3]["text"]

dataset[3]["text"]

학습 전 테스트입니다.

당연히 <think> 과정은 없을 텐데요 그래도 수학 문제를 풀렸더는 무언가 사고의 과정은 관찰이 되는군요.

• 학습 후에 이 사고의 과정이 어떻게 변하는 지,
• 그리고, 그래서 진짜 정답을 잘 맞추는지가 가장 중요하겠습니다.

In [ ]:

FastLanguageModel.for_inference(model) # Enable native 2x faster inference
inputs = tokenizer(
[
    "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nCompute the mean molecular speed v in the heavy gas radon (Rn) in m/s<|im_end|>\n<|im_start|>assistant\n"]
                    , return_tensors = "pt").to("cuda")

from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 1024)

FastLanguageModel.for_inference(model) # Enable native 2x faster inference inputs = tokenizer( [ "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nCompute the mean molecular speed v in the heavy gas radon (Rn) in m/s<|im_end|>\n<|im_start|>assistant\n"] , return_tensors = "pt").to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 1024)

데이터 길이 확인¶

데이터 길이를 확인할 것입니다, 너무 길다 보니 발생할 수 있는 문제점 들을 한번 생각해보겠습니다.

In [15]:

# get token length of this
token_length = len(tokenizer(dataset[5]["text"]).input_ids)

# for all dataset's "text", get token length and get distribution.
lengths = [len(tokenizer(data["text"]).input_ids) for data in dataset]

# get token length of this token_length = len(tokenizer(dataset[5]["text"]).input_ids) # for all dataset's "text", get token length and get distribution. lengths = [len(tokenizer(data["text"]).input_ids) for data in dataset]

In [ ]:

#visualize lengthes distribution
import matplotlib.pyplot as plt
plt.hist(lengths, bins=100)
plt.show()

#visualize lengthes distribution import matplotlib.pyplot as plt plt.hist(lengths, bins=100) plt.show()

In [16]:

학습 시키기¶

• 학습은 하이퍼 파라미터는

In [ ]:

from trl import SFTTrainer
from transformers import TrainingArguments, DataCollatorForSeq2Seq
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer),
    dataset_num_proc=2,
    packing=False,  # Can make training faster for short sequences if needed
    args=TrainingArguments(
        # 요청 사항: per_device_train_batch_size = 16
        per_device_train_batch_size = 4,
        gradient_accumulation_steps = 4,

        # 요청 사항: 5 에포크
        #num_train_epochs=5,
        num_train_epochs=2,

        # 요청 사항: learning_rate = 1e-5
        learning_rate=1e-5,

        # 요청 사항: warmup은 전체 스텝의 5% (315 스텝 중 5%면 약 16 스텝)
        # 정확히 16 스텝으로 설정
        #warmup_steps=16,
        warmup_steps=5,

        # 요청 사항: 스케줄은 코사인
        lr_scheduler_type="cosine",

        # 요청 사항: AdamW + betas = (0.9, 0.95), weight decay = 1e-4
        # huggingface Trainer에서는 adam_beta1, adam_beta2 로 설정
        # weight_decay 값도 변경
        optim="adamw_8bit",
        adam_beta1=0.9,
        adam_beta2=0.95,
        weight_decay=1e-4,

        # precision 관련: bfloat16이 지원되면 bf16=True, 아니면 fp16
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),

        # 로깅 스텝은 원하는 대로
        logging_steps=1,

        # 재현성
        seed=2503,

        # 결과 출력 경로
        output_dir="outputs",

        # 보고 설정 (wandb를 쓰면 "wandb" 등으로 변경)
        report_to="none",
    ),
)

from trl import SFTTrainer from transformers import TrainingArguments, DataCollatorForSeq2Seq from unsloth import is_bfloat16_supported trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=dataset, dataset_text_field="text", max_seq_length=max_seq_length, data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer), dataset_num_proc=2, packing=False, # Can make training faster for short sequences if needed args=TrainingArguments( # 요청 사항: per_device_train_batch_size = 16 per_device_train_batch_size = 4, gradient_accumulation_steps = 4, # 요청 사항: 5 에포크 #num_train_epochs=5, num_train_epochs=2, # 요청 사항: learning_rate = 1e-5 learning_rate=1e-5, # 요청 사항: warmup은 전체 스텝의 5% (315 스텝 중 5%면 약 16 스텝) # 정확히 16 스텝으로 설정 #warmup_steps=16, warmup_steps=5, # 요청 사항: 스케줄은 코사인 lr_scheduler_type="cosine", # 요청 사항: AdamW + betas = (0.9, 0.95), weight decay = 1e-4 # huggingface Trainer에서는 adam_beta1, adam_beta2 로 설정 # weight_decay 값도 변경 optim="adamw_8bit", adam_beta1=0.9, adam_beta2=0.95, weight_decay=1e-4, # precision 관련: bfloat16이 지원되면 bf16=True, 아니면 fp16 fp16=not is_bfloat16_supported(), bf16=is_bfloat16_supported(), # 로깅 스텝은 원하는 대로 logging_steps=1, # 재현성 seed=2503, # 결과 출력 경로 output_dir="outputs", # 보고 설정 (wandb를 쓰면 "wandb" 등으로 변경) report_to="none", ), )

In [17]:

학습은 답변에만!¶

system prompt 와 user 질문는 학습하지 않도록 마스킹을 해줍니다.

질문으로 들어온 문제를 학습하는 일은 없어야 합니다, 답변 방식만 reasoning 을 하도록 수정할 것이니까요.

In [ ]:

from unsloth.chat_templates import train_on_responses_only

trainer = train_on_responses_only(
    trainer,
    instruction_part=(
        "<|im_start|>user\n"         # 사용자 메시지 시작
    ),
    response_part=(
        "<|im_end|>\n"               # 사용자 메시지 종료
        "<|im_start|>assistant\n"    # 어시스턴트 메시지 시작
    )
)

from unsloth.chat_templates import train_on_responses_only trainer = train_on_responses_only( trainer, instruction_part=( "<|im_start|>user\n" # 사용자 메시지 시작 ), response_part=( "<|im_end|>\n" # 사용자 메시지 종료 "<|im_start|>assistant\n" # 어시스턴트 메시지 시작 ) )

In [ ]:

gpu_stats = torch.cuda.get_device_properties(0)
start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.")
print(f"{start_gpu_memory} GB of memory reserved.")

gpu_stats = torch.cuda.get_device_properties(0) start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3) max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3) print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.") print(f"{start_gpu_memory} GB of memory reserved.")

In [ ]:

trainer_stats = trainer.train()

trainer_stats = trainer.train()

In [ ]:

# Merge to 16bit
if True: model.save_pretrained_merged("qwen-7b-s1.1-2epoch", tokenizer, save_method = "merged_16bit")
if True: model.push_to_hub_merged("jonhpark/qwen-7b-s1.1-2epoch", tokenizer, save_method = "merged_16bit", token = "")

# Merge to 16bit if True: model.save_pretrained_merged("qwen-7b-s1.1-2epoch", tokenizer, save_method = "merged_16bit") if True: model.push_to_hub_merged("jonhpark/qwen-7b-s1.1-2epoch", tokenizer, save_method = "merged_16bit", token = "")

In [21]:

결과 테스트¶

Math-500 으로 학습해보지 않은 문제를 풀어봅시다

In [ ]:

from datasets import load_dataset

dataset = load_dataset("HuggingFaceH4/MATH-500")

from datasets import load_dataset dataset = load_dataset("HuggingFaceH4/MATH-500")

In [ ]:

level = "1"
questions = [q for q in dataset['test'] if str(q['level']) == level] if level != "all" else dataset['test']
len(questions)

level = "1" questions = [q for q in dataset['test'] if str(q['level']) == level] if level != "all" else dataset['test'] len(questions)

In [ ]:

from unsloth import FastLanguageModel
import torch
max_seq_length = 32768 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "jonhpark/qwen-7b-s1.1-2epoch", # or choose "unsloth/Llama-3.2-1B-Instruct"
    max_seq_length = max_seq_length,
    dtype=torch.bfloat16,
    token = "",
)

from unsloth import FastLanguageModel import torch max_seq_length = 32768 # Choose any! We auto support RoPE Scaling internally! dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ model, tokenizer = FastLanguageModel.from_pretrained( model_name = "jonhpark/qwen-7b-s1.1-2epoch", # or choose "unsloth/Llama-3.2-1B-Instruct" max_seq_length = max_seq_length, dtype=torch.bfloat16, token = "", )

In [95]:

#model, tokenizer = FastLanguageModel.from_pretrained(
#    model_name = "Qwen/Qwen2.5-7B-Instruct", # or choose "unsloth/Llama-3.2-1B-Instruct"
#    max_seq_length = max_seq_length,
#    dtype=torch.bfloat16,
#    token = "",
#)

FastLanguageModel.for_inference(model) # Enable native 2x faster inference
from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)

#model, tokenizer = FastLanguageModel.from_pretrained( # model_name = "Qwen/Qwen2.5-7B-Instruct", # or choose "unsloth/Llama-3.2-1B-Instruct" # max_seq_length = max_seq_length, # dtype=torch.bfloat16, # token = "", #) FastLanguageModel.for_inference(model) # Enable native 2x faster inference from transformers import TextStreamer text_streamer = TextStreamer(tokenizer)

In [ ]:

generated_outputs = []
for question in questions:
  if len(generated_outputs) >= 5:
    break

  print("------------------------")
  inputs = tokenizer(
    [
#    f"<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nAnswer the following question.\n{question['problem']}<|im_end|>\n<|im_start|>assistant\n",
    f"<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nAnswer the following question.\n{question['problem']}<|im_end|>\n<|im_start|>assistant\n<think>\n",
    ]
                    , return_tensors = "pt").to("cuda")


  outputs = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 10000)
  generated_outputs.append(outputs)

generated_outputs = [] for question in questions: if len(generated_outputs) >= 5: break print("------------------------") inputs = tokenizer( [ # f"<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nAnswer the following question.\n{question['problem']}<|im_end|>\n<|im_start|>assistant\n", f"<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nAnswer the following question.\n{question['problem']}<|im_end|>\n<|im_start|>assistant\n\n", ] , return_tensors = "pt").to("cuda") outputs = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 10000) generated_outputs.append(outputs)

결과 저장¶

풀이와 정답들은 다시 채점을 해보거나, 정성적으로 답변을 살펴보기 위해 저장합니다.

In [97]:

# for all elem in genereated outputs, apply tokenizer.decode()
for i in range(len(generated_outputs)):
  generated_outputs[i] = tokenizer.decode(generated_outputs[i][0])

# save generated_outputs in json format
import json
with open(f'generated_outputs_{level}.json', 'w') as f:
  json.dump(generated_outputs, f)

# for all elem in genereated outputs, apply tokenizer.decode() for i in range(len(generated_outputs)): generated_outputs[i] = tokenizer.decode(generated_outputs[i][0]) # save generated_outputs in json format import json with open(f'generated_outputs_{level}.json', 'w') as f: json.dump(generated_outputs, f)

평가와 LLM as Judge¶

문제, 풀이, 답변을 읽고 채점을 시켜봅시다.

In [98]:

# read generated_outputs_1.json
import json

with open('generated_outputs_1_qwen2.5-7b-instruct.json', 'r') as f:
  generated_outputs_base = json.load(f)

with open('generated_outputs_1.json', 'r') as f:
  generated_outputs = json.load(f)

# read generated_outputs_1.json import json with open('generated_outputs_1_qwen2.5-7b-instruct.json', 'r') as f: generated_outputs_base = json.load(f) with open('generated_outputs_1.json', 'r') as f: generated_outputs = json.load(f)

In [ ]:

In [ ]:

# read token size for all item in generated_outputs and generated_ouputs_base
base_token_lens = [len(tokenizer.encode(output)) for output in generated_outputs_base]
token_lens = [len(tokenizer.encode(output)) for output in generated_outputs]

#visualize of all len values of both, not histogram line chart
import matplotlib.pyplot as plt

plt.plot(base_token_lens, label='base')
plt.plot(token_lens, label='new')
plt.legend()
plt.show()

# read token size for all item in generated_outputs and generated_ouputs_base base_token_lens = [len(tokenizer.encode(output)) for output in generated_outputs_base] token_lens = [len(tokenizer.encode(output)) for output in generated_outputs] #visualize of all len values of both, not histogram line chart import matplotlib.pyplot as plt plt.plot(base_token_lens, label='base') plt.plot(token_lens, label='new') plt.legend() plt.show()

In [ ]:

# check if "</think>" text exists in item of generated_output
count = 0
for output in generated_outputs:
  if "</think>" in output:
    count = count +1

print(count)

# check if "" text exists in item of generated_output count = 0 for output in generated_outputs: if "" in output: count = count +1 print(count)

In [102]:

from openai import OpenAI
import os

os.environ["openai_api_key"] = ""
openai_client = OpenAI(api_key=os.getenv("openai_api_key"))

def judge_call(problem, answer, model_response):

    user_message_to_judge = f"""Problem: {problem}
Answer: {answer}
Model Response: {model_response}

---

If Model Response is correct, write 'CORRECT'. If not, write 'INCORRECT'."""
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a Judge grading a answer of problem"},
            {"role": "user", "content": user_message_to_judge}
        ]
    )

    output = response.choices[0].message.content

    print(output)

    if 'INCORRECT' in output:
      return False
    elif 'CORRECT' in output:
      return True
    else:
      return False

from openai import OpenAI import os os.environ["openai_api_key"] = "" openai_client = OpenAI(api_key=os.getenv("openai_api_key")) def judge_call(problem, answer, model_response): user_message_to_judge = f"""Problem: {problem} Answer: {answer} Model Response: {model_response} --- If Model Response is correct, write 'CORRECT'. If not, write 'INCORRECT'.""" response = openai_client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": "You are a Judge grading a answer of problem"}, {"role": "user", "content": user_message_to_judge} ] ) output = response.choices[0].message.content print(output) if 'INCORRECT' in output: return False elif 'CORRECT' in output: return True else: return False

In [ ]:

count = 0
for i in range(len(questions)):
  question = questions[i]
  problem = question['problem']
  answer = question['answer']


  model_response = generated_outputs[i]
  # if </think> is in the model_response, strip and get sting afterh the </think>
  if "</think>" in model_response:
    model_response = model_response.split("</think>")[1]

  if judge_call(problem, answer, model_response):
    count = count + 1

print(f"{count} / {len(questions)}")

count = 0 for i in range(len(questions)): question = questions[i] problem = question['problem'] answer = question['answer'] model_response = generated_outputs[i] # if is in the model_response, strip and get sting afterh the if "" in model_response: model_response = model_response.split("")[1] if judge_call(problem, answer, model_response): count = count + 1 print(f"{count} / {len(questions)}")

In [ ]:

In [ ]:

count = 0
for i in range(len(questions)):
  question = questions[i]
  problem = question['problem']
  answer = question['answer']
  model_response = generated_outputs_base[i]
  if judge_call(problem, answer, model_response):
    count = count + 1

print(f"{count} / {len(questions)}")

count = 0 for i in range(len(questions)): question = questions[i] problem = question['problem'] answer = question['answer'] model_response = generated_outputs_base[i] if judge_call(problem, answer, model_response): count = count + 1 print(f"{count} / {len(questions)}")

In [49]:

In [ ]: