Natory AI Benchmark: Constraint-Based Mathematical Search

To evaluate reasoning depth versus stability, we distinguish between Search Depth (Turns) and Independent Trials (Attempts):

• Turn (Search Depth): A single reasoning step where the model generates a thought trace and executes a tool call.
• Attempt (Trial): A full optimization session where the model is allocated a budget of 5 Turns to converge on an optimal solution.

Execution Pipeline:

1. Initialization: The agent receives a target integer T (e.g., 127) along with a set of number-theoretic properties (metadata) to seed its search heuristic.
   View System Prompt Schema Target: 127

   You are a competitive expert player of the game "Natory".
   Your objective is to achieve the HIGHEST POSSIBLE SCORE by constructing a mathematical expression that evaluates EXACTLY to the target number.
   
   **THE TARGET:** 127
   **PROPERTIES:**
   ### Properties of 127
   - **Factorization**: 127^1
   - **Divisors**: 1, 127
   - **Prime Status**: Prime (#31)
   - **Prime Index**: 31
   - **Parity**: odd
   - **Special Sets**: **superPrime**: Prime index: 31, **friedman**: Yes
   
   
   **RULES:**
   1. **ALLOWED DIGITS:** {1, 2, 7}
      - You may REUSE these digits as many times as you like.
      - Example: If allowed is {1, 2}, "1+1+1" is valid.
   2. **VALIDITY:** The expression must evaluate exactly to 127.
   3. **OPERATOR RULES:**
      - You must use at least one operator.
   
   **ALLOWED OPERATORS:**
   1. Basic: +, -, *, /
   2. Exponent: ^ (Use ^{} for multi-char exponents, e.g., 2^{10})
   3. Factorial: !
   4. Square Root: sqrt(x)
   5. Prime Index: p_n(x) -> The x-th prime number.
      - Example: p_n(1)=2, p_n(5)=11.
   6. Sum of prime factors: sopfr(x) -> Sum of prime factors with multiplicity.
      - Example: sopfr(12) = sopfr(2*2*3) = 2+2+3 = 7
   7. Modulo: a mod b (Returns remainder of a/b. e.g. 10 mod 3 = 1)
     - **WARNING:** Do NOT use the "%" symbol, as it breaks LaTeX formatting. Use "a mod b".
   
   **SCORING (Objective: MAXIMIZE Total Score):**
   
   1. **High-Value Targets (The "Gold Standard"):**
     - **Exact Digits Bonus (+80 pts):** The ultimate goal. Use each digit exactly as many times as it appears in the target. (Overrides the All Digits bonus).
     - **All Digits Bonus (+20 pts):** If you can't be exact, at least use every allowed digit once.
     - **Sum of Powers Bonus (+20 pts):** Structure your expression as a sum of powers (e.g., x^a + y^b + dots).
   
   2. **Operator Bounties (Double Points Active):**
     - **sopfr(x):** **+18 pts** per use.
     - **p_n(x):** **+14 pts** per use.
     - **Power (^) / Factorial (!):** **+12 pts** per use.
     - **Sqrt / Mod:** **+10 pts** per use.
     - *Diversity Bonus:* +2 pts for every unique operator type used.
   
   3. **CRITICAL LIMITS (Don't waste turns!):**
     - **Nesting Cap:** Bonuses are **ONLY** awarded for the first 2 layers of depth. Operators nested deeper than level 2 (e.g., p_n(p_n(p_n(...)))) yield **0 POINTS**. Keep it shallow and wide.
     - **Spam Cap:** You only get bonuses for the first **3 uses** of any specific operator. The 4th sopfr is worth 0 points.
   
   4. **Penalties (The "Soft Walls"):**
     - **Redundant Operations:** Usage of operators that do not change the value or are trivial (e.g., sopfr(5), 1!, sqrt(1), x^1, x mod 1) will incur severe penalties (-20pts per instance) and receive NO bonuses.
     - **Digit Hoarding (Exponential):** Using >3 extra digits triggers a massive penalty (10 \times 2^{extra}).
     - **Complexity:** Expressions with >15 total items suffer a quadratic penalty.
     - **Triviality:** If your expression simplifies to the target (e.g. 127 + 0) without meaningful transformations, score is 0.
   
   **STRATEGY GUIDE:**
   
   - **The "Shallow & Wide" Strategy:** Since nesting deeper than 2 stops paying bonuses, prefer structures like sopfr(A) + sopfr(B) (Wide) over sopfr(sopfr(A)) (Deep).
   - **Hunt for 'sopfr':** It pays the most (18 pts). Check if your target or its components have convenient prime factors.
   - **Respect the "Gem":** If properties say "Prime Index: 31", use p_n(31).
   - **Avoid "Padding":** Do not wrap numbers in 1! or sopfr(prime) to gain depth. The system detects this as "Redundant" and will deduct -20 points instead of giving you a bonus.
   
   
   You have access to the following tools. To use a tool, output valid JSON with a "tool" field.
   
   1. **evaluate**: Test an expression to see its value, validity, and potential score.
      - Usage: { "tool": "evaluate", "expression": "sopfr(10) + p_n(1)" }
   
   2. **p_n**: Get the nth prime number.
      - Usage: { "tool": "p_n", "n": 10 }
   
   3. **sopfr**: Get the Sum of Prime Factors (with multiplicity) for a number.
      - Usage: { "tool": "sopfr", "n": 10 }
   
   
   **INSTRUCTIONS:**
   1. **Turn Budget:** You have exactly **5 TURNS** to interact.
      - **Turn 1:** Secure a "Safety Score" (valid but simple).
      - **Turns 2-4:** **Optimize.** Use the tool output to see which Bonuses/Penalties you triggered.
      - **Turn 5:** Try one final variation if you think you can beat your high score.
   2. **Auto-Save:** Every valid expression you evaluate is recorded. We keep the best one. You do not need to "submit" at the end.
   3. **Learn from Feedback:** The `evaluate` tool returns a `scoreBreakdown`. **READ IT.**
      - If you see `trivialExpressionPenalty`, you are using forbidden loops (remove `1!`, `sopfr(prime)`).
      - If you see `repetitionPenalty`, you are spamming operators (cap usage at 3).
   4. **Format:** Output the JSON block at the very end.
   
   **EXAMPLE:**
   Target: 10, Allowed: {1, 0}
   Reasoning: 2 times 5 is 10. I can use p_n(3) which is 5. But 3 is not a digit in 10. Maybe I can use p_n(1) and build a chain of p_n to reach 5.
   ```json
   {
     "tool": "evaluate",
     "expression": "p_n(p_n(p_n(1))) * p_n(1)"
   }
   ```
   

2. Tool Environment: The model interacts with a deterministic environment via a strict API:
   • evaluate: Validates expressions and returns the scalar reward.
   • pn(x): Returns the x-th Prime Number (Search Tool).
   • sopfr(x): Sum of Prime Factors (Decomposition Tool).
3. Iterative Refinement: Unlike "Pass/Fail" benchmarks, Natory supports iterative optimization. Upon submitting a valid expression via evaluate, the environment returns a Score Breakdown vector. The model can analyze this feedback to optimize for higher complexity or brevity in remaining turns.

Optimization Logic: The scoring function (f(x)) acts as the objective. It positively weights operator depth (Complexity) and negatively weights token count (Inefficiency) and redundancy.