Fate Partner

RELATIONSHIP & BAZI JOURNAL · BaZi basics

2026 5-Model BaZi Benchmark (v1.1): From 78% Pillar Error to 105k Thinking Tokens

We audited 5 zero-tool LLMs across 100 universal-UTC astronomical boundary birthdays covering Start of Spring, 12 solar terms, True Solar Time, late Zi hour, and historical DST

Using Fate Partner's deterministic perpetual calendar engine, we built an open-source benchmark of 100 universal-UTC astronomical boundary birth cases (v1.1) covering Start of Spring, the 12 month-boundary solar terms, True Solar Time shifts in Tokyo and Taipei, late Zi hour day transitions, and historical Daylight Saving Time. Auditing 5 zero-tool LLMs on November 22, 2026 revealed a stark tier divide: non-reasoning DeepSeek-Chat failed 78.0% of charts, fast-reasoning Gemini 3.8 Flash failed 44.0%, Claude Sonnet 5.5 failed 16.0%, flagship Claude Opus 5.5 still failed 4.0%, and Gemini 4 Carbon reached 100/100 only by burning 105,418 thinking tokens across roughly 18 minutes.

1. Five-Model 100-Case Leaderboard (v1.1 Universal-UTC Ground Truth)

Even under a generous upper-bound prompt that explicitly supplied each city's decimal longitude, IANA timezone, True Solar Time flag, and historical DST rules, zero-tool results diverged dramatically: (1) DeepSeek-Chat (non-reasoning REST API, temp 0): 22.0% exact match, 78.0% error rate (22% wrong Day Masters, 52% wrong Hour Pillars); (2) Gemini 3.8 Flash (Medium thinking, ~53k output tokens): 56.0% exact match, 44.0% error rate (19% wrong Day Masters, 95% error on True Solar Time); (3) Claude Sonnet 5.5 (Medium thinking, ~49k tokens): 84.0% exact match, 16.0% error rate (32% error on 12-Jie month boundaries); (4) Claude Opus 5.5 (Medium thinking, ~38k tokens): 96.0% exact match, 4.0% error rate (12% error on 12-Jie month boundaries); (5) Gemini 4 Carbon (Medium thinking, 105,418 tokens, ~18 minutes): 100.0% exact match; (6) Fate Partner deterministic engine and MIT MCP server: 100.0% exact match in under 2 milliseconds and 0 LLM tokens. Transparency note: open-weight GPT-OSS 120B (Medium) refused zero-tool computation on Attempt 1 and collapsed into repeated placeholder output under 50-case batching on Attempt 2, so it was excluded from the leaderboard.

2. Failure Breakdown Across All 5 Astronomical Trap Categories

Examining each category shows where everyday chatbots break down: (1) Start of Spring transitions (25 cases): DeepSeek-Chat erred on 72.0%, Claude Sonnet 5.5 on 16.0%, and Gemini 3.8 Flash on 12.0%; (2) 12 month-boundary solar terms (25 cases): the hardest frontier barrier, where DeepSeek and Gemini 3.8 Flash both erred on 48.0%, Claude Sonnet 5.5 erred on 32.0%, and even flagship Claude Opus 5.5 erred on 12.0% (for example, 1992-07-07 05:20 in Taipei occurred 40 minutes after Minor Heat, yet Opus stayed in the prior month); (3) True Solar Time across Japan, Taiwan, and global cities (20 cases): DeepSeek erred on 100.0% (0/20) and Gemini 3.8 Flash erred on 95.0% (1/20), routinely missing Tokyo's +35 minute longitude and equation-of-time shift on 1992-11-03 00:35; (4) Late Zi 23:00 day boundary (15 cases): DeepSeek erred on 86.7% and Gemini 3.8 Flash erred on 33.3%; (5) Historical DST in Japan (1948-1951), Taiwan (1946-1979), and Mainland China (1986-1991) (15 cases): DeepSeek erred on 100.0% and Gemini 3.8 Flash erred on 33.3%.

3. 105,000 Thinking Tokens vs 2 Milliseconds: Why Tool Separation Wins

The real lesson is architectural economics rather than dismissing AI progress. Everyday users asking free or fast-tier chat apps for BaZi readings never paste decimal city longitudes or historical DST rules, putting real-world casual error rates in the 44% to 78%+ band. And even when a frontier model like Gemini 4 Carbon solves all 100 boundary charts mentally, it spends over 1,050 thinking tokens and roughly 11 seconds per single chart doing manual Julian Day arithmetic — more than 5,000 times slower than a deterministic 2-millisecond calendar program that uses zero tokens and never drifts.

4. Open-Source Audit Dividend (v1.1 Timezone Fix) & Hybrid Architecture

Running cross-model blind verification also paid an immediate engineering dividend: inspecting disagreements in v1.0 caught a 1-hour cross-timezone solar-term bug affecting 10 Japan (Asia/Tokyo, UTC+9 vs UTC+8 ephemeris) boundary cases, which we fixed and documented transparently in v1.1. The complete universal-UTC dataset, all 5 audited model transcripts, and our standalone MIT MCP server (bazi-solar-term-mcp) are open-sourced on GitHub (github.com/liu17513567-blip/bazi-solar-term-mcp). Fate Partner separates deterministic astronomy from plain-language relationship guidance so language models focus solely on thoughtful interpretation.

← Back to blog