Close Menu
    Trending
    • Creating AI that matters | MIT News
    • Scaling Recommender Transformers to a Billion Parameters
    • Hidden Gems in NumPy: 7 Functions Every Data Scientist Should Know
    • Is RAG Dead? The Rise of Context Engineering and Semantic Layers for Agentic AI
    • ChatGPT Gets More Personal. Is Society Ready for It?
    • Why the Future Is Human + Machine
    • Why AI Is Widening the Gap Between Top Talent and Everyone Else
    • Implementing the Fourier Transform Numerically in Python: A Step-by-Step Guide
    ProfitlyAI
    • Home
    • Latest News
    • AI Technology
    • Latest AI Innovations
    • AI Tools & Technologies
    • Artificial Intelligence
    ProfitlyAI
    Home » A Chinese firm has just launched a constantly changing set of AI benchmarks
    AI Technology

    A Chinese firm has just launched a constantly changing set of AI benchmarks

    ProfitlyAIBy ProfitlyAIJune 23, 2025No Comments3 Mins Read
    Share Facebook Twitter Pinterest LinkedIn Tumblr Reddit Telegram Email
    Share
    Facebook Twitter LinkedIn Pinterest Email


    Growth of the benchmark at HongShan started in 2022, following ChatGPT’s breakout success, as an inside device for assessing which fashions are value investing in. Since then, led by associate Gong Yuan, the crew has steadily expanded the system, bringing in exterior researchers and professionals to assist refine it. Because the venture grew extra subtle, they determined to launch it to the general public.

    Xbench approached the issue with two totally different techniques. One is just like conventional benchmarking: an educational check that gauges a mannequin’s aptitude on numerous topics. The opposite is extra like a technical interview spherical for a job, assessing how a lot real-world financial worth a mannequin would possibly ship.

    Xbench’s strategies for assessing uncooked intelligence at present embrace two elements: Xbench-ScienceQA and Xbench-DeepResearch. ScienceQA isn’t a radical departure from current postgraduate-level STEM benchmarks like GPQA and SuperGPQA. It consists of questions spanning fields from biochemistry to orbital mechanics, drafted by graduate college students and double-checked by professors. Scoring rewards not solely the correct reply but additionally the reasoning chain that results in it.

    DeepResearch, against this, focuses on a mannequin’s skill to navigate the Chinese language-language net. Ten subject-matter consultants created 100 questions in music, historical past, finance, and literature—questions that may’t simply be googled however require important analysis to reply. Scoring favors breadth of sources, factual consistency, and a mannequin’s willingness to confess when there isn’t sufficient knowledge. A query within the publicized assortment is “What number of Chinese language cities within the three northwestern provinces border a international nation?” (It’s 12, and solely 33% of fashions examined received it proper, if you’re questioning.)

    On the corporate’s web site, the researchers stated they wish to add extra dimensions to the check—for instance, points like how inventive a mannequin is in its drawback fixing, how collaborative it’s when working with different fashions, and the way dependable it’s.

    The crew has dedicated to updating the check questions as soon as 1 / 4 and to keep up a half-public, half-private knowledge set.

    To evaluate fashions’ real-world readiness, the crew labored with consultants to develop duties modeled on precise workflows, initially in recruitment and advertising. For instance, one job asks a mannequin to supply 5 certified battery engineer candidates and justify every choose. One other asks it to match advertisers with acceptable short-video creators from a pool of over 800 influencers.

    The web site additionally teases upcoming classes, together with finance, authorized, accounting, and design. The query units for these classes haven’t but been open-sourced.

    ChatGPT-o3 once more ranks first in each of the present skilled classes. For recruiting, Perplexity Search and Claude 3.5 Sonnet take second and third place, respectively. For advertising, Claude, Grok, and Gemini all carry out nicely.

    “It’s actually tough for benchmarks to incorporate issues which can be so laborious to quantify,” says Zihan Zheng, the lead researcher on a brand new benchmark known as LiveCodeBench Professional and a pupil at NYU. “However Xbench represents a promising begin.”



    Source link

    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    Previous ArticleLLMs factor in unrelated information when recommending medical treatments | MIT News
    Next Article Building AI-Powered Low-Code Workflows with n8n
    ProfitlyAI
    • Website

    Related Posts

    AI Technology

    Why AI should be able to “hang up” on you

    October 21, 2025
    AI Technology

    From slop to Sotheby’s? AI art enters a new phase

    October 17, 2025
    AI Technology

    Future-proofing business capabilities with AI technologies

    October 15, 2025
    Add A Comment
    Leave A Reply Cancel Reply

    Top Posts

    How a Research Lab Made Entirely of LLM Agents Developed Molecules That Can Block a Virus

    August 5, 2025

    LLM in Banking and Finance: Key Use Cases, Examples, and a Practical Guide

    April 7, 2025

    LLM Optimization: LoRA and QLoRA | Towards Data Science

    May 30, 2025

    Run Your Python Code up to 80x Faster Using the Cython Library

    July 8, 2025

    Beyond Code Generation: Continuously Evolve Text with LLMs

    June 19, 2025
    Categories
    • AI Technology
    • AI Tools & Technologies
    • Artificial Intelligence
    • Latest AI Innovations
    • Latest News
    Most Popular

    And Why Does It Matter? » Ofemwire

    April 4, 2025

    LLM Optimization: LoRA and QLoRA | Towards Data Science

    May 30, 2025

    3 Greedy Algorithms for Decision Trees, Explained with Examples

    September 2, 2025
    Our Picks

    Creating AI that matters | MIT News

    October 21, 2025

    Scaling Recommender Transformers to a Billion Parameters

    October 21, 2025

    Hidden Gems in NumPy: 7 Functions Every Data Scientist Should Know

    October 21, 2025
    Categories
    • AI Technology
    • AI Tools & Technologies
    • Artificial Intelligence
    • Latest AI Innovations
    • Latest News
    • Privacy Policy
    • Disclaimer
    • Terms and Conditions
    • About us
    • Contact us
    Copyright © 2025 ProfitlyAI All Rights Reserved.

    Type above and press Enter to search. Press Esc to cancel.