Measuring What Matters: Project Athena Innovation Program’s Real-World Testing

By Sudha Ganesh, Jim Scovell and Clem Wong

BUILT IN - ARTICLE INTRO SECOND COMPONENT

For years, the PC industry has faced a central question: what’s the ‘right’ way to test laptops?

Traditional benchmarks have succeeded in measuring raw computing power with highly repeatable tests in lab conditions. But after extensive human-centered research, Intel has found that these benchmarks don’t completely reflect the criteria that matter most to people. The right benchmarks and pure processing speed are still important, but not a complete view of the best laptop experience. People want a laptop that’s always ready, helps them focus and stays responsive as they multi-task throughout the day—all with stellar battery life.

People may care about pure processing speed only to the extent that it impacts their experience, but that does not define the full expectations of a great laptop experience. That’s why, when we launched the Project Athena innovation program in 2019, we anchored our testing approach in solving these human, real-world needs.

And it’s why we regularly update this testing methodology with each successive Project Athena specification to reflect new research on the latest trends. For example, laptops that feature the Intel® Evo™ brand must meet all the testing requirements of the 2nd edition Project Athena spec that aims to match real-world user experience needs.

Intel believes the “right” way to test laptops is by using metrics that measure real-world user experience. Testing results then reflect true user utility and value—which is ultimately what matters to people.

To help the industry deliver a new class of advanced laptops, Intel has developed a one-of-a-kind, automated testing tool that:

1. Focuses on key experiences
2. Reflects real-world workflows and conditions
3. Equips OEMs to optimize platform designs

Key Experiences That Matter

Project Athena rigorously tests designs for three foundational key experiences: consistent responsiveness1, instant wake and a worry-free day of battery life.2 These experiences, drawn from ethnographic research and measured by a set of Key Experience Indicators (KEIs), provide the best basis for impactful laptop innovation.

Starting with People and Their Experiences

Project Athena’s real-world testing is based on extensive, in-depth research to identify the experiences that matter most to people, especially the ambitious and demanding people who expect the best from their laptops, without compromise:

Human-centered innovation: Intel’s research teams boast more than 15 years of global research around human-centered innovation, helping to provide a detailed picture of how people use their laptops, what their pain points are and where innovations can help. In particular, the research focused on deep immersion sessions with people who depend on their laptops to manage their dynamic, fast-paced personal and professional lives.

Identifying key experiences: This research revealed that people want their laptops to help them focus, adapt and always be ready and connected. These are essential to help people stay in their flow, so they can work, connect and play without interruption. To achieve this, we determined that laptops need to wake in less than a second; respond quickly and efficiently to people’s inputs; and have worry-free battery life and charge rapidly—the key experiences for Project Athena.

Defining the Key Experience Indicators (KEIs)

Intel’s technical experts then translated these key experiences into a set of quantitative KEIs—the backbone of our real-world testing methodology:

Focusing on impactful interactions: To select the KEIs, Intel zeroed in on the tasks that have the greatest impact and are the most meaningful to a person’s laptop experience. For the second edition, which is represented by the Evo brand, Intel put a specific emphasis on the connection between local and cloud-based tasks to better reflect today’s agile working environment. Launching email, starting video calls, creating charts and loading browser videos are the kinds of essential interactions that are included in the 25 KEIs in Project Athena’s second edition.

Technical decomposition: Drawing on more than 15 years of Intel’s user testing, Intel’s experts defined time targets—down to the millisecond—for each KEI. This testing uses MOS (mean opinion score) models to determine the exact threshold for a best-in-class experience for each interaction. For example, in Project Athena’s second edition, a laptop must be able to wake in an instant, play a YouTube video quickly or launch Outlook in a matter of seconds. Battery life must last for more than 9 hours under real-world conditions3, while being able to charge 4 hours of power in just 30 minutes with full HD displays.4

Real-World Workflows and Conditions

The Project Athena innovation program measures the KEIs with a realistic “day-in-the-life” workflow in conditions that mirror how people typically use their laptops. This provides a more practical view of performance, as opposed to using only synthetic workflows and artificial lab conditions.

A Rigorous, Realistic Workflow

Unlike benchmarks that are designed to tax specific components such as the CPU or the disk, Project Athena testing is designed to reflect a “day in the life” of a person who depends on their laptop to get things done. The testing tool repeatedly runs a workflow of more than 200 tasks, including the KEIs, which model how people use their laptops every day, based on ethnographic research and telemetry data.

This research revealed that people constantly switch between four “computing modes”: Communication and sharing; personal organization; expert production; and casual creation. This includes everything from less performance-demanding activities, like web browsing, online shopping or reading the news, to more demanding tasks, such as Zoom calls, YouTube streaming or working in the cloud with Google Suite. All of this is incorporated into the workflow with Project Athena’s second edition. 

The testing tool also runs with multiple applications open at once, including constant companions like Outlook, Google Chrome, Spotify, Slack and more. This realistic concurrency tests the laptop’s responsiveness and battery life in the same way as a person who is constantly multi-tasking, browsing the Internet and streaming videos throughout the day.

To ensure the battery-life test is as realistic as possible, we use real telemetry data on communication and sharing, casual creation, organization and expert production workflows. Each test takes about two hours to complete, and we require three unique iterations of the test to ensure consistency.

Intel integrated these real-world insights into the second edition of Project Athena:

Real-World Insights

Improvements in Second Edition Testing

More people doing more multi-tasking Added more constant companions, including Spotify, LinkedIn, Slack and Google Drive
Growing expectations for responsiveness Increased number of KEIs from 15 to 25; increased the responsiveness requirements for all KEIs
Growing expectations for battery life Intensified the workload to make the 9 hours of battery life more meaningful; added delays in workflow that reflect a person’s pauses during day
The rise of the cloud Added KEIs for Google Drive tasks, editing Google Sheets files; added Google Drive as constant companion, along with interacting with Office files on the cloud via Office 365
Adoption of tools for work collaboration and personal connection Added KEIs for Zoom tasks; added Slack as constant companion

Real-World Usage

Percentage of Time Tested based on User-Telemetry

Web Browsing 23%
Productivity 19%
Media Streaming 6%
Collaboration 6%
Idle Time 45%

Real-World Conditions

Project Athena tests in real-world conditions for a more accurate picture of performance. This differs from most benchmarks, which are typically measured in artificial lab conditions—for example, the laptop is plugged into the wall, with a dimmed screen and no Wi-Fi connection. Instead, with Project Athena, the laptop is tested on battery power, on a live network, with common background applications open, the latest software updates, logged into multiple cloud accounts and with the display set to ≥250 nits brightness on LCD panels.

The real-world testing tool runs 15 times to ensure a consistent result for the responsiveness KEIs. Each iteration of the workflow captures changes in responsiveness as the battery drains, and the laptop must exceed a stringent threshold to be verified. If a laptop does not meet these requirements, it undergoes further tuning by Intel and the OEM. In every workflow, the laptop is unplugged, so that Project Athena-verified laptop designs are responsive whenever and wherever they’re needed.

OEMs Empowered to Optimize Platform Designs

Project Athena’s real-world testing provides OEMs with an automated tool that they can use to assess, tune and improve their laptop designs. This equips OEMs—and the entire PC ecosystem—to deliver meaningful innovation, from balanced performance to best-in-class platform technologies.

Driving Innovation Across the PC Platform

OEMs use Intel’s real-world testing tool to optimize performance and battery life, as well as make key decisions about the entire PC platform:

• Drilling into the results: The automated testing tool enables an OEM to view its results for each of the KEIs across multiple runs. Engineers can use this information to identify pain points, understand the problem and tune the system for better performance and battery life.

Beyond the CPU: OEMs can also use the tool to make decisions about components and the overall PC platform, because the test requires more than just processing speed. Laptops must also deliver reliable connectivity for online and cloud-enabled tasks, sophisticated memory and storage for responsiveness, and platform optimization to meet battery life requirements.

Project Athena-verified: Ultimately, each laptop design must meet these KEIs and the spec in order to be verified as a Project Athena laptop5. This ensures that people can buy any Project Athena-verified laptop design with the confidence that it will deliver on their most important key experiences.

Just the Beginning

Project Athena and its unique approach to real-world testing is already helping the ecosystem create complete PC platforms tailored for people’s needs. These laptop designs are tuned, tested and verified to deliver on key experiences under real-world conditions, with a comprehensive set of KEIs that measure performance to the millisecond.

Intel will continue to evolve Project Athena’s testing to reflect trends in how people use their laptops, what they expect and prioritize, and what they don’t even know they need yet.

The Project Athena Innovation Program


Notices and Disclaimers6 7

产品和性能信息

1

在包括多个基于云的应用和本地应用以及 Web 页面(包括 Google* Chrome、Google* G-Suite、Microsoft* Office 365、YouTube 和 Zoom)的典型使用环境下进行 25 个工作流程的性能预测时测量的基于高级版 Windows 操作系统的设计的响应速度。测试在直流电池电量 ≥ 30% 的笔记本电脑上进行,采用 802.11 无线连接和自带的硬件配置,包括 Windows* 10 和 250 尼特 LCD 屏幕亮度。基于 2020 年 8 月的测试结果,且不能保证个别笔记本电脑的性能。功率和性能根据使用、配置和其他因素而有所不同。

2

有关性能和基准测试结果的更完整信息,请访问 intel.cn/Evo

3

在真实环境中执行典型工作流程时电池电量从 100% 消耗至临界值所需的时间。有关性能和基准测试结果的更完整信息,请访问 intel.cn/Evo。 

4

从 OEM 默认关闭级别获取的电源。有关性能和基准测试结果的更完整信息,请访问 intel.cn/Evo

5

基于集成的英特尔® Wi-Fi 6 (Gig+) 和 Thunderbolt™ 4 技术。有关性能和基准测试结果的更完整信息,请访问 intel.cn/Evo

6

参加英特尔笔记本电脑创新计划 Project Athena,通过高端规格和关键经验指标的验证、测量和测试。基于 2020 年 8 月的测试结果,且不能保证个别笔记本电脑的性能。功率和性能根据使用、配置和其他因素而有所不同。有关性能和基准测试结果的更完整信息,请访问 Intel.cn/Evo

7

性能因用途、配置和其他因素而异。请访问 www.Intel.cn/PerformanceIndex 了解更多信息。

性能结果基于截至配置中所示日期的测试,并且可能无法反映所有公开的更新。有关详细配置信息,请参阅 www.intel.cn/Evo

没有任何产品或组件能够做到绝对安全。

您的成本和结果可能会有所不同。

英特尔® 技术可能需要支持的硬件、软件或服务激活。