On May 21, Microsoft's Build 2024 Global Developer Conference was held in Seattle, USA. Microsoft CEO Satya Nadella delivered a keynote speech, focusing on the introduction of Windows and AI, and released more than 60 new products and solutions consecutively.

Firstly, Nadella proposed two core questions that run through the entire event:

1. In this era of information explosion, how can PCs help us utilize this information for quick reasoning, planning, and action?
2. Can computers actively understand us without us needing to understand computers?

"Copilot+PC" Preheat

On the eve of this conference, Microsoft held a preheating event on "Surface and Windows AI", unveiling the new generation Surface and a brand-new AI feature called Recall, officially announcing the debut of Copilot+PC.

It is understood that the new generation Surface is divided into two models: the seventh-generation Surface Laptop and the eleventh-generation Surface Pro. They are equipped with the new Qualcomm Snapdragon X Elite chip and Prism technology to fully transition to the ARM camp. Recall integrates Copilot's "retrospection" feature (learning, understanding, and reasoning abilities) to track user operations using large models, supporting the "replay" of relevant content and operations in a timeline format.

For Copilot+PC, its main OEM partners include AMD, Intel, and Qualcomm, as well as well-known manufacturers such as Acer, Asus, Dell, HP, Lenovo, and Samsung. The former three are responsible for manufacturing chips for Microsoft PCs, while the latter are Microsoft's device partners whose PC products are expected to be equipped with Microsoft's AI models.

Yusuf Mehdi, Microsoft's Corporate Vice President and Chief Marketing Officer for Consumer Goods, introduced that Windows 11 AI PC is the most powerful Windows PC ever, integrating a variety of groundbreaking AI experiences, and it is expected that the sales of this AI PC could reach 50 million units next year.

Nadella also stated, "Apple has excelled in many ways, and we are now looking forward to Windows and Mac engaging in a true showdown."

Copilot Product Line

It is obvious that Copilot holds a significant position at this developer conference: on the one hand, it can help individual users handle complex tasks more smoothly; on the other hand, Copilot makes team collaboration effortless.

Team Copilot

For professionals, Team Copilot makes Copilot more "personified". Copilot is no longer a mere "observer" but directly becomes a team member, directly callable in collaboration applications such as Teams, Loop, Planner, and others.

For example, Copilot can act as a meeting host, managing agendas and real-time recording of key points; or act as a collaborator, extracting important information and resolving lingering issues; or even serve as a project manager, driving team collaboration by creating and assigning tasks, tracking deadlines, and more... Its preview version will be released later this year.

Copilot Agent Proxy Functionality

With this feature, developers in Microsoft Copilot Studio can build Copilots that can actively respond to data and events based on specific tasks and functionalities. It can understand context through memory and knowledge, reason about operations and inputs, learn based on user feedback, and autonomously manage complex, long-running business processes, with the potential to evolve into fully automated AI agents.

Charles Lamanna, Corporate Vice President of Microsoft Business Applications and Platforms, said, "We quickly realized that a Copilot limited to conversation has very limited capabilities in today's context. Instead of letting Copilot wait for someone to chat with it, we'd rather make it more proactive in executing automation tasks in the background."

GitHub Copilot Extensions

During the conference, Nadella referred to GitHub Copilot as "the first popular product of this AI era." As one of the most widely adopted AI development tools, GitHub Copilot's subscription user base has exceeded 1.8 million.

Now, Microsoft has further collaborated with over 100 partners to launch GitHub Copilot Extensions, transforming from the previous norm of "code completion" to an efficiency-boosting tool—integrating all development processes through conversation, reducing context switching, and allowing developers to focus on core code content.

Whether it's voice or text input, whether it's Java or Python, there are no language restrictions. GitHub Copilot can provide developers with the code they need as long as the requirements are stated. Moreover, it can also answer questions about the development process and support various development tools and platforms.

Copilot Stack & Fabric Real-Time Intelligence

In 2023, Microsoft successfully built Microsoft Copilot and updated it over 150 iterations, and developed the Copilot Stack, giving developers greater freedom.

Building on this, Microsoft has created the Copilot Stack for developers this year, allowing them to build their own AI applications, solutions, and diverse experiences. According to reports, the Windows Copilot library contains over 40 edge AI models, including Windows-compatible APIs and algorithms.

Additionally, Nadella announced the launch of Real-Time Intelligence on Microsoft Fabric, an AI-driven analytics platform that provides organizations with real-time decision-making and SaaS services, helping data analysts gain simple low-code or no-code experiences and also benefiting professional developers through code-rich user interfaces.

GPT-4o & Phi-3-vision

As the largest investor in OpenAI, Microsoft also has priority access to all AI models developed by OpenAI.

Last week, OpenAI's latest multimodal model, GPT-4o, was trained on Azure and is now available as an API in Azure AI Studio, supporting multimodal inputs and outputs, providing more creative space for enterprise users and developers. Microsoft CTO Kevin Scott also joked that GPT-4o is about 12 times cheaper than the original model.

At the end of this grand event, OpenAI CEO Sam Altman made a surprise appearance and revealed that Microsoft is developing a supercomputer capable of hosting the high compute demands of GPT-5.

In addition to GPT-4o, Microsoft has released a new multimodal model, Phi-3-vision, as part of the Phi-3 series of AI small language models. Together with its predecessors Phi-3-mini and Phi-3-medium, Phi-3-vision, through Azure AI's MaaS product, is aimed at users.

It is understood that Phi-3-vision has audio and visual capabilities, can read text and analyze images, and its smaller scale (4.2 billion parameters) makes it suitable for mobile devices. However, unlike DALL-E and Stable Diffusion, Phi-3-vision does not generate images; it is primarily used to understand the content of images and provide analysis for users. This model is currently in the preview stage.

Expanding AI Collaborative Networks

With NVIDIA

Microsoft has announced its collaboration with NVIDIA to drive the digitalization of global manufacturing. Leveraging NVIDIA's Omniverse Cloud API on Microsoft Azure, the collaboration aims to bring crucial functionalities such as data interoperability, collaboration, and physically-based visualization to software used for designing, building, and operating industrial digital twin tools.

With Meta

Microsoft has announced the introduction of Windows Volumetric Apps to Meta Quest headsets, enabling developers to extend their applications into 3D space. This extension will allow users to stay within the applications that support their work dependencies while enhancing spatial understanding capabilities.

With Khan Academy

The focus of this diverse partnership is to leverage AI technology to support educational materials. Microsoft will provide Khanmigo for Teachers, an AI education assistant, free of charge to all K-12 educators in the United States, along with donating Azure AI-optimized infrastructure access.

Khan Academy, on the other hand, will explore economically viable, scalable, and adaptable ways to improve math tutoring using the latest version of Phi-3 developed by Microsoft. They also plan to integrate more Khan Academy teaching content into Copilot and Microsoft Teams Education Edition to provide more learning resources.

Other Highlights

Edge Real-time Video Translation

This feature will support real-time voice translation for mainstream websites such as YouTube, LinkedIn, Reuters, and Coursera, but only supports bidirectional translation for English, Hindi, German, Russian, Italian, and Spanish. Microsoft also stated that more languages and video platforms will be added in the future.

Teams Custom Emoji

In July, Microsoft Teams will fully roll out custom emoji functionality, allowing users to express themselves more creatively and authentically. Enterprise IT administrators will be able to restrict which users can upload or delete custom emojis or disable the feature entirely. Custom emojis will be visible only within the same organizational domain.

Windows 11‘s Advanced Paste

Reportedly, the "Advanced Paste" feature has been introduced in PowerToys version 0.81, allowing users to invoke the feature with "Windows+Shift+V". Once enabled, users can perform format conversions such as plain text, markdown, or JSON when pasting content.