From the author of this article: Li Dan

Source: Hard AI

One day after announcing the launch of Copilot's Windows 11 AI PC fully integrated into the Windows system, Microsoft also sent Copilot's multi-functional “gift package” to further promote the integration of advanced artificial intelligence (AI) assistants with Microsoft's business and partners.

At the annual Build Developers Conference, which opened on Tuesday, May 21, EST, Microsoft CEO Satya Nadella first mentioned Microsoft's two major dreams for decades: one is whether computers can understand humans rather than being forced to understand computers; the other is whether computers can help humans reason, plan, and use all information more effectively in a world where information is increasing. He believes that the current wave of AI can help Microsoft find answers to these two major dreams.

Nadella then focused on Copilot. He mentioned the Windows Copilot Library, a Copilot library with ready-to-use local APIs, saying that there are more than 40 end-side AI models “out of the box,” showing how the new PC Copilot+ PC, which fully integrates Copilot into a personal computer (PC), can be closely integrated with AI applications (apps).

On Monday, Microsoft introduced the Windows Copilot Runtime, which includes more than 40 AI models. On Tuesday, Nadella compared Windows Copilot Runtime to the blockbuster Windows API Win32 because it makes it easier for developers to use AI to build their own Windows apps.

Azure AI is now combined with GPT-4o Azure will be the first to use Nvidia Blackwell AI chips

Nadella then mentioned AI infrastructure. He said that Microsoft has “the most comprehensive artificial intelligence infrastructure in the world,” and it is expected that 100% of its data centers will be powered by renewable energy by 2025, and the number of countries where Microsoft Cloud Azure's AI services have been implemented has tripled.

He said Microsoft will be one of the first cloud service providers to provide Nvidia's latest Blackwell AI chips and has established a “deep partnership” with Nvidia.

Today, more than 50,000 organizations use Azure AI, according to Nadella. With Azure AI, Microsoft offers the broadest selection of cutting-edge and open source models, including the Big Language Model (LLM) and the Small Language Model (SLM).

Microsoft said on Monday that the latest flagship model GPT-4O released by OpenAI last week will soon be able to support Copilot. On Tuesday, Microsoft announced that GPT-4O, a multi-modal model that integrates text, image, and audio processing, can now be used in Azure AI Studio and also as an API.

Nadella revealed at the developer conference that GPT-4O has been integrated into Azure AI, saying that OpenAI's GPT-4O demonstration last week was “amazing” and that OpenAI is Microsoft's “most strategic and most important partner.”

Microsoft said that on the same day that OpenAI officially announced GPT-4O, Microsoft opened the model for testing on the Azure OpenAI service. This Tuesday, Microsoft announced that it had obtained full GPT-4O API access.

On social media X, some netizens were excited about this. One highly praised comment saying, “This is a brand new world, hello to Copilot.”

Some netizens called for showing native apps for writing Python code, etc. If you want to be able to talk and encode/copy content at the same time, and you don't need to make any special commands or press any buttons, just like someone is in the same room as you.

Azure AI can be used as a small text-to-image model PHI-3-Vision OPPO's new phone is testing Azure AI text-language interchange technology

On Tuesday, the AI SLM Phi-3 series developed by Microsoft released a new multi-modal small model, PHI-3-Vision, which can be used in Azure.

According to Microsoft, the Phi-3 model is powerful, affordable, and optimized for personal devices. PHI-3-Vision has 4.2 billion parameters, can input images and text, and receive text responses. For example, users can ask questions about diagrams or ask open-ended questions about specific images.

Microsoft also revealed that Chinese mobile phone manufacturer OPPO is piloting Azure AI speech-to-text, rapid transcription, and Azure AI text-to-speech technology on its new smartphones.

Team Copilot turns personal assistants into team meeting hosts, chat collaborators, and project managers

Microsoft has launched a variety of Copilot's AI feature extensions. One of the major extensions announced on Tuesday is Team Copilot.

As the latest expansion of the Copilot series of generative AI technology, Team Copilot expanded Copilot for Microsoft 365 from a personal AI assistant behind the scenes to a new important member of the team. You'll be able to invoke Copilot in collaboration tools like Teams, Loop, and Planner.

Team Copilot can act as a meeting host, manage the meeting agenda, track meeting time, and record meeting highlights; it can also act as a collaborator in a chat to provide key information, track action items, and resolve outstanding issues. It can also act as a project manager, help ensure each project progresses smoothly, and keep the team informed for input. These initial experiences will be available in preview later this year.

According to a comment on social media X, the Copilot extension looked interesting, and Microsoft previewed some features apparently copied from ChatGPT: Builder UI, custom functions, GPT Mentions.

Copilot Studio upgrades user-customizable AI agents

Microsoft Copilot Studio was upgraded and a new agent agent function was introduced, enabling developers to build Copilots that actively respond to data and events according to specific tasks and functions.

Based on these new features, Copilot independently manages complex, long-running business processes by understanding context, reasoning operations, and input through memory and knowledge, learning based on user feedback, and seeking help when it doesn't know how to handle it.

From purchasing IT equipment to customer reception for sales and service, users can make Copilot built on Copilot Studio work for them. Before deployment, users can test the Copilot AI agent of their own design and then use it on multiple channels.

GitHub Copilot extensions to customize the Copilot experience

As one of Microsoft's first major applications of generative AI, GitHub Copilot currently has 1.8 million paying users. GitHub launched the first set of GitHub Copilot extensions developed by Microsoft and third-party partners this Tuesday, and an invited preview is now open.

The GitHub Copilot extension is used to create an intelligent third-party ecosystem. Its new features allow developers and businesses to customize their GitHub Copilot experience directly in GitHub Copilot Chat through their preferred services such as Azure, Docker, and Sentry.

GitHub Copilot for Azure shows how to use natural language and a wider range of features to speed up development. Using the extension through Copilot Chat, developers can explore and manage Azure resources while troubleshooting and finding relevant logs and code.

Edge can watch videos on websites such as YouTube and Bloomberg in real time in English, German, Italian, Spanish, and Hindi

On Tuesday, Microsoft announced a new AI feature in the browser that allows users to obtain translated content in real time through dubbing and subtitles when watching videos. Microsoft's Edge browser will “soon” provide real-time video translation on websites such as YouTube, LinkedIn, and Coursera.

The feature will support translation from Spanish to English, as well as English to German, Hindi, Italian, Russian, and Spanish. Edge will also support real-time translation of videos from news sites such as Reuters, CNBC, and Bloomberg. Microsoft plans to add more language translations and expand the range of supported websites in the future.

Media reviews say that in addition to providing a simple way to translate videos into users' native languages, Edge's new AI feature should also make it easier for deaf or hearing-impaired users to watch videos.

Some netizens commented that if this kind of real-time video translation feature were introduced, it would definitely be worth paying attention to. The question is how accurate and fluent the translation is, and how to preserve the tone and tone of the original speaker.

The preview version of the first Arm-based Azure Cobalt virtual machine was officially released, Azure ND MI300X v5 virtual machine

In November of last year, Microsoft released the first self-developed AI chip Azure Maia 100 and the first self-developed cloud processor Azure Cobalt 100. On Tuesday, Microsoft announced that it will launch a preview version of the Cobalt 100 virtual machine (VM) customized based on the Arm architecture based on the Azure Cobalt 100. It is the first generation virtual machine using Microsoft's new Cobalt processor, optimized for efficiency and performance when running general purpose workloads and cloud-native workloads.

According to Nadella, compared to Azure virtual machines of the same type, the Arm-based Cobalt 100 virtual machine can improve performance by up to 40%. It is the most efficient computing product launched by Microsoft Azure.

Rani Borkar, head of Microsoft Azure hardware systems and infrastructure, also told the media that initial tests showed that the performance of Azure Cobalt was up to 40% higher than the performance of the commercial Arm servers currently used in its data centers. According to the media, Microsoft is currently testing Cobalt CPUs on workloads such as Microsoft Teams and SQL Server, and plans to provide customers with virtual machines to handle various workloads next year.

Maia 100 is currently being tested on GPT 3.5 Turbo, which is also a model that supports ChatGPT, Bing AI workloads, and GitHub Copilot. Similar to the Cobalt CPU, Microsoft is currently in the early stages of deployment of the Maia 100 and has not disclosed its specific specifications and performance.

Microsoft is the first cloud service provider to use AMD's MI300X AI accelerator chip for customer AI training and AI inference. On Tuesday, Microsoft announced that the Azure ND MI300X v5 virtual machine series has been officially released, optimized for high-demand AI and high-performance computing (HPC) workloads such as Azure OpenAI Service.

Data analysis platform Microsoft Fabric launches real-time intelligence

Microsoft's AI-enabled data analysis platform Microsoft Fabric has also released new features. Users of the platform can make immediate decisions using real-time intelligence (Real-Time Intelligence) and provides an end-to-end software-as-a-service (SaaS) solution, enabling customers to act on massive, time-sensitive, and high-granular data to make business decisions faster and more intelligently.

Real-time intelligence is now in preview. It not only helps data analysts get a simple low-code/no-code experience, but also benefits professional developers through a code-rich user interface.

Microsoft Fabric has launched a workload development kit that enables independent software vendors (ISVs) and developers to extend applications within Fabric and create a unified user experience.

Microsoft partners with Khan Academy to provide free AI assistant Khanmigo to American primary and secondary school teachers

On Tuesday, Microsoft and Khan Academy (Khan Academy) announced a multi-faceted partnership aimed at turning AI's transformative potential into reality.

At the beginning of the partnership, Microsoft will assist Khan Academy in providing free Khanmigo for Teachers to all K-12 educators equivalent to domestic primary and secondary education in the US. This AI-powered educational assistant frees up teachers' time and allows them to focus on what matters most — engaging and supporting students. Microsoft will also donate Azure AI-optimized infrastructure rights to improve the usability of Khanmigo for Teachers. The service will be powered by Azure OpenAI Service.

Khan Academy will also work with Microsoft to explore ways to improve math tutoring in an affordable, scalable, and adaptable way using the latest version of the SLM series PHI-3 developed by Microsoft. They also plan to introduce more Khan Academy teaching content to Copilot and Microsoft Teams education editions to provide more resources for learners.