Microsoft Releases Phi-2

PLUS: The EU AI Act

Author

AI For All
December 15, 2023

Hello readers,

Welcome to another edition of This Week in the Future! Small, open-source AI models are becoming more powerful. Microsoft released Phi-2 to the public, which outperforms Meta’s Llama 2, and French AI startup Mistral released Mixtral 8×7B. Plus, the EU AI Act is finally coming together.

As always, thanks for being a subscriber! We hope you enjoy this week’s content — for a video breakdown, check out the episode on YouTube.

Let’s get into it!

David and Goliath

Generated by DALL·E 3

In the realm of artificial intelligence, smaller language models are beginning to demonstrate capabilities that defy the conventional belief that bigger is always better. Microsoft's Phi-2 and Mistral AI's Mixtral 8x7B are prime examples of this trend, showcasing impressive capabilities despite their relatively smaller sizes, especially when compared to giants like Meta's Llama 2.

Microsoft's Phi-2 Model

Microsoft's Phi-2 is a standout in the small language model (SLM) category. With 2.7 billion parameters, it is a notable reduction compared to larger models like GPT-4, which boasts 1.7 trillion parameters. Despite its smaller size, Phi-2 demonstrates remarkable capabilities in various domains, including common sense reasoning, language understanding, math, coding, and specific benchmarks like Bigbench-hard.

A key aspect of Phi-2's effectiveness is its high-quality training data. Microsoft focused on "textbook-quality" data designed to impart knowledge, complemented by techniques to transfer learned insights from smaller to larger models. This approach allows Phi-2 to not only match but in some instances outperform models up to 25 times its size, such as the 7B Mistral, 13B Llama-2, and even 70B Llama-2 in select benchmarks.

Remarkably, Phi-2 achieves its strong performance without undergoing alignment techniques like reinforcement learning from human feedback or instructional fine-tuning. Yet, it still demonstrates superior safety with regard to mitigating toxicity and bias compared to other available open-source models that did utilize alignment. This achievement highlights the potential of developing capable yet safer models through data selection alone.

Mistral AI's Mixtral 8x7B

Mistral AI's Mixtral 8x7B model represents a significant advancement in the realm of small language models. This model is characterized by its "mixture of experts" approach, which integrates various sub-models, each specializing in different tasks. This design has led to its impressive performance, equating or surpassing other notable models like GPT-3.5 and Llama 2 in various benchmarks.

Performance: Mixtral matches or outperforms other high-profile models like Llama 2 70B and GPT-3.5 on most benchmarks. Notably, it exhibits more truthful responses and presents less bias on certain benchmarks. It also demonstrates strong performance in code generation and supports multiple languages including English, French, Italian, German, and Spanish.

Instructed Models: Alongside the standard Mixtral 8x7B, there is also Mixtral 8x7B Instruct, which has been optimized through supervised fine-tuning and direct preference optimization for careful instruction following. On MT-Bench, it reaches a score of 8.30, making it comparable to GPT-3.5 in performance.

Adaptability and Flexibility: Mixtral Instruct shows remarkable performance on industry standards such as MT-Bench and AlpacaEval. It excels in the instruct and chat model domain, asserting its dominance despite having a smaller scale compared to some models with higher parameter counts.

Why This Matters

If models can be competitive with less parameters simply due to higher quality training data, this could make models more energy-efficient and easier to deploy. With smaller, more capable LLMs that are also open source, more businesses can consider a native implementation of AI into their products and services.

The EU AI Act

Generated by DALL·E 3

The European Union's AI Act represents a significant step in regulating artificial intelligence, aiming to ensure that AI is safe, respects fundamental rights, and fosters innovation. Here's a breakdown of just some of the key aspects and implications:

Risk-Based Classification of AI Systems: The AI Act classifies AI systems according to the risk they pose to users. Systems are analyzed and categorized into different levels, each subject to varying degrees of regulation.

Bans on Certain AI Applications: The Act prohibits specific uses of AI that are deemed to threaten citizens' rights and democracy. This includes biometric categorization systems that use sensitive characteristics, scraping of facial images to create recognition databases, emotion recognition in workplaces and educational institutions, social scoring based on personal characteristics, AI systems manipulating human behavior, and AI used to exploit vulnerabilities of people.

Law Enforcement Exemptions: There are narrow exceptions for law enforcement's use of biometric identification systems, subject to judicial authorization and specific conditions. This includes targeted searches for serious crimes such as terrorism and human trafficking.

Obligations for High-Risk AI Systems: High-risk AI systems, due to their significant potential harm, must adhere to strict obligations. These include mandatory fundamental rights impact assessments, applicable to sectors like insurance and banking. Systems influencing elections and voter behavior are also classified as high-risk.

General AI Systems and Foundation Models: General-purpose AI systems, and the models they are based on, must comply with transparency requirements, including technical documentation and compliance with EU copyright law. For high-impact models, there are additional obligations like model evaluations, systemic risk assessments, and reporting on cybersecurity and energy efficiency.

Sanctions and Implementation Timeline: Non-compliance can lead to significant fines. The Act stipulates a two-year implementation period for tech companies, with a shorter timeline for the application of certain bans.

🔥 Rapid Fire

🎙️ The AI For All Podcast

This week’s episode featured Jerome Pesenti, the CEO and founder of Sizzle AI and former VP of AI at Meta, who discussed AI's enormous potential to disrupt education by providing personalized learning experiences. Jerome also shared how Sizzle acts as an AI tutor, adapting to student's interests and providing real-time assistance.

📖 What We’re Reading

This week’s handpicked content includes insightful articles on the benefits of applied AI as well as how you can start developing fitness and health monitoring applications with AI. Plus, Qualcomm shares 6 consumer tech trends for 2024.

The Benefits of Applying AI to Complex Industries (link)

“Enterprises and organizations that have become early adopters of Applied AI are already seeing the benefits and the huge potential for future applications across their operations. Some of the greatest initial innovations are taking place in sectors such as oil and gas production, healthcare, and manufacturing, across a wide variety of applications including predictive maintenance, optimization of operational processes, and enhanced safety practices.”

Source: AI For All
AI Technologies Used in Smart Fitness Applications (link)

“AI has great potential in fitness software. However, incorporating this technology into your fitness applications can be confusing, especially if you're new to working with such projects. Today, we'll provide a summary of what AI technology is critical to the development of fitness applications, along with some real-world examples.”

Source: AI For All
The 6 most important consumer tech trends for next year (link)

“The generative AI conversation in 2023 was predominantly about the cloud, but privacy, latency and cost will increasingly be choke points that on-device AI capabilities can help solve. As generative AI becomes more integrated in our lives, our personal devices like our smartphones, PCs, vehicles, and even IoT devices will become the hubs for multi-modal generative AI models.”

Source: Qualcomm OnQ

💻️ AI Tools and Platforms

  • Armada → Bringing AI to the edge

  • Braintrust Data → Rapidly ship AI products

  • GoSearch → AI-powered enterprise search and discovery

  • Lightning AI → All-in-one AI development cloud platform

  • Citrusˣ → Make AI/ML models transparent and explainable