Your browser does not support JavaScript!

Advancements and Collaborations in AI Technologies and Large Language Models

GOOVER DAILY REPORT July 5, 2024
goover

TABLE OF CONTENTS

  1. Summary
  2. Partnerships and Collaborations
  3. Developments in Large Language Models (LLMs)
  4. Technological Advancements and Innovations
  5. Competitive Landscape and Strategic Investments
  6. Microsoft's AI Initiatives and Competitive Dynamics
  7. Conclusion

1. Summary

  • The report titled 'Advancements and Collaborations in AI Technologies and Large Language Models' explores recent developments and partnerships in the AI sector, with a particular focus on large language models (LLMs). Major AI companies such as Dassault Systèmes, Mistral AI, Naver, Microsoft, Google DeepMind, and Nvidia are keenly analyzed. Key highlights include the Dassault Systèmes and Mistral AI partnership aimed at delivering AI-powered industry solutions and Naver's strategic investment in Mistral AI to promote 'Sovereign AI.’ The report also discusses the advancements in large language models by Google DeepMind and Mistral AI, Nvidia's technological contributions, SAP’s AI innovations, significant funding efforts by Lambda Labs, and the competitive dynamics among leading tech firms influencing the AI landscape.

2. Partnerships and Collaborations

  • 2-1. Dassault Systèmes and Mistral AI partnership

  • On July 1, 2024, Dassault Systèmes and Mistral AI announced their partnership aimed at delivering AI-powered industry-grade solutions. This collaboration leverages Dassault Systèmes' expertise in scientific modeling, simulation, and AI with Mistral AI's advanced large language models (LLMs). Mistral AI's enterprise-grade 'Large' model provides a balanced combination of accuracy, responsiveness, and sustainability, which aligns with Dassault Systèmes' requirements. The partnership aims to offer solutions that empower the workforce of the future, enhance industry knowledge, and maintain intellectual property security. The collaboration underscores the commitment of both companies to drive the use of generative AI in a trusted, sovereign environment, reinforcing high standards of performance, efficiency, security, and privacy.

  • 2-2. Naver’s investment in Mistral AI

  • Naver invested in Mistral AI through Corelia Capital's Europe-specific 'K-Fund 2', positioning itself as a significant ally in the Sovereign AI strategy. Mistral AI is recognized as a strong contender in the global AI market, which is predominantly dominated by U.S. companies like OpenAI and Google. This investment aims to accelerate Naver’s strategy of developing AI models tailored to the cultural and linguistic specifics of different regions. Mistral AI, known for its AI sovereignty mission, aligns with Naver’s goal of preventing a monopolistic 'Big Brother' scenario in AI. Significant support from French authorities, including President Emmanuel Macron, highlights Mistral AI's role as a pillar of European AI. The partnership promises potential collaborations beyond equity investment, potentially increasing the value of Naver’s stake as Mistral AI's global influence grows.

3. Developments in Large Language Models (LLMs)

  • 3-1. Overview of large language models

  • A large language model (LLM) is a computational model notable for its ability to achieve general-purpose language generation and other natural language processing tasks such as classification. LLMs acquire these abilities by learning statistical relationships from vast amounts of text during a self-supervised and semi-supervised training process. Utilizing the transformer architecture, invented in 2017, LLMs are built to process and generate large-scale text data efficiently. Notable LLMs include OpenAI's GPT series, Google's Gemini, Meta's LLaMA, Anthropic’s Claude models, and Mistral AI's models. Historically, fine-tuning was the primary method for specific tasks, but larger models like GPT-3 have shown the effectiveness of prompt engineering for achieving similar results. However, LLMs also inherit inaccuracies and biases from the data they are trained on.

  • 3-2. Google DeepMind's Gemma 2 evaluation

  • On June 27, 2024, Google DeepMind introduced the Gemma 2 model, available in 9 billion (9B) and 27 billion (27B) parameter sizes. These models offer high performance and efficiency, rivaling much larger closed-source models. The 27B variant competes with models more than twice its size, while the 9B model outperforms other models in its category like Llama 3 8B. Gemma 2 models are optimized for full precision inference, requiring significantly reduced deployment costs. These models are available under a commercially friendly license and are compatible with major AI frameworks. Gemma 2's evaluation showed exceptional performance in multilingual understanding, outperforming both open-source and several closed-source models such as GPT-3.5-turbo, Google’s Gemini, and Anthropic’s Claude. The 27B model sets a new standard for self-hosting multilingual LLMs.

  • 3-3. Mistral AI's large language models

  • Mistral AI, a French startup initiated by former employees of Google DeepMind and Meta, has been influential with its advanced AI solutions. In September 2023, Mistral introduced the 7B model, outperforming the best open-source models like Llama 2. Mistral offers two model types: open-weight models and commercial models. The Mistral Large model, released in February 2024, is tailored for text generation tasks and features strong reasoning skills, multilingual capabilities, and in-depth programming knowledge. Mistral Large performs well on benchmarks such as Arc Challenge, MMLU, HellaSwag, MBPP, HumanEval, and GSM8K. Unique features include an extended 32K token context window and inherent support for function calling. Mistral models excel in efficiency, cost-effective deployment, and high performance, making advanced AI more accessible.

4. Technological Advancements and Innovations

  • 4-1. Nvidia's advancements and tools

  • Nvidia has made significant advancements with several tools and models. The Deepseek Coder v2, available as an NVIDIA NIM microservice, enhances project-level coding and infilling tasks. Additionally, the NVIDIA NeMo T5-TTS model advances text-to-speech technology by addressing hallucinations in speech synthesis LLMs. The H100 Tensor Core GPUs combined with TensorRT-LLM achieve high Mixtral 8x7B performance. Security for large language models has been advanced with Edgeless Systems’ Continuum AI framework, which keeps prompts encrypted at all times. The Phi-3-Medium model available on the NVIDIA API Catalog accelerates research with its logic-rich features. For code generation, summarization, and documentation, the StarCoder2-15B, trained on 600+ programming languages, is now a NIM inference microservice. Brev.dev and the NVIDIA NGC Catalog facilitate deploying GPU-optimized AI software with one click. Google’s Gemma 2 model has also been optimized with TensorRT-LLM and is now available as an NVIDIA NIM inference microservice. The DoRA model provides a high-performing alternative to LoRA for fine-tuning tasks. Additionally, robust and scalable RAG (Retrieval Augmented Generation) applications can be created using NVIDIA NIM and Haystack on Kubernetes. The NVIDIA Nemotron-4 340B model supports synthetic data generation, and new Grouped GEMM APIs in the cuBLAS library offer enhanced performance for deep learning and high-performance computing tasks. Lastly, Nvidia has set new generative AI performance and scale records in MLPerf Training v4.0.

  • 4-2. SAP’s AI innovations

  • SAP has introduced SAP GenAI HUB and SAP JOULE, focusing on innovations related to AI. The SAP JOULE system integrates company data from SAP systems and Microsoft applications using Retrieval Augmented Generation (RAG). This process, also known as 'grounding,' enhances the context provided in AI applications. For detailed technical explanations on large language models (LLMs), trainings and resources are available, including an introductory video on YouTube.

5. Competitive Landscape and Strategic Investments

  • 5-1. Lambda Labs' Funding Efforts

  • According to a Financial Times report, Lambda Labs, a Silicon Valley startup specializing in AI infrastructure, is engaging in discussions to raise $800 million. This move aims to address the escalating demand for AI computing, driven largely by the surge in generative AI. The proposed funding would significantly elevate Lambda Labs' valuation, positioning it among the best-funded startups in Silicon Valley. The funds will be utilized to acquire more Nvidia GPUs and associated cloud networking software, as well as to hire additional staff. Lambda Labs has already secured a $500 million loan backed by Nvidia Corp chips to expand its cloud services. The startup's close relationship with Nvidia is pivotal, as Nvidia's market value has recently reached $3 trillion. Previous funding rounds for Lambda Labs include $44 million raised in March 2023 with support from notable names such as Quora founder Adam D'Angelo and OpenAI president Greg Brockman.

  • 5-2. AI's Influence on the Nasdaq-100 Index

  • The proliferation of artificial intelligence (AI) has significantly impacted the Nasdaq-100 index, which comprises the 100 largest non-financial companies listed on the Nasdaq stock exchange. In 2024, five major AI-focused companies—Microsoft, Apple, Nvidia, Alphabet, and Amazon—accounted for 35.9% of the entire index's value. These companies have delivered an average return of 47.3% year to date, largely driving the Nasdaq-100 index's 18.9% rise as of mid-2024. This is in stark contrast to the Nasdaq-100 Equal Weighted index, which has seen a much smaller increase of 6.2%. The difference in performance is due to the substantial influence these five companies exert on the index. Each company has committed substantial resources to AI, which is expected to continue shaping their performance and influence over the Nasdaq-100.

6. Microsoft's AI Initiatives and Competitive Dynamics

  • 6-1. Microsoft and OpenAI Collaborations

  • Microsoft has secured an observer role on OpenAI’s board, similar to Apple. In a landmark AI deal, Apple also managed to secure an observer role through Phil Schiller, the head of Apple’s App Store. As a board observer, Schiller won't possess voting rights but will gain insights into OpenAI's decision-making processes. This role is expected to start later this year. The board arrangement might lead to new collaborations, but also potential tensions, as Microsoft's significant investments in OpenAI allow it to stay closely involved in these discussions. Microsoft's observer role aims to foster deeper integration and collaboration in advancing AI technologies.

  • 6-2. Microsoft AI Unit Chief on Competition with OpenAI

  • Microsoft AI CEO Mustafa Suleyman has acknowledged the competitive nature between Microsoft and OpenAI. Despite their deep partnership, Suleyman highlighted that OpenAI operates independently and is not controlled by Microsoft, emphasizing that the company does not have any board members at OpenAI. He addressed the intensity of the competition during an interview at the Aspen Ideas Festival, mentioning that while they compete 'ferociously', there is mutual respect and trust between the companies. Suleyman also advocated for regulated and measured AI development, valuing meaningful dialogue among technologists even if it may slow down AI progress by up to 18 months.

7. Conclusion

  • The report underscores significant advancements and strategic collaborations driving the AI and large language model (LLM) sectors. Dassault Systèmes' partnership with Mistral AI, and Naver's investment in Mistral AI highlight important strides toward region-specific AI solutions and ethical considerations in AI. Technological contributions from companies like Nvidia and SAP further showcase the fast-evolving landscape. The competitive interactions between giants like Microsoft and Google DeepMind, alongside strategic investments by startups like Lambda Labs, reveal a dynamic and robust competitive environment. It emphasizes the need for continued collaboration, ethical practices, and a balanced approach to competition and strategic alliances. The findings suggest promising future prospects where AI technologies will continually evolve, potentially offering even broader applications and improving efficiency and productivity in various sectors. However, challenges such as managing biases in AI models and ensuring privacy and security need to be persistently addressed.

8. Glossary

  • 8-1. Dassault Systèmes [Company]

  • A French software company that has partnered with Mistral AI to offer AI-powered industry-grade solutions for a generative economy, focusing on high performance, efficiency, security, and privacy.

  • 8-2. Mistral AI [Company]

  • An AI company noted for its large language models and partnerships with Dassault Systèmes and Naver. It aims to provide AI solutions tailored to specific cultures and languages and has received support from European leaders.

  • 8-3. Naver [Company]

  • A South Korean online platform developer that has invested in Mistral AI to promote ‘Sovereign AI,’ focusing on building AI models unique to different regions.

  • 8-4. Nvidia [Company]

  • An American technology company that leads in AI GPU development and offers various AI tools and technologies. Nvidia’s advancements continue to push the boundaries of AI innovation.

  • 8-5. Google DeepMind [Company]

  • A subsidiary of Google that has developed Gemma 2, an open-source multilingual language model that excels in performance and efficiency.

  • 8-6. Lambda Labs [Company]

  • A Silicon Valley startup in AI computing, currently seeking to raise additional funding to expand its AI infrastructure and meet the growing demand for Nvidia's GPUs.

  • 8-7. Microsoft [Company]

  • An American technology company heavily involved in AI development, particularly through collaborations with OpenAI. It offers various AI services and competes fiercely in the AI market.

9. Source Documents