Your browser does not support JavaScript!

Comparative Analysis of AI Chatbots: ChatGPT, Gemini, and Meta AI

GOOVER DAILY REPORT September 3, 2024
goover

TABLE OF CONTENTS

  1. Summary
  2. Subscription Offerings and Functionalities
  3. Performance Comparisons in Practical Tasks
  4. Comparison with Meta AI
  5. Conclusion

1. Summary

  • The report titled 'Comparative Analysis of AI Chatbots: ChatGPT, Gemini, and Meta AI' presents a comprehensive comparison among the leading AI chatbots, highlighting their performance, subscription offerings, and practical applications. The primary focus is on OpenAI’s ChatGPT, Google’s Gemini, and Meta AI, evaluating their strengths and weaknesses based on various tasks such as coding, natural language processing, creative writing, reasoning, ethical dilemmas, and translation. Key findings reveal that ChatGPT Plus excels in natural language processing and problem-solving, Gemini Advanced is superior in creative writing and ethical reasoning, and Meta AI performs reliably in tasks including email generation, recipe provision, and math problem solving. Each product’s unique offerings and subscriptions, such as Google’s integration with Google One and Microsoft's seamless integration with Office applications via Copilot Pro, are also discussed.

2. Subscription Offerings and Functionalities

  • 2-1. Google's Gemini Advanced

  • Google offers Gemini Advanced as a subscription service priced at $20 per month. This package includes access to Gemini Ultra 1.0, Google's best AI model, and additionally encompasses a Google One subscription which provides 2 terabytes of cloud storage. As part of this subscription, users can expect further integrations with Gmail and Docs in the future. A new Gemini model, Gemini Pro 1.5, that can process more data than the current version has been announced, but it is not yet available to the public.

  • 2-2. OpenAI's ChatGPT Plus

  • OpenAI's ChatGPT Plus, which is also offered at $20 per month, provides users with access to GPT-4 and Dall-E 3. While it does not include additional benefits like cloud storage, it features the GPT store, allowing users to build and share custom versions of ChatGPT optimized for various scenarios. This transition to ChatGPT Plus is designed to be seamless for existing users of ChatGPT.

  • 2-3. Microsoft's Copilot Pro

  • Copilot Pro from Microsoft is available for a subscription fee of $20 per month and offers unrestricted access to GPT-4 and Dall-E 3. Its primary distinction is the integration with Microsoft’s productivity software, enabling users to utilize AI tools directly within applications such as Excel, Outlook, and PowerPoint, provided they are also subscribers of Microsoft 365.

3. Performance Comparisons in Practical Tasks

  • 3-1. Coding Capabilities

  • Both ChatGPT (GPT-3.5) and Google Gemini (Gemini Pro 1.0) successfully created a fully functional Python program for tracking personal expenses based on the prompt. Gemini provided extra functionality, including adding labels within expense categories and offering more granular reporting options, which led to it being declared the winner in this category.

  • 3-2. Natural Language Processing

  • In the assessment of natural language understanding, ChatGPT excelled by clearly demonstrating its thought process when solving a Cognitive Reflect Test question involving a bat and a ball's total cost. While both chatbots provided the correct answer, ChatGPT's explanation was deemed to be clearer, resulting in it winning this comparison.

  • 3-3. Creative Writing

  • For the creative writing test, both chatbots were tasked with writing a short story about a futuristic city. Although both produced good stories, Gemini was recognized for better adherence to the rubric provided, leading to it being viewed as the superior output in this category.

  • 3-4. Reasoning and Problem-Solving

  • In a classic reasoning challenge about choosing between two doors guarded by a truth-teller and a liar, both AI models provided correct answers. However, ChatGPT offered more detailed explanations and clarity, thus winning this category.

  • 3-5. Ethical Dilemmas

  • When faced with a scenario involving ethical decision-making regarding autonomous vehicles, both AI systems discussed various considerations. While neither provided a personal opinion, Gemini's response demonstrated more nuanced ethical reasoning, making it the winner in this area.

  • 3-6. Explanation of Concepts to Children

  • In the task of explaining a complex concept to a five-year-old, both chatbots managed to simplify their responses effectively. However, Gemini's bullet-point format and practical experiment suggestion for children made its response more engaging, leading to its designation as the winner.

  • 3-7. Cross-Lingual Translation

  • Both AI tools were tested on their ability to translate a paragraph about Thanksgiving with a focus on cultural nuances. Gemini provided a more nuanced translation alongside an explanation of its approach, which led it to be recognized as the superior performer in this test.

4. Comparison with Meta AI

  • 4-1. Email Generation

  • All three chatbots, Meta AI, ChatGPT, and Google Gemini, successfully generated a well-written email requesting a project extension. Each generated email was polite and professional, allowing users to personalize the content with relevant information. This task was relatively simple, resulting in all three chatbots receiving perfect scores for their performance.

  • 4-2. Recipe Provision

  • When asked to provide a recipe for chili, all chatbots provided accurate recipes with minor variations. However, the significant difference lay in their sourcing methods. Meta AI and Gemini included proper sourcing at the bottom of their recipes, linking to the websites used. In contrast, ChatGPT did not provide any sources for its recipe, raising potential concerns regarding trustworthiness, especially for novice cooks relying on safe cooking instructions.

  • 4-3. News Summarization

  • Each chatbot was tasked with summarizing the latest news and produced a bulleted list quickly. However, the quality of sourcing varied significantly. ChatGPT and Meta AI linked to the actual news outlets they cited, thereby enhancing their reliability. In comparison, Gemini mentioned various news sites but failed to provide direct links, which could detract from its credibility in news sourcing.

  • 4-4. Math Problem Solving

  • The chatbots were presented with two sets of math problems, one focused on algebra and the other on geometry. Although all chatbots had different approaches to the algebra problem, they reached the same conclusion. For the geometry problem, only Meta AI correctly solved it, while ChatGPT and Gemini did not provide final answers. Thus, Meta AI emerged as the best option for solving math problems among the three.

  • 4-5. Programming Tasks

  • Each chatbot was asked to provide code for a complex variant of tic-tac-toe. Both Meta AI and ChatGPT responded with complete programming code in HTML and JavaScript as requested. However, Gemini provided JavaScript but substituted HTML with CSS, indicating a lack of understanding of the different roles of these languages. Therefore, for programming tasks, Meta AI and ChatGPT were the preferred choices.

  • 4-6. Mock Interviews

  • In the assessment of mock interviews, all three chatbots simulated interview scenarios effectively for a computing staff writer position. They generated mock questions and answers, providing a solid starting point for users to understand potential interview dynamics. Each bot approached this task with different styles but achieved satisfactory results overall.

  • 4-7. Information Sourcing and Citations

  • In various tasks, the effectiveness of each chatbot in sourcing information and providing citations was notably different. Meta AI and ChatGPT excelled in linking back to their sources, enhancing the credibility of the information provided. In contrast, Gemini often lacked proper citations, which raised concerns about its reliability and the traceability of the information it provided.

5. Conclusion

  • The findings underscore that each AI chatbot—Gemini Advanced, ChatGPT Plus, and Meta AI—has distinct capabilities that cater to different user needs. ChatGPT Plus demonstrates proficiency in natural language understanding and complex problem-solving, making it suitable for applications requiring detailed reasoning and explanations. Gemini Advanced stands out in creative storytelling and ethical dilemmas, indicating its strength in nuanced and engaging outputs. Meta AI, with its reliable information sourcing and accurate task execution, emerges as the most dependable option for routine tasks such as email generation and problem-solving in math. Despite these strengths, the report identifies limitations such as Gemini’s occasional lack of proper citations, which may affect credibility. Future research should aim to explore each tool's long-term performance across various domains and enhance their practical applicability. Users are advised to choose a chatbot that best aligns with their specific requirements, leveraging the unique strengths of each platform for optimal outcomes.

6. Glossary

  • 6-1. Gemini Advanced [Product]

  • An AI chatbot by Google that includes access to advanced AI models and a Google One subscription with 2TB of cloud storage.

  • 6-2. ChatGPT Plus [Product]

  • A subscription offering by OpenAI providing access to the advanced capabilities of GPT-4, including the GPT store for custom model versions.

  • 6-3. Copilot Pro [Product]

  • A Microsoft product that integrates AI capabilities smoothly with Microsoft Office applications for enhanced productivity.

  • 6-4. Meta AI [Product]

  • Meta's AI chatbot known for its reliability in generating emails, providing recipes, and solving math problems with proper information sourcing and citations.

7. Source Documents