Your browser does not support JavaScript!

Comparative Analysis of AI Chatbots: Gemini, ChatGPT, and Meta AI

GOOVER DAILY REPORT September 12, 2024
goover

TABLE OF CONTENTS

  1. Summary
  2. Overview of AI Chatbot Subscriptions
  3. Performance Comparisons
  4. Task-Specific Evaluations
  5. Conclusion

1. Summary

  • This report presents a detailed comparison of three prominent AI chatbots: Google's Gemini, OpenAI's ChatGPT, and Meta AI. The objective is to provide a comprehensive understanding of each chatbot's strengths and weaknesses based on their functionalities, subscription benefits, performance across various tasks, and overall reliability. Key findings indicated that Gemini Advanced and ChatGPT Plus both offer subscriptions at $20 per month, each with unique supplementary features. Performance assessments revealed that Gemini excelled in creative text generation, ethical reasoning, and cross-lingual translation, while ChatGPT provided superior explanations in natural language understanding and was commendable in coding proficiency. Meta AI was particularly noted for its reliable performance in email writing, recipe creation, math problem-solving, and programming tasks. Each chatbot showed competence in generating professional emails and useful interview responses.

2. Overview of AI Chatbot Subscriptions

  • 2-1. Subscription Costs and Features of Gemini and ChatGPT

  • As of February 2024, both Google’s Gemini Advanced and OpenAI’s ChatGPT Plus are available for $20 per month. Gemini Advanced offers access to Google’s AI model, Gemini Ultra 1.0, and includes additional perks through a Google One subscription, providing 2 terabytes of cloud storage. In contrast, ChatGPT Plus focuses on delivering a powerful chatbot experience with access to GPT-4 and Dall-E 3, but lacks supplementary features like cloud storage.

  • 2-2. Comparison of Advanced and Free Versions

  • For the average user, both Gemini and ChatGPT provide free versions that are effective for tasks such as writing emails and casual content generation. However, users with specific needs, like coding or utilizing advanced AI capabilities, may find the paid subscriptions worthwhile. The free versions are competent and far exceed the capabilities of older tools.

  • 2-3. Additional Benefits of Gemini Advanced and ChatGPT Plus

  • Gemini Advanced not only provides advanced AI features but also integrates future functionalities such as Gemini features for Gmail and Docs. ChatGPT Plus, while lacking in supplemental benefits, has an innovative GPT store for users to create and share custom bot versions optimized for different tasks.

  • 2-4. User Guidance on Subscription Choices

  • Users are advised to consider their specific needs before deciding on a subscription. If basic functionalities meet their requirements, the free versions are quite sufficient. For those looking to explore advanced features or have specialized tasks, Gemini Advanced and ChatGPT Plus subscriptions may offer enhanced capabilities that justify their monthly cost.

3. Performance Comparisons

  • 3-1. Coding Proficiency: Gemini vs ChatGPT vs Meta AI

  • In a head-to-head coding proficiency test, both Gemini and ChatGPT were tasked with creating a Python program that functions as a personal expense tracker. The prompt required functionalities to allow users to input expenses along with categories and dates. While both chatbots produced working solutions, Gemini's version included additional functionality, such as labels within a category and more granular reporting options. Therefore, Gemini was declared the winner in this category based on its superior performance.

  • 3-2. Creative Text Generation: Analysis of Chatbot Capabilities

  • For testing creative text generation, both chatbots were prompted to write a short story set in a futuristic city that explores themes of freedom and dependence. Both entries were deemed good but Gemini's story was regarded as better for its adherence to thematic elements and overall narrative quality. This led to Gemini being awarded the win in this category.

  • 3-3. Natural Language Understanding and Reasoning Tasks

  • In assessing natural language understanding, both chatbots were presented with a Cognitive Reflect Test question concerning the pricing of a bat and a ball. Both provided the correct answer; however, ChatGPT's explanation was clearer, leading it to win this segment.

  • 3-4. Ethical Decision-Making and AI Consensus

  • In the ethical reasoning test, both Gemini and ChatGPT faced a scenario involving an autonomous vehicle deciding between two harmful actions. Neither chatbot provided a definitive opinion but outlined considerations that could guide decision-making. Nonetheless, Gemini's response exhibited a more nuanced understanding, securing its victory.

  • 3-5. Cross-Lingual Translation Strengths

  • When tested on cross-lingual translation, Gemini outshone ChatGPT by translating a paragraph about Thanksgiving into French, presenting additional nuances and a detailed explanation of its translation process. Thus, Gemini was recognized as the stronger performer in this area.

4. Task-Specific Evaluations

  • 4-1. Email Writing and Recipe Creation Performance

  • All three chatbots, Meta AI, ChatGPT, and Google Gemini, performed exceptionally well in generating emails, successfully crafting professional and polite requests for work project extensions. Each generated well-structured emails suitable for personalization. In terms of recipe creation, when prompted to provide a recipe for chili, all chatbots delivered accurate and thorough recipes. However, Meta AI and Gemini differentiated themselves by sourcing their recipes with links, while ChatGPT did not provide any sources, raising concerns about potential plagiarism and reliability in food safety.

  • 4-2. News Summarization and Math Problem-Solving Accuracy

  • For news summarization, each chatbot was tasked with generating a bulleted list of the latest news. They managed to deliver content relatively quickly but primarily copied headlines with minimal context. Meta AI and ChatGPT excelled by linking to their sources, unlike Google Gemini, which only mentioned various news sites without direct links. In the math problem-solving task, all chatbots were presented with two challenges—one algebraic and one geometric. Meta AI was the only one that accurately solved the problems, with ChatGPT failing to provide a final answer for one, and Gemini offering a theoretical answer without inserting numeric values.

  • 4-3. Programming Capabilities and HTML Submission

  • When given a programming challenge to create a 12-by-12 tic-tac-toe game in HTML and JavaScript, both Meta AI and ChatGPT provided complete and correct code in the required languages. In contrast, Google Gemini produced JavaScript code but mistakenly opted for CSS instead of HTML, indicating a significant error in fulfilling the prompt requirements.

  • 4-4. Mock Interviews: Effectiveness and Response Quality

  • In the mock interview scenario, where each chatbot simulated an interview for a role as a computing staff writer, all three demonstrated adequate performance by generating relevant questions and answers. This approach effectively served as a useful starting point for potential interviewees, showcasing their capability to provide insightful practice.

5. Conclusion

  • The comparative analysis highlights the diverse capabilities of the three AI chatbots—Google Gemini, OpenAI's ChatGPT, and Meta AI—each showing unique strengths. Meta AI emerged as the most reliable, especially excelled in math problem-solving and programming tasks, demonstrating overall accuracy and reliability. OpenAI's ChatGPT stood out for its coding proficiency and natural language reasoning, making it highly suitable for technical tasks. Google's Gemini, while feature-rich and powerful in certain areas like ethical reasoning and nuanced translations, displayed inconsistencies that need addressing. These observations underscore the significance of understanding each chatbot's specialty when making subscription choices. Future improvements should target enhancing Gemini's consistency and expanding ChatGPT’s supplementary features. These findings suggest that the ideal chatbot depends heavily on user-specific needs and the nature of tasks, with Meta AI being recommended for precision tasks, ChatGPT for technical scenarios, and Gemini for multilingual and ethical applications.

6. Glossary

  • 6-1. Google Gemini [Product]

  • Google's AI chatbot offering advanced features with a $20/month subscription. Known for its cloud storage benefits and nuanced ethical decision-making. Evaluated as inconsistent compared to competitors.

  • 6-2. OpenAI's ChatGPT [Product]

  • An AI chatbot by OpenAI with a subscription option offering enhanced functionalities. Praised for coding proficiency, reasoning, and the introduction of features like the GPT store for custom versions.

  • 6-3. Meta AI [Product]

  • Meta's AI chatbot, evaluated highly for accurate math problem-solving, sourcing capabilities, and overall reliability in diverse tasks including programming and mock interviews.

7. Source Documents