Your browser does not support JavaScript!

Comparative Analysis of AI Chatbots: ChatGPT, Google Gemini, and Meta AI

GOOVER DAILY REPORT August 10, 2024
goover

TABLE OF CONTENTS

  1. Summary
  2. AI Chatbot Subscription Features
  3. Comparative Performance Analysis
  4. Task-Specific Evaluations
  5. Conclusion

1. Summary

  • The report, titled 'Comparative Analysis of AI Chatbots: ChatGPT, Google Gemini, and Meta AI,' provides an in-depth comparison of the three AI chatbots across various criteria including subscription features, coding proficiency, natural language understanding, creative text generation, and more. The objective is to help readers understand which AI chatbot excels in specific tasks and to highlight the strengths and weaknesses of each model. Key findings suggest that Meta AI is generally the most dependable, particularly in email writing and math problem-solving. ChatGPT exhibited strengths in natural language understanding and reasoning capabilities, while Google Gemini offered unique features such as cloud storage and showed prowess in coding and ethical decision-making but lagged slightly in consistency. The report also provides detailed user recommendations, pricing information, and performance in specific tasks such as email writing, recipe sourcing, and programming.

2. AI Chatbot Subscription Features

  • 2-1. Pricing and Subscription Plans

  • Google’s Gemini Advanced, OpenAI’s ChatGPT Plus, and Microsoft’s Copilot Pro each offer their premium services for $20 a month. While Google and OpenAI provide access to their latest AI models (Gemini Ultra 1.0 and GPT-4, respectively) for this price, Microsoft’s Copilot Pro also includes integration with its suite of productivity software. The basic versions of both Gemini and ChatGPT are free and sufficient for most users.

  • 2-2. Advanced Features

  • Gemini Advanced from Google includes 2 terabytes of cloud storage as part of the Google One subscription and is expected to integrate with Gmail and Docs. ChatGPT Plus includes access to DALL-E 3 and, uniquely, the GPT store, where users can build and share custom versions of ChatGPT. Microsoft’s Copilot Pro also features GPT-4 and DALL-E 3 integrations, which can be utilized within Excel, Outlook, and PowerPoint for Microsoft 365 subscribers.

  • 2-3. User Recommendations

  • For typical users crafting emails or fan fiction, the free versions of ChatGPT and Gemini provide powerful capabilities. Those with specialized needs, such as coding or using advanced AI features, may find the $20 a month subscription for ChatGPT Plus or Gemini Advanced worthwhile. It's important to verify the outputs, as chatbots can produce misleading results, such as incorrectly identifying objects.

3. Comparative Performance Analysis

  • 3-1. Coding Proficiency

  • In the domain of coding proficiency, the free versions of ChatGPT and Google Gemini were evaluated based on their ability to develop a simple Python program for tracking personal expenses. Both chatbots produced fully functional code. However, Gemini's script included additional features like labels within categories and more detailed reporting options, making it the winner in this test.

  • 3-2. Natural Language Understanding

  • To test natural language understanding, both chatbots were asked a common Cognitive Reflect Test (CRT) question about the price of a bat and a ball. While both chatbots provided the correct answer, ChatGPT was able to articulate its reasoning more clearly, making it the winner in this category.

  • 3-3. Creative Text Generation

  • For creative text generation, the chatbots were prompted to write a short story set in a futuristic city. Both ChatGPT and Gemini produced solid stories, but Gemini's story adhered more closely to the given rubric and was judged to be slightly better overall.

  • 3-4. Reasoning Capabilities

  • The reasoning capabilities of the two chatbots were tested with a classic puzzle involving two doors and two guards—one always lying and the other always telling the truth. Both chatbots provided the correct answer and reasonable explanations; however, ChatGPT's detailed explanation gave it an edge in this round.

  • 3-5. Simplification for Children

  • Chatbots were challenged to explain how airplanes stay in the sky to a five-year-old. Both ChatGPT and Gemini used simple language and relatable analogies involving birds. Gemini's response included a series of bullet points and a practical experiment for children, giving it a slight advantage.

  • 3-6. Ethical Decision-Making

  • The AI's ethical decision-making was evaluated through a scenario involving an autonomous vehicle. Both ChatGPT and Gemini provided frameworks for making ethical decisions without offering concrete recommendations. Gemini's response was more nuanced and careful, which made it the winner in this category.

  • 3-7. Cross-Lingual Translation

  • When asked to translate a paragraph about Thanksgiving in the United States from English to French, both chatbots demonstrated near-equivalent capabilities. However, Gemini offered more nuance in its translation and an explanation of its approach, giving it a slight edge.

4. Task-Specific Evaluations

  • 4-1. Email Writing

  • In a test where all three AI chatbots (Meta AI, ChatGPT, and Google’s Gemini) were asked to 'write an email for work asking for a project extension,' they all performed exceptionally. Each chatbot generated a well-written, polite, and professional email template that could be personalized. This indicates that Meta AI, ChatGPT, and Google Gemini are equally competent in writing professional emails.

  • 4-2. Recipe Sourcing

  • When asked to 'Give me a recipe for chili,' all three chatbots provided accurate and thorough recipes, though with slight variations. However, a significant difference was noted in how the recipes were sourced. Meta AI and Google Gemini listed the source of the recipes at the bottom and provided links to the websites used, while ChatGPT did not source the recipe, raising concerns about potential plagiarism or inaccuracies in cooking instructions. Therefore, for recipe sourcing, Meta AI and Google Gemini are more reliable.

  • 4-3. Math Problem-Solving

  • In solving two sets of math problems—one algebra and the other geometry—each chatbot used different methods but arrived at the correct answer for the algebra problem. For the geometry problem, ChatGPT and Google Gemini struggled to provide a final answer, while Meta AI was the only one to deliver a correct and solid solution. Hence, Meta AI excels in solving math problems accurately.

  • 4-4. Programming

  • The chatbots were asked to generate a variant of the tic-tac-toe game, coded in simple HTML and JavaScript. Both Meta AI and ChatGPT successfully provided complete and correct code in the specified programming languages. On the other hand, Google Gemini provided the code in JavaScript but incorrectly substituted HTML with CSS. Thus, Meta AI and ChatGPT are the preferred choices for generating reliable programming code.

  • 4-5. Mock Interviews

  • When tasked with simulating a mock interview for a computing staff writer role, all three chatbots generated relevant questions and answers, approaching the task in different but effective ways. This capability makes each of the three AI chatbots useful for preparing for job interviews, with no single chatbot significantly outperforming the others in this category.

5. Conclusion

  • The comparative analysis underscores Meta AI's reliability across diverse tasks, notably in email writing and solving complex math problems, making it the most dependable of the three. ChatGPT demonstrated significant strengths in natural language understanding and reasoning, making it particularly useful for tasks requiring deep comprehension and critical thinking. Although Google Gemini shines in areas like coding proficiency and ethical decision-making, it shows inconsistencies that impact its overall reliability. These distinctions are crucial for users aiming to choose the best AI chatbot for their specific needs. However, the report's context-specific nature limits the generalizability of the findings. Future research should explore additional use cases and broader situational contexts to provide a more comprehensive understanding. The insights from this report can help users apply these AI chatbots effectively in real-world scenarios, tailoring their usage to the strengths of each respective technology.

6. Glossary

  • 6-1. ChatGPT [Technology]

  • OpenAI's popular AI chatbot known for its superior natural language understanding and reasoning capabilities. It offers a unique GPT store for customized chatbot versions through its subscription plan.

  • 6-2. Google Gemini [Technology]

  • Google’s AI chatbot which excels in coding and ethical decision-making scenarios. Despite some inconsistencies, it uniquely offers cloud storage benefits within its subscription plan.

  • 6-3. Meta AI [Technology]

  • An AI chatbot from Meta, distinguished by its high reliability across diverse tasks, particularly in email writing and math problem-solving. It has been identified as the most dependable amongst the compared chatbots.

7. Source Documents