Your browser does not support JavaScript!

A Comparative Analysis of AI Chatbot Subscriptions and Performance: ChatGPT, Google Gemini, and Meta AI

GOOVER DAILY REPORT September 1, 2024
goover

TABLE OF CONTENTS

  1. Summary
  2. Subscription Models and Key Features
  3. Performance Comparison Across Various Tasks
  4. Overall Comparison and Ranking
  5. Conclusion

1. Summary

  • The report, titled 'A Comparative Analysis of AI Chatbot Subscriptions and Performance: ChatGPT, Google Gemini, and Meta AI,' provides a detailed comparison of AI chatbots from OpenAI, Google, and Meta. It evaluates subscription benefits and costs, coding proficiency, natural language understanding, creative text generation, ethical decision-making, translation skills, and other performance aspects. Key findings indicate that Google Gemini excels in coding and ethical decisions, ChatGPT is superior in natural language understanding and reasoning, and Meta AI shows consistent performance across various tasks. Microsoft’s Copilot Pro is presented as an alternative, integrating with Microsoft 365 productivity tools.

2. Subscription Models and Key Features

  • 2-1. Subscription costs and included features

  • The subscription costs for AI chatbots are set at $20 per month for both Google’s Gemini Advanced and OpenAI’s ChatGPT Plus. Gemini Advanced includes access to the Gemini Ultra 1.0 AI model, as well as Google One subscription benefits, which provide 2 terabytes of cloud storage. In contrast, ChatGPT Plus grants access to the GPT-4 model and Dall-E 3, but lacks additional perks like cloud storage. Microsoft’s Copilot Pro also charges $20 per month and offers similar access to GPT-4 and Dall-E 3, seamlessly integrating with Microsoft productivity applications like Excel, Outlook, and PowerPoint.

  • 2-2. Casual versus advanced user requirements

  • Based on the assessment, the average user may find the free versions of ChatGPT and Gemini sufficient for general usage like crafting emails or creative writing. However, for users with specialized needs such as coding or those who want to access advanced AI features, the paid subscriptions of Gemini Advanced or ChatGPT Plus may provide necessary enhancements.

  • 2-3. Comparison of Gemini Advanced and ChatGPT Plus

  • Both Gemini Advanced and ChatGPT Plus have been compared based on their functionality and output quality. Notably, they are built on the same generative AI models, leading to similar results in productivity tasks. Gemini Advanced is noted for its additional benefits like cloud storage and future integrations, whereas ChatGPT Plus allows for creating custom versions through the GPT store.

  • 2-4. Microsoft’s Copilot Pro as an alternative

  • Microsoft’s Copilot Pro serves as an alternative to both Gemini Advanced and ChatGPT Plus, providing access to GPT-4 and Dall-E 3 integrated into Microsoft 365 tools. This integration enables users to utilize AI functionality directly within widely used applications like Excel and PowerPoint, making it appealing for users already engaged with Microsoft products.

3. Performance Comparison Across Various Tasks

  • 3-1. Coding proficiency

  • In evaluating coding proficiency, a task was proposed to each AI to develop a personal expense tracker in Python. The prompt required creating a script that allows users to input expenses by category, summarize them, and include comments explaining the code. Both ChatGPT and Google Gemini were able to produce fully functional code. However, Google Gemini provided additional functionality, including more granular reporting options and labels within categories, ultimately leading to its designation as the winner in this category.

  • 3-2. Natural language understanding

  • The analysis of natural language understanding involved a Cognitive Reflect Test (CRT) question about the costs of a bat and a ball. The prompt asked how much a ball costs when the bat costs £1.00 more than the ball, totaling £1.10. Both AIs provided the correct answer of 5 pence for the ball, but ChatGPT was recognized for demonstrating clearer reasoning in its response, thus being declared the winner.

  • 3-3. Creative text generation

  • For the creative text generation task, both AI models were tasked with writing a short story. The prompt required a narrative set in a futuristic city where technology controls life, focusing on themes of freedom and dependence. Although both models produced satisfactory stories, Google Gemini provided narratives that aligned better with the prompt's requirements. Therefore, Google Gemini was announced as the winner for its superior adaptability and creativity.

  • 3-4. Reasoning capabilities

  • In assessing reasoning capabilities, both AI chatbots were challenged with a classic logic puzzle involving two doors and two guards, where one guard always lies. The correct inquiry to discern the safe door involves asking one guard what the other would say. Both ChatGPT and Google Gemini arrived at the correct conclusion, but ChatGPT's explanation was more detailed and clearer, leading to ChatGPT winning this round.

  • 3-5. Simplification for children

  • The task of simplifying information for children involved explaining how airplanes stay up in the sky to a five-year-old. Both chatbots produced effective explanations using engaging language. However, Google Gemini’s response was structured as bullet points and included a practical experiment for a child, earning it the title of the winner in this category.

  • 3-6. Ethical decision-making

  • An ethical reasoning scenario was presented wherein an autonomous vehicle must choose between harming a pedestrian or risking its passengers' safety. Both AIs did not express personal opinions but discussed various ethical frameworks and considerations. While both offered insightful responses, Google Gemini was deemed the winner for its more nuanced understanding and thorough examination of the ethical dilemmas.

  • 3-7. Translation skills

  • For translation skills, the prompt involved translating a paragraph on Thanksgiving in the U.S. into French, emphasizing cultural nuances. Both AI models performed admirably in this task; however, Google Gemini excelled by providing a more nuanced translation and explanation of its process, thus emerging as the winner in translation.

4. Overall Comparison and Ranking

  • 4-1. Email writing

  • All three AI chatbots, Meta AI, ChatGPT, and Google Gemini, were tasked with generating a work-related email requesting a project extension. Each chatbot produced a well-written email that effectively addressed the prompt in a polite and professional manner, receiving perfect marks for this task.

  • 4-2. Recipe provision

  • When asked to provide a recipe for chili, all chatbots provided accurate and thorough recipes. However, Meta AI and Google Gemini included sources for their recipes, while ChatGPT did not, which raised concerns about plagiarism and reliability for novice cooks.

  • 4-3. News summarization

  • Each chatbot was asked to produce a bulleted list of the latest news. While all three provided quick responses, they mainly copied headlines without offering much context. ChatGPT and Meta AI included links to their sources, advantageous for credibility, while Gemini mentioned various news sites but did not provide direct links.

  • 4-4. Math problem-solving

  • The chatbots were presented with two sets of math problems. In the first task, all three chatbots successfully solved the problem using different methods. However, for the second math question, only Meta AI provided a complete, correct answer, while ChatGPT and Gemini failed to do so.

  • 4-5. Programming capabilities

  • For a programming task to create a variant of tic-tac-toe, both Meta AI and ChatGPT delivered complete code in HTML and JavaScript as requested. However, Gemini provided JavaScript code but incorrectly substituted CSS for HTML, demonstrating a lack of understanding of these programming languages.

  • 4-6. Mock interviews

  • Each chatbot simulated a mock interview for a computing staff writer role. All three approached the task differently but produced positive outcomes, offering a good starting point for interview preparation.

  • 4-7. Overall reliability and consistency

  • In the overall assessment, Meta AI emerged as the most reliable AI chatbot across various tasks, demonstrating consistent performance. ChatGPT performed adequately but had some inconsistencies, while Google Gemini was found to be the least reliable of the three, struggling to keep up with the competition.

5. Conclusion

  • The analysis underscores the unique strengths and suitability of each AI chatbot subscription based on user needs. Google Gemini's prominence in coding and ethical decision-making makes it ideal for technical users. OpenAI's ChatGPT offers robust natural language understanding and reasoning, helpful for users focused on communication and logic tasks. Meta AI's broad and consistent performance across tasks portrays it as the most reliable option. Users should be aware of potential inaccuracies, especially in translations requiring cultural context. Future developments in AI chatbots should enhance their applicability further, aiming for improved accuracy, ethical reasoning, and functionality integrations. This assessment aids users in selecting the most appropriate AI tool tailored to their specific needs and preferences, ensuring optimal utility and experience.

6. Glossary

  • 6-1. ChatGPT [AI Chatbot]

  • Developed by OpenAI, ChatGPT is an AI chatbot offering advanced natural language understanding, reasoning capabilities, and customizable versions. It is part of the ChatGPT Plus subscription model, priced at $20 per month.

  • 6-2. Google Gemini [AI Chatbot]

  • Google's AI chatbot, Gemini, offers a subscription model that includes the powerful Gemini Ultra 1.0 model and Google One cloud storage. Notable for its coding proficiency and ethical decision-making capabilities.

  • 6-3. Meta AI [AI Chatbot]

  • Meta AI is another advanced chatbot, known for its consistent performance across a variety of tasks, from email writing to solving math problems. It demonstrated overall reliability in multiple comparative tests.

  • 6-4. Microsoft’s Copilot Pro [AI Integration]

  • Microsoft's Copilot Pro integrates AI functionalities with Microsoft 365, providing access to GPT-4 and offering an alternative for users seeking AI-driven productivity tools.

7. Source Documents