Your browser does not support JavaScript!

Comparative Analysis of AI Chatbot Subscriptions: ChatGPT, Google Gemini, and Meta AI

GOOVER DAILY REPORT September 15, 2024
goover

TABLE OF CONTENTS

  1. Summary
  2. Testing AI Chatbot Subscriptions: ChatGPT Plus vs. Google Gemini Advanced
  3. Comparative Performance Analysis: Google Gemini vs. OpenAI ChatGPT
  4. Comprehensive Evaluation: Meta AI, ChatGPT, and Google Gemini

1. Summary

  • This report provides an in-depth evaluation of three leading AI chatbots—OpenAI's ChatGPT, Google's Gemini, and Meta AI. Focused on their performance across a range of tasks such as coding, creative text generation, and natural language processing, the report compares their pricing, usability of free versus paid versions, and unique subscription perks. Key findings indicate that while both Google Gemini and ChatGPT perform excellently in their respective domains, Meta AI emerges as the overall best performer due to its consistent reliability in various tasks. The findings suggest that user needs should guide the choice of AI chatbot.

2. Testing AI Chatbot Subscriptions: ChatGPT Plus vs. Google Gemini Advanced

  • 2-1. Pricing and Subscription Features

  • As of February 2024, Google offers the Gemini Advanced subscription for $20 per month, similar to OpenAI's ChatGPT Plus, which is also priced at $20 per month. The Google Gemini Advanced subscription provides access to its best AI model, Gemini Ultra 1.0, along with all features included in the Google One subscription, such as 2 terabytes of cloud storage. In contrast, ChatGPT Plus includes access to GPT-4 and Dall-E 3 but does not offer additional perks like cloud storage.

  • 2-2. General Usability: Free vs. Paid Versions

  • Both Google Gemini and ChatGPT provide free versions that are competent for users who may need basic AI functionalities, such as crafting emails or generating creative content. However, users with more specialized needs, such as coding or advanced AI features, may benefit from the paid subscriptions. Evaluations indicate that most users find the free options adequate for everyday tasks.

  • 2-3. Unique Subscription Perks and Enhancements

  • Google's Gemini Advanced comes with additional perks, including an integration for Gmail and Google Docs, planned for the future. In contrast, ChatGPT Plus offers the unique feature of the GPT store, allowing users to build and share custom versions of ChatGPT optimized for various situations. This distinct feature sets ChatGPT Plus apart from Gemini Advanced, which focuses on including storage capabilities.

3. Comparative Performance Analysis: Google Gemini vs. OpenAI ChatGPT

  • 3-1. Coding Proficiency

  • In this comparative analysis, coding proficiency was assessed by asking both Google Gemini and OpenAI ChatGPT to develop a Python script for a personal expense tracker. The script was required to allow users to input expenses along with categories and the date of the expense, and to provide a summary of expenses by category and total spend over a given time period. According to the testing data, both chatbots produced a fully functional script, but Gemini outperformed ChatGPT by adding extra functionality, including labels within a category and more granular reporting options. Hence, Gemini was declared the winner in this category.

  • 3-2. Creativity in Text Generation

  • For the creativity test, both AI chatbots were tasked with writing a short story set in a futuristic city where technology controls life, ultimately revealing a hidden society that lives without modern technology. The evaluation placed emphasis on originality, adherence to themes, and narrative consistency. While both outputs were deemed good, Gemini demonstrated better adherence to the evaluation criteria and produced a more cohesive and engaging story. Therefore, Gemini was the winner in the creativity test.

  • 3-3. Reasoning and Natural Language Processing

  • The ability of each AI to understand natural language was tested using a reasoning challenge involving a bat and a ball costing £1.10 in total, with the bat costing £1.00 more than the ball. This posed a Cognitive Reflect Test (CRT) question aimed at evaluating the AI's understanding of ambiguity. Although both chatbots correctly solved the problem, ChatGPT showed a clearer explanation of its reasoning steps. Consequently, ChatGPT was awarded the win in this category.

  • 3-4. Performance in Ethical Decision-making

  • In assessing ethical reasoning and decision-making, both chatbots were given a scenario regarding an autonomous vehicle that must choose between hitting a pedestrian or swerving and risking passengers' lives. Both provided discussions on various points to consider without offering a definitive opinion. However, Gemini was found to provide a more nuanced response, which was further affirmed through an A/B trial with other AI models that favored Gemini. Thus, Gemini was deemed the winner in this ethical reasoning category.

  • 3-5. Translation and Cultural Awareness

  • The final test analyzed cross-lingual translation and cultural awareness by having both chatbots translate a paragraph about Thanksgiving in the United States into French, focusing on cultural nuances. Both provided high-quality translations, but Gemini offered more nuanced understanding and an explanation of its translation approach, leading to its selection as the winner for this category.

4. Comprehensive Evaluation: Meta AI, ChatGPT, and Google Gemini

  • 4-1. General Task Handling: Email Writing and Recipe Generation

  • All three AI chatbots, Meta AI, ChatGPT, and Google Gemini, were tested on their capability to assist with email writing and recipe generation. Each chatbot successfully generated a well-written email that was polite and professional in response to a request for an extension on a project, scoring perfect marks on this task. In terms of recipe generation, when prompted for a chili recipe, all chatbots provided accurate recipes. However, Meta AI and Google Gemini excelled in sourcing their recipes, providing links to the websites used, whereas ChatGPT failed to provide any sources, raising concerns about reliability in culinary instructions.

  • 4-2. Differences in Sourcing Practices

  • The evaluation highlighted significant differences in how the chatbots sourced information. Both Meta AI and ChatGPT provided direct links to news outlets when summarizing current events, allowing users to verify the information. Google Gemini, however, mentioned various sites but did not link to specific pages. In terms of recipe sourcing, only Meta AI and Gemini linked back to credible sources, while ChatGPT did not, indicating that it may be less trustworthy in this area.

  • 4-3. Math Problem Solving Capabilities

  • When presented with algebra and geometry problems, each chatbot demonstrated varying levels of proficiency. All three chatbots successfully solved the first algebra problem, but Meta AI was the only one to correctly answer the more complex geometry question. Neither ChatGPT nor Google Gemini provided final answers for the second problem, with Meta AI outperforming the others in math problem-solving capabilities.

  • 4-4. Programming Task Performance

  • In programming tasks, both Meta AI and ChatGPT provided complete code in HTML and JavaScript as requested for a complex variant of tic-tac-toe. In contrast, Google Gemini provided JavaScript code but mistakenly used CSS instead of HTML, failing to meet the requirements of the task. This inconsistency shows that for programming tasks, Meta AI and ChatGPT are the more reliable choices.

  • 4-5. Overall Performance Ranking

  • Based on the overall performance across the various tasks tested, Meta AI emerged as the top AI chatbot due to its consistent results and reliability across a wide range of prompts. ChatGPT ranked second, showing noticeable improvement from its previous 3.5 model, while Google Gemini was rated lowest in performance, indicating it has not yet caught up with its competitors.

5. Glossary

  • 5-1. ChatGPT [AI Chatbot]

  • A product of OpenAI, ChatGPT offers both free and subscription options with enhanced features like access to a GPT store for custom AI versions. It excels in reasoning and natural language processing tasks.

  • 5-2. Google Gemini [AI Chatbot]

  • Google's AI chatbot solution, Gemini, includes subscription benefits like 2 terabytes of Google One cloud storage. It is noted for its superior performance in creative and technical coding tasks.

  • 5-3. Meta AI [AI Chatbot]

  • Developed by Meta, this AI chatbot is proficient in a variety of tasks including math problem solving and advanced programming. It also maintains a high level of consistency in providing accurate responses.

6. Source Documents