Your browser does not support JavaScript!

Comparative Analysis of AI Chatbots: ChatGPT, Gemini, and Meta AI

GOOVER DAILY REPORT August 31, 2024
goover

TABLE OF CONTENTS

  1. Summary
  2. Subscription-Based AI Chatbots
  3. Performance Comparison
  4. Comprehensive Task Evaluations
  5. Comparative Analysis and Results
  6. Conclusion

1. Summary

  • This report provides a comprehensive comparative analysis of three prominent AI chatbots: OpenAI's ChatGPT, Google's Gemini, and Meta AI. The primary objective is to evaluate their performance across various tasks, subscription benefits, and unique features to help determine their suitability for different user requirements. The key findings indicate that Meta AI exhibits the highest overall reliability, particularly in math problem-solving and coding tasks. ChatGPT excels in natural language understanding and reasoning clarity, while Gemini demonstrates superior performance in creative text generation and coding. However, there are concerns about ChatGPT's sourcing accuracy, contrasted with Meta AI's more reliable information sourcing.

2. Subscription-Based AI Chatbots

  • 2-1. Google’s Gemini Advanced

  • Google’s Gemini Advanced is offered as a subscription product for $20 a month, providing access to the company's best AI model, Gemini Ultra 1.0. The subscription also includes all features available through the Google One subscription, which encompasses 2 terabytes of cloud storage. Furthermore, Google is expected to integrate Gemini with Gmail and Docs in the future to enhance user experience. Recently, a new version, Gemini Pro 1.5, has been announced, which processes more data than the previous iterations but is not yet available to the public.

  • 2-2. OpenAI’s ChatGPT Plus

  • OpenAI's ChatGPT Plus is also priced at $20 a month, providing users with access to GPT-4 and DALL-E 3. Unlike Gemini, ChatGPT Plus does not offer additional benefits like cloud storage. However, it features an innovative aspect called the GPT store, where users can create and share custom versions of ChatGPT tailored for specific needs. This integration facilitates a smoother transition for users who are already familiar with the standard ChatGPT.

  • 2-3. Subscription costs vs. Free versions

  • Despite the availability of subscription options like Gemini Advanced and ChatGPT Plus at $20 per month, many users find that the free versions sufficiently meet their requirements. The free versions of both services deliver competent capabilities for a variety of tasks. Users who may benefit from the additional features of the subscription models, such as coding or experimenting with advanced AI functionalities, might consider subscribing; however, for the average user, the free tiers might be adequate.

  • 2-4. Value-added benefits

  • The value-added benefits differ between the subscription models. Gemini Advanced offers substantial additional features, including significant cloud storage as part of Google One, whereas ChatGPT Plus emphasizes user customization through the GPT store. These benefits cater to different user needs, reflecting the versatility in the application of these AI chatbots across various tasks and user environments.

3. Performance Comparison

  • 3-1. Coding proficiency

  • In the evaluation of coding proficiency, both ChatGPT and Gemini were tasked with writing a Python program that serves as a personal expense tracker. The specifications included the ability to input expenses along with categories and dates, and to provide a summary of expenses. Both chatbots successfully produced functional codes, but Gemini displayed superior functionality by including more granular reporting options. Therefore, the winner for coding proficiency is Gemini.

  • 3-2. Natural language understanding

  • For the natural language understanding test, a Cognitive Reflect Test (CRT) question was presented: 'A bat and a ball cost £1.10 in total. The bat costs £1.00 more than the ball. How much does the ball cost?' While both ChatGPT and Gemini arrived at the correct answer of 5 cents for the ball, ChatGPT demonstrated superior clarity and detail in its explanation. Hence, the winner is ChatGPT.

  • 3-3. Creative text generation

  • In assessing creative text generation, both chatbots were prompted to write a short story set in a futuristic city controlled by technology. While both produced commendable stories, Gemini was noted for better adherence to the narrative prompt and showcased superior creativity, making it the winner in this category.

  • 3-4. Reasoning and ethical decision-making

  • The reasoning and ethical decision-making abilities were tested with the classic question about two doors and guards where one always tells the truth and the other always lies. Both ChatGPT and Gemini provided correct answers, but ChatGPT illustrated its reasoning with greater clarity and detail. Hence, the winner is ChatGPT.

  • 3-5. Translation capabilities

  • For the translation capabilities evaluation, the chatbots were tasked with translating a paragraph about Thanksgiving in the United States from English to French. Gemini provided a more nuanced translation and an explanation of its approach, thereby securing the win in this category.

4. Comprehensive Task Evaluations

  • 4-1. Email writing

  • All three AI chatbots—Meta AI, ChatGPT, and Google Gemini—were tasked with composing a work-related email requesting a project extension. Each chatbot successfully generated a well-structured and polite email that fulfilled the prompt's requirements. They all achieved perfect scores in this task due to their ability to craft professional templates that users could personalize with relevant details.

  • 4-2. Recipe generation

  • The chatbots were prompted to provide a recipe for chili. Each chatbot produced accurate and detailed recipes, but there were notable differences in sourcing information. Both Meta AI and Gemini cited their sources at the end of the recipe, with Gemini providing additional links to more recipes. In contrast, ChatGPT did not cite any sources, which raised concerns about the trustworthiness of its recipe. Due to these sourcing discrepancies, it is recommended to use Meta AI or Gemini for recipe generation as they offer verifiable origin information, enhancing food safety.

  • 4-3. Math problem-solving

  • A series of math problems were presented to all three chatbots, including algebra and geometry questions. In the first problem involving nonnegative integers, all chatbots reached the same conclusion using different methods. However, in the geometry problem, ChatGPT started well but failed to deliver the final answer, while Gemini provided theoretical insights without numeric conclusions. Only Meta AI delivered a correct and complete answer, making it the most reliable choice for solving math problems.

  • 4-4. Programming tasks

  • The chatbots were asked to program a variant of tic-tac-toe in HTML and JavaScript. Both Meta AI and ChatGPT successfully generated the requested code in both languages. However, Gemini provided JavaScript code but substituted CSS for HTML, which is incorrect as both serve different purposes in web development. Consequently, Meta AI and ChatGPT are identified as superior options for programming tasks.

  • 4-5. Mock interviews

  • Each chatbot conducted a mock interview for a computing staff writer role, generating a set of questions and answers. While their approaches varied, all three chatbots delivered satisfactory interview simulations. They can serve as valuable resources for users preparing for real interviews, although more detailed scenarios would enhance the role-playing experience.

5. Comparative Analysis and Results

  • 5-1. Meta AI vs. ChatGPT vs. Gemini

  • This section provides a comprehensive comparison of three leading AI chatbot services: Meta AI, OpenAI's ChatGPT, and Google's Gemini. The analysis focuses on their performance across a variety of tasks, revealing distinct capabilities and limitations inherent to each service. The competition among these chatbots illustrates the evolving landscape of generative AI.

  • 5-2. Strengths and weaknesses

  • Each AI chatbot presents unique strengths and weaknesses. Meta AI is recognized for its overall reliability across multiple tasks, particularly in areas like math problem-solving. ChatGPT excels in natural language understanding, demonstrating clarity in reasoning and explanation. Gemini showcases creativity in text generation and offers additional features like cloud storage but falls short in consistency compared to its competitors.

  • 5-3. Sourcing and accuracy

  • In terms of sourcing information, both Meta AI and Gemini effectively link to external references while generating content, enhancing trustworthiness. Meta AI and Gemini also provided accurate recipe sourcing, unlike ChatGPT, which failed to acknowledge its sources, raising concerns about the potential for misinformation.

  • 5-4. Overall reliability and consistency

  • Overall reliability is a notable distinguishing factor among the three chatbots. Meta AI was deemed the most consistent performer, particularly in resolving math queries accurately. ChatGPT showed significant improvement in its latest iteration, performing well across various tasks, but with less reliability compared to Meta AI. Gemini, while inventive, was assessed as the least consistent of the trio.

6. Conclusion

  • The comparative analysis highlights that while Meta AI stands out as the most reliable AI chatbot, each of the three services—ChatGPT, Gemini, and Meta AI—presents unique strengths tailored to different user needs. ChatGPT's superior natural language understanding and reasoning clarity make it an exceptional tool for complex language processing tasks. Gemini's creativity in text generation and additional features like cloud storage position it well for users needing more innovative outputs. Meta AI's consistent performance across various tasks, particularly in math problem-solving and coding, underscores its dependable reliability. Despite their strengths, the report also notes limitations such as Gemini's inconsistencies and ChatGPT's issues with sourcing accuracy. Future advancements in these AI services may focus on enhancing consistency and reliability while maintaining their unique strengths, thus broadening their practical applicability in real-world scenarios.

7. Glossary

  • 7-1. ChatGPT [Technology]

  • OpenAI's AI chatbot known for its strong natural language understanding capabilities and reasoning clarity. It is also available as a subscription service (ChatGPT Plus), offering features like a custom GPT store.

  • 7-2. Gemini [Technology]

  • Google's AI chatbot, particularly strong in coding and creative text generation. The subscription service (Gemini Advanced) includes additional benefits like cloud storage.

  • 7-3. Meta AI [Technology]

  • Meta's AI chatbot, noted for its reliability across various tasks, including math problem-solving and accurate code generation. Meta AI often provides well-sourced content, improving the accuracy of its outputs.

8. Source Documents