Your browser does not support JavaScript!

Comparative Analysis of AI Chatbots: ChatGPT, Google Gemini, and Meta AI

GOOVER DAILY REPORT August 13, 2024
goover

TABLE OF CONTENTS

  1. Summary
  2. Subscription Options and Pricing
  3. Performance Comparison in Various Tasks
  4. Evaluation of General Use Cases
  5. Comparative Analysis and Final Verdict
  6. Conclusion

1. Summary

  • The report titled 'Comparative Analysis of AI Chatbots: ChatGPT, Google Gemini, and Meta AI' provides an in-depth evaluation of three prominent AI chatbots. It compares their subscription options, performance across various tasks, and unique features. Key tasks assessed include coding proficiency, natural language understanding, creative text generation, and problem-solving. The findings indicate that Google Gemini excels in creative tasks and coding, ChatGPT in natural language understanding and logical reasoning, and Meta AI in consistent programming performance and mathematical problem-solving. Evaluations also extend to general use cases like email drafting, recipe generation, and news summarization, showing distinct strengths and weaknesses among the chatbots.

2. Subscription Options and Pricing

  • 2-1. Overview of subscription services

  • ChatGPT Plus, Google Gemini Advanced, and Microsoft Copilot Pro represent the subscription offerings for AI chatbots discussed. Each of these subscriptions provides access to advanced AI models, with ChatGPT Plus and Copilot Pro leveraging OpenAI's GPT-4, while Google Gemini Advanced offers access to Gemini Ultra 1.0. These subscriptions cater to users with specific needs such as coding, productivity, and enhanced AI features.

  • 2-2. Pricing details

  • The subscription plans for ChatGPT Plus, Google Gemini Advanced, and Microsoft Copilot Pro are priced at $20 per month each. This standard pricing allows users access to advanced tools and functionalities that are not available in the free versions of these chatbots. These price points reflect the enhanced capabilities and features that come with each subscription.

  • 2-3. Additional perks and features

  • Google Gemini Advanced subscription includes additional benefits such as 2 terabytes of cloud storage through Google One and the potential future integration of Gemini with Gmail and Google Docs. ChatGPT Plus does not include these ancillary perks but offers access to the innovative GPT store, where users can build and share custom versions of ChatGPT. Microsoft Copilot Pro integrates seamlessly with Microsoft 365 suite applications, enabling AI functionalities directly within Excel, Outlook, and PowerPoint, making it a handy tool for productivity tasks.

3. Performance Comparison in Various Tasks

  • 3-1. Coding Proficiency

  • The coding proficiency of ChatGPT and Google Gemini was tested by asking both chatbots to write a simple Python program for a personal expense tracker. The program needed to allow users to input their expenses along with categories (such as groceries, utilities, entertainment) and the date of the expense, then provide a summary of expenses by category and total spend over a given time period. Both chatbots created fully functional expense trackers. However, Gemini added extra functionality, including labels within a category and more granular reporting options, which made it the winner in this category.

  • 3-2. Natural Language Understanding

  • To evaluate natural language understanding, the AI chatbots were given a common Cognitive Reflect Test (CRT) question about the price of a bat and a ball. The correct response should be that the ball costs 5 cents and the bat $1.05. Both ChatGPT and Google Gemini answered correctly. However, ChatGPT provided a clearer explanation of its reasoning, making it the winner in this category.

  • 3-3. Creative Text Generation

  • For creative text generation, the chatbots were tasked with writing a short story set in a futuristic city where technology controls every aspect of life, but the main character discovers a hidden society living without modern tech. The story needed to incorporate themes of freedom and dependence. Both chatbots produced good stories, but Gemini's adherence to the rubric and overall narrative quality were better, making it the winner in this category.

  • 3-4. Reasoning and Decision-Making

  • The reasoning and decision-making capabilities were tested with a classic logic problem involving two doors and two guards. Both chatbots provided the correct answer and a clear explanation. However, ChatGPT provided a slightly more detailed and clearer response, which made it the winner in this category.

  • 3-5. Cross-Lingual Translation

  • For cross-lingual translation, the chatbots were asked to translate a short paragraph about celebrating Thanksgiving in the United States from English to French, emphasizing cultural nuances. Both chatbots performed well, but Gemini excelled in providing a nuanced translation along with an explanation of its approach, making it the winner in this category.

4. Evaluation of General Use Cases

  • 4-1. Email drafting

  • In the evaluation of email drafting, all three AI chatbots—Meta AI, ChatGPT, and Google Gemini—performed excellently. When asked to generate a work email requesting a project extension, each chatbot produced a well-written, polite, and professional email. All responses were in template style, allowing users to personalize with relevant information. This indicates that all three chatbots are equally effective for drafting professional emails.

  • 4-2. Recipe generation

  • For recipe generation, notable differences emerged between the chatbots. When prompted for a chili recipe, Meta AI and Google Gemini provided detailed recipes with sources and links to the websites used. Google Gemini also included additional recipes for further exploration. In contrast, ChatGPT did not provide any sources, raising concerns about the origin and accuracy of its recipe, which could be problematic for food safety. Therefore, Meta AI and Google Gemini are more reliable for recipe generation due to their sourcing practices.

  • 4-3. News summarization

  • Regarding news summarization, the evaluation showed that both ChatGPT and Meta AI linked directly to the news outlets they cited, ensuring transparency and reliability. ChatGPT even linked to multiple sources for each headline. Google Gemini mentioned various news sites but did not include direct links to their pages. This makes ChatGPT and Meta AI more trustworthy for news updates, as they offer verifiable sources.

  • 4-4. Math problem-solving

  • In solving math problems, Meta AI demonstrated superior capabilities. When given algebra and geometry problems, all three chatbots arrived at the same correct answer for the algebra question using different methods. However, in the geometry problem, only Meta AI provided a correct and complete answer. ChatGPT nearly completed the problem but failed to post a final result, while Google Gemini gave a theoretical answer without numeric values. Thus, Meta AI is the best option for math problem-solving.

  • 4-5. Programming and coding

  • For programming and coding tasks, both Meta AI and ChatGPT delivered complete and correct code in the requested HTML and JavaScript languages. Google Gemini, however, provided JavaScript code but substituted CSS for HTML, which deviates from the prompt and indicates a misunderstanding of the task requirements. Therefore, Meta AI and ChatGPT are preferable for programming due to their adherence to the coding task specifications.

  • 4-6. Mock interviews

  • In the mock interview scenario, all three AI chatbots—Meta AI, ChatGPT, and Google Gemini—produced effective simulations of an interview for a role as a computing staff writer at a tech publication. Each chatbot provided mock questions and answers, serving as good starting points for interview preparations. This demonstrates that all three chatbots can be useful tools for practicing and preparing for job interviews.

5. Comparative Analysis and Final Verdict

  • 5-1. Performance Highlights of Each Chatbot

  • The comparative analysis of ChatGPT, Google Gemini, and Meta AI demonstrates various performance highlights across different tasks. ChatGPT excels in natural language understanding and reasoning capabilities, showing superior clarity in explanations and logical problem-solving. Google Gemini shows a strong capability in coding proficiency and creative text generation, producing functional code and adhering well to creative writing prompts. Meta AI stands out for its consistent performance in programming tasks and solving complex mathematical problems. All three chatbots provide impressive outputs in email generation, but Meta AI leads in accuracy for mathematical problem-solving and offers a reliable source for recipe-related queries.

  • 5-2. Strengths and Weaknesses

  • Each chatbot showcased distinct strengths and weaknesses. ChatGPT's strengths include natural language understanding and clear logical reasoning. It also performed well in conversational fluency, detecting sarcasm effectively. Its weaknesses are noted in areas such as recipe sourcing, where it lacked proper citation, raising concerns about information reliability. Google Gemini's strengths include coding proficiency, creative writing, and ethical reasoning. However, its weaknesses are evident in mathematical problem-solving and improperly substituted HTML with CSS in programming tasks. Meta AI's strengths lie in mathematical accuracy, properly sourcing recipes, and consistent performance in programming. Despite this, it showed some limitations in conversational fluency, especially in handling sarcasm.

  • 5-3. Best Use Case Scenarios

  • The best use cases for each AI chatbot differ based on their strengths. ChatGPT is best suited for natural language processing tasks, logical problem-solving, and scenarios requiring detailed explanations. It is ideal for users needing assistance in understanding complex concepts or engaging in nuanced conversations. Google Gemini is most effective in creative tasks, programming, and ethical reasoning, making it suitable for developers and creative professionals. Meta AI excels in academic and technical fields, particularly in solving complex mathematical problems, detailed programming tasks, and reliable information retrieval. Users who require precise and well-sourced responses for technical or mathematical queries will find Meta AI most beneficial.

6. Conclusion

  • The comparative analysis highlights significant variations in the strengths of each AI chatbot, making their suitability dependent on specific user requirements. Meta AI emerges as the top performer for tasks requiring consistent accuracy, especially in coding and math problem-solving. ChatGPT stands out with its superior natural language processing and clear logical reasoning, making it ideal for detailed explanations and nuanced conversations. Google Gemini excels in creative and ethical reasoning tasks, suited for developers and creative professionals. Users should leverage these insights to select the chatbot that best aligns with their needs. The report indicates that future iterations and improvements in each AI chatbot could yield even more specialized functionalities, enhancing their practical applicability in diverse fields. While each AI has demonstrated notable performance, addressing current limitations and expanding capabilities will be crucial for their development.

7. Glossary

  • 7-1. ChatGPT [AI Chatbot]

  • Developed by OpenAI, ChatGPT offers a subscription service known as ChatGPT Plus. It integrates with Microsoft productivity tools through Copilot Pro and includes features like a unique GPT store for custom chat versions. Notable for its natural language understanding and clear reasoning capabilities.

  • 7-2. Google Gemini [AI Chatbot]

  • Google Gemini Advanced is priced similarly to ChatGPT Plus and includes additional perks such as Google One cloud storage. It excels in creative tasks, ethical reasoning, and coding proficiency, making it a strong competitor in the AI chatbot market.

  • 7-3. Meta AI [AI Chatbot]

  • Meta AI is evaluated as the best overall performer in this comparative study. Known for its consistency in solving math problems and generating accurate content for various tasks, Meta AI establishes itself as a reliable choice for diverse applications.

  • 7-4. Copilot Pro [Product]

  • A Microsoft service that integrates GPT-4 with Microsoft productivity tools, enhancing the functionality of ChatGPT for professional use. It facilitates seamless interaction with AI for tasks like summarizing meetings and drafting emails.

8. Source Documents