Deep Research: OpenAI's Advanced AI and an Open-Source Implementation

OpenAI introduced Deep Research, an AI tool capable of producing PhD-quality research with comprehensive citations and multimodal data processing, albeit at a premium subscription cost. An open-source implementation, developed by David, offers identical research capabilities using a significantly more affordable pay-as-you-go OpenAI API key.

Key Points Summary

• OpenAI Deep Research Capabilities and Cost

  OpenAI's Deep Research performs in-depth AI-driven investigations, delivering PhD-quality answers within 5 to 30 minutes, a task that would take human researchers hours or days. This tool is multimodal, processing various data types including pictures, text, and PDFs, and provides verifiable citations for its findings, enhancing trust in AI-generated information. Access to OpenAI Deep Research requires a ChatGPT Pro subscription costing $200 per month.

• Open-Source Deep Research as an Alternative

  An open-source implementation of Deep Research, created by David, offers the same robust capabilities as OpenAI's version but at a substantially lower cost. This alternative uses an OpenAI API key, which operates on a pay-as-you-go model, making it significantly more affordable than the $200 monthly ChatGPT Pro subscription.

• Underlying Technology and Setup Requirements

  Both the OpenAI and open-source Deep Research versions utilize OpenAI's `GPT-3.5-turbo` model (referred to as `mini-3` or `03 mini High`) for their operations. The open-source setup requires an OpenAI API key and a Fircrawl API key (with a free tier up to 500 credits), along with a Node.js environment, preferably managed within a Docker container for secure and separate operation.

• Installation and Configuration Steps

  Setting up the open-source Deep Research involves cloning the GitHub repository, creating a `.env.local` file with the Fircrawl and OpenAI API keys, defining a `Dockerfile`, and configuring a `docker-compose.yml` file. A minor adjustment to the `package.json` file is also necessary to ensure proper JSON formatting for environment variable handling before running the Docker container.

• Research Process and Customization

  During a research task, Deep Research prompts for additional context to refine the query, then generates a research plan. Users can control the breadth and depth of the research, which influences how wide-ranging and deep the AI's investigation becomes. The tool also displays its thought process or ongoing actions during research, providing transparency into its operations.

• Comparative Testing and Results

  Testing both OpenAI and the open-source Deep Research with questions like 'Which animal is better, cats or dogs?' and 'Which OS is better for IT professionals?' yielded similar, comprehensive conclusions. While OpenAI's interface is visually polished with inline links, the open-source version provides results in a markdown file with URLs, offering comparable quality and customizability in research parameters.

• AI's Role in Human Productivity and Future Outlook

  The longer response times of Deep Research, compared to typical fast AI answers, indicate a more thorough, multi-step reasoning process, akin to human thinking, which some see as a step towards Artificial General Intelligence (AGI). AI's ability to provide citations addresses the trust issue common with other AI outputs, making its research more reliable. The increasing prevalence of AI highlights its potential to significantly enhance human job performance, prompting IT professionals to integrate AI into their workflows.

• Emerging Agentic AI Frameworks

  Beyond OpenAI's offering, other major players like Hugging Face are developing their own agentic frameworks for deep search, indicating a growing trend in the AI community towards sophisticated, autonomous research agents.

Under Details