Python course ยท Module 11: RAG and Multi-Agent Systems
CrewAI - Multi-Agent Framework
In this lesson10
In the previous lesson we wrote the agent class, the orchestrator and the message chain ourselves. It worked, but every new feature, such as tools, memory or task delegation, would have to be added by hand. When the expedition grows, it is better to use a ready-made camp rulebook.
CrewAI is a Python framework for creating teams of AI agents that collaborate to solve complex tasks. It is like assembling an expedition crew where everyone has their own role. The whole framework rests on three concepts: Agent (who), Task (what) and Crew (how together).
Installation and Basics
We install two packages and import the core classes. The language model is configured with the LLM class from CrewAI itself:
# pip install crewai crewai-tools
from crewai import Agent, Task, Crew, Process, LLM
# LLM model
llm = LLM(model="gpt-4o-mini", temperature=0.7)Watch out for a change in the library: older examples online use LangChain's ChatOpenAI here, but current CrewAI versions do not depend on LangChain and reject such an object when validating the agent. LLM(model="gpt-4o-mini") uses the key from the OPENAI_API_KEY variable under the hood.
Creating Agents
An agent in CrewAI is three sentences about itself: a role (role), a goal (goal) and a backstory (backstory). The model receives them as the description of the character it should play. Here is the researcher:
# Researcher agent
researcher = Agent(
role="Senior Research Analyst",
goal="Conduct thorough research and provide reliable information",
backstory="""You are an experienced analyst with 15 years of experience
in researching technology trends. You specialize in AI and ML analysis.
You always verify facts and look for reliable sources.""",
verbose=True,
allow_delegation=False,
llm=llm
)verbose=True prints the agent's reasoning to the console, and allow_delegation=False forbids it from passing work to others. In current versions False is the default anyway.
The writer and the editor look similar, only the character description changes:
# Writer agent
writer = Agent(
role="Content Writer",
goal="Create engaging and educational content about technology",
backstory="""You are a talented technical writer with a passion for
translating complex concepts into simple language. Your articles
are always well-organized and enjoyable to read.""",
verbose=True,
allow_delegation=True,
llm=llm
)
# Editor agent
editor = Agent(
role="Senior Editor",
goal="Ensure the highest quality of published content",
backstory="""You are an experienced editor with an eye for detail.
You care about style consistency, factual accuracy, and clarity of message.
You are not afraid to suggest changes that will improve the text.""",
verbose=True,
allow_delegation=False,
llm=llm
)The writer has allow_delegation=True, so when it does not know something, it can ask a crewmate. That is the only technical difference, the rest is character written into backstory.
Defining Tasks
A Task describes the work to be done. Every task has a description, an expected output (expected_output) and an assigned agent:
# Research task
research_task = Task(
description="""Conduct thorough research on the topic: {topic}
Your task:
1. Find the latest information and trends
2. Identify key players and technologies
3. Analyze potential applications
4. Assess future development directions
Deliver a detailed research report.""",
expected_output="Detailed research report with citations and sources",
agent=researcher
)The {topic} notation in the description is a placeholder for a variable. CrewAI fills it with the value from the inputs parameter when the crew starts.
The writing task uses the research result. That is what the context parameter is for:
# Writing task
writing_task = Task(
description="""Based on the research report, write a blog article.
Requirements:
1. Engaging introduction
2. Clear structure with headings
3. Practical examples
4. Summary with key takeaways
5. Length: 1000-1500 words
Use accessible language for technical readers.""",
expected_output="Complete blog article ready for publication",
agent=writer,
context=[research_task] # Uses the result of the previous task
)context=[research_task] hands the researcher's report to the writer, so we do not have to glue texts manually the way our orchestrator did.
The editor receives the writer's article as context:
# Editing task
editing_task = Task(
description="""Review and edit the article.
Check:
1. Grammar and style correctness
2. Argument consistency
3. Message clarity
4. Engaging tone
Make necessary corrections.""",
expected_output="Edited, final article",
agent=editor,
context=[writing_task]
)Description, expected output, agent and context are the four elements you build every task from.
Creating and Running the Crew
A Crew connects agents and tasks. The process parameter decides how the crew works:
# Create the team
content_crew = Crew(
agents=[researcher, writer, editor],
tasks=[research_task, writing_task, editing_task],
process=Process.sequential, # or Process.hierarchical
verbose=True
)
# Run with inputs
result = content_crew.kickoff(inputs={"topic": "RAG in enterprise AI"})
print(result)Process.sequential runs the tasks one after another, in list order. CrewAI supports two process types: sequential and hierarchical. kickoff returns a CrewOutput object whose text is in the raw field, and print shows it directly.
Tools for Agents
An agent without tools knows only what the model learned. The crewai-tools package provides ready-made tools for search, websites and files:
from crewai_tools import (
SerperDevTool, # Google search
WebsiteSearchTool, # Website search
FileReadTool, # File reading
PDFSearchTool, # PDF search
)
# Configure tools
search_tool = SerperDevTool()
web_tool = WebsiteSearchTool()
file_tool = FileReadTool()
# Agent with tools
researcher_with_tools = Agent(
role="Research Analyst",
goal="Gather and analyze information from various sources",
backstory="Expert in information retrieval...",
tools=[search_tool, web_tool, file_tool],
verbose=True,
llm=llm
)SerperDevTool needs a key in the SERPER_API_KEY variable. The agent decides by itself when to reach for which tool, based on their descriptions.
A custom tool is a class that inherits from BaseTool. Today we import it from crewai.tools, not from crewai_tools as in older examples:
# Custom tool
from crewai.tools import BaseTool
from pydantic import BaseModel, Field
class DatabaseQueryInput(BaseModel):
query: str = Field(description="SQL query to execute")
class DatabaseQueryTool(BaseTool):
name: str = "Database Query"
description: str = "Executes SQL queries on the database"
args_schema: type[BaseModel] = DatabaseQueryInput
def _run(self, query: str) -> str:
# Query implementation
return f"Query result: {query}"args_schema is a Pydantic model describing the arguments, and the _run method contains the actual logic. The language model reads name and description, so write them clearly.
Hierarchical Process
In a hierarchical process, a manager distributes the tasks. We create it like an ordinary agent:
# Coordinating manager
manager = Agent(
role="Project Manager",
goal="Coordinate teamwork and ensure timely delivery",
backstory="Experienced PM with AI team management skills",
verbose=True,
llm=llm
)
# Crew with hierarchy
hierarchical_crew = Crew(
agents=[researcher, writer, editor],
tasks=[research_task, writing_task, editing_task],
process=Process.hierarchical,
manager_agent=manager, # or manager_llm=llm
verbose=True
)The manager does not go into the agents list but into manager_agent. Instead of your own manager you can pass manager_llm, and CrewAI creates one itself. One of the two is required in a hierarchical process.
Memory and Context
The crew can remember earlier findings. We enable memory with the memory=True flag, and embedder points to the embedding model CrewAI uses to store memories:
# Crew with memory
crew_with_memory = Crew(
agents=[researcher, writer],
tasks=[research_task, writing_task],
process=Process.sequential,
memory=True, # Enable memory
embedder={
"provider": "openai",
"config": {"model": "text-embedding-3-small"}
},
verbose=True
)
# Agent with memory
agent_with_memory = Agent(
role="Assistant",
goal="Help the user",
backstory="...",
memory=True,
llm=llm
)These are the same embeddings we learned two lessons ago, only used to store the crew's memories instead of documents.
Callbacks and Monitoring
With longer tasks you want to know what is going on. Crew accepts two callback functions: task_callback after every task and step_callback after every agent step:
from crewai.tasks.task_output import TaskOutput
def on_task_end(output: TaskOutput) -> None:
"""Called after each task finishes."""
print(f"Completed: {output.description[:50]}")
print(f" Output: {output.raw[:100]}...")
def on_step(step) -> None:
"""Called after each agent step."""
print(f"Agent step: {str(step)[:100]}")
# Usage
crew = Crew(
agents=[researcher, writer],
tasks=[research_task, writing_task],
task_callback=on_task_end,
step_callback=on_step,
verbose=True
)TaskOutput carries the task description and the raw result in the raw field. Do not look for a CrewCallback class or a crewai.callbacks module: CrewAI has neither, and you pass callbacks exactly through these two parameters.
Asynchronous Execution
You can run several independent crews at the same time. That is what kickoff_async is for:
import asyncio
async def run_crew_async(topic: str) -> str:
"""Asynchronous crew execution."""
crew = Crew(
agents=[researcher, writer],
tasks=[research_task, writing_task],
process=Process.sequential
)
result = await crew.kickoff_async(inputs={"topic": topic})
return result
# Run multiple crews in parallel
async def run_multiple_crews():
topics = ["AI in medicine", "AI in finance", "AI in education"]
tasks = [run_crew_async(topic) for topic in topics]
results = await asyncio.gather(*tasks)
return dict(zip(topics, results))
results = asyncio.run(run_multiple_crews())asyncio.gather waits for all three crews at once, and dict(zip(...)) pairs the topics with the results. In production, create separate agent and task objects for each crew, because here they all share the same research_task and writing_task.
Example: Content Pipeline
Finally we put everything together into one pipeline with an extra SEO specialist:
def create_content_pipeline(topic: str) -> str:
"""Complete content creation pipeline."""
# Agents
seo_analyst = Agent(
role="SEO Specialist",
goal="Optimize content for search engines",
backstory="SEO expert with content marketing experience",
tools=[SerperDevTool()],
llm=llm
)
# Tasks
seo_task = Task(
description=f"Analyze keywords for the topic: {topic}",
expected_output="List of 10 keywords with their potential",
agent=seo_analyst
)
# Crew
pipeline = Crew(
agents=[seo_analyst, researcher, writer, editor],
tasks=[seo_task, research_task, writing_task, editing_task],
process=Process.sequential,
verbose=True
)
return pipeline.kickoff(inputs={"topic": topic})
# Run
article = create_content_pipeline("Machine Learning in 2025")The seo_task description uses an f-string, but the researcher's and writer's tasks still contain {topic}, which is why kickoff receives inputs. Without it, the agents would see the literal text "{topic}". I recommend starting with the sequential process and turning on the hierarchy only for a bigger crew, because the manager means extra model calls.
CrewAI is a powerful framework for building AI teams. In the next lesson you will learn about production RAG systems - how to deploy these technologies in the enterprise!
Remember: in CrewAI an agent is a crew member, a task is their assignment, and the crew is the whole expedition.
Docs entries used in this lesson
Spotted a mistake in this lesson?
Check yourself
Answer the questions from this lesson. Pick an answer to see right away whether it is correct.
1. What is CrewAI?
2. What process types does CrewAI support?
Hands-on tasks in the game
- Horizontal ordering
Arrange the elements:
- Code editor
Define a researcher agent in CrewAI
- Horizontal ordering
Arrange the Crew creation in CrewAI:
- Click in order
Arrange the Task definition elements in CrewAI: