Features

• Aggregates company and social data from Crunchbase, LinkedIn, and Reddit.
• Automates web scraping via CrewAI agents and Bright Data MCP (handles JS rendering, proxy rotation, and CAPTCHAs)
• AI-powered analysis and insight generation with Gemini models
• Real-time visualization in a Streamlit dashboard
• Modular architecture for easy addition of new data sources or workflows

How it works

1. Initiate: Data-retrieval with specialized CrewAI agents.
2. Orchestrate: API calls & dynamic scraping with Bright Data MCP handles JS, proxies, rate limiting, CAPTCHA, more.
3. Normalize: Store and send data to Gemini AI for instant analysis.
4. Display: Results and actionable insights in Streamlit dashboard.

Tech Stack

• CrewAI:
• Bright Data MCP:
• Gemini:
• Streamlit:

Data Sources

Use Cases

• Competitive intelligence & company research
• Market trend analysis
• Real-time sentiment & topic monitoring
• Investment/M&A targeting
• Multi-source data discovery for analysts

Prerequisites

• Python3 & Node.js
• Bright Data account with API key
  • PLUS Web Unlocker and Browser API zones
• Gemini API access

Installation

# Clone the repo
git clone https://github.com/brightdata/trendscan.git
cd trendscan

# Create and activate Python virtual environment
uv venv

# Install dependencies
uv pip install -r requirements.txt

Configuration

1. Open the .env file in your projects root directory.
2. Fill in your real credentials and API keys.
3. (Optional) To use a different Gemini model, set LLM_MODEL in .env.
   See Gemini model docs.

Usage

Launch the Streamlit web interface with:

streamlit run streamlit_trendscan.py

Youll be able to initiate data retrieval, run the pipeline, and interact with the dashboard in your browser.

Contributing

PRs, feedback, and feature suggestions are welcome!

• Open an issue for bugs and improvements.
• For large contributions, please open a discussion first.

License

Distributed under the MIT License.

Have fun scanning trends and supercharging company intelligence!