research-brightdata

# Bright Data Research Skill

Advanced web research powered by Bright Data MCP - perform market analysis, competitive intelligence, data extraction, and comprehensive web research with anti-bot protection.

## Overview

This skill provides complete workflows for automated web research using Bright Data MCP. Handle search discovery, content collection, structured data extraction, and comprehensive analysis with browser automation support.

## When This Skill Applies

Activate this skill when the user's request involves:
- Web scraping and data collection
- Market research and competitive analysis
- Price monitoring and comparison
- Product information extraction
- Search engine result analysis
- Large-scale web data gathering
- Research requiring anti-bot protection

## Core Capabilities

### Search and Discovery

Use `search_engine` tool to find relevant sources:

```json
{
  "tool": "search_engine",
  "parameters": {
    "query": "site:etsy.com nba merchandise",
    "engine": "google",
    "cursor": "0"
  }
}
```

**Search strategies:**
- Use site operators: `"site:etsy.com keywords"`
- Use exact phrases: `"machine learning in healthcare"`
- Exclude terms: `"iphone -case -cover"`
- Paginate with cursor: "0", "1", "2" for more results

### Content Collection

Three collection modes based on research depth:

**Quick Mode** (3-5 URLs, serial processing):
- Use `scrape_as_markdown` for each URL
- Best for: Fast overviews, fact-checking

**Standard Mode** (10-20 URLs, parallel batch):
- Use `scrape_batch` for up to 10 URLs concurrently
- Best for: Market research, competitive analysis

**Deep Mode** (20-50 URLs, browser automation):
- Use `scraping_browser_navigate` for JavaScript-rendered pages
- Use `scraping_browser_links` to discover page links
- Use `scraping_browser_click` for interactions
- Best for: Dynamic content, multi-page extraction

### Data Extraction

Use `extract` tool for AI-powered structured data extraction:

```json
{
  "tool": "extract",
  "parameters": {
    "url": "https://example.com/product",
    "extraction_prompt": "Extract: product name, price as number, rating (0-5), number of reviews, seller name, availability status"
  }
}
```

**Common extraction schemas:**
- **E-commerce**: name, price, rating, reviews, seller, availability
- **Articles**: title, author, date, summary, key points
- **Companies**: name, industry, founded, headquarters, employee count

### Output Formats

Three report formats for different use cases:

**Report Format** (default):
- Executive summary
- Key findings with evidence
- Detailed analysis
- Methodology and recommendations
- Source references

**JSON Format**:
- Structured data for API integration
- All raw and processed data
- Metadata and provenance
- Statistical analysis

**Markdown Format**:
- Clean, readable content
- Tables and lists
- Source links
- Minimal formatting

## Research Workflow

### Phase 1: Query Analysis

Understand the research intent:
- **Scope**: How broad/deep should research be?
- **Key entities**: Products, companies, topics
- **Target sources**: Which sites/platforms?
- **Data needed**: What fields to extract?

### Phase 2: Source Discovery

Use `search_engine` to find URLs:
1. Execute initial search
2. Extract URLs from SERP
3. Filter irrelevant domains
4. Paginate if needed
5. Prioritize by relevance

### Phase 3: Content Collection

Choose appropriate mode:
- **Quick**: `scrape_as_markdown` per URL
- **Standard**: `scrape_batch` 10 URLs at once
- **Deep**: `scraping_browser_navigate` + browser tools

Handle errors gracefully:
- Retry failed URLs with alternative methods
- Log errors for transparency
- Continue with available data

### Phase 4: Data Extraction

Apply extraction schema:
- Use `extract` with custom prompts
- Validate extracted data
- Handle missing/malformed data
- Ensure data quality

### Phase 5: Analysis & Synthesis

Process and analyze:
- Clean and normalize data
- Perform statistical analysis
- Identify patterns and trends
- Cross-reference sources
- Validate findings

### Phase 6: Report Generation

Generate output:
- **Report**: Comprehensive document with all sections
- **JSON**: Structured data for processing
- **Markdown**: Clean, readable content

## Best Practices

### Search Strategy
- Start broad, then narrow down
- Use site operators for targeted searches
- Try multiple search engines if needed
- Set realistic limits (10-20 URLs usually sufficient)

### Performance
- Use `scrape_batch` for parallel processing (10x faster)
- Only use `deep` mode when necessary (much slower)
- Set appropriate timeouts
- Monitor success rates
- **Avoid token limits**: Batch 1-2 URLs at a time for large pages (Etsy, Amazon, etc.)

### Data Quality
- Always validate extracted data
- Cross-reference multiple sources
- Check for outliers and anomalies
- Normalize formats (dates, currencies, units)

### Error Handling
- Implement retry logic
- Have fallback strategies
- Log errors for debugging
- Don't fail on individual URL errors

### Ethical Considerations
- Respect robots.txt
- Don't overwhelm servers
- Rate limit requests
- Cite sources properly
- Don't misuse personal data

## Common Research Scenarios

### E-commerce Market Research

```
Query: "site:etsy.com nba merchandise"
Mode: standard
Extract: product name, price, rating, reviews, seller
Output: report
```

Expected: Price analysis, popular products, top sellers

### Price Comparison

```
Query: "iphone 15 pro max 256GB price comparison"
Mode: standard
Extract: retailer, price, availability, shipping
Output: json
```

Expected: Structured comparison with best deal identified

### Academic Research

```
Query: "machine learning in healthcare 2024 papers"
Mode: standard
Extract: title, authors, date, key findings, methodology
Output: report
```

Expected: Literature review with trends and insights

### Competitive Intelligence

```
Query: "competitor.com features pricing"
Mode: deep
Extract: feature name, description, pricing tier, availab