my-websearch

<div align="center">

# MyWebSearch

**[🇨🇳 中文](./README-zh.md) | 🇺🇸 English**

![npm version](https://img.shields.io/npm/v/my-websearch)
![npm downloads](https://img.shields.io/npm/dm/my-websearch)
![license](https://img.shields.io/npm/l/my-websearch)
![GitHub stars](https://img.shields.io/github/stars/wtznicy/my-websearch)

</div>

`my-websearch` provides an MCP server, CLI, and local daemon, and can also be paired with skill-guided agent workflows for live web search and content retrieval without API keys.


## Features

- Web search using multi-engine results
    - Domestic engines (no proxy needed): bing, baidu, csdn, juejin, sogou
    - Overseas engines (⚠️ **proxy required in mainland China**): duckduckgo, exa, brave, startpage
- HTTP proxy configuration support for accessing restricted resources
- No API keys or authentication required
- Returns structured results with titles, URLs, and descriptions
- Configurable number of results per search
- Customizable default search engine
- Support for fetching individual article content
    - csdn
    - github (README files)
    - generic HTTP(S) page / Markdown content

## Choose the Right Path

- `MCP`
  - Best when you want to connect `my-websearch` to Claude Desktop, Cherry Studio, Cursor, or another MCP client.
- `CLI`
  - Best for one-shot local commands, shell scripts, and direct terminal usage.
- `Local daemon`
  - Best when you want a reusable long-lived local HTTP service exposing `status`, `GET /health`, and `POST /search` / `POST /fetch-*`. Start it explicitly with `my-websearch serve` and check it with `my-websearch status`.
- `Skill`
  - Best as an agent-facing guidance layer for setup and usage. A skill does not replace MCP, CLI, or the local daemon; it typically works together with the CLI and/or local daemon to help an agent discover, activate, and use the smallest working path.

## Use with a Skill

Install the `my-websearch` skill for your agent first:

```bash
npx skills add https://gitee.com/wtznicy/my-websearch --skill my-websearch
```

On first use, the skill typically follows this path: detect whether a usable `my-websearch` path already exists, guide setup/enablement if it does not, validate that the capability is active, and only then continue with search or fetch through the smallest working path.

If the current environment cannot complete setup or activation automatically, you can explicitly have the agent start the local daemon first:

```bash
my-websearch serve
my-websearch status
```

Keep installation proxy settings separate from runtime proxy settings:

- Installation proxy / mirror
  - Use this when the skill or agent is installing `my-websearch`, `playwright`, or other npm packages.
  - In restricted networks, npm-specific flags or npm config often work better than generic shell proxy variables, for example:

```bash
npm --proxy http://127.0.0.1:7890 --https-proxy http://127.0.0.1:7890 install -g my-websearch
```

- Runtime proxy
  - Use this when the daemon is already installed and is about to perform live `search` / `fetch` work.
  - This affects the `my-websearch` network traffic after `serve` starts, for example:

```bash
USE_PROXY=true PROXY_URL=http://127.0.0.1:7890 my-websearch serve
```

If the agent can only get through the package-install step with npm proxy settings, but live search/fetch also needs a proxy after startup, those are two separate configuration steps and should be handled separately.

## CLI and Local Daemon

CLI is for one-shot execution. The local daemon is a long-lived local HTTP service for repeated calls with lower startup friction. Use `my-websearch serve` as the explicit daemon start command and `my-websearch status` as the explicit daemon status command.

Action commands such as `search` and `fetch-web` try the default local daemon first when it is available. If you pass `--daemon-url`, that daemon path becomes explicit and silent fallback to direct execution is disabled.

Build first:

```bash
npm run build
```

Start the local daemon:

```bash
npm run serve
# globally installed: my-websearch serve
```

Check status:

```bash
npm run status -- --json
# globally installed: my-websearch status --json
```

Run a one-shot local CLI search:

```bash
npm run search:cli -- "open web search" --json
```

Notes:
- Bare `my-websearch` is the MCP server compatibility entrypoint, not the recommended daemon start command for agent automation.
- For content extraction, prefer searching first and then fetching a more specific result page. Some homepages and JS-heavy landing pages may not expose readable article text through `fetch-web`.
- `--min-results N` on `search` auto-runs additional engines (not already requested) until at least N results come back; defaults to off.
- Bing's HTTP mode is the most anti-bot-prone engine. If you hit verification pages often: (a) spread load with `engines: ["duckduckgo", "brave"]` on Bing-unrelated queries, or (b) set `BING_PLAYWRIGHT_FALLBACK=false` plus `--min-results` so a blocked Bing automatically cascades to lighter engines instead of launching a Playwright browser.
- `cache-clear` clears the in-memory search TTL cache — useful after an engine recovers from an outage or when a stale anti-bot page got cached:
  ```bash
  my-websearch cache-clear
  ```

For the local daemon HTTP API (`serve`, `status`, `GET /health`, `POST /search`, `POST /fetch-*`, `POST /cache/clear`), see [docs/http-api.md](docs/http-api.md).

## Installation Guide

If you are using `my-websearch` as an MCP server, continue with the MCP-oriented setup below.

### NPX Quick Start (Recommended)

The fastest way to get started:

```bash
# Basic usage
npx my-websearch@latest

# With environment variables (Linux/macOS)
DEFAULT_SEARCH_ENGINE=bing ENABLE_CORS=true npx my-websearch@latest

# Windows PowerShell
$env:DEFAULT_SEARCH_ENGINE="bing"; $env:ENABLE_CORS="true"; npx my-websearch@latest

# Windows CMD
set MODE=stdio && set DEFAULT_SEARCH_ENGINE=bing && npx my-websearch@latest

# Cross-platform (requires cross-env, Used for local development)
npm install -g my-websearch
npx cross-env DEFAULT_SEARCH_ENGINE=bing ENABLE_CORS=true my-websearch
```

**Environment Variables:**

| Variable | Default                 | Options | Description |
|----------|-------------------------|---------|-------------|
| `ENABLE_CORS` | `false`                 | `true`, `false` | Enable CORS |
| `CORS_ORIGIN` | `*`                     | Any valid origin | CORS origin configuration |
| `DEFAULT_SEARCH_ENGINE` | `bing`                  | `bing`, `duckduckgo`, `exa`, `brave`, `baidu`, `csdn`, `juejin`, `startpage`, `sogou` | Default search engine |
| `USE_PROXY` | `false`                 | `true`, `false` | Enable HTTP proxy |
| `PROXY_URL` | `http://127.0.0.1:7890` | Any valid URL | Proxy server URL |
| `PROXY_ENGINES` | empty (all engines) | Comma-separated engine names | With `USE_PROXY=true`, **only** the engines in this whitelist route through the proxy; others stay direct. Empty = all engines proxied (legacy global behavior). Recommended for mainland China: `PROXY_ENGINES=duckduckgo,exa,brave,startpage` (overseas engines via proxy, domestic engines direct) |
| `FAKE_IP_CIDRS` | empty | Comma-separated CIDR list | Treat DNS answers in these CIDRs as synthetic fake-IP results and do not block them as private-network DNS answers. Literal private/local targets and other private-network DNS answers remain blocked |
| `FETCH_WEB_INSECURE_TLS` | `false` | `true`, `false` | Disable TLS certificate verification for `fetchWebContent` only. Use only when a target site has a broken certificate chain |
| `MODE` | `both`                  | `both`, `http`, `stdio` | Server mode: both HTTP+STDIO, HTTP only, or STDIO only |
| `PORT` | `3211`                  | 1-65535 | Server port (MCP HTTP/S; CLI daemon uses 3210 by default) |
| `ALLOWED_SEARCH_ENGINES` | empty (all available) | Comma-separated engine names | Limit which search engines can be used; if the default engine is not in this list, the first allowed engine becomes the default |
| `SEARCH_MODE` | `auto` | `request`, `auto`, `playwright` | Search strategy. Currently only affects Bing: request only, request then Playwright fallback, or force Playwright |
| `BING_IMPERSONATE_TARGET` | `chrome131` | Any curl-cffi-node impersonate target (e.g. `chrome131`, `chrome124`, `chrome116`) | Browser fingerprint target for Bing's HTTP mode. Bing soft-degrades pure-HTTP requests by TLS/HTTP2 fingerprint; this enables Chrome fingerprint impersonation (2/3 requests return full results vs stable degradation otherwise). Fallback to the default HTTP client is automatic if the native module is unavailable |
| `BING_PLAYWRIGHT_FALLBACK` | `true` | `true`, `false` | In auto mode, when Bing's request mode hits an anti-bot page: `true` = launch a Playwright browser (slow, ~400MB); `false` = surface the error so the search service can cascade to lighter engines (e.g. duckduckgo/brave) via `minResults` |
| `STARTPAGE_PLAYWRIGHT_FALLBACK` | `true` | `true`, `false` | `false` = do not launch a Playwright browser for Startpage's Anubis anti-bot warmup (saves ~400MB memory and seconds of latency); surface the error so `minResults` can cascade to lighter engines instead |
| `PLAYWRIGHT_PACKAGE` | `auto` | `auto`, `playwright`, `playwright-core` | Which Playwright client package to resolve when browser mode is enabled |
| `PLAYWRIGHT_MODULE_PATH` | empty | Absolute path or project-relative path | Reuse an existing Playwright client package outside this project |
| `PLAYWRIGHT_EXECUTABLE_PATH` | empty | Any valid browser binary path | Launch an existing Chromium/Chrome executable without installing bundled browsers |
| `PLAYWRIGHT_WS_ENDPOINT` | empty | Valid Playwright `ws://` / `wss://` endpoint | Connect to an existing remote Playwright browser server |
| `PLAYWRIGHT_CDP_ENDPOINT` | empty | Valid Chromium CDP endpoint | Connect to an existing Chromium instance over CDP |
| `PLAYWRIGHT_HEADLESS` | `true` |