media-crawler
Install, authenticate, configure, operate, and troubleshoot the external MediaCrawler client shared by Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu collectors. Xiaohongshu uses the separate browser-first xiaohongshu-mcp Collector. Use when auditing this client, onboarding a supported platform account, selecting search/detail/creator modes, enabling comments or media, locating outputs, or diagnosing crawler failures.
git clone --depth 1 https://github.com/tsingyuai/growth-lab /tmp/media-crawler && cp -r /tmp/media-crawler/collectors/media-crawler ~/.claude/skills/media-crawlerSKILL.md
# MediaCrawler
This is the shared tool layer. Read [operations.md](references/operations.md) before changing the external checkout. Then invoke exactly one platform Skill:
- [Douyin](../media-crawler-douyin/SKILL.md)
- [Kuaishou](../media-crawler-kuaishou/SKILL.md)
- [Bilibili](../media-crawler-bilibili/SKILL.md)
- [Weibo](../media-crawler-weibo/SKILL.md)
- [Tieba](../media-crawler-tieba/SKILL.md)
- [Zhihu](../media-crawler-zhihu/SKILL.md)
MediaCrawler does not support Twitter/X or Reddit. Do not imply otherwise.
## Contract
1. If install or authentication is missing, invoke [onboard-growth-lab](../../models/onboard-growth-lab/SKILL.md). Do not duplicate the global audit here.
Onboarding must obtain the user's explicit ban-risk acknowledgement before login or crawling, require existing-Chrome CDP with no browser or Cookie fallback, and verify each enabled platform with a non-empty minimal real read. Installation, a persisted profile, or a visible login alone is not readiness.
2. Treat `${MEDIACRAWLER_DIR:-${GROWTHLAB_CLIENT_ROOT:-$HOME/.growth-lab/clients}/MediaCrawler}` as an external checkout. Never vendor it or commit its browser profile, cookies, databases, or downloaded data.
3. Before a run, record upstream commit, platform, crawl type, keywords/IDs, config changes, login type, comment/media flags, and destination.
4. Modify only the documented platform config and `config/base_config.py`; show the diff before running. Restore unrelated example values.
5. Run serially and conservatively. Never silently retry risk-control or authentication errors.
6. Copy the required output into the invoking Model's `memory/<model>/...`; leave source provenance beside it. A Collector does not invent a new Memory owner.
7. Apply the upstream non-commercial learning license and each target platform's terms.
## Standard invocation
```bash
cd "${MEDIACRAWLER_DIR:-${GROWTHLAB_CLIENT_ROOT:-$HOME/.growth-lab/clients}/MediaCrawler}"
uv run main.py --platform <dy|ks|bili|wb|tieba|zhihu> --lt qrcode --type <search|detail|creator>
```
Use `--lt qrcode` with CDP and an existing Chrome session. Do not fall back to standard Playwright, a newly launched clean browser, or Cookie injection.
## Completion report
Return: exact source query/URLs, run time, upstream commit, raw and copied paths, record/media/comment counts, filters, partial failures, and any risk-control signal. Never report a search-card excerpt as full detail.Collect Bilibili video and creator evidence with MediaCrawler through search, exact BV detail, comments, dynamics, contacts, and optional media. Use for topic, format, title, creator, or audience research with explicit time-range and quality controls.
Collect Douyin competitive evidence with MediaCrawler using keyword search, exact video detail, comments, media, and creator profiles. Use for trend, hook, format, audience-language, or creator research that needs reproducible raw evidence and a documented selection method.
Collect Kuaishou competitive evidence with MediaCrawler using keyword search, exact video detail, comments, media, and creator profiles. Use for trend, format, audience-language, or creator research requiring reproducible source records and explicit collection limits.
Collect Baidu Tieba thread and user evidence with MediaCrawler using keyword or bar discovery, exact thread detail, replies, and creator pages. Use for community pain-point, vocabulary, objection, topic, or user research with thread-context preservation.
Collect Weibo posts and creator evidence with MediaCrawler using search, exact post IDs, comments, optional media, and creator IDs. Use for discourse, trend, messaging, audience-language, or account research requiring preserved provenance and risk-aware detail enrichment.
Collect Zhihu answers, articles, videos, comments, and creator evidence with MediaCrawler through search and exact URLs. Use for expert discourse, problem framing, objections, terminology, topic, or creator research where content type and question context must remain explicit.
渐进式研究当前产品，并把已经稳定、可追溯的产品认知增量写入根目录 SOUL.md。首次接入产品代码、原型或线上 URL，需要确认产品形态与已有能力，或某个增长 loop 在执行中发现新的产品事实、用户场景、问题与价值证据时使用。不得一次性臆造完整产品画像，也不得仅凭代码功能推断用户、问题或价值。
完整调研并确定 SEO 机会：从产品理解、领域词表拆解、关键词矩阵扩展、Bing 真实热度验证、竞品发现、实时 SERP 抓取，到头部页面的搜索引擎层、用户层和质量层拆解。需要查热词、验证搜索量、判断搜索意图、研究竞品页面、寻找信息增益缺口或决定应该做什么 SEO 页面时使用。