Generated with sparks and insights from 2 sources

img6

img7

img8

img9

img10

img11

Introduction

  • Striver-Ing Wechat-Spider is an open-source tool designed to scrape WeChat public account articles.

  • It can capture article information, read counts, likes, and comments.

  • The tool supports multiple platforms including Android, iPhone, Mac, and Windows.

  • Data is stored in MySQL and tasks are managed using Redis.

  • The tool is easy to deploy and continuously maintained.

Features [1]

  • Article Scraping: Captures all articles from a WeChat public account.

  • Engagement Metrics: Retrieves read counts, likes, and comments for each article.

  • Platform Support: Compatible with Android, iPhone, Mac, and Windows.

  • Automated Operation: Automatically monitors and scrapes new articles daily.

  • Data Storage: Uses MySQL for data storage and Redis for task management.

  • No Installation Required: Executable files available for direct use without installation.

  • Distributed Scraping: Supports multiple WeChat accounts for simultaneous data collection.

  • Persistent Links: Converts temporary links to permanent ones.

img6

Installation [1]

  • MySQL Installation: Required for data storage.

  • Redis Installation: Used for task caching to reduce MySQL operations.

  • Certificate Installation: Necessary for mitmproxy to intercept WeChat traffic.

  • Platform-Specific Instructions: Detailed steps for Windows, Mac, Android, and iPhone.

  • Proxy Configuration: Ensure the device and the computer running Wechat-Spider are on the same network.

Configuration [1]

  • Config.yaml: Main configuration file for MySQL and Redis connection settings.

  • Database Creation: Create a 'wechat' database in MySQL.

  • Auto Table Creation: Set 'auto_create_tables' to true for initial setup.

  • Proxy Settings: Configure proxy settings on the device to route traffic through mitmproxy.

Usage [1]

  • Start Wechat-Spider: Run the executable file to start the tool.

  • Task Assignment: Add public account tasks to the 'wechat_account_task' table.

  • Data Verification: Check the database for scraped data after running the tool.

  • Daily Monitoring: The tool automatically scrapes new articles daily.

img6

Common Issues [1]

  • MySQL Connection: Ensure correct configuration in config.yaml.

  • Certificate Issues: Reinstall mitmproxy certificate if expired.

  • No Tasks: Verify tasks are correctly added to 'wechat_account_task' table.

  • Packet Capture: Check proxy settings and port usage if no data is captured.

Technical Exchange [1]

  • Discussion Group: Join the QQ group for technical discussions.

  • Support: Deployment support and Q&A available for contributors and donors.

  • Contact: Reach out via WeChat for further assistance.

img6

<br><br>