Twitter / X Data Without the API: Scraping Public Tweets by Keyword or Handle

In early 2023 Twitter (now X) shut down free API access and moved all developer access to a paid tier starting at $100/month for the Basic level. For brand monitoring, competitive research and trend tracking — use cases that don’t generate revenue from the API — the cost is difficult to justify. Many teams stopped collecting X data entirely.

The Twitter / X Scraper provides an alternative path: it uses a real browser with residential proxy to access publicly visible X pages and extract tweet data without a Twitter developer account. Pay per tweet delivered, nothing charged on failed requests.

What You Get Per Tweet

• tweetId + tweetUrl
• authorName + authorHandle + authorUrl + authorVerified
• text — full tweet body including hashtags and mentions
• date — ISO 8601 timestamp
• likes, retweets, replies, views
• isRetweet + isReply
• mediaUrls — image/video URLs attached to the tweet
• hashtags + mentions — extracted from the tweet text
• source — the search term or handle that produced this tweet

An Honest Assessment of Non-Login Twitter Access

X enforces strict login gates on automated browser sessions. The visibility of tweets to non-authenticated users varies significantly by:

• Residential IP geography — some IP ranges are treated as authentic visitors, others are immediately classified as bots
• Request pattern — fresh sessions that navigate directly to search are more likely to be challenged
• Time of day — enforcement appears heavier during US business hours

In practice: some runs return 10–20 tweets per search term; others hit the login redirect on first page load and return zero. The actor detects the login gate, exits cleanly and does not charge for empty runs.

For use cases where this variance is acceptable — one-off brand monitoring sweeps, ad-hoc trend checks — the actor provides a no-subscription path. For continuous high-volume X data collection, a paid API plan or a cookie-injection approach (outside the scope of this actor) provides more consistent access.

Collect Tweets in Python

import os
from apify_client import ApifyClient

apify = ApifyClient(os.environ["APIFY_TOKEN"])

run = apify.actor("themineworks/twitter-x-scraper").call(run_input={
    "searchTerms": ["#AI agents", "Claude Anthropic", "GPT-4o"],
    "handles": [],
    "maxTweets": 20,
    "searchFilter": "top",
})

tweets = list(apify.dataset(run["defaultDatasetId"]).iterate_items())

for t in tweets:
    if t.get("_type") == "summary":
        continue
    if not t.get("text"):
        continue  # login-wall record — no tweet content
    print(f"@{t['authorHandle']} ({t['date'][:10]})")
    print(f"  {t['text'][:140]}")
    print(f"  {t.get('likes', 0)} likes | {t.get('retweets', 0)} RTs | {t.get('views', 0)} views")

Profile Timeline Mode

run = apify.actor("themineworks/twitter-x-scraper").call(run_input={
    "searchTerms": [],
    "handles": ["OpenAI", "AnthropicAI", "Google"],
    "maxTweets": 10,
    "searchFilter": "top",
})

Use It as an MCP Tool in Claude

{
  "mcpServers": {
    "twitter-x-scraper": {
      "command": "npx",
      "args": ["-y", "@apify/mcp-client"],
      "env": {
        "APIFY_TOKEN": "your-token-here",
        "MCP_ACTOR_URL": "https://mcp.apify.com/?tools=themineworks/twitter-x-scraper"
      }
    }
  }
}

Ask Claude to search X for recent mentions of your brand, pull the top tweets about a competitor product launch, or summarise the sentiment around a hashtag. Claude calls the actor and reasons over whatever data is returned.

Use Cases

Ad-hoc brand mention checks. Run a one-off sweep for your brand name or product to see what’s being said. Works reliably for brands with recent tweet activity — the top-tweet ranking tends to surface non-stale content even with limited session access.

Competitor launch monitoring. When a competitor announces a product or price change, pull the top tweets about it within the hour to gauge public reaction before writing your response.

Hashtag trend research. For campaign planning, pull the top-performing tweets under a hashtag to understand the vocabulary, tone and content types that are resonating.

Influencer research. Pull a handle’s recent tweets to review their content style and engagement metrics before a potential collaboration. Public profile data is more stable than search results.

Pricing

Each tweet that successfully delivers costs $0.002. Empty runs (login-wall blocks) are not charged.

Frequently Asked Questions

Why am I getting zero tweets?

X’s server-side login gate redirected the session before any tweets loaded. The actor logs this, pushes a no_results record with status: "login_blocked", and exits cleanly with no charge. Try re-running — a different residential proxy IP may have better access. Access rates vary by geography and time of day.

Does this require a Twitter developer account?

No. The actor uses a real browser (Chromium via Playwright) with residential proxy, not the Twitter API. No developer account, OAuth tokens or paid API plan needed.

Can I filter by top tweets vs latest?

Yes. Set searchFilter to "top" for the default ranked results or "latest" for reverse-chronological. Top tweets are generally more stable to access; latest tweets may be behind a tighter login gate.

Why is the view count sometimes missing?

X renders view counts via a separate async call after the main timeline loads. If the session is interrupted or the page is partially loaded before extraction, views may be null. Likes and retweets load earlier and are more reliably captured.