Scweet v5 — Full Documentation¶

Scweet is an API-only Twitter/X scraper built on the web GraphQL endpoints. It handles account pooling, rate limiting, cooldowns, resume, and output persistence — all backed by a local SQLite database.

pip install -U Scweet

from Scweet import Scweet, ScweetConfig, ScweetDB

The constructor reaches no network by default. It reads the local state file and returns, in about one second. Two things change that:

manifest_scrape_on_init=True reads the JavaScript bundles of X while the object is built, which takes a few seconds more.
The first call of a read method builds the x-client-transaction-id header, and that reads a page of X. The object does not build it before then.

Account Setup¶

Scweet needs the cookie of a logged-in X account to make authenticated requests. The first path uses one value; the other forms below add scale.

The first path: one auth_token¶

Log into X in your browser.
Open DevTools (F12) > Application > Cookies > https://x.com.
Copy the auth_token value.

s = Scweet(auth_token="YOUR_AUTH_TOKEN")

That one value is enough. Scweet builds the ct0 (CSRF) token from it. Use a dedicated account, never your personal one.

More accounts, more scale: cookies.json¶

For several accounts, and for a proxy per account, list them in a cookies.json file:

[
  { "username": "account1", "cookies": { "auth_token": "..." }, "proxy": "http://user1:pass1@host1:port1" },
  { "username": "account2", "cookies": { "auth_token": "..." }, "proxy": "http://user2:pass2@host2:port2" }
]

s = Scweet(cookies_file="cookies.json")

ct0 is optional in each entry; Scweet bootstraps it from the auth_token. A file exported by a browser cookie extension (a list of {"name": ..., "value": ...} objects) is accepted too. The other credential forms — an inline cookies= dict, a colon-separated accounts_file=, and a .env file through env_path= — all reach the same store; see Configuration reference.

The state file¶

After the first run, the accounts live in a SQLite state file (scweet_state.db by default). A later run reuses them with no credential:

s = Scweet()                 # reads scweet_state.db
tweets = s.search("ethereum", limit=100)

Pass db_path="path/to/state.db" to choose the file. db_path is the location of that state, not a place where the tweets go; a run returns the tweets and, with save=True, writes them to the output directory.

A global proxy¶

s = Scweet(auth_token="YOUR_AUTH_TOKEN", proxy="http://user:pass@proxy.example.com:8000")

# Or through ScweetConfig — the same effect
s = Scweet(auth_token="YOUR_AUTH_TOKEN", proxy="http://user:pass@proxy.example.com:8000")

Proxy modes¶

You control how each account reaches the network. There are four modes:

No proxy. Requests go direct.
One global proxy. Pass proxy="http://user:pass@host:port". Every account shares it.
One proxy session per account. Put {session} in the URL: proxy="http://user,session-{session}:pass@host:port". Each account session replaces the placeholder with a unique token (letters and digits only), so a rotating provider pins one exit IP per account. A session that is built again after a transport failure gets a new token and a new exit IP, so a retry recovers instead of reaching the same dead exit. The replacement is a plain string substitution, so it works with any provider that names the session inside the URL — write your provider's own syntax around {session} (for example ...-session-{session} or ...-sessid-{session}). A provider that pins a session by port instead of by name needs mode 4.
A proxy per account. Set "proxy" on the account record (in the cookies JSON, or with set_account_proxy). It overrides the global proxy for that account, and it can also contain {session}.

Option C: inline cookies¶

Pass cookies directly — useful for scripts and one-off runs:

# Single account
s = Scweet(cookies={"auth_token": "...", "ct0": "..."})

# Multiple accounts
s = Scweet(cookies=[
    {"auth_token": "tok1", "ct0": "ct0_1"},
    {"auth_token": "tok2", "ct0": "ct0_2"},
])

How provisioning works¶

When you create a Scweet instance, it provisions your accounts into a local SQLite database (scweet_state.db by default). This means your cookies are imported, validated, and stored so Scweet can manage them — tracking rate limits, cooldowns, daily caps, and lease state across requests.

This happens automatically on init (provision=True by default). You provide cookies once, and Scweet handles the rest. If an account already exists in the DB (matched by username or auth_token), it's updated rather than duplicated.

Reuse existing DB¶

Because accounts are persisted in SQLite, you don't need to provide cookies every time. After the first run, you can just point to the DB:

s = Scweet(db_path="scweet_state.db")

This reuses your previously provisioned accounts with all their state (daily counters, cooldowns, etc.) intact.

You can also skip provisioning entirely if you only want to work with accounts already in the DB:

s = Scweet(db_path="scweet_state.db", provision=False)

Controlling Limits¶

Every method that paginates (search, search_users, get_profile_tweets, get_profile_media, get_tweet_replies, get_reposters, get_followers, get_following, get_verified_followers) accepts a limit parameter — the maximum number of items to collect in that call. If omitted (None), scraping continues until results are exhausted or account daily caps are hit.

Always set a limit to avoid burning through your account quota unexpectedly:

tweets = s.search("python", limit=200)              # stop after 200 tweets
tweets = s.get_profile_tweets(["elonmusk"], limit=100)
users  = s.get_followers(["elonmusk"], limit=500)
users  = s.get_following(["OpenAI"], limit=500)

There are two layers of limits:

Layer	Where	What it controls
Per-call `limit`	Method argument	Max items returned by a single call
Account daily caps	`ScweetConfig`	Max API requests / tweets per account per UTC day

The per-call limit is what you should set in normal usage. The account daily caps (daily_requests_limit, daily_tweets_limit) are safety nets that protect your account from over-use across all calls in a day — see Rate Limiting in the config reference.

get_user_info does not paginate (one API call per user), so it has no limit parameter.

Search API¶

Basic search¶

s = Scweet(cookies_file="cookies.json")

# Defaults to last 30 days if since/until omitted — always set explicit dates for reproducibility
tweets = s.search("python programming", limit=100)

# Explicit date range
tweets = s.search("python programming", since="2024-01-01", until="2024-02-01", limit=500)
print(f"Found {len(tweets)} tweets")

Both since and until are optional. since defaults to 30 days ago, until defaults to today. Always set explicit dates for reproducible results.

Structured filters¶

All filters are optional and merge with the query string:

tweets = s.search(
    since="2024-01-01",
    from_users=["elonmusk"],
    min_likes=100,
    has_images=True,
    lang="en",
    limit=200,
)

Combining a query string with filters:

tweets = s.search(
    "AI tools",
    since="2024-01-01",
    from_users=["OpenAI"],
    min_likes=50,
    limit=100,
)

Available filters¶

Parameter	Type	Description
`all_words`	`list[str]`	All words must appear (AND)
`any_words`	`list[str]`	Any word can appear (OR)
`exact_phrases`	`list[str]`	Exact phrase match
`exclude_words`	`list[str]`	Exclude tweets with these words
`hashtags_any`	`list[str]`	Match any of these hashtags
`hashtags_exclude`	`list[str]`	Exclude these hashtags
`from_users`	`list[str]`	Tweets from these users
`to_users`	`list[str]`	Tweets to these users
`mentioning_users`	`list[str]`	Tweets mentioning these users
`tweet_type`	`str`	`all`, `originals_only`, `replies_only`, `retweets_only`, `exclude_replies`, `exclude_retweets`
`verified_only`	`bool`	Verified accounts only
`blue_verified_only`	`bool`	Blue verified only
`has_images`	`bool`	Must contain images
`has_videos`	`bool`	Must contain videos
`has_links`	`bool`	Must contain links
`has_mentions`	`bool`	Must contain mentions
`has_hashtags`	`bool`	Must contain hashtags
`min_likes`	`int`	Minimum likes
`min_replies`	`int`	Minimum replies
`min_retweets`	`int`	Minimum retweets
`place`	`str`	Place filter
`geocode`	`str`	Geocode filter (e.g., `"37.7749,-122.4194,10km"`)
`near`	`str`	Near location
`within`	`str`	Within radius (e.g., `"15mi"`)

Standard parameters¶

Parameter	Type	Default	Description
`query`	`str`	`""`	Raw query string (Twitter search operators)
`since`	`str`	30 days ago	Start date (`YYYY-MM-DD`)
`until`	`str`	today	End date (`YYYY-MM-DD`)
`lang`	`str`	`None`	Language filter (e.g., `"en"`)
`display_type`	`str`	`"Latest"`	`"Latest"` (chronological, fills a volume order) or `"Top"` (a ranked selection with far fewer tweets)
`limit`	`int`	`None`	Max tweets to collect. `None` = no cap (scrapes until exhausted). Recommended to always set.
`max_empty_pages`	`int`	config value	Stop after N consecutive empty pages
`resume`	`bool`	`False`	Resume from last checkpoint
`save`	`bool`	`False`	Save results to disk
`save_format`	`str`	config value	`"csv"`, `"json"`, or `"both"`
`save_name`	`str`	auto-generated	Base filename for saved output (without extension)

Async variant¶

tweets = await s.asearch("query", since="2024-01-01", limit=100)

Profile Tweets¶

Fetch tweets from user timelines:

tweets = s.get_profile_tweets(["elonmusk", "OpenAI"], limit=100)

# With options
tweets = s.get_profile_tweets(
    ["elonmusk"],
    limit=500,
    max_empty_pages=2,
    save=True,
    save_format="json",
)

Parameter	Type	Default	Description
`users`	`list[str]`	required	Usernames or profile URLs
`limit`	`int`	`None`	Max tweets to collect. `None` = no cap. Recommended to always set.
`max_empty_pages`	`int`	config value	Stop after N consecutive empty pages
`resume`	`bool`	`False`	Resume from last checkpoint
`save`	`bool`	`False`	Save results to disk
`save_format`	`str`	config value	`"csv"`, `"json"`, or `"both"`
`save_name`	`str`	auto-generated	Base filename for saved output (without extension)

Two variants of the timeline:

# The tweets AND the replies of the profile
tweets = s.get_profile_tweets(["elonmusk"], limit=100, include_replies=True)

# Only the tweets that carry an image or a video
media = s.get_profile_media(["nasa"], limit=100)

Async: await s.aget_profile_tweets(["elonmusk"], limit=100) / await s.aget_profile_media(["nasa"], limit=100)

Tweet Lookup & Replies¶

Read full data for tweet ids you already hold — from an earlier scrape, a dataset, or a URL:

# Up to 50 ids in one request
tweets = s.get_tweet_info(["1866123456789", "1866123456790"])

# The replies under one tweet
replies = s.get_tweet_replies("1866123456789", limit=200)

# The accounts that reposted one tweet
reposters = s.get_reposters("1866123456789", limit=500)

get_tweet_info returns the same tweet record as search(). A deleted or missing id gives no row and no error.
get_tweet_replies returns tweet records; the focal tweet itself is not in the list.
get_reposters returns user records with type: "reposters".

Each method takes raw_json, save, save_format and save_name; the paginating two also take limit and max_empty_pages. Async: aget_tweet_info, aget_tweet_replies, aget_reposters.

Followers / Following¶

# Followers
users = s.get_followers(["elonmusk"], limit=1000)

# Following
users = s.get_following(["OpenAI"], limit=500)

# Verified followers only
users = s.get_verified_followers(["elonmusk"], limit=500)

All three methods share the parameters below and return the same user record.

Parameter	Type	Default	Description
`users`	`list[str]`	required	Usernames or profile URLs
`limit`	`int`	`None`	Max users to collect. `None` = no cap. Recommended to always set.
`max_empty_pages`	`int`	config value	Stop after N consecutive empty pages
`resume`	`bool`	`False`	Resume from last checkpoint
`raw_json`	`bool`	`False`	Include full Twitter user payload under `raw` key
`save`	`bool`	`False`	Save results to disk
`save_format`	`str`	config value	`"csv"`, `"json"`, or `"both"`
`save_name`	`str`	auto-generated	Base filename for saved output (without extension)

raw_json option¶

By default, follower/following records contain curated fields. With raw_json=True, each record includes the full Twitter user payload under a raw key (CSV output stays curated regardless):

users = s.get_followers(["elonmusk"], limit=100, raw_json=True)
# users[0]["raw"] contains the full GraphQL user object

Async: await s.aget_followers(["elonmusk"], limit=500) / await s.aget_following(["OpenAI"], limit=500) / await s.aget_verified_followers(["elonmusk"], limit=500)

CLI: scweet ... verified-followers elonmusk --limit 500 --save

User Info¶

Fetch profile information for one or more users:

profiles = s.get_user_info(["elonmusk", "OpenAI"])
# Returns list of dicts with profile fields

# By numeric id, up to 100 in one request; both inputs combine
profiles = s.get_user_info(user_ids=["44196397", "783214"])

Parameter	Type	Default	Description
`users`	`list[str]`	`None`	Usernames or profile URLs
`user_ids`	`list[str]`	`None`	Numeric rest ids; combines with `users`
`save`	`bool`	`False`	Save results to disk
`save_format`	`str`	config value	`"csv"`, `"json"`, or `"both"`
`save_name`	`str`	auto-generated	Base filename for saved output (without extension)

Async: await s.aget_user_info(["elonmusk"])

X sometimes refuses the id lookup for a whole connection (HTTP 403). The call then raises an error that names the refusal instead of returning an empty list. Retry later, or look the same profiles up by username.

Search Users¶

Find accounts by keyword — the "People" tab of a search:

people = s.search_users("python developer", limit=50)
# User records, with type: "search_users"

Takes limit, max_empty_pages, raw_json and the save parameters. Async: asearch_users.

The trends of the explore page, one request:

trends = s.get_trending()
# [{"name": "...", "context": "1 day ago · Sports · 12K posts", "trend_id": "...", "raw": {...}}]

Async: aget_trending.

Manifest Refresh¶

X rotates its GraphQL query ids without notice, and a stale id answers 404 for every request from every account. Scweet reads fresh ids from X's own bundles — including the operations that live in lazy chunks, which a plain main.js scrape misses:

s = Scweet(auth_token="...", manifest_scrape_on_init=True)   # refresh at startup

changes = s.refresh_manifest()   # refresh at any moment
# {"search_timeline": {"old": "KPSo2_UW...", "new": "aB3xY9..."}} — empty when current

scweet refresh-manifest

When every search suddenly answers 404 while the accounts look healthy, refresh the manifest first: that 404 describes the request, not the credentials.

Saving Results¶

By default, results are returned in-memory only (save=False). To persist to disk:

# Save as CSV (default format)
tweets = s.search("query", since="2024-01-01", limit=200, save=True)

# Save as JSON
tweets = s.search("query", since="2024-01-01", limit=200, save=True, save_format="json")

# Save both CSV and JSON
tweets = s.search("query", since="2024-01-01", limit=200, save=True, save_format="both")

Output files are written to the save_dir directory (default: "outputs"). File names are based on the operation type (search.csv, profile_tweets.json, followers.csv, etc.).

The default format can be set globally via ScweetConfig(save_format="json").

Output Schemas¶

Tweet record¶

Returned by search() and get_profile_tweets(). One real row, with raw and the long text removed:

{
    "tweet_id": "2090227249551483321",
    "text": "The text of the tweet",
    "timestamp": "Wed Aug 19 23:59:45 +0000 2026",
    "tweet_url": "https://x.com/bitcoinvaccine/status/2090227249551483321",
    "user": {"screen_name": "bitcoinvaccine", "name": "bitcoin vaccine", "followers_count": 527},
    "likes": 11, "retweets": 0, "comments": 3, "quotes": 0, "bookmarks": 1, "views": 355,
    "lang": "ko",
    "media": {"image_links": ["https://pbs.twimg.com/media/HQH6W7zaMAAurOm.jpg"], "video_links": []},
    "hashtags": [], "mentions": [], "urls": [],
    "is_quote": False, "is_retweet": False,
    "quoted_tweet": None, "retweeted_tweet": None,
    "in_reply_to_tweet_id": None, "in_reply_to_user": None,
    "embedded_text": None, "emojis": None,
    "raw": {"...": "the full answer of X"},
}

Each method returns a list of these. The table below holds every field.

Field	Type	Description
`tweet_id`	`str`	Tweet ID
`timestamp`	`str`	Post time — Twitter date string, e.g. `"Thu Mar 20 22:25:15 +0000 2025"`
`user`	`dict`	Author: `{"screen_name": "elonmusk", "name": "Elon Musk"}`
`text`	`str`	Full tweet text
`likes`	`int`	Like count
`retweets`	`int`	Retweet count
`comments`	`int`	Reply count
`tweet_url`	`str`	Permalink, e.g. `"https://x.com/user/status/123"`
`media`	`dict \\| None`	`{"image_links": [...], "video_links": [...]}` — `video_links` holds the highest-bitrate MP4 of each video
`embedded_text`	`str \\| None`	Text of the quoted or retweeted tweet — `None` for plain tweets
`emojis`	`str \\| None`	Always `None`. The field stays for a script written against version 4. Read the emoji from `text`.
`raw`	`dict`	Full GraphQL payload
`views`	`int \\| None`	View count — `None` when X sends none
`quotes`	`int \\| None`	Quote count
`bookmarks`	`int \\| None`	Bookmark count
`lang`	`str \\| None`	Language code X assigned, e.g. `"en"`
`hashtags`	`list[str]`	Hashtags, without the `#`
`mentions`	`list[str]`	Mentioned screen names
`urls`	`list[str]`	Expanded links in the tweet
`in_reply_to_tweet_id`	`str \\| None`	The tweet this one answers
`in_reply_to_user`	`str \\| None`	The screen name it answers
`is_quote`	`bool`	The tweet quotes another tweet
`is_retweet`	`bool`	The tweet is a retweet
`quoted_tweet`	`dict \\| None`	The quoted tweet, as a tweet record one level deep
`retweeted_tweet`	`dict \\| None`	The retweeted tweet, as a tweet record one level deep

The fields from views down arrived in 5.6.0. Every one is additive: a script written against an earlier version reads the same values from the fields above them. A count is None and never 0 when X sends no value, so a real zero stays a zero.

quoted_tweet and retweeted_tweet nest one level only, and their own raw is None, because the raw of the parent already holds the nested payload. embedded_text keeps its old value: it reads the truncated text of X, so it can be shorter than the text of the nested record.

CSV output flattens user and media: user_screen_name, user_name, and image_links become their own columns (in that order). JSON output and the Python return value preserve the nested structure above.

User record¶

Returned by get_followers(), get_following(), get_verified_followers(), and get_user_info().

Field	Type	Description
`user_id`	`str`	Twitter user ID
`username`	`str`	Screen name / handle
`name`	`str`	Display name
`description`	`str`	Bio text
`location`	`str \\| None`	Self-reported location
`created_at`	`str`	Account creation date (Twitter date string)
`followers_count`	`int`	Follower count
`following_count`	`int`	Following count
`statuses_count`	`int`	Total tweets posted
`favourites_count`	`int`	Total likes given
`media_count`	`int`	Media tweet count
`listed_count`	`int`	List membership count
`verified`	`bool`	Legacy verified badge
`blue_verified`	`bool`	Twitter Blue / paid verification
`protected`	`bool`	Protected (private) account
`profile_image_url`	`str`	Profile photo URL
`profile_banner_url`	`str`	Banner image URL
`url`	`str \\| None`	Website URL set in bio
`identity_verified`	`bool`	X confirmed the identity of the person
`pinned_tweet_ids`	`list[str]`	The pinned tweet of the profile
`description_urls`	`list[str]`	Expanded links inside the bio
`raw`	`dict`	Full GraphQL payload (only present when `raw_json=True`)

The counts above were 0 for every profile before 5.6.0. X stopped sending the flat legacy object and moved the counts into other nodes of the answer, and the parser read only the old place. Version 5.6.0 reads both, so an older answer of X keeps working.

Followers / following only: each record also has type ("followers", "following" or "verified_followers") and target (info about the queried account).

User info only: each record has input (the queried input) instead of type/target. The raw field is omitted by default (not present unless the underlying engine returns it).

Resume Interrupted Searches¶

Resume a search from where it left off using SQLite cursor checkpoints:

# First run — gets interrupted or completes partially
tweets = s.search("query", since="2024-01-01", until="2024-06-01", limit=1000)

# Resume — picks up from last saved checkpoint
tweets = s.search("query", since="2024-01-01", until="2024-06-01", limit=1000, resume=True)

Resume works by matching a hash of the query parameters. The same since, until, query, lang, and display_type must be provided to resume correctly.

Configuration Reference¶

All fields have sensible defaults. Override with ScweetConfig:

from Scweet import Scweet, ScweetConfig

s = Scweet(
    cookies_file="cookies.json",
    config=ScweetConfig(
        concurrency=3,
        proxy="http://user:pass@host:port",
        min_delay_s=2.0,
    ),
)

Core¶

Field	Type	Default	Description
`db_path`	`str`	`"scweet_state.db"`	SQLite state file path
`proxy`	`str \\| dict \\| None`	`None`	HTTP proxy for API calls
`concurrency`	`int`	`5`	Number of parallel workers

Output¶

Field	Type	Default	Description
`save_dir`	`str`	`"outputs"`	Default output directory
`save_format`	`str`	`"csv"`	Default format: `"csv"`, `"json"`, or `"both"`

HTTP Tuning¶

Field	Type	Default	Description
`api_http_mode`	`str`	`"auto"`	HTTP mode: `"auto"`, `"async"`, `"sync"`
`api_http_impersonate`	`str \\| None`	`"chrome"`	Browser impersonation target for curl_cffi. `"chrome"` follows the newest Chrome fingerprint that the installed curl_cffi supports.
`api_user_agent`	`str \\| None`	`None`	Custom User-Agent string
`request_404_retries`	`int`	`1`	Retries of a request that answers 404, after Scweet builds a fresh transaction id. X answers 404 when the id is stale, so one retry usually repairs the request. `0` stops the retry.
`transaction_init_attempts`	`int`	`3`	Attempts to build the transaction id when the client starts. X refuses a request with no `x-client-transaction-id` header, so a failure here stops every request.
`transaction_init_backoff_s`	`float`	`1.5`	Seconds between two attempts to build the transaction id. Each attempt waits this value multiplied by the number of the attempt.

Batch Sizes¶

One request of X carries several ids. These values hold the largest batch that each endpoint accepts.

Field	Type	Default	Description
`tweet_lookup_batch_size`	`int`	`50`	Tweet ids in one request of `get_tweet_info()`. X accepts 50.
`user_lookup_batch_size`	`int`	`100`	User ids in one request of `get_user_info(user_ids=…)`. X accepts 100.
`trending_count`	`int`	`20`	Trends that `get_trending()` asks for.

Rate Limiting¶

Field	Type	Default	Description
`daily_requests_limit`	`int`	`300`	Max API requests per account per day
`daily_tweets_limit`	`int`	`6000`	Max tweets per account per day
`max_empty_pages`	`int`	`3`	Stop only after N consecutive empty result pages. X sends a stray empty page mid-stream, so a value of 1 loses the rest of the results.
`api_page_size`	`int`	`20`	Tweets per API page (1-100)
`window_request_limit`	`int`	`50`	Requests allowed per account per rate-limit window. X counts the total in a window, so a short run bursts this budget and waits nothing.
`relationship_window_request_limit`	`int`	`45`	Window budget for the followers/following paths. X allows 50 there and restricts an account more easily than at a search, so this budget keeps a margin of 5.
`rate_limit_window_s`	`float`	`900.0`	Length of the rate-limit window in seconds (X uses ~15 minutes)
`min_delay_s`	`float`	`1.0`	Floor between two requests of one account, so a burst does not arrive at wire speed. `0` removes the floor; the window limit still respects X.
`rate_limit_min_remaining`	`int`	`2`	Hand off an account when `x-rate-limit-remaining` falls to this value, and rest it until `x-rate-limit-reset`. The margin stops the account a few requests before X answers `429`, which loses a page. `0` hands off only when the window is fully spent.
`requests_per_min`	`int`	`30`	Deprecated. The limiter paces to `window_request_limit` over `rate_limit_window_s`, not to a per-minute rate. Kept so an old config still loads.

Advanced¶

Field	Type	Default	Description
`enable_wal`	`bool`	`True`	SQLite WAL mode
`busy_timeout_ms`	`int`	`5000`	SQLite busy timeout
`lease_ttl_s`	`int`	`120`	Account lease time-to-live
`lease_heartbeat_s`	`float`	`30.0`	Heartbeat interval for active leases
`cooldown_default_s`	`float`	`120.0`	Default cooldown after rate limit
`transient_cooldown_s`	`float`	`120.0`	Cooldown for transient errors (e.g., 404/stale query IDs)
`auth_cooldown_s`	`float`	`2592000.0`	Cooldown when an account is proven dead (30 days). A 401/403 from a page gives only a short cooldown; the long block applies only after a self-lookup of the account's own handle also fails.
`cooldown_jitter_s`	`float`	`10.0`	Random jitter added to cooldowns
`locked_cooldown_s`	`float`	`3600.0`	Rest time for an account that X locked behind a human challenge (code 326 in a 200 answer). The log names the unlock page; the account retries after each rest until you unlock it.
`pool_wait_max_s`	`float`	`120.0`	When every account is on cooldown, wait up to this long for one to expire before failing. Set `0` to fail immediately.
`pool_wait_poll_s`	`float`	`5.0`	How often to retry leasing while waiting for a cooldown to expire
`task_retry_base_s`	`int`	`1`	Base delay for task retry backoff
`task_retry_max_s`	`int`	`30`	Max delay for task retry backoff
`max_task_attempts`	`int`	`3`	Max retry attempts per task
`max_fallback_attempts`	`int`	`3`	Max fallback attempts on failure
`max_account_switches`	`int`	`2`	Max account switches per task
`scheduler_min_interval_s`	`int`	`300`	Minimum time interval split (seconds)
`n_splits`	`int`	`5`	Number of time interval splits for search. The run raises this to the account count when there are more accounts, so every account does work.
`max_interval_depth`	`int`	`100`	How many times the run may continue a `Latest` interval when a cursor chain ends while tweets remain; it continues from the oldest tweet. A `Top` search is ranked, not in time order, so the run does not continue a `Top` interval (use `Latest` for a large volume). This is a safety backstop; the floor `scheduler_min_interval_s` and the result limit are the real bounds. `0` turns the continuation off.
`priority`	`int`	`1`	Task priority
`proxy_check_on_lease`	`bool`	`True`	Verify proxy connectivity before leasing
`proxy_check_url`	`str`	`"https://x.com/robots.txt"`	URL for proxy check
`proxy_check_timeout_s`	`float`	`10.0`	Timeout for proxy check
`profile_timeline_allow_anonymous`	`bool`	`False`	Allow anonymous profile timeline requests

Manifest (Query IDs)¶

Field	Type	Default	Description
`manifest_url`	`str \\| None`	`None`	Remote manifest URL for query IDs
`manifest_ttl_s`	`int`	`3600`	Cache TTL for remote manifest
`manifest_update_on_init`	`bool`	`False`	Fetch remote manifest on init
`manifest_scrape_on_init`	`bool`	`False`	Scrape fresh query IDs from X on init

Account Management (ScweetDB)¶

ScweetDB provides direct access to the SQLite state for account inspection and management:

from Scweet import ScweetDB

db = ScweetDB("scweet_state.db")

accounts_summary()¶

summary = db.accounts_summary()
# {"db_path": "...", "total": 5, "eligible": 3, "unusable": 1, "cooling_down": 1, ...}

list_accounts()¶

accounts = db.list_accounts(limit=10, eligible_only=True)
# Returns list of account dicts with redacted secrets (fingerprints only)

# Include cookie keys
accounts = db.list_accounts(include_cookies=True)

# Reveal full secrets (use with caution)
accounts = db.list_accounts(reveal_secrets=True)

get_account(username)¶

account = db.get_account("my_account")

repair_account(username)¶

Reset cooldowns, clear leases, and optionally refresh auth tokens:

result = db.repair_account("my_account")
# {"updated": 1, "changes": ["cooldown_cleared", "lease_cleared", ...], ...}

# Force token refresh even if auth material looks valid
result = db.repair_account("my_account", force_refresh=True)

reset_account_cooldowns()¶

# Reset all account cooldowns
db.reset_account_cooldowns()

# Reset specific accounts
db.reset_account_cooldowns(usernames=["account1", "account2"])

# Include unusable accounts (reactivates them)
db.reset_account_cooldowns(include_unusable=True)

clear_leases()¶

# Clear expired leases only (safe)
db.clear_leases(expired_only=True)

# Clear all leases
db.clear_leases(expired_only=False)

reset_daily_counters()¶

db.reset_daily_counters()

Other methods¶

delete_account(username) — Remove an account from the pool.
set_account_proxy(username, proxy) — Set or clear a per-account proxy override.
mark_account_unusable(username) — Mark an account as unusable (won't be leased).
import_accounts_from_sources(...) — Import accounts from files/cookies into the DB.
collapse_duplicates_by_auth_token(dry_run=True) — Find and merge duplicate accounts.
get_checkpoint(query_hash) / clear_checkpoint(query_hash) / clear_all_checkpoints() — Manage resume checkpoints.
list_runs(limit=50) / last_run() / runs_summary() — Inspect run history.

Logging¶

Scweet uses Python's standard logging module under the "Scweet" logger namespace. By default no output is produced (NullHandler — standard library practice). To see logs, configure a handler on the "Scweet" logger in your application:

import logging

logging.basicConfig(level=logging.INFO)
# or target only Scweet:
logging.getLogger("Scweet").setLevel(logging.INFO)
logging.getLogger("Scweet").addHandler(logging.StreamHandler())

The CLI automatically sets up INFO-level logging to stderr. Pass -v / --verbose to the CLI for DEBUG-level output.

Async Usage¶

All public methods have async variants. Use them in async contexts:

import asyncio
from Scweet import Scweet

async def main():
    s = Scweet(cookies_file="cookies.json")

    tweets = await s.asearch("query", since="2024-01-01", limit=100)
    profiles = await s.aget_user_info(["elonmusk"])
    followers = await s.aget_followers(["elonmusk"], limit=500)

asyncio.run(main())

The sync methods (search, get_followers, etc.) wrap their async counterparts with asyncio.run(), so they cannot be called from within an already-running event loop.

No close needed¶

Scweet and ScweetDB don't require explicit closing. HTTP sessions are created and closed per-request internally, and SQLite connections are scoped per-operation. You can create a Scweet instance, use it, and let it go out of scope — no resource leaks.

Error Handling¶

All Scweet methods raise exceptions on failure. Wrap calls in try/except to handle errors explicitly:

from Scweet import Scweet, AccountPoolExhausted, RateLimitError, AuthError, NetworkError, RunFailed

s = Scweet(cookies_file="cookies.json")

try:
    tweets = s.search("query")
except AccountPoolExhausted as e:
    print(f"No accounts available: {e}")
except RateLimitError:
    print("Rate limited — wait for cooldowns and retry")
except AuthError:
    print("Credentials expired — refresh your auth_token/ct0")
except NetworkError:
    print("Network issue — check your connection or proxy")
except RunFailed as e:
    print(f"Scrape failed: {e}")

This applies to all methods: search, get_profile_tweets, get_followers, get_following, and get_user_info.

Exception hierarchy¶

All Scweet exceptions inherit from ScweetError, so you can catch everything with a single handler:

ScweetError                          # Base — catch-all
  AccountPoolExhausted               # No eligible accounts (all cooled down / at daily limits)
  ConfigError                        # A setting is invalid — raised before any request
  ManifestError                      # The GraphQL query ids could not be loaded or validated
  ResumeError                        # The checkpoint of a resumed run is missing or unreadable
  AccountSessionBuildError           # An account session could not be built
    AccountSessionAuthError          # The cookies of that account are missing or invalid
    AccountSessionTransientError     # A temporary fault — the account stays in the pool
    AccountSessionRuntimeError       # An unexpected fault while the session started
  EngineError                        # Engine-level runtime error
    RunFailed                        # Run completed but couldn't produce results
      RateLimitError                 # All accounts rate-limited (429) — wait and retry
      AuthError                      # Credentials invalid or expired (401/403)
      NetworkError                   # Network/connectivity failure
      ProxyError                     # Proxy misconfiguration or connectivity failure

The AccountSession* group tells a dead account from a temporary fault. AccountSessionAuthError means the cookies of that one account are finished, so Scweet cools it for a long period. AccountSessionTransientError means the fault was temporary, so the account stays available.

These eight are importable from the top-level package:

from Scweet import (
    ScweetError, AccountPoolExhausted,
    RunFailed, RateLimitError, AuthError, NetworkError, ProxyError, EngineError,
)

The rest come from the module of the exceptions:

from Scweet.exceptions import (
    ConfigError, ManifestError, ResumeError,
    AccountSessionBuildError, AccountSessionAuthError,
    AccountSessionTransientError, AccountSessionRuntimeError,
)

Troubleshooting¶

Empty results / fewer tweets than expected - Check your date range — Twitter search is often shallow on older dates - Check display_type — the default "Latest" is chronological; "Top" is a ranked selection and holds far fewer tweets - Your account may have hit its daily cap (daily_requests_limit / daily_tweets_limit in ScweetConfig). Check with ScweetDB("scweet_state.db").accounts_summary() - Run with logging enabled to see what's happening: logging.basicConfig(level=logging.INFO)

AccountPoolExhausted - All accounts are cooling down or at daily limits. The error message includes counts: total=N, unusable=M, cooling_down=K - Wait for cooldowns to expire, or add more accounts - Reset cooldowns manually: ScweetDB("scweet_state.db").reset_account_cooldowns()

RateLimitError - X has rate-limited your accounts. Wait for cooldowns (usually a few minutes) and retry - Add more accounts to spread the load

AuthError - Your auth_token or ct0 cookie has expired — refresh them from your browser - Use ScweetDB("scweet_state.db").repair_account("username", force_refresh=True) to trigger token refresh

RunFailed / NetworkError - Check your internet connection and proxy configuration - X may have rotated GraphQL query IDs — pass manifest_scrape_on_init=True to Scweet() (or --manifest-scrape-on-init in the CLI) to auto-fetch fresh ones - 404 errors in logs mean stale query IDs (transient) — not bad auth

CLI¶

Scweet ships with a scweet command-line tool installed automatically alongside the package. No extra setup required — just pip install -U Scweet.

Usage pattern¶

scweet [auth options] [config options] <subcommand> [subcommand options]

Global options¶

Auth:

Flag	Description
`--auth-token TOKEN`	`auth_token` cookie value
`--cookies-file FILE`	Path to a cookies JSON file
`--env-file FILE`	Path to a `.env` file
`--db-path PATH`	SQLite state file (default: `scweet_state.db`)

Config:

Flag	Description
`--proxy PROXY`	Proxy URL or JSON string
`--concurrency N`	Worker concurrency (default: `5`)
`--manifest-scrape-on-init`	Scrape fresh query IDs from X's main.js on startup

Output options¶

Available on every subcommand:

Flag	Description
`--save`	Save results to file
`--save-format {csv,json,both}`	File format (default: `csv`)
`--save-dir DIR`	Output directory (default: `outputs`)
`--save-name NAME`	Base filename for saved output
`--pretty`	Print results as indented JSON to stdout

By default the CLI runs silently — no output is printed. Use --save to write results to a file, --pretty to print to stdout, or both together.

Subcommands¶

`search`¶

scweet --auth-token TOKEN search [QUERY] [options]

QUERY is optional — you can use filters alone.

Flag	Description
`--since DATE`	Start date `YYYY-MM-DD`
`--until DATE`	End date `YYYY-MM-DD`
`--limit N`	Max tweets to return
`--lang CODE`	Language code (e.g. `en`)
`--display-type {Top,Latest}`	Default: `Latest`
`--from USER [USER ...]`	Tweets from these users
`--to USER [USER ...]`	Tweets sent to these users
`--mention USER [USER ...]`	Tweets mentioning these users
`--all-words WORD [WORD ...]`	Tweets containing ALL of these words (AND)
`--any-words WORD [WORD ...]`	Tweets containing ANY of these words (OR)
`--exact-phrases PHRASE [PHRASE ...]`	Tweets containing these exact phrases
`--hashtags-any TAG [TAG ...]`	Tweets containing any of these hashtags
`--hashtags-exclude TAG [TAG ...]`	Exclude tweets with these hashtags
`--exclude-words WORD [WORD ...]`	Exclude tweets with these words
`--tweet-type {originals-only,replies-only,retweets-only,exclude-replies,exclude-retweets}`	Filter by tweet type
`--min-likes N`	Minimum likes
`--min-replies N`	Minimum replies
`--min-retweets N`	Minimum retweets
`--has-images`	Must contain images
`--has-videos`	Must contain videos
`--has-links`	Must contain links
`--has-mentions`	Must contain @mentions
`--has-hashtags`	Must contain hashtags
`--verified-only`	Verified accounts only
`--blue-verified-only`	Blue verified accounts only
`--place PLACE`	Place filter
`--geocode GEOCODE`	Geocode filter (e.g. `40.7,-74.0,10km`)
`--near PLACE`	Near this location (e.g. `"San Francisco"`)
`--within RADIUS`	Radius for `--near` (e.g. `15km` or `10mi`)
`--resume`	Resume from last checkpoint
`--max-empty-pages N`	Stop after N consecutive empty pages

`profile-tweets`¶

scweet --auth-token TOKEN profile-tweets USER [USER ...] [options]

Flag	Description
`--limit N`	Max tweets to return
`--include-replies`	Read the tab "Posts and replies" instead of "Posts", so the result holds the replies of the user
`--resume`	Resume from last checkpoint
`--max-empty-pages N`	Stop after N consecutive empty pages

`followers` / `following` / `verified-followers`¶

scweet --auth-token TOKEN followers USER [USER ...] [options]
scweet --auth-token TOKEN following USER [USER ...] [options]
scweet --auth-token TOKEN verified-followers USER [USER ...] [options]

Flag	Description
`--limit N`	Max users to return
`--resume`	Resume from last checkpoint
`--max-empty-pages N`	Stop after N consecutive empty pages
`--raw-json`	Return raw API JSON instead of normalized dicts

`user-info`¶

scweet --auth-token TOKEN user-info USER [USER ...]
scweet --auth-token TOKEN user-info --ids ID [ID ...]

No pagination — one API call per user, or one call per 100 ids.

`tweet-info` / `tweet-replies` / `reposters`¶

scweet --auth-token TOKEN tweet-info ID [ID ...]           # up to 50 ids per request
scweet --auth-token TOKEN tweet-replies ID --limit 200
scweet --auth-token TOKEN reposters ID --limit 500

`search-users` / `trending` / `profile-media`¶

scweet --auth-token TOKEN search-users "python developer" --limit 50
scweet --auth-token TOKEN trending --pretty
scweet --auth-token TOKEN profile-media nasa --limit 100

`refresh-manifest`¶

scweet refresh-manifest
# reposters: ROjiuYUeo... -> iH7h2J19n...   (or: "Every query id is current.")

Examples¶

# Search (defaults to last 30 days — set explicit dates for reproducibility)
scweet --auth-token TOKEN search "ChatGPT" --limit 200 --pretty

# Search with date range and engagement filters
scweet --auth-token TOKEN search "AI tools" \
  --since 2026-01-01 --until 2026-06-01 \
  --min-likes 100 --has-images --limit 500

# Tweets from specific accounts containing a hashtag
scweet --auth-token TOKEN search \
  --from elonmusk naval sama \
  --hashtags-any AI startups --limit 100

# Pull a user's timeline, save to JSON
scweet --cookies-file cookies.json profile-tweets elonmusk \
  --limit 200 --save --save-format json

# Get followers and pipe to jq
scweet --auth-token TOKEN followers elonmusk --limit 1000 --pretty | jq '.[].username'

# Lookup multiple profiles
scweet --auth-token TOKEN user-info elonmusk OpenAI sama --pretty

# Resume a previously interrupted search
scweet --auth-token TOKEN search "python" --since 2026-01-01 --resume

Help¶

scweet --help
scweet search --help
scweet followers --help

Migration from v4¶

v4	v5
`Scweet.from_sources(...)`	`Scweet(cookies_file=...)`
`scweet.scrape(words=["bitcoin"], ...)`	`s.search("bitcoin", ...)`
`scweet.ascrape(...)`	`s.asearch(...)`
`scweet.profile_tweets(usernames=[...])`	`s.get_profile_tweets([...])`
`scweet.get_user_information(usernames=[...])`	`s.get_user_info([...])`
`ScweetConfig.from_sources(overrides={...})`	`ScweetConfig(field=value)`
Nested config (`pool.concurrency`)	Flat config (`concurrency`)
`from Scweet.scweet import Scweet`	`from Scweet import Scweet`

MIT License