GitHub scraper API: search repos and profiles in Python
By the AxioAPI team · Published
GitHub holds the best public data on developers and open-source projects, but the official REST API makes you manage tokens and hourly rate limits. A GitHub scraper API returns the same public data as clean JSON behind one key, which suits research tools, recruiting dashboards and star trackers.
What you need
- An AxioAPI key, stored as the AXIOAPI_KEY environment variable.
- Python 3 with the requests package.
- A repository search query or a GitHub username.
Search repositories and read a profile
GET /api/v1/social/github/repos takes a q query, an optional sort (stars, forks or updated) and a limit. GET /api/v1/social/github/users/{username} returns the profile of a user. The script below uses both.
import os
import requests
API = "https://axioapi.com/api/v1/social/github"
HEADERS = {"Authorization": f"Bearer {os.environ['AXIOAPI_KEY']}", "Accept": "application/json"}
def search_repos(query, sort="stars", limit=10):
res = requests.get(f"{API}/repos", headers=HEADERS, timeout=60,
params={"q": query, "sort": sort, "limit": limit})
res.raise_for_status()
return res.json()["data"]["items"]
def user_profile(username):
res = requests.get(f"{API}/users/{username}", headers=HEADERS, timeout=60)
res.raise_for_status()
return res.json()["data"]
if __name__ == "__main__":
for repo in search_repos("web scraping language:python")[:5]:
print(repo["title"], repo["stats"]["likes"], "stars", repo["stats"]["shares"], "forks", repo["extra"]["language"])
profile = user_profile("torvalds")
print(profile["username"], profile["followers"], "followers", profile["posts_count"], "public repos")Read the response
- Each repository in data.items has a title (owner/name), a text description, stats.likes for stars, stats.shares for forks and extra.language.
- A profile has username, followers, posts_count (public repositories) and a link to the profile.
- Missing values come back empty instead of being guessed.
Public activity and repository details
Add GET /api/v1/social/github/users/{username}/activity to see what a developer pushed recently, or GET /api/v1/social/github/repos/{owner}/{repo} to read one repository on a schedule and track its stars over time.
# Search repositories, sorted by stars
curl -s "https://axioapi.com/api/v1/social/github/repos?q=web%20scraping&sort=stars&limit=5" \
-H "Authorization: Bearer $AXIOAPI_KEY"
# Public activity of a user
curl -s "https://axioapi.com/api/v1/social/github/users/torvalds/activity" \
-H "Authorization: Bearer $AXIOAPI_KEY"Compared with the official GitHub API
The official GitHub REST API allows 60 unauthenticated requests per hour and 5,000 per hour per token, and you create and rotate the tokens. With this API you call AxioAPI with one key and pay per request. It returns public data only, never private repositories.