Python Removed Wikis for 8 Months to Escape AI Crawlers, Relaunches on GitHub Pages (blogspot.com) 16
In February the Python Software Foundation made its wikis read-only "due to excessive automated traffic and security concerns... that traffic turned out to be almost entirely AI bots that were trying to scrape the entire internet."
But their infrastructure was also outdated, and this week they announced new wikis using GitHub Pages "that address the automated traffic and security concerns." The new wikis [for Python, Jython, and the PSF] will have a more modern front-end, as well as a permissions-based content management system for community members to make updates.
With the old wiki structure, we received a large volume of alerts at all hours, each serious enough to page and wake our on-call infrastructure staff, which is very limited (just one person currently, although this will be changing soon!). The quantity of scraping overloaded our servers, causing outages, which happened often enough that these issues required a significant amount of effort for the PSF staff to manage...
Additionally, some portions of the wikis were still built with Python 2. Python 2 reached end of life on January 1, 2020, and no longer receives upstream security fixes. Moving to a static site makes it very fast and reduces security risks, while also having the benefit of a more modern user interface... GitHub was chosen for the home of the new wikis as this is where CPython, python.org, and other properties the PSF hosts currently live. The hope is to continue to streamline the PSF's hosting and stewardship of services to simplify for both PSF staff and the community folks who contribute to them. Previously, the Python and Jython wikis were user-editable by invitation of an existing editor and the PSF wiki was permissions based. Moving forward, all three of the wikis will be permissions based via the GitHub repository they are hosted on, and changes can be requested through pull requests by anyone in the community...
GitHub allows us to distribute granular control, review changes through pull requests, and maintain the uptime this service deserves. Also, since the site is now generated statically, it removes any worry about AI bots overwhelming the hosts that the wikis were previously hosted on.
But their infrastructure was also outdated, and this week they announced new wikis using GitHub Pages "that address the automated traffic and security concerns." The new wikis [for Python, Jython, and the PSF] will have a more modern front-end, as well as a permissions-based content management system for community members to make updates.
With the old wiki structure, we received a large volume of alerts at all hours, each serious enough to page and wake our on-call infrastructure staff, which is very limited (just one person currently, although this will be changing soon!). The quantity of scraping overloaded our servers, causing outages, which happened often enough that these issues required a significant amount of effort for the PSF staff to manage...
Additionally, some portions of the wikis were still built with Python 2. Python 2 reached end of life on January 1, 2020, and no longer receives upstream security fixes. Moving to a static site makes it very fast and reduces security risks, while also having the benefit of a more modern user interface... GitHub was chosen for the home of the new wikis as this is where CPython, python.org, and other properties the PSF hosts currently live. The hope is to continue to streamline the PSF's hosting and stewardship of services to simplify for both PSF staff and the community folks who contribute to them. Previously, the Python and Jython wikis were user-editable by invitation of an existing editor and the PSF wiki was permissions based. Moving forward, all three of the wikis will be permissions based via the GitHub repository they are hosted on, and changes can be requested through pull requests by anyone in the community...
GitHub allows us to distribute granular control, review changes through pull requests, and maintain the uptime this service deserves. Also, since the site is now generated statically, it removes any worry about AI bots overwhelming the hosts that the wikis were previously hosted on.