Create a new Python virtual environment. You can use venv, virtualenv or preferably pyenv virtualenv, which requires installing pyenv first).

If you don’t have pyenv, try installing with homebrew

If you don’t have pyenv-virtualenv, try installing with homebrew

Note: you may have to manually update ~/.bashrc for virtual env commands to work

Create the environment (with pyenv virtualenv):

Note: you may have to specify the patch version e.g. 3.7.9

Activate the environment

Then load the requirements.txt into the virtual environment:

Copy server_py/flatgov/.env-sample to server_py/flatgov/.env, and change the SECRET_KEY defined in that file.

Also, obtain an API key from ProPublica and add it as PROPUBLICA_CONGRESS_API_KEY. This is used to get Press Statement data from the ProPublica API.

Use Django manage.py commands to download the data and populate the database (see DATABASE).

The core of the application is a bill. This is described in BILL_MODEL, and the model itself is set up in Django in server_py/flatgov/bills/models.py. We model bills at the level of the billnumber, e.g. 116hr1500 is a bill in the 116th Congress, in the House of Representatives, bill number 1500. This bill may have many versions, which may differ significantly from each other (e.g the Introduced version may have just a few sections, while the Reported in House version has an entirely new thousand section bill substituted in its place). Where there are differences, we attempt to process the latest version of any bill (e.g to calculate bill similarity).

To download and process data from earlier congresses, see details in DATA_BACKGROUND. There are ~50Gb of data, total for Congresses 110-117, including processed json files, and `DATA_BACKGROUND`describes options for downloading and processing this data. For a 'quick start', you can use data from only the most recent Congress:

Download data from the most recent Congress

Deployment instructions are in DEPLOYMENT. The application is served on a Linux server (currently Ubuntu Ubuntu 18.04.5 LTS on AWS).

The components of the system are:

• Linux server on AWS (Ubuntu 18.04.5 LTS)

• Nginx web server

• Postgresql server (see DATABASE)

• Elasticsearch server for search and bill similarity processing (see ES_SIMILARITY)

• Python/Django application (this repository)

• uwsgi Python server running the Django application, proxied by Nginx above

• Bill metadata and xml, downloaded using scrapers from unitedstates/congress

• Scrapers: other data scraped from public sources, including:

  -Statements of Administration Policy
  -Press statements
  -Congressional Budget Office reports
  -Congressional Research Service reports
  -Calendar information from various congressional sources

These are described in more detail in SCRAPING.

Bills that are related to each other are identified in three ways:

1. Metadata (in billstatus XML) from the Congressional Research Service identifies bills as identical or related (e.g through a Committee process). We show these in the Related Bills table of the application.

2. Same or similar titles. Two bills are considered related if they have exactly the same title, or differ only in the year (e.g. 'The Very Important Information Act of 2022' and 'The Very Important Information Act of 2023').

3. Calculation of text similarity between bills. We calculate similarity between bills using the bill_similarity module (see below).