Big Data, Open Government, and Sunlight
Barbie E. Keiser
Posted On February 28, 2013
Open States is the latest in a collection of Big Data-open government analysis tools from Sunlight Foundation. It has taken Sunlight Labs 4 years to scrape legislative data from 50 state websites (plus the District of Columbia and Puerto Rico), collect, sift, standardize, and make it all available in a single, comprehensive database that can be queried online by anyone via Open States API.
Individuals can use Open States at its most basic level to find who represents them in their state legislature, explore a legislator’s position on any bill, and get alerts when related legislation is introduced. Beyond contact information, Open States lists committee memberships, bill sponsorship, campaign contributions in the two most recent election cycles, coverage in the media (news and blogs), and recent votes. By clicking the link at the bottom of each Open States legislator profile, users will be presented the legislator’s profile from InfluenceExplorer providing details for all who have donated to the legislator’s campaign, with analysis of in-state versus out-of-state contributions; individuals versus PACs; top industries; and top contributors. InfluenceExplorer highlights and connects the dots on public and private influence (connections and relationships) from a number of different sources, including NIMSP—National Institute on Money in State Politics and CRP—Center for Responsive Politics.
The source of Open States data is listed at the bottom of each webpage, elegantly designed for visual clarity. The code for Open States is open source, and the public is encouraged to use the API and download bulk data (JSON or CSV) for additional, in-depth analysis. A number of organizations have used this data to create their own rich analyses, such as StateImpact from NPR and Pension Games from the Chicago Tribune.
Open States makes it easy to follow what’s going on with issues of concern in Washington, D.C., and across the nation. Those interested in following a bill, lawmaker, or issue can create a free Open States account and receive email alerts via Sunlight’s notifications platform, Scout. These customized email or text (SMS) alerts indicate actions Congress or state legislatures take on an issue or a specific bill.
Using Scout, users can search phrases within bills, federal regulations as they are passed, or speeches of legislators in a particular state, for example. Results are returned with source data provided in the sidebar, along with filtering options to limit the results by state or action (e.g., bills passed by the House, speeches by a member of Congress, phrases used by party affiliation). To obtain additional information on any bill, users need to simply click on the title in the search results to view what’s happening with the bill, when it’s coming up for debate on the floor, and links to original source material (e.g., THOMAS). Click on FOLLOW and Scout will notify you as soon as there is any movement on the bill.
Sources for Scout beyond Open States include the Congressional Record, THOMAS, GovTrack, GAO Reports, and the Federal Register. That means that users can use Scout to search proposed and final regulation from every federal agency, updated throughout the day. Search beyond a keyword or phrase is possible (e.g., use the minus sign to exclude terms from search results). Sunlight creates RSS feeds for every alert; users can import other data feeds from anywhere on the internet (e.g., Electionlawblog.org) letting Scout create an alert for any information in that feed. Tagging alerts allows users to aggregate those tags, grabbing the RSS feed, perhaps to make that feed public.
Related Tools From Sunlight
In January 2013, Sunlight Foundation launched Docket Wrench that allows the public to track influence and uncover trends in the federal rulemaking process. The service monitors comments of 10,000 organizations, lobbyists, and campaign contributors as they weigh in on federal rules and proposed regulations from 300-plus federal agencies. The data—pulled from more than 3.5 million OCRed PDF documents—can be searched by agency, company, organization, number, or phrase. Data on the site refreshes daily via the Regulations.gov API. Using Docket Wrench, individuals can do the following:
- Track rulemaking dockets, drill down into the rules and notices they contain, and the comments on those rules, including who commented when and on what
- Investigate similar comments within a rulemaking docket, identifying common language across commenters, thereby determining relationships among the commenters (e.g., evidence of letter writing campaigns)
- Explore a legislator’s position on a bill and get alerts when related legislation is introduced
Capitolwords permits anyone to explore the Congressional Record, parsing the data and visualizing mentions (number) and actual quotes (i.e., phrases most often used by each legislator, including dates), hyperlinked to the full text, and an indication of “popularity” (i.e., frequency of use by state or party). In addition to reviewing “hot words” by state, users can compare phrases used by a party over time, for example, “illegal immigrants” and “dreamers.”
Information and links to other tools developed by Sunlight can be found on the foundation’s website. These include the following:
- Partytime announces fundraising events, “leaked” to Sunlight Foundation from party insiders.
- Clearspending is a scorecard that analyzes how well U.S. government agencies are reporting their spending data on USASpending.gov.
- Open Congress brings together official government data with news and blog coverage, social networking, public participation tools, and more to give you the real story behind what’s happening in Congress.
- OpeningParliament is a global initiative bringing together more than 80 international organizations to develop best practices on how the world's legislatures should be transparent and accountable.
- PublicMarkup allows the public to review and comment on proposed bills before they’re introduced—or while they’re pending—in Congress.
- Roku Open Government Apps stream audiovisual content from the White House, Congress, and the Supreme Court to any Roku player.
- Sunlight Academy is an interactive training portal designed for those wishing to learn more about how to search government databases and use tools and websites that expose money and influence in politics and amplify publicly available data.
- Foreign Lobbying Influence Tracker digitizes information that representatives of foreign governments, political parties, and government-controlled entities must disclose to the U.S. Justice Department when they seek to influence U.S. policy.
- Checking Influence. By securely analyzing your online bank or credit card statement, Checking Influence shows your spending alongside corporate spending on lobbying activities and campaign contributions, helping you to be a more informed consumer and citizen.
- Political Ad Sleuth indicates who is buying airtime for political ads on the top four stations in the top 50 television markets.
- Follow the Unlimited Money, a product of the Sunlight Foundation Reporting Group, is a searchable database tracking organizations that are paying for ads and other forms of political communication (i.e., SuperPACs), with links to Federal Election Commission (FEC) filings.
- Inbox Influence allows you to see the political contributions of the people and organizations that are mentioned in emails you receive. This easy-to-use tool can be used for researching influence background on corporate correspondence, adding context to emailed articles, or discovering who is behind political fundraising solicitations.
- Lobbyist Registration Tracker is a database that allows users to see lobbying registrations as they’re submitted and the trends in issues and registrations over time
- Poligraft is a website and utility that allows anyone to uncover levels of influence in federal and state-level politics and the news coverage of it. Using data from Sunlight’s Influence Explorer, Poligraft allows you to connect the dots between money and politics in Congress. Simply paste the URL or text of a news article, blog post, or press release and Poligraft will create an enhanced view of the people, organizations, and relationships described within it.
- House Expenditure Database is a searchable database of each House member’s expenditures.
- The House Staff Directory empowers the public to better communicate with their elected representatives by making available staff names and (public) phone numbers and addresses. In addition to legislative offices, the directory also includes staffers from offices that support House activities.
- Politwoops is the only comprehensive collection of deleted tweets by U.S. politicians. From quick typos to major gaffes, Politwoops offers a window into what they hoped you didn’t see.
- Call on Congress is a free telephone service available in English and Spanish. Those not comfortable using the web-based tools can dial 1-888-907-6886 to learn more about how representatives in Congress are voting on bills and raising campaign money. Citizens can also ask where they can vote.
- Mobile Apps include Congress for Android, Sitegeist, Congress for Windows Phone 7, Ad Hawk, Open States for iPhone and iPad, Real Time Congress, Sunlight Health, and Upwardly Mobile.
The Sunlight Foundation is a nonpartisan nonprofit that uses cutting-edge technology and ideas to make government transparent and accountable.