CSV,JSON
Now you can request additional data and/or customized columns!
Try It Now! Certified
Files | Size | Format | Created | Updated | License | Source |
---|---|---|---|---|---|---|
2 | 178kB | csv zip | 5 years ago | 5 years ago | Open Data Commons Public Domain Dedication and License v1.0 | Robert Shiller |
Download files in this dataset
File | Description | Size | Last changed | Download |
---|---|---|---|---|
data | 107kB | csv (107kB) , json (375kB) | ||
s-and-p-500_zip | Compressed versions of dataset. Includes normalized CSV and JSON data with original data and datapackage.json. | 130kB | zip (130kB) |
Signup to Premium Service for additional or customised data - Get Started
This is a preview version. There might be more data in the original version.
Field Name | Order | Type (Format) | Description |
---|---|---|---|
Date | 1 | date (%Y-%m-%d) | |
SP500 | 2 | number | Level ('price') of the S&P 500 index |
Dividend | 3 | number | |
Earnings | 4 | number | |
Consumer Price Index | 5 | number | |
Long Interest Rate | 6 | number | 10 year interest rate (gov bonds) |
Real Price | 7 | number | |
Real Dividend | 8 | number | |
Real Earnings | 9 | number | |
PE10 | 10 | number | Cyclically Adjusted Price Earnings Ratio P/E10 or CAPE |
Use our data-cli tool designed for data wranglers:
data get https://datahub.io/core/s-and-p-500
data info core/s-and-p-500
tree core/s-and-p-500
# Get a list of dataset's resources
curl -L -s https://datahub.io/core/s-and-p-500/datapackage.json | grep path
# Get resources
curl -L https://datahub.io/core/s-and-p-500/r/0.csv
curl -L https://datahub.io/core/s-and-p-500/r/1.zip
If you are using R here's how to get the data you want quickly loaded:
install.packages("jsonlite", repos="https://cran.rstudio.com/")
library("jsonlite")
json_file <- 'https://datahub.io/core/s-and-p-500/datapackage.json'
json_data <- fromJSON(paste(readLines(json_file), collapse=""))
# get list of all resources:
print(json_data$resources$name)
# print all tabular data(if exists any)
for(i in 1:length(json_data$resources$datahub$type)){
if(json_data$resources$datahub$type[i]=='derived/csv'){
path_to_file = json_data$resources$path[i]
data <- read.csv(url(path_to_file))
print(data)
}
}
Note: You might need to run the script with root permissions if you are running on Linux machine
Install the Frictionless Data data package library and the pandas itself:
pip install datapackage
pip install pandas
Now you can use the datapackage in the Pandas:
import datapackage
import pandas as pd
data_url = 'https://datahub.io/core/s-and-p-500/datapackage.json'
# to load Data Package into storage
package = datapackage.Package(data_url)
# to load only tabular data
resources = package.resources
for resource in resources:
if resource.tabular:
data = pd.read_csv(resource.descriptor['path'])
print (data)
For Python, first install the `datapackage` library (all the datasets on DataHub are Data Packages):
pip install datapackage
To get Data Package into your Python environment, run following code:
from datapackage import Package
package = Package('https://datahub.io/core/s-and-p-500/datapackage.json')
# print list of all resources:
print(package.resource_names)
# print processed tabular data (if exists any)
for resource in package.resources:
if resource.descriptor['datahub']['type'] == 'derived/csv':
print(resource.read())
If you are using JavaScript, please, follow instructions below:
Install data.js
module using npm
:
$ npm install data.js
Once the package is installed, use the following code snippet:
const {Dataset} = require('data.js')
const path = 'https://datahub.io/core/s-and-p-500/datapackage.json'
// We're using self-invoking function here as we want to use async-await syntax:
;(async () => {
const dataset = await Dataset.load(path)
// get list of all resources:
for (const id in dataset.resources) {
console.log(dataset.resources[id]._descriptor.name)
}
// get all tabular data(if exists any)
for (const id in dataset.resources) {
if (dataset.resources[id]._descriptor.format === "csv") {
const file = dataset.resources[id]
// Get a raw stream
const stream = await file.stream()
// entire file as a buffer (be careful with large files!)
const buffer = await file.buffer
// print data
stream.pipe(process.stdout)
}
}
})()
S&P 500 index data including level, dividend, earnings and P/E ratio on a monthly basis since 1870. The S&P 500 (Standard and Poor’s 500) is a free-float, capitalization-weighted index of the top 500 publicly listed stocks in the US (top 500 by market cap).
The data provided here is a tidied and CSV’d version of that collected and prepared by the Economist Robert Shiller and made available on his website.
Details of the data construction as described on Shiller’s website (and slightly reformatted):
Stock market data used in my book, Irrational Exuberance [Princeton University Press 2000, Broadway Books 2001, 2nd ed., 2005] are available for download, Excel file (xls). This data set consists of monthly stock price, dividends, and earnings data and the consumer price index (to allow conversion to real values), all starting January 1871.
The price, dividend, and earnings series are from the same sources as described in Chapter 26 of my earlier book (Market Volatility [Cambridge, MA: MIT Press, 1989]), although now I use monthly data, rather than annual data. Monthly dividend and earnings data are computed from the S&P four-quarter totals for the quarter since 1926, with linear interpolation to monthly figures. Dividend and earnings data before 1926 are from Cowles and associates (Common Stock Indexes, 2nd ed. [Bloomington, Ind.: Principia Press, 1939]), interpolated from annual data.
Stock price data are monthly averages of daily closing prices through January 2000, the last month available as this book goes to press. The CPI-U (Consumer Price Index-All Urban Consumers) published by the U.S. Bureau of Labor Statistics begins in 1913; for years before 1913 1 spliced to the CPI Warren and Pearson’s price index, by multiplying it by the ratio of the indexes in January 1913. December 1999 and January 2000 values for the CPI-U are extrapolated. See George F. Warren and Frank A. Pearson, Gold and Prices (New York: John Wiley and Sons, 1935). Data are from their Table 1, pp. 11–14.
For the Plots, I have multiplied the inflation-corrected series by a constant so that their value in january 2000 equals their nominal value, i.e., so that all prices are effectively in January 2000 dollars.
No exact statement on license of original data but given size and factual nature believe one can assume these are public domain (and I, the maintainer, explicitly license under the ODC Public Domain Dedication and License (PDDL)).
That said, it would be natural to credit Robert Shiller for preparing this dataset and kindly making it publicly available.
Notifications of data updates and schema changes
Warranty / guaranteed updates
Workflow integration (e.g. Python packages, NPM packages)
Customized data (e.g. you need different or additional data)
Or suggest your own feature from the link below