{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "

Scraping Fidelity.com

", "In this exercise, you scrape data from fidelity.com. The goal is to get the latest sector performance data from the US markets, and to get the total market capitalization for each sector. ", " ", "You need to write one function: get_us_sector_performance() that will return a list of tuples. Each tuple should correspond to a sector and should contain the following data: ", "

the sector name ", "

the amount the sector has moved ", "

the market capitalization of the sector ", "

the market weight of the sector ", "

a link to the fidelity page for that sector ", " ", "

", "The data should be sorted by decreasing order of market weight. I.e., the sector with the highest weight should be in the first tuple, etc." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Process

", "

Get a list of sectors and the links to the sector detail pages from the url (see function) ", "

Loop through the list and call the function get_sector_change_and_market_cap(sector_page_link) for each sector ", "

Accumulate the name, the change, the capitalization, the weight and the link for each sector in output_list (see function) ", "

Sort the list by market weight" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notes: ", "

Note that the market weight is a string with a % sign at the back. You will need to get rid of the % and convert the string into a float before you can sort it ", "

Your starting data is the url listed below. You need to extract all data, including links to the sector pages, from the page at this url ", "

To sort a list of tuples by an arbitrary element, use the example at the bottom of this notebook" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def get_us_sector_performance(): ", " output_list = list() ", " url = \"https://eresearch.fidelity.com/eresearch/goto/markets_sectors/landing.jhtml\" ", " ", " ", " **** Your code goes here **** ", " ", " return output_list ", " " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def get_sector_change_and_market_cap(sector_page_link): ", " ", " **** Your code goes here **** ", " ", " return sector_change,sector_market_cap,sector_market_weight" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#Test get_sector_change_and_market_cap() ", "link = \"https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=25\" ", "get_sector_change_and_market_cap(link) ", "#Should return (-0.40, $3.58T, 6.80) ", "#Note that neither the -0.40, nor the 6.80, end with a %sign" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#Test get_us_sector_performance() ", "get_us_sector_performance() ", "#Should return (example: obviously the results will vary over time!) ", "\"\"\" ", "[('Telecommunication Services', ", " 0.21, ", " '$1.74T', ", " 2.0, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=50'), ", " ('Materials', ", " 0.22, ", " '$1.95T', ", " 2.49, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=15'), ", " ('Real Estate', ", " -0.45, ", " '$1.22T', ", " 2.7, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=60'), ", " ('Utilities', ", " -0.33, ", " '$1.25T', ", " 2.86, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=55'), ", " ('Energy', ", " 0.76, ", " '$3.90T', ", " 5.83, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=10'), ", " ('Consumer Staples', ", " -0.32, ", " '$3.58T', ", " 6.8, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=30'), ", " ('Industrials', ", " 0.72, ", " '$4.31T', ", " 9.83, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=20'), ", " ('Consumer Discretionary', ", " 1.03, ", " '$5.76T', ", " 12.9, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=25'), ", " ('Financials', ", " 0.39, ", " '$7.45T', ", " 13.71, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=40'), ", " ('Health Care', ", " 0.76, ", " '$5.70T', ", " 14.71, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=35'), ", " ('Information Technology', ", " 0.81, ", " '$9.84T', ", " 26.17, ", " 'https://eresearch.fidelity.com/eresearch/markets_sectors/sectors/sectors_in_market.jhtml?tab=learnor=45')] ", " ", "\"\"\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Sorting

" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "x = [('a',23.2,'b'),('c',17.4,'f'),('d',29.2,'z'),('e',1.74,'bb')] ", "x.sort() #Sorts by the first element of the tuple ", "x" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "x = [('a',23.2,'b'),('c',17.4,'f'),('d',29.2,'z'),('e',1.74,'bb')] ", "x.sort(key=lambda k: k[1]) #Sorts by the element at position 1 ", "x" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python [py36]", "language": "python", "name": "Python [py36]" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 2 }