How Developers Can Get Symbols From Stock Indexes With Python
source: Photo by Nataliya Vaitkevich from Pexels
If you are a software developer who is interested in day trading and stock investment, then you are definitely the one who is also looking for efficient ways to automate stock analysis and even develop your tools for custom trading or investment strategies.
With a software engineering background, I’m not an exception for this!
These days accurate data is the most precious asset for financial market participants. Hence, it’s not cheap.
No longer Yahoo Finance and Google Finance allow developers to access their full APIs. So, now when dealing with financial reference data, a significant part of the struggle is to get decent constituent data of stock indexes. For example, for my purposes, I can’t easily pull the list of stock symbols of the US indexes below anymore:
If you day trade regularly or from time to time or use more advanced trading strategies with derivatives, you understand how it’s important to screen the market daily and view the list of companies in the major stock indexes for analysis.
In my case, I need the tickers/symbols of US stocks to implement my stock screener. That will allow me to select the best stocks in play for the upcoming intraday. The problem with the existing solutions:
‣ they are usually pricey (from $95 per month)
‣ they have lots of data that my strategies do not require
Unfortunately, there are very few free real-time data providers left. Still, Alpha Vantage is one of the data providers that allows sending 500 requests via their API per day before going premium, and it’s enough for me to implement the tools I needed.
If you want to learn more about what kind of tools I will implement or purchase for my investment and trading strategies, you can join my regular newsletter (+bonus: Top-15 tips to free more cash monthly).
My stock screener is a program that can filter stocks based on specific criteria. To loop through all stocks, I need to fetch the list of most US stock symbols.
There are several ways to scrape this data from financial websites, keeping it hardcoded as a list, or simply requesting the index data from Wikipedia or other index websites. Before fetching from web-pages, I inspected where the data is present in their HTML tags to find enclosed table tags inside. Despite that, I decided to make it easier and just downloaded the .html pages of S&P500, NASDAQ, etc., with the stock symbols into the tables:
import pandas as pd
sp_table = pd.read_html('sp500.html')
sp = sp_table[0]
sp.to_csv('sp500-symbols.csv', columns=['Symbol'])
sp = pd.read_csv('sp500-symbols.csv')
sp_symbols = sp['Symbol'].to_list()
print(sp_symbols)
The read_html function is scraping the stored web-page related to the S&P500 data and return a list of DataFrame objects. In my case, I am only willing to get the data from the “Symbol” column: the DataFrame object at index 0. Then I extract the table with results to my project directory using the to_csv function. Next, the to_list function converts the table’s data to the list type.
Do the same for the list of stocks in the NASDAQ:
nasdaq_table = pd.read_html('nasdaq.html')
nasdaq = nasdaq_table[0]
nasdaq.to_csv('nasdaq-symbols.csv', columns=['Symbol'])
nasdaq = pd.read_csv('nasdaq-symbols.csv')
nasdaq_symbols = nasdaq['Symbol'].to_list()
print(nasdaq_symbols)
You can continue pulling symbols of different stock indexes from the stored web-pages. The last step is to concatenate the stock symbols into one list:
all_symbols = sp_symbols + nasdaq_symbols
print(all_symbols)
This code should be capable of scraping the details of most symbols in the stock indexes.
The concatenated data could be saved as a CSV or a JSON file. Instead of writing the data to any file, you can also connect it to a MySQL database.
If you are in the world of trading, investment, and finance, you probably should keep an eye on thousands of stocks on various stock markets. Sometimes getting the “stock symbol for the company ABC” is not what you only need and that most APIs allow you to do for free.
In the next post, I will show you how the fetched stock symbols from the indexes have been applied to the tool I have implemented to find well-defined entry and exit points for one of my trading strategies.
Disclaimer: Author’s opinions are their own and do not constitute financial advice in any way whatsoever. Nothing published by Author constitutes an investment recommendation, nor should any data or content published by Author be relied upon for any investment activities.
