Market Data 101 [Part 2]

This is Part 2 of a 3-part blog series that breaks down the basics of market data. It covers the different ways you can access market data, how licenses and fees work, and where you can buy market data. If you work in the fintech space as an engineer, quant, or founder, this is a great resource to gain a base understanding of how and where to access market data.

Different ways you can access stock market data

There are three primary ways that you can access Market Data: Colocation, Web Methods, and Non-streaming solutions.

Colocation

Colocation is a complex, infrastructure-heavy way to connect directly to a stock exchange for the most up-to-date market data. It involves putting your own servers in close proximity to the exchanges. The sheer expense of co-locating your servers makes this type of solution inaccessible for most investors. However, colocation is necessary for enterprises that are engaging in systematic or high frequency trading. The closer your servers are to the exchange, the sooner you will have access to critical stock price changes.  

Most stock exchanges offer Co-location or (CoLo) services which enable clients to put their servers and equipment directly within an exchange’s data center. Unless you are starting a quant fund or planning on executing a strategy that relies on extremely low latency market changes, I would not recommend co-location as an access method for market data.

Web Methods

Web methods, including RESTful Api or WebSocket Api delivery, are the most popular, affordable, and accessible ways to access market data. API access to market data functions like a “pull” whereas WebSocket access to market data functions like a “push”. With an API, you make or schedule API calls (using code) to query a stock price. When the API call is executed, the stock price is returned from the database of whichever company’s API you are using. With a WebSocket, you set up a “channel” to stream prices that are “pushed” to you. Once this channel is open, you’ll only get updated data on your end when a price has changed, which can be an efficient benefit.

Non-Streaming Solutions

Non-streaming solutions provide market data with a delay. These solutions are quite common and are easier to set up and maintain and cheaper options when your use case doesn’t require the most up to date data.

Typically, non-streaming solutions are referred to as end of day or EOD data because they update once per day after markets close and include the final prices for securities at the close of trading, generally around 4 to 5 PM eastern time. There are non-streaming solutions that update more frequently, such as hourly, but the vast majority of charting, back-testing, and historical applications for market data across asset classes updates once per day.  

Snowflake

Snowflake is a relative newcomer in the non-streaming data space, however this publicly traded company is quickly gaining acceptance as a delivery method for numerous data types but for market data in particular. Snowflake allows data to be stored in a single location with numerous users being granted access which is a big improvement in storage and transfer costs compared to other non-streaming solutions.  

Its very easy to understand why snowflake is growing in popularity when large historical market datasets, such as historical tick data for options, are considered. These datasets can easily generate several gigabytes of data daily- with snowflake that data can update and be queried in one database without data consumers needing to transfer such large files to their own servers on a regular basis.

File Transfer Protocol (FTP)

On the other end of the spectrum for non-streaming solutions is the file transfer protocol or FTP.

This access method has been around for decades and despite being definitively old school, is still one of the most popular methods for institutions to access non-streaming data like end of day stock prices. Setting up an FTP connection allows files to be transferred on a set schedule and many developers are comfortable with this method. A sender provides credentials and instructions to a data consumer and then, at regular intervals, the receiver updates their local files with new data from the sender’s servers.

Bulk Download

Bulk downloads are one of the easiest methods for transferring non-streaming files.  

With this method, a data consumer simply downloads a file, either by manually clicking on a link or by using an API endpoint that is capable of downloading CSV, JSON, or some other structured file format. This access method can be automated but it is a common method for less-technical users that want to update a large historical dataset at irregular intervals.

‍
Licensing & Fees Overview: What is an Exchange Fee?

When exchanges transact securities, they create data about the level or price of those securities as well as bids, asks, and other information that is useful for investors. When an individual or organization wants to access this data the exchange can charge a fee. Not all exchanges charge a fee to access their data but larger exchanges, especially those that are heavily regulated by organizations such as the SEC, do have exchange fees.  

These fees can be set independently by the exchange, but it is more common that a regulating body will need to approve the fees an exchange charges. This approval process generally comes with rules exchanges must follow when charging fees. These rules are intended to protect investors and ensure exchanges don’t discriminate or play favorites.  

For many exchanges and data types this means that exchange fees are set in stone. Depending on the use case, or type of data, the exchange must charge everyone the same fee. Exchange fees range from 0 to tens of thousands of dollars per month. There is some controversy around the high cost of exchange fees- in many cases, exchange fees are too expensive for individuals or small businesses. This high cost can result in the need to find alternative sources of data with lower or no exchange fees.  

In many cases, however, there is no way to avoid paying exchange fees and it is important to find out exactly what the exchange fees will be and what alternatives exist before committing to a solution. Delaying data, choosing a lower volume exchange, or using data internally rather than displaying it are all common ways to reduce or eliminate exchange fees.

What is a per user fee?

In addition to set monthly exchange fees to access data, exchanges typically charge per user fees that vary depending on the number of users that will view or access data.

If, for example, a business is building a trading terminal that shows the stock price for a particular security, the company building the terminal will pay an exchange fee to access that data. Then, the company will pay a per-user fee for each user that has access to the terminal.

Not all data types or exchanges have per user fees- frequently, delayed data or end of day data come with no per user fees. In some cases, per user fees are charged on a monthly basis- a flat fee is paid for every month a user had access to data. Sometimes, companies charge on a per query basis each time data is accessed. Companies that are accessing exchange data will need to determine if monthly per user fees or per query fees will be more cost effective for their application.

If an exchange has per user fees, data consumers will need to track and report each user that has access to market data and report that access to the exchange.  

This creates an administrative burden, but in many cases there is no way to avoid these fees. The SEC mandates that brokerages, for example, display data when investors buy or sell stock on their platforms- this data has per user fees. One way to reduce the cost of per user fees is to choose a data source without them or to supplement sources with per user and exchange fees with lower cost alternatives whenever possible.

What is “Vendor of Record”?

A vendor of record is a company that takes responsibility for tracking and reporting per user fees on behalf of a data user.

In many cases, companies have specialized technology and business processes that make it easier for them to track entitlements for clients than it is for clients to build their own systems. Vendors of record typically charge a fixed fee or a small per user fee to take on the responsibility of authenticating, entitling, and reporting how many users have access to market data at an organization. If a company cannot take this responsibility on themselves, they need to purchase this service from a vendor of record if the exchange data they are using has per user fees and reporting requirements.

What is a “Technology Fee”?

Often market data is distributed through vendors rather than directly from the exchanges that create that data. It can be difficult for many businesses that need market data to get that data from exchanges if the exchanges require servers to be collocated or direct connections to be made in physical data centers. Many market data consumers can’t or don’t want to set up the physical operations required to obtain market data.

In some cases, vendors have superior technology, service, and tools that make it simpler or cheaper to access data from them instead of setting up direct exchange access. SDKs, interval bars, APIs, normalization of data, security masters, derivative data, server monitoring, documentation, and licensing support are just a few of the benefits that party market data vendors can provide that many exchanges don’t.

Third party vendors charge a fee to access market data through their platforms. This fee is often referred to as a technology fee but is also called a vendor fee, or licensing fee. These costs are paid in addition to exchange fees and per user fees and it is important for data consumers to know the difference between these costs. A common deceptive tactic used by third party vendors is to quote their vendor fee to data consumers without making it clear what the additional exchange fees will be.  

This makes their service look like a cheaper alternative but after exchange fees are added in the total cost is typically more expensive. Despite increasing upfront costs, technology fees to access market data from third party vendors can save data buyers money in the long run. Often times a vendor’s fees save so much development time that they pay for themselves many times over. The added help  party distributors of market data can provide to businesses is often times invaluable.

Display vs Internal Use

One way that third party vendors can help data buyers reduce costs is by advising them in ways to reduce exchange fees.  

There are usually multiple architectures or data sources that will work for a data consumer but the solutions will have wildly different costs. One thing that can increase the cost of a solution are display fees. Display fees can apply when market data will be viewed outside of the company that is licensing it. If a company is using market data internally so that only their employees can see it, these fees can be avoided. Display outside of the firm, on a public website, to clients, or behind a paywall, often results in additional exchange, per user, or vendor fees.  

Not all data sources or vendors have display fees but it is important to find out before committing to a solution so that they can be avoided or minimized.

Real-time vs Delayed vs EOD fees

Exchanges and  party vendors often reduce their fees if market data is delayed.  

Typically, a delay of 15 minutes can reduce or eliminate many fees. End of day non-streaming data often has no fees or even further reduced fees than 15-minute delayed data. In some cases, data needs to be delayed only a few seconds or milliseconds to no-longer qualify as real time. In other cases, it needs to be delayed by more than a day.

When licensing market data be sure to ask if there are price breaks for delayed or end of day data. If your use case doesn’t truly require real time data, delayed data can be a more affordable alternative.

Exchanges are closely regulated & lock down on misuse!

It is also important to remember that the SEC and exchanges are very strict about how market data is licensed. A single violation can result in large fines or the inability to license market data in the future.  

To avoid misusing financial data, make sure to be upfront with regulators and exchanges about what you are using the data for and where you are getting it. Serious investors and businesses don’t do themselves any favors by violating licensing agreements or using data without the proper approvals.

Where can you buy this Data?

Data consumers have many options for licensing market data. Individual users can get data from B2C platforms such as brokerages or apps that offer APIs for individual users. Businesses can license data directly from exchanges, either over the web via methods such as FTP, API, or by physically collocating a server in the exchange’s data center. This method often incurs exchange fees and can be difficult to implement without the resources necessary for physical operations. Traditional vendors make market data available in terminals or via APIs.  

These vendors often have strict licensing requirements and high technology fees that are inaccessible to startups or individuals.  party tech companies like Intrinio try to fill a gap between traditional vendors and direct exchange integrations by offering easier to implement and license access to market data that lowers total cost of ownership.

So remember, although this may feel complicated, companies like Intrinio can help you match your use case to the right affordable solution. In fact, that’s what Part 3 of this video series is all about. Market data is complicated but necessary for all investors – we can't wait to see how you use it! Reach out to one of our data specialists today to find out more about our equities market data packages. And to learn more about all our financial services please visit intrinio.com.  

And don’t forget to check out our channel on YouTube for more informational, fintech videos and our complete Market Data 101 series!