A reckoning for Market and Financial Data APIs

This year, we're seeing a reckoning in the financial data space. The "lag" – the time it takes for users to realize they can't rely on cheap data or for providers to realize their models are unsustainable – has caught up to us. Smaller data companies are shutting their doors, financial data users are fed up with poor quality, and fees for market data are increasing.

The truth is, reliable data costs money. Building and delivering data sets of any quality requires resources—there’s no shortcut to that.

When you think about financial data, it might seem simple to get a quote or chart, but behind the scenes, it’s an incredibly complex process. 

What it takes to build and deliver market data

1. Getting Data from the Source: some text
   1. Data providers have to source and pull data directly from exchanges like the NYSE, regulatory bodies like the SEC, as well as public sources such as FRED. 
   2. These are the 'raw materials' of the financial world.
2. Storing the Data: some text
   1. Once financial data is collected, it has to be stored securely in massive databases, ensuring it's accessible and organized.
3. Cleaning the Data: some text
   1. The raw data often contains errors, outliers, or structural issues. It must be cleaned, correcting these problems and ensuring the data is accurate.
4. Normalizing the Data: some text
   1. This step makes the data consistent and comparable. 
   2. Different sources present data in different formats, and everything must be made uniform so users can analyze it properly.
5. Documenting the Data: some text
   1. No data set is complete without thorough documentation, guides, and metadata.
   2. These materials help users understand how to access and utilize the data effectively.
6. Delivering the Data: some text
   1. Now comes the delivery. Data providers must build APIs, WebSockets, and provide downloadable formats to get the data into the hands of users quickly and efficiently.
7. Supporting the Data: some text
   1. The job doesn’t end at delivery. 
   2. Almost all data providers have to offer support, requiring them to hire staff, develop workflows, build ticketing systems, and even chatbots to ensure users can solve issues as they arise.
8. Marketing the Data: some text
   1. Lastly, data providers have to invest in marketing so that users—whether professional firms or individuals—know about the data and can access it.

This data product development process - sourcing, storing, cleaning, normalizing, documenting, delivering, supporting, and marketing - is lengthy, time-consuming, and resource-heavy. Any reliable data provider HAS to invest in each of these steps to get the data to you.

Now, let’s talk about the evolution of financial market data providers. Back in the day, major players like Bloomberg were pioneers. They built massive data sets by hand, relying on manual processes and legacy code, which was labor-intensive and incredibly costly.

Then innovators, like us at Intrinio, stepped into the space. Using cutting-edge technology—AI, machine learning, and cloud infrastructure—we’ve modernized the process, bringing down costs while improving the speed and accuracy of data delivery.

But there’s also been a rise in cheap, ‘too good to be true’ providers. These companies scrape data from unreliable sources, violate exchange rules, and can’t invest in the infrastructure needed to deliver quality or support. Because they don’t pay to license data properly, provide support, or document the sources of their data, they can sell it at extremely low prices.

Here’s where the problem comes in for data consumers.

We’ve seen a flood of newer, cheap providers who cut corners. They scrape data illegally, violate regulations, and, in order to keep costs low, fail to deliver reliable or high-quality data. The problem is, many users don’t realize how bad and unreliable the data is—until it’s too late.

Then, there are the faulty innovators. These companies try to introduce unique marketplace models or payment plans like pay-per-API-call, bundling data sets in ways that seem like a good idea, but their business models are unsustainable and often unprofitable.

And there’s a lag—a delay between when people start using these cheap services and when they realize they’re dealing with inaccurate data, faulty delivery, unlicensed providers, or terrible support.

These cheaper providers skip a lot of steps, so let’s talk about the real cost of delivering quality financial data. I want to show you a visual representation of what it takes to provide reliable data.

• First, you have the baseline cost of sourcing data.some text
  • This includes the cost of staff and any fees required to access raw data.
• Next, you add the cost of storing data.some text
  • This includes the cost of staff, database costs (which can be incredibly high as data sets grow), and other technical infrastructure costs to safely store and keep the data secure.
• Then, there’s the cost of cleaning and normalizing the data, which is essential for accuracy.some text
  • This includes the cost of staff as well as technical infrastructure and compute cost for processing data.
• After that comes the cost of documenting the data so that users can understand and access it.some text
  • This includes the cost of staff, website infrastructure, and hosting.
• Delivering the data through APIs, WebSockets, or other means adds another layer of cost.some text
  • These APIs and delivery tools have to be built, which includes the cost of staff and technical infrastructure along with hosting and storage expenses.
• Finally, supporting that data with customer service, documentation updates, and infrastructure maintenance isn’t free either.some text
  • You’ve got to add in the hefty cost of support representatives and staffing time to build out and maintain support systems.
• And of course, there’s the cost of marketing the data, so people know it exists.some text
  • This cost can vary, but it’s important to recognize that data products have to be marketed as part of the process.

There’s a key moment, what we call an innovation trigger, where technology can help reduce costs, particularly in areas like cleaning, normalizing, and documenting data. For example, Intrinio leverages machine-learning and ai-processes to normalize and clean data. This is a MAJOR innovation trigger that has helped us DRASTICALLY reduce costs in this area.

There have also been technical innovations over the years that have reduced the cost of storing, documenting, and supporting data. Modern databases, auto-generated docs, and chatbots are a few examples, but these innovation triggers result in marginal cost reductions.

But even with all this innovation, there’s still a baseline cost for financial and market data: You should see our cloud server bill! There’s no way around it: delivering high-quality and reliable data takes real investment.

Here’s the bottom line. 

This year, we’re seeing a reckoning in the financial data space. The “lag” we mentioned earlier - the time it takes for users to realize they can’t rely on cheap data or for providers to realize their models are unsustainable - has caught up to us. Smaller data companies are shutting their doors, financial data users are fed up with poor quality, and fees for market data are increasing.

The truth is, reliable data costs money. Building and delivering data sets of any quality requires resources—there’s no shortcut to that.

Cheap providers and faulty innovators are not just a nuisance. They’re dangerous. 

Working with unreliable providers can lead to faulty investment decisions for large institutions AND individual investors, jeopardizing personal wealth, and putting institutional portfolios at massive risk.

For example - if you use data from a provider who does not disclose the source, you may find out later that you owe MASSIVE fees because the provider didn’t tell you about the licenses you needed to legally use the data.

Be sure to ask where the data comes from, and double check the disclosures at the bottom of provider websites for language indicating they don’t guarantee the source or quality.

So, what’s the solution?

At Intrinio, we’re continuing to innovate across the data pipeline—from sourcing and standardizing to storing and supporting. 

But, users need to understand that there’s a minimum cost for high-quality data. 

Financial and market data users should budget accordingly and even consider raising capital to cover these expenses.

And for institutions like universities or research bodies, it’s time to invest in organization-wide data subscriptions. 

Giving students and researchers access to accurate, reliable data should be a priority, and it’s worth the investment to avoid relying on faulty, cheap sources.

Remember: when it comes to data, you truly get what you pay for. Don’t compromise your financial decisions with low-quality data. Choose a provider that delivers accuracy, reliability, and innovation—choose Intrinio.

‍