We are witnessing a shift in the way financial institutions focus on data management and analytics. Historically, the two disciplines have been largely separate. The data-management process has typically involved activities such as data sourcing, cross-referencing and ironing out any discrepancies via reconciliations and data-cleansing processes.
Data-analytics processes have usually been carried out afterward in a variety of desk-level tools and libraries, close to users and typically operating on separately stored subsets of data. This has created problems for many financial institutions, with the separation impacting time to insight and acting as a brake on the decision-making processes that drive business success.
Data was typically held, and often still is, and siloed in data stores and legacy systems, where accessing it was difficult. The metadata surrounding the data was often not updated frequently, making data lineage and understanding of the relevant permissions around the data difficult to understand fully. Judging whether data was fit for purpose was complex, and frequently models came to suboptimal results because they were based on stale, incomplete or otherwise inappropriate data.
Today, though, this situation is starting to change. Many financial-services firms have come to realise that they need a better way to provision their data scientists and other key users with clean price and market data. The role of analytics has become much more pervasive and ingrained in everyday business processes. Apart from this, the bar has been raised regarding tracking dataflows and managing model risks.
A shift to the cloud and the adoption of cloud-native technologies are helping firms transition to a more integrated approach to data management and analytics. Apache Cassandra, for example, has emerged as a highly scalable, open-source, distributed database that makes it easier to store and manage large volumes of financial time-series data securely. Apache Spark is a unified data engine for big-data processing. Taken together, the two, along with other associated tools, are helping to facilitate the integration of data and analytics.
At least in part, as a result of this, we see data management and analytics increasingly joined at the hip. This integration is also blurring the line between the source of primary data (the stock exchange, for instance) and what is derived and calculated (for example, interest-rate curves, correlations or volatilities). Data management and analytics are today two sides of the same coin in user workflows.
We still see quantitative analysts (quants) at the endpoints of data flows, carrying out modelling, forecasting and pricing complex products. But, in addition, recurring business processes in risk assessment, analysis and reporting as well as other modelling have become more data-intensive.
Also, increasingly the focus is on bringing analytics to the data rather than vice versa. In other words, it is about moving the analytics capability to where the data resides rather than moving large stores of often siloed data over to the analytics function, which has typically led to inconsistent copies over time and lots of analyst time spent verifying and gathering data before the actual analysis could start
Data quality matters
For analytics to work effectively and efficiently, however, the data that fuels it has to be of the highest quality. Good quality input data makes analytics more reliable. Conversely, even the best model will produce useless results when fed poor-quality data. This drive towards efficiency and accuracy as businesses look to turbo-charge their analytics function is one reason why data quality matters to the finance sector today. The other is that regulators are increasingly scrutinising data quality—and, in particular, the quality of the data that feeds into models.
Financial businesses will often need to explain the results—not only the mathematics of the model itself but also the data that went into it, what the quality issues were, what the sources were and who touched it on the way. That can be difficult if they treat data management and analytics as separate disciplines. Without having the ability to analyse the data and the oversight of where it came from and is going to, businesses struggle to gain transparency over how they are provisioning their models with data. They also need the data-quality capability in place to ensure that their data is consistent and validated and that the burden of reconciliation is reduced.
User enablement and enhanced decision-making
While regulatory reporting has long been an essential discipline for financial organisations and while the coming together of data and analytics has supported it, it is far from the only benefit that this new trend delivers. As we have seen, the role of integrated data and analytics in supporting user enablement and enhanced decision-making is key. As the cycle of managing and processing data extends further to take in analytics, users increasingly want to be empowered by the process and leverage these new capabilities to drive better-informed decision-making. This move to data-as-a-service (DaaS), when combined with the latest analytics capabilities, is making this happen for financial organisations today.
By adopting this approach, they can gain access to multiple data sources and data types, from pricing and reference data to curves and benchmark data to ESG (environmental, social and corporate governance) and alternative data. With the help of the latest open-source technology such as Python, users can share these analytics across their entire data supply chains and, thanks to the power of the analytics engine, develop a common approach to risk management, performance management and compliance.
Quants and data scientists benefit from this. But this approach to user enablement is also helping to democratise analytics, bringing it into the orbit of those who are not data experts. Today, thanks to the contextualisation provided alongside analytics, it is not just the preserve of the quant or the data scientist but a key tool that those who are less expert in data can use to drive business decisions.
This in itself promotes business agility, but the combination of data and analytics can also help businesses to optimise costs. It does this by supporting greater agility with the data and selecting only those elements strictly needed to help drive the business forward. It is, however, also about centralising data more efficiently and removing duplication.
Looking ahead, we are on the cusp of a new age in financial-data management. Today, technology, processes, macro-economic factors and business awareness are all joining forces to bring data analytics and management together. The result for financial institutions is a new world of opportunity in which they can optimise costs, drive user enablement and maximise the value they get from data.