Tracking spend down to the SKU: How OnCorps AI uses Sigma and Databricks to keep costs down

I’m the head of OnCorps’ Data Team, which is responsible for algorithm development and production pipelines. We use AI to extract insights from the large amounts of data accumulated by financial services institutions in order to reduce labor and human errors.

We use AI to identify anomalies in mutual fund accounting data over time, allowing us to catch costly errors. The biggest question that fund administrators have is, “What caused this anomaly?” We are using generative AI to reduce the time to answer that question. Additionally, we are using AI to perform reconciliations on the shareholder reports that fund administrators are required to release. As these are legally binding documents, the question we answer is “Does this financial report match the actual holdings and operating patterns of this fund?”

We had used QuickSight, but found the feature set wasn’t complete enough for the complex visualizations we need to build for financial services clients. We then built customized dashboards and embedded visualizations from scratch using Plotly and Dash. However, building this out required data extraction services and the total cost of ownership was much higher than we had anticipated, including maintenance and infrastructure management.

We set up a Sigma dashboard to perform real-time monitoring of our Databricks spend in order to control costs. This allows Databricks administrators to quickly visualize changes in cost in response to optimizations and cost-cutting measures, as well as to receive real-time alerting in response to sudden and unexpected increases in costs. Sigma visualizations allow us to dissect the total Databricks spend by SKU and by Databricks workspace, which saves us a lot of time getting to the root cause of sudden movements in spend.

The primary usage of Sigma is for us to create customized dashboards for our users to embed in the applications that they subscribe to. One example is a dashboard containing over-the-counter derivatives for a major financial institution that performs a lot of these trades. These trades are much more complicated and labor-intensive than exchange-traded derivatives, and as a result, they can take longer to execute. We provide analysis of time-to-completion for both open and completed trades, which is important for our client’s SLAs. Additionally, we break that down by counterparty and the steps within the confirmation process, allowing our client to plan fine-grained actions to reduce the time to trade confirmation.

We are able to add our own branding to Sigma dashboards and also to embed so seamlessly that the dashboards are perfectly integrated into the page. A user can’t tell that a portion of a page is actually a Sigma embed.

Our CFO has a focus this year on enhanced analytics. This is a team of Excel wizards that have extremely detailed tasks surrounding payroll, budgeting, and forecasting, and one of their yearly goals is to leverage BI tooling to take the work out of this process. We realized Sigma could be a good fit due to Sigma’s new financial feature suite. This is a work in progress, but my team is in the planning phase with our finance team and I think Sigma could be a great fit. It’s similar to how we use Sigma to forecast and monitor Databricks operations.