The Importance of both Business and Data Acumen for the Business Intelligence Analyst
Having knowledge of how a leader perceives both the operations and the goals of the organization are critical for a business intelligence (BI) analyst. Without an understanding of the desired outcome, as well as the teams process, it will be difficult for both the analyst to understand what processes metrics can easily be measured, as well as for business leaders and team members to accept the results. In the Project Management Institute’s “Disciplined Agile” framework the “Govern Delivery Team” process blade focuses on the importance of measuring a team’s performance based on the result desired for that team, rather than organization wide metrics. The preferred analytical tool is called “Goal Question Metrics” (GQM) in which the metrics for a team are decided based on a particular team’s process, goals, makeup, and operations, and recognizes that each team in an organization may have different KPI metrics. (Ambler & Lines, 2020, p. 398) In order to successfully apply GQM so that both the team as well as the stakeholders/management have metrics by which to make decisions; the Analyst needs to have a thorough understanding of the processes and objectives of both the team and management, and the technical analytical skills to be able to measure progress against these processes and objectives.
BI analysts need to be purple people
In the book Secrets of Analytical Leaders Eckerson refers to the analysts needed as “Purple People,” who have both a deep understanding of business operations (“Blue” people) and advanced mathematical technical skills (“Red” People). (Eckerson, 2012, p. 1) According to Eckerson, “The best analytical leads are proverbial switch hitters. They have strong credentials and a solid reputation in either a business or technology discipline—and then they switch sides.” (Eckerson, 2012, p. 2) A BI Analyst must examine operations through two different eyes, the eyes of a modeling engineer, where mathematical and statistical reasoning attempts to draw order from chaos, and the eyes of a business leader, who recognizes that businesses are dynamic entities comprised of real living people guided by different and often competing values, personalities, emotions, egos, intentions, and objectives.
Understanding our stakeholders’ vision
Program managers must not only constantly ensure that projects and programs are aligned with the vision of our stakeholders (sponsors, clients, and team members alike), but also accurately understand the variance between our intended outcome and our current trajectory. To make accurate predictions about the capacity of our developers and team members, identify our assumptions and our constraints, account for risk, and develop a solution which meets the needs of the client (which are often not clearly articulated) you must study the minds of your stakeholders so that you can not only communicate with all stakeholders—clients, business leaders, and developers alike—but deliver a solution which meets their expectations. “For analytics to work, modelers need to build models that reflect business managers’ perceptions of business realities — and they need to make those connections clear.” (Eckerson, 2013) The modeler should be utilizing “design thinking” concepts by taking under consideration both the end-user(s) for whom the model is being built as well as the technical problem for which the model is being built to solve. The most beautifully engineered solution is not a viable solution simply because it solves the analysts understanding of the problem; a solution is only a solution if it solves the business clients problem, “[communicated] with business executives in business terms.”(Eckerson, 2013)
Applying the Principal of Parsimony to Modeling for Business Stakeholders
Business solutions do not just need to work on paper; they need to be able to be applied in the real business settings in which the problem exists. Our solutions need to be compatible with the different paradigms and ways of working of the users for which it is designed. Eckerson notes that “Communicating the results to executives so they understand what the model discovered and how it can benefit the business is critical but challenging — it’s the “last mile” in the whole analytical modeling process and often the most treacherous.”(Eckerson, 2013) This is why I think that the Principal of Parsimony is of great importance for the BI modeler. Often the completeness of a solution is correlated with its complexity resulting in a tradeoff between precision/accuracy and utility/value; as models become more accurate, they also become more complex, computationally expensive, and unintelligible to the normal individual, resulting in diminishing returns of value. The Principal of Parsimony (also referred to as Ockham’s Razor) says that given multiple viable explanations the simpler one with less assumptions and constraints is often the better one.
The value formula
In a BI context, what does it mean for one solution to “be better?” I suggest that we apply the value equation where value is equal to the ratio of quality to cost:
Value=\ \ Quality/Cost
What is quality for an organization? Although it is the goal of leadership to help a team or organization latch on to a single coherent vision to which every action, initiative, and individual becomes aligned, the motivations of even the highest performing teams never reach full convergence. Even if all members of an organization were to be perfectly aligned to the same vision, each person would still be operating out of their own contextual paradigm defined by their unique knowledge, understanding, education, worldview, language, etc., and each of those paradigms have their own measurements of cost and quality.
Quality. The highest quality solution is what meets the needs of the organization, considering each of the individual stakeholders needs, including their ability to understand and apply the solution. It is the convergence of the model solution with the mental models of the stakeholders, the perceived utility of the solution, etc. All of this is in addition to the accuracy of the solution.
Cost (Burden). Likewise, the cost is the burden which the solution places upon the organization, including the cost to implement the model, the data requirements, impact on team’s functioning process, etc. Our goal is to maximize the value of analytics, which is done by maximizing the utility of our model while minimizing the burden.
Model quality versus model grade
For the BI Analyst business quality is the degree to which a solution increases the benefit to the stakeholders, cost is the extent to which a solution increases the burden, and value is the maximization of the ratio of quality to cost. If we apply the Principal of Parsimony, we want to minimize the complexity of the solution, while still solving the business problem (i.e., the requirements). In project management, this is referred to ‘Grade’. The Project Management Body of Knowledge (PMBOK) defines grade as “a category or rank used to distinguish items that have the same functional use but do not share the same requirements for quality” (Project Management Institute, 2013, p. 542) Thus, for the Analyst we ultimately need to ensure that our model solutions take into account both the value and grade: they must meet the needs of the business while maximizing the value, or the ratio of the quality to the cost, by setting the upper limit of the solution based on the analytical maturity of the stakeholders. Building a model that will solve for the problem at the intersect of value for the business and grade of the solution is an artform which requires high levels of technical/mathematical skills, business acumen, and stakeholder/emotional understanding.
Practical use cases and tradeoffs of parsimonious modeling
For each organization, and even each individual, the tradeoff between quality and cost to maximize value will change based on the context, arguably even depending on the Analytical Maturity of the team or organization itself, which determines the appropriate model grade. A solution is only a solution if the stakeholder understands it to be such, and a higher grade is only valuable if its increased cost is less than the increased benefit realized by the business owner. For instance, in the following use cases, the modeler must consider both the context within which a solution is being deployed as well as account for users from which the model derives its value.
- Decision trees versus random forests for use by a sales team. Decision trees and random forests can both be used by a sales team to help predict sales closing and inform approach based on known customer dimensions. Although random forests (high grade) often provide better predictions, they are impossible for someone to visually understand and require computation to arrive at practical solutions (The algorithm must be run to make a prediction). Conversely, the lower grade decision trees, especially ones which are well pruned, can be printed out and read by sales agent, posted on the wall of their cubical, etc. If the accuracy and precision for a decision tree meets the minimum requirement of the business, choosing a decision tree over a random forest maximizes the parsimony. The tradeoff for predictability is the ability for the practical business user to not only understand the logic, perhaps even intuitively within their contextual paradigm, but memorize it and apply it to business solutions without the need of a computer.
- Linear regression over logistic regression when performing valuations. Both linear and logistic regression models are useful for predicting the value of an asset such as a property or a stock based on the valuation of similar assets. Although logistic regression often provides a much better fit, it comes at the cost of model explainability for many general business users. The degree of the tradeoff between linear or logistic regression is likely dependent on the analytical maturity of the team or organization for which the solution is derived as well as the solution being delivered. If the solution will only be accessed by means of an application the use of logistic regression might be preferred. If, on the other hand, the equation is being used to inform the evaluators understanding of the impact of various variables on valuation, and not simply being used to obtain the output of the computation, a linear regression model is likely preferred at the cost of fit. Some teams might be comprised of all individuals who are comfortable thinking in terms of log odds, but for most organizations fluency with log ratios should not be assumed. Further, the increased complexity of logistic regression models may increase the complexity of incorporating these models into other models. For many business applications the lower grade but tried-and-true linear regression model has more than enough accuracy to solve the business problem.
- Innumeracy (the equivalent of illiteracy) and Classification in marketing. We expect that both business users and analysts are both confident and comfortable with the relationship and magnitude of numbers, but for many the accuracy of an individual’s “mental math” decreases as the order of operations increases: more people can add or subtract accurately than people who can multiply and divide, etc. When performing clustering or segmentation business users will more easily be able to understand classification models, even if this in many cases comes at the expense of predictive power.
Choosing proper grade
If the business unit is unable to realize the benefits which offsets the rise in cost, using a higher-grade model than necessary potentially decreases the value of the solution. As the complexity of a solution increases, that universality of understanding and general applicability decreases. Although the level of complexity sustainable by an organization is relative to how analytically advanced the organization is, we want to minimize the variance between the complexity of the solution and the organizations ability to understand how the solution works. For the modeler, this means choosing a grade appropriate to the solution and the business customer. Avoid diminishing returns in quality due to increases in cost for unrealized benefits from increased grade, thereby minimizing the solution value. (e.g., when minimal increases in precision or accuracy come at the expense of model understanding, ROI, universality, cost, etc., unnecessarily increasing the grade above what the business problem requires lowers the value to the business unit.)
Works Cited
Ambler, S., & Lines, M. (2020). Choose your Wow! A Disciplined Agile delivery handbook for optimizing your way of working. Project Management Institute.
Eckerson, W. (2012). Secrets of analytical leaders: insights from information insiders. Technics Publications.
Eckerson, W. (2013). Analytical modeling is both science and art. Retrieved 2021-02-21, from https://searchbusinessanalytics.techtarget.com/opinion/Analytical-modeling-is-both-science-and-art
Project Management Institute. (2013). A guide to the project management body of knowledge (PMBOK Guide) (Fifth Edition ed.). Project Management Institute.
Andrew is a General Partner at Northwestern Analytics and is the managing partner responsible for Business Operations and Project Management. Andrew holds his Project Management Professional certification and also serves as the Chief of Staff of Business Outreach for the Project Management Institute Chicagoland Chapter.