Just weeks after our exciting deal with the Japanese entertainment conglomerate Avex, we are thrilled to announce another significant partnership, this time with Bandai Namco Music Live, a key player in Japanese Anime music. This marks a major milestone in our expansion into Japan and entry into the thriving anime music industry.
As Bandai Namco Music Live’s rights management partner on YouTube, we will leverage our AI technology to oversee, optimize, and monetize their extensive catalog on the platform.
Introducing Bandai Namco Music Live
Bandai Namco Music Live, a leading player in the Japanese anime music industry, has an extensive collection of songs and represents over 3,000 talented creators. They have contributed to numerous beloved entertainment moments, from the popular Love Live! series to the action-packed One-Punch Man.
As the popularity of digital platforms such as YouTube continues to skyrocket, the global reach of anime is also increasing. Recognizing this opportunity, Bandai Namco Music Live chose to partner with Orfium for its superior technology, which optimizes the management and monetization of music used in UGC which will enable them to continually invest into new productions and reach more fans.
As Akiho Matsuo, Rights Department General Manager at Bandai Namco Music Live, put it, “By partnering with Orfium, whose superior technology allows for the management and monetization of music used in user generated content, we can ensure that artists and creators make their music available on a popular global platform and one that, importantly, is also optimized for monetization.”
The Rising Tide of UGC in Japan’s Booming Music Market
Our partnership with Bandai Namco Music Lives sees Orfium move into anime music, a genre with its roots firmly in Japan but which has grown in popularity around the world as anime (short for animation) has become a global phenomenon. According to research by Morgan Stanley, the market reached a value of US$ 25 billion in 2023 and is expected to grow to US $ 35 billion by 2030. Japan is the major contributor in the anime market, accounting for over 40% of the global anime market in 2020.
This growth provides a perfect opportunity for those in the entertainment and music industry to utilize new solutions to capitalize on international growth and fan appetite – a natural alignment with Orfium’s expansion into the Japanese and Asian markets. Leveraging our market-leading AI-powered technology, Orfium aims to help clients optimize and monetize user-generated content (UGC) in this rapidly growing market.
The Path Ahead: Future Opportunities for Orfium in Japan
Japan has held a pivotal position in Orfium’s global expansion strategy since 2022 when we acquired Breaker and established Orfium Japan, led by Alan Swarts. This latest deal builds on our partnerships with Avex, Warner Music Japan, and Victor Entertainment, highlighting our commitment to driving global digital transformation through local collaborations.
With the strength and expertise of our dedicated local team and the latest in cutting edge AI-powered technology, we are excited to continue forging more partnerships in Japan and across Asia’s dynamic music and entertainment landscape to power revenues for rights holders and creators across digital music and broadcast platforms.
As of the writing of this article, the W3C Design Tokens Standard is still in development. Its goal is to establish a universal framework for using tokens that will be compatible with any tool or platform. This has the potential to be a game-changer and is worth keeping an eye on.
If you want to integrate design tokens into your design system but don’t quite know where to begin, you’re not alone. Adding design tokens to an existing system calls for good organization and consistent use. This already challenging task is not made any easier by the lack of standard naming conventions.
In this article, we’ll delve into how our team integrated design tokens into our design system, Ictinus. We’ll discuss the challenges we encountered, the solutions we came up with, and the impact design tokens had on our workflow. Finally, we’ll share insights gained from our experience.
What are design tokens?
A design token is a piece of data that encapsulates a single design decision.
The term ‘design token’ was first coined by Jina Anne at Salesforce in 2014. They can represent a wide range of properties and link (‘alias’) to other tokens, making them scalable by nature.
Sound familiar? It’s similar to how programming variables work: you declare a specific value which is then propagated to many different instances. These values and relationships can be represented in JSON format.
What are the benefits of design tokens?
In our preliminary research, we analyzed the approach of other Design Systems (e.g. Polaris, Carbon, etc.) and read up on token literature and best practices. Our early findings were enough to convince our team: tokens were worth the time, cost and effort of integration. Our biggest challenges at the time were gaps in communication and insufficient change tracking. Tokens promised to help with these issues as well as do the following:
1. Resolve gaps in communication
Design decisions often got ‘lost in translation’ due to inadequate documentation and time pressure. Discrepancies between design and implementation were not just common but expected. This translated to much more time and effort spent in QA – and still, mistakes could slip through.
2. Fix issues with change tracking
This gap in communication had a ripple effect. Changes in design documentation were not always reflected in current implementation. That made it hard to confirm whether a specific section of documentation was updated or not. When should we implement QA?
As a result, we had to freeze our design system to a specific version (V4); a last resort solution that opened a whole new can of worms. Last but not least, our current workflow all but ensured that the team ran at different speeds.
The more we learned about tokens, the more they seemed like a winning bet. They would help us resolve some of our biggest issues, become more efficient and enrich our design language.
3. Improve efficiency
Before Figma (our tool of choice) introduced variables, there was no way to natively link styles. Design tokens would allow us to mass-update multiple styles by editing a single aliased token. This would simplify maintenance and reduce cognitive load, allowing the team to focus on the bigger picture instead.
4. Extensive customization support
Design tokens can represent a wide range of properties beyond the usual color, typography, and layer effect styles. Tokens also support the following:
- Spacing
- Sizing
- Border radius
- Border width
- Animation styles
The challenges of implementing design tokens
It was not all sunshine and rainbows though. As mentioned before, a standardized framework for creating and using design tokens doesn’t exist just yet. In this article, Brian Heston addresses the most common design token implementation concerns.
Of special note is the mention that there is no common standard for naming and applying tokens. This lack of consistency poses challenges when combining components from different libraries. It also makes it hard to outline and follow a common set of best practices.
What token tiers are, and how we used them (aka layers, groups or sets…)
Before diving into token creation, we first needed to lay down the basics. What would we call different token levels, for example? And how many levels would we need?
Tokens can be organized into different levels of abstraction. Common names for different levels include ‘levels’, ‘types’, ‘tiers’, ‘layers’, ‘groups’, and ‘sets’. These levels help organize tokens based on their purpose, scope, or usage. We ended up going with ‘tiers’, as it was both the most unique-sounding and descriptive.
Naming settled, we shifted our focus to tier structure. How many tiers should we have? What should each tier contain? How would it relate to other tiers?
In the many faces of themeable design systems, Brad Frost suggests a three-tiered model that uses a vertical hierarchy.
Tier-1, the lowest tier, contains tokens that encapsulate raw visual design materials.
Tier-2 tokens are contextual, and link Tier-1 tokens to high-level usage.
Finally, Tier-3 tokens are specific to components and link to Tier-2 tokens.
How we named our tiers
We settled on the name ‘global’ for Tier-1. It would contain every single tokenized value in our design system. ‘blue.3’, a token containing a single color value with no context, is an example of a ‘global’ tier.
Tier-2 would draw from ‘global’ to create contextual tokens. For example, ‘blue.3’, a global token, may link to the ‘sem.textColor.active’ semantic token to create a token describing text color for an ‘active’ state. We named this tier ‘semantic’ (‘sem’ for short).
You can think of Tier-2, the ‘semantic’ layer, as one of your theme layers. If you want to be able to switch to dark mode down the line, all you need is identical ‘semantic’ layers that point to different aliases.
Finally, Tier-3 became ‘component’ (abbreviated to ‘comp’). This final tier would link to Tier-2, drawing contextual tokens in component-specific combinations.
Token name conventions and semantic attributes
With the tiers in place, it was finally time to think about token name structure. What should we include in a design token name, and in what order? This well-known, oft-referenced talk by the Asana team on Schema 2021 served as a great starting point. The whole video is worth a watch, but of special interest is the hierarchical naming convention the Asana team uses to name design tokens.
Inspired by Asana and mindful of HTML/CSS standards, we created our own set of naming rules as follows:
- A token name is composed of the following semantic attributes: tier.category.property.state
- Tier: Indicates the tier the token can be found in.
- The ‘Global’ tier is omitted from the token name for the sake of simplicity.
- Tokens found in the ‘semantic’ tier start with ‘sem.’
- Tokens found in the ‘component’ layer begin with ‘comp.’
- Category: Chooses a general token category (e.g. ‘spacing’). When used in a ‘comp’ token, ‘category’ displays the component name.
- Group: provides context on a given category (e.g. ‘background’ as a subcategory to ‘color’).
- Property: Contextual value of token. May also be state or variant. May also reference alternative visual variants for non-interactive components.
- Tier: Indicates the tier the token can be found in.
- Semantic attributes are separated by period (.)
- Token names that include more than one word are formatted in camelCase.
How we use Tokens Studio for Figma (our not so secret weapon)
When we were implementing design tokens, Figma didn’t offer support for token creation as variables had not been released yet. Even now, variables do not support properties like border width (and, as of the writing of this article, cannot be exported in JSON without a plugin). This meant we would need to use a third-party tool. We chose Tokens Studio, a popular choice that we could easily integrate into our Figma workflow.
Tokens Studio for Figma represents tokens in JSON as a list of objects. You can create and edit tokens either via the user interface provided by the plugin or by directly editing the JSON code.
The ability to store the generated tokens as JSON in GitHub branches was a game-changer for our team. It allowed development to review design work as it was being done. Upon approval, typescript code would be auto-generated from said JSON. Finally, small issues were addressed and corrected in a fraction of the usual time. This was a big step towards bridging the aforementioned communication gap.
Importing global tier tokens
Tokens Studio allows you to import existing Figma styles and convert them into tokens; this saved us a lot of time. After we imported our preexisting color, typography, and layer effect styles into Tokens Studio, it was time for the good stuff. We created extra categories such as spacing, sizing, and border radius.
Semantic tier tokens
It was then time to begin work on the ‘semantic’ tier tokens. These are tokens with contextual meaning that link to context-less ‘global’ tokens. Semantic style examples include text color, typography styles, icon sizes, and interaction palettes. These styles are then combined in many different ways to make the final tier, ‘comp’.
Component design tier tokens
The component tier is the final layer (Tier-3). Each token created in this tier links to a ‘sem’ token. This allows for easy tracking of individual component tokens while still maintaining scalability.
The ‘category’ semantic attribute for ‘Component’ tier tokens is always the component name. This level of granularity allows for better control when working with large design systems.
Here’s an example of a ‘comp’ token:
comp.button.compact.iconSize
- ‘comp’: indicates ‘component’ token tier
- ‘button’: the component’s name
- ‘compact’: applies to compact button
- ‘iconSize’: sets icon size property
Results and Impact
The adoption and utilization of design tokens significantly enhanced our team’s workflow. Design tokens helped our team create a common language spoken by all and allowed for a unified mindset. This fostered a better understanding between designers and developers regarding each other’s constraints. It also led to significant delivery time and quality improvements.
Internal design-to-development average time was reduced by roughly 30%. This has a lot to do with the fact that token structure is similar for both designers and front-end developers. As a result, understanding them and applying them correctly became a more intuitive process for both disciplines. This was also a big help towards streamlining communication and reducing excess noise.
It reduced visual discrepancies between design and implementation. As a result, time normally spent on QA was instead used to better develop the components and pad the documentation. This affected onboarding of users and products to the design system, speeding up the process by approximately 65%. This allowed us to create better, more thorough documentation for component behavior and patterns. This led to a sharp decline in bug fix tickets (to the tune of approximately 55%).
The team’s upcoming steps involve introducing the highly-anticipated theming to ORFIUM products. Our previous framework made this a pipe dream, but with tokens, sky’s the limit!
Takeaways
Here are some practical tips to help your team adopt and use design tokens:
- Define a clear tier structure that your team can understand at a glance and use with ease. Each tier should have a distinct name, abbreviation, and purpose.
- Create naming conventions for your design tokens as a team. Make sure the naming conventions are easy to remember and use.
- Document everything token related. Documentation should be accessible and understandable by both design and development team members.
- Simplify token creation and export with third party tools like Tokens Studio for Figma.
That’s all for now, folks! We encourage you to share with us your own findings and concerns at @LifeatOrfium.
To learn more about design tokens, check out Jina Anne’s 2021 YouTube video or read the video summary by Amy Lee.
References
https://uxdesign.cc/naming-design-tokens-9454818ed7cb
https://medium.com/@amster/wtf-are-design-tokens-9706d5c99379
https://medium.com/fast-design/evolution-of-design-tokens-and-component-styling-part-1-f1491ad1120e
The Many Faces of Themeable Design Systems
https://medium.muz.li/unlocking-the-power-of-design-tokens-to-create-dark-mode-ui-18c0802b094e
We’re excited to share news of our latest partnership with Avex in Japan. Avex is a powerhouse in the world of entertainment and this partnership signals a significant step in Orfium’s growth as the first deal of its kind in Asia.
Japan is the world’s second largest music market, yet UGC monetization is still in its infancy. Recognising the opportunity to support the market there, Orfium expanded to Japan in 2022 with the acquisition of Breaker (now Orfium Japan) and has been growing solidly in the market since then.
Orfium is set to become Avex’s partner for music catalog management on YouTube, utilizing our AI-based matching technology to maximize revenue for Avex’s rights holders, artists, writers, and composers.
Who is Avex?
Picture a goliath in the world of entertainment, a company that weaves music, animation, film, and digital prowess into the iconic music landscape in Japan. The sounds behind a whole world of anime, filmmaking, Japan’s iconic and thriving DJ scene and so much more. That’s Avex Inc.
Founded in 1988, they started in the music and entertainment industry as a wholesale business of imported music records. They’ve since expanded to be a powerhouse in record labels, live music performances, animation, and filmmaking.
Avex’s mission is most aptly put by Avex’s own Representative Director and CEO Katsumi Kuroiwa, who says they are “Making impossible entertainment, possible.”, and they have been a pioneer in doing just that. Projects like Justin Bieber’s world tour to Japan, and the first fully remotely animated film ‘She Sings, As If It Were Destiny’ are just two shining examples of Avex’s innovation that proves that they are a driving force and strong representation of the desire in the Japanese market to consume and contribute to the music and entertainment industry.
The Growth of UGC & the Japanese Music Market
Ranking as YouTube’s third-largest revenue contributor, Japan’s digital music scene is flourishing. Although YouTube thrives with 70 million monthly active users, UGC monetization is just taking flight, with many Japanese music companies only now transitioning from restrictions to embracing the vast potential of sharing and licensing.
In 2022, despite YouTube’s global payout of over $6 billion, Japan received a modest 6% – showing clearly the opportunity for artists and rights holders to gain more for their art. There’s a huge opportunity here to fill the gaps by many platforms’ own services for tracking and monetizing music which leave potentially millions of dollars worth of revenue unfound and unclaimed – and as more music is shared, that opportunity is only growing.
Having already established ourselves as the leader in other major markets for UGC rights management and monetization, Orfium is excited to be a part of reclaiming those missed revenue opportunities by partnering with more Japanese clients, and supporting the unique industry there. We already work with the world’s leading Music Publishers, Record Labels, Broadcasters and Production Music Companies in the US and UK, and have delivered unrivaled results for them; and we’re excited to demonstrate the same value for the Japanese and Asian markets.
Our local team, led by the experienced and passionate Alan Swarts, has been instrumental in our swift rise in the Japanese market – behind every great achievement, is a dedicated and skilled team, and Orfium Japan is no exception. With their deep industry knowledge and commitment, they’ve ensured that Orfium Japan is positioned to deliver the full potential and degree of success we’ve seen around the world for our clients.
Looking Ahead: What Lies Beyond the Horizon
Orfium’s partnership with Avex follows agreements with Warner Music Japan and Victor Entertainment, and we’re looking forward to sharing news of more Japanese partner deals very soon. Driven by our company goal to deliver optimized UGC revenue generation, we are continuing to improve the industry for the better for creators, artists and rights owners, one partnership at a time. As we grow internationally, we look forward to more partnerships across the music and entertainment landscape of Japan and Asia
Be a Part of the Journey with Orfium Japan
If you’re intrigued by our journey and want to explore opportunities with Orfium, don’t hesitate to reach out to our team.
We’re delighted to announce Orfium’s official sponsorship of All That Matters 2023!
As Orfium rolls out our full suite of tools and services into the Japanese and Asian markets, we are proud to announce our entry into the region with the confidence and clientele of some of the largest broadcasters, video game publishers and music publishers already.
With a global team of over 600 specialists, including dedicated teams in Taiwan, Japan, and South Korea, we have already established ourselves as a major player in solving key challenges in catalog and rights management in the entertainment industry. Our participation in the All That Matters event, which aligns perfectly with our core vision of driving innovation in the entertainment industry, provides us with the perfect platform to introduce our innovative solutions and to network with other professionals in the region.
Our sponsorship of the event reflects our unwavering dedication to driving innovation in the industry. We are excited to connect with other professionals, artists, and visionaries in the industry to explore the latest trends and innovations shaping the future.
About All That Matters 2023
The event blends business insights with fan enthusiasm and will take place in Singapore from September 11-13, 2023. This annual gathering has become a highlight on the industry calendar where the entertainment, media, and technology professionals come together to focus on current and future challenges, opportunities and innovation. An impressive lineup of professionals, artists, experts, and visionaries will come together to explore the latest trends and innovations shaping the future of these dynamic fields.
What’s in Store
The agenda is filled with an engaging array of keynotes, interactive panels, workshops, music showcases, and networking opportunities that are designed to foster the exchange of knowledge and ideas. This event is a platform to uncover the ever-evolving trends, challenges, and opportunities in music, gaming, sports, marketing, and beyond.
The Orfium team will be there, including Alan Swarts, CEO of Orfium Japan, Bonna Choi, Head of Soundmouse by Orfium Korea, John Possman, Director of Orfium Japan, and Shiro Hosojima, Chief Operations Officer in Japan.
Orfium to Discuss Online Media Monetization
Orfium will participate in a panel discussion on online media monetization at All That Matters 2023. The panel, titled “Online Media Matters: A Comprehensive Guide for Content Owners and Creators,” will be held on Tuesday, September 12, 2023, at 2:20 PM – 2:50 PM GMT+8.
Alan Swarts, CEO of Orfium Japan, will be one of the panelists and will share his insights on the latest growth monetization strategies for content owners and creators in the Asia Pacific region. Some of the topics to be discussed will include different revenue streams for content owners, from ad revenue to brand collaborations, as well as legal considerations and future trends in online media monetization. Alan will be joined on stage by other panelists including Hao Zhang, Head of Western Content Partnerships & Licensing at Tencent Music Entertainment Group (TME), and Reta Lee, Head of Commerce at Yahoo.
Orfium’s Vision and Contribution
Orfium’s partnership with All That Matters perfectly aligns with our core mission: driving innovation in entertainment through technology. We offer a suite of services that empower music creators, labels, broadcasters, and more to effortlessly manage and monetize their content. Our involvement in this event is a testament to our dedication to forward-thinking discussions, industry insights, and collaborative endeavors.
Shaping the Future Together with All That Matters 2023
This sponsorship showcases our ongoing commitment to nurturing growth, creativity, and progress in music and entertainment. We’re confident that All That Matters will serve as an invaluable platform for forging connections, sparking fresh ideas, and advancing our collective understanding of this ever-evolving landscape.
A Special Thanks
Our heartfelt gratitude goes out to the organizers and participants who have crafted an environment that encourages exploration, collaboration, and insightful conversations. We’re eagerly looking forward to contributing to the event’s success and engaging in meaningful exchanges that will undoubtedly shape the future of entertainment and technology.
For more details about the event, including registration information and the schedule, please visit the official All That Matters website.
Come Meet us at All That Matters 2023!
If you’re curious to learn more about Orfium and how we’re using machine learning, AI, and innovative algorithms to streamline and boost client revenue streams, feel free to drop by our Orfium VIP Lounge at the event, and leave your details here.
We’re back
As described in a previous blog post from ORFIUM’s Business Intelligence team, the set of tools and software used varied as time progressed. A shorter list of software was used when the team was two people strong, a much longer and more sophisticated one is being currently used.
2018-2020
As described previously, the two people in the BI team were handling multiple types of requests from various customers from within the company and from ORFIUM’s customers. For the most part, they were dealing with Data Visualization, as well as a small part of data engineering and some data analysis.
Since the team was just the two of them, tasks were more or less divided in engineering and analysis vs visualization. It is simple to guess that in order to combine data from Amazon Athena and Google Spreadsheets or ad-hoc csv’s, a lot of scripting was used in Python. Data were retrieved from these various sources and after some (complex more often than not) transformations and calculations the final deliverables were some csv’s to either send to customers, or load on a Google Sheet. In the latter case a simple pivot table was also bundled in the deliverable in order to jump start any further analysis from the, usually internal, customer.
In other cases where the customer was requesting graphs or a whole dashboard, the BI team was just using Amazon Athena’s SQL editor to run the exploratory analysis, and when the proper dataset for analysis was eventually discovered, we were saving the results to a separate SCHEMA_DATASET in Athena itself. The goal behind that approach was that we could make use of Amazon’s internal integration of their tools, so we provided our solutions into Amazon Quicksight. This seemed at that moment the decision that would provide deliverables in the fastest way, but not the most beautiful or the most scalable.
Quicksight offers a very good integration with Athena due to both being under the Amazon umbrella. To be completely honest, at that point in time the BI Analyst’s working experience was not optimal. From the consumer side, the visuals were efficient but not too beautiful to look at, and from ORFIUM’s perspective a number of full AWS accounts was needed to share our dashboards externally, which created additional cost.
This process slightly changed when we decided to evaluate Tableau as our go-to-solution for the Data Visualization process. One of the two BI members at that time leaned pretty favorably towards Tableau, so they decided to pitch it. Through an adoption proposal for Tableau, which was eventually approved by ORFIUM’s finance department, Tableau came into our quiver. Tableau soon became our main tool of choice for Data Visualization. It allows better and more educated decisions to be made from management, and is able to showcase the value that our company can offer to our current and potential future clients.
This part of BI’s evolution led to the deprecation of both Quicksight and python usage, as pure SQL queries and DML were developed in order to create tables within Athena, and some custom SQL queries were embedded on the Tableau connection with the Data Warehouse. We focused on uploading ad-hoc csv’s or data from GSheets on Athena, and from there the almighty SQL took over.
2021- 2022
The team eventually grew larger and more structured, and the company’s data vision shifted towards Data Mesh. Inevitably, we needed a new and extended set of software.
A huge initiative to migrate our whole data warehouse from Amazon Athena to Snowflake started, with BI’s main data sources playing the role of the early adopters. The YouTube reports were the first to be migrated, and shortly after the Billing reports were created in Snowflake. That was it, the road was open and well paved for the Business Intelligence team to start using the vast resources of Snowflake and start building the BI-layer.
A small project of code migration so that we use the proper source and create the same tables that Tableau was expecting from us, turned into a large project of restructuring fully the way we worked. In the past, the python code used for data manipulation and the SQL queries for the creation of the datasets to visualize were stored respectively in local Jupyter notebooks and either within View definitions in Athena or Tableau Data source connections. There was no version control; there was a Github repo but it was mainly used as a code storage for ad-hoc requests, with limited focus on keeping it up to date, or explaining the reasoning of updates. There were no feature branches, and almost all new commits on the main one were adding new ad-hoc files in the root folder and using the default commit message. This situation, despite being a clear pain point of the team’s efficiency, emerged as a huge opportunity to scrap everything, and start working properly.
We set up a working guide for our Analysts: training on usage of git and Github, working with branches, PullRequest templates, commit message guidelines, SQL formatting standards, all deriving from the concept of having an internal Staff Engineer. We started calling the role Staff BI Analyst, and we indeed currently have one person setting the team’s technical direction. We’ll discuss this role further in a future blog post.
At the same time we were exploring options on how to combine tools so that the BI Analysts are able to focus on writing proper and efficient SQL queries, without having to either be fully dependent on Data Engineers for building the infrastructure for data flows, or requiring python knowledge in order to create complex DAGs. dbt and Airflow surfaced from our research and, frankly, the overall hype, so we decided to go with the combination of the two.
Initially the idea was to just use Airflow, where an elegant loop would be written so that the dags folder would be scanned, and using folder structures and naming conventions on sql files, only a SnowflakeOperator would be needed to transform a subfolder in the dags folder to a DAG on the AirflowUI, with each file from the folder would be a SnowflakeOperator task, and the dependencies would be handled by the naming convention of the files. So, practically a folder structure as the one shown to the right would automatically create a Dynamic Dag as shown on the left.
No extra python knowledge needed, no Data Engineers needed, just store the proper files with proper names. A brief experimentation with DAGfactory was also implemented but we soon realized that the airflow should just be used as the orchestrator of the Analytics tasks, and the whole analytics logic should be handled by something else. All this was very soon abandoned when dbt was fully onboarded to our stack.
Anyone who works in the Data and Analytics field must have heard of dbt. And if they haven’t already, they should. This is why there is nothing too innovative to describe about our dbt usage. We started using dbt from early on in its development, having first installed v0.19.1, and with an initial setup period with our Data Engineers, we combined Airflow with dbt-cloud for our production data flows, and core dbt CLI for our local development. Soon after that, and in some of our repos, we started using github actions in order to schedule and automate runs of our Data products.
All of the BI Analysts in our team are now expected to attend all courses from the Learn Analytics Engineering with dbt program offered at dbt Learn regarding its usage. The dbt Analytics Engineering Certification Exam remains optional. However, we are all fluent with using the documentation and the slack community. Generic tests dynamically created through yml, alerts in our instant messaging app in case of any DAG fails, and snapshots are just some of the features we have developed to help the team. As mentioned also above, our Staff BI Analyst plays a leading role in creating this culture of excellence.
There it was. We endorsed the Analytics engineering mindset, we reversed the ETL and implemented ELT, thus finally decoupling the absolute dependency on Data Engineers. It was time to enjoy the fruits of Data Mesh: Data Discovery and Self Service Analytics.
2023-beyond
Having more or less implemented almost all of ORFIUM’s Data Products on Snowflake with proper documentation we just needed to proceed to the long awaited data democratization. Two key pillars of democratizing data is to make them discoverable and available for analysis by non-BI Analysts too.
Data Discovery
As DataMesh principles dictate, each data product should be discoverable by its potential consumers, so we also needed to find a technical way to make that possible.
We first needed to ensure that data were discoverable. For this, we started testing out some tools for data discovery. Among the ones tested was Select Star Select Star,, which turned out to be our final choice. During the period of trying to find the proper tool for our situation, Select Star was still early in its evolution and development so, after realizing our sincere interest, they invested in building a strong relationship with us, consulting us closely when building their roadmaps, while having a very frequent communication looking to get our feedback as soon as possible. The CEO herself Shinji Kim was attending our weekly call helping us make not just our data discoverable to our users, but the tool itself easily used by our users in order to increase adoption.
Select Star offered most of the features we knew we wanted at that time, and it offered a quite attractive pricing plan which went in line with our ROI expectations.
Now, more than a year after our first implementation, we have almost 100 users active on Select Star, which is a pretty part of the internal Data consumer base within ORFIUM, given that we have a quite large operations department of people who do not need to access data or metadata.
We are looking to make it the primary gateway of our users to our data. All analysis, even thoughts, should start by using Select Star to explore if data exist.
Now, data discovery is one thing, and documentation coverage is another thing. There’s no point in making it easy for everyone to search for table and column names. We need to add metadata on our tables and columns so that the search results of Select Star parse that content too, and provide all available info to seekers. Working in this direction we have established within the Definition of Done of a new table in the production environment a clause that there should be documentation on the table and the columns too. Documentation on the table should include not only technical stuff like primary and foreign keys, level of granularity, expected update frequency etc, but also business information like what is the source for the dataset, as this varies between internal and external producers. Column documentation is expected to include expected values, data types and formats, but also business logic and insight.
The Business Intelligence team uses pre-commit hooks in order to ensure that all produced tables contain descriptions for all the columns and the tables themselves, but we cannot always be sure of what is going on in other Data products. As Data culture ambassadors (more on that on a separate post too), BI has set up a data coverage monitoring dashboard, in order to quantify the Docs coverage of tables produced by other Products, raising alerts when the coverage percentage falls below the pre-agreed threshold.
Tags and Business and Technical owners are also implemented through Select Star, making it seamless for data seekers to ask questions and start discussions on the tables with the most relevant people available to help.
Self-Service Analytics
The whole Self-Service Analytics initiative in ORFIUM, as well as Data Governance, will be depicted in their very own blog posts. For now, let’s focus on the tools used.
Having all ORFIUM Data Products accessible on Snowflake and discoverable through Select Star, we were in position to launch the Self-Service Analytics project. A decentralization of data requests from BI was necessary in order to be able to scale, but we could not just tell our non-analysts “the data is there, knock yourself out”.
We had to decide if we wanted Self-Service Analysts to work on Tableau or if we could find a better solution for them. It is interesting to tell the story of how we evaluated the candidate BI tools, as there were quite a few on our list. We do not claim this is the only correct way to do this, but it’s our take, and we must admit that we’re proud of it.
We decided to create a BI Tool Evaluation tool. We had to outline the main pillars on which we would evaluate the candidate tools. We then anonymously voted on the importance of those pillars, averaging the weights and normalizing them. We finally reached a total of 9 pillars and 9 respective weights (summing up to 100%). The pillars list contain connectivity effectiveness, sharing effectiveness, graphing, exporting, among other factors.
These pillars were then analyzed in order to come up with small testing cases, using which we would assess the performance in each pillar, not forgetting to assign weights on these cases too, so that they sum up to 100% within each pillar. Long story short we came up with 80 points to assess each one of the BI tools.
We needed to be as impartial as possible on this, so we assigned two people from the BI team to evaluate all 5 tools involved. Each BI tool was also evaluated by 5 other people from within ORFIUM but outside BI, all of them potential Self-Service Analysts.
Coming up with 3 evaluations for each tool, averaging the scores, and then weighting them with the agreed weights, led us to an amazing Radar Graph.
Though there is a clear winner in almost all pillars, it performed very poorly in the last pillar, which contained Cost per user and Ease of Use/Learning Curve.
We decided to go for the blue line which was Metabase. We found out that it would serve >80% of current needs of Self-Service Analysts, with very low cost, and almost no code at all. In fact we decided (Data Governance had a say on this too) that we would not allow users to be able to write SQL queries on Metabase to create Graphs. We wanted people to go on the Snowflake UI to write SQL, as those people were few and SQL-experienced, as they usually were backend engineers.
We wanted Self Service Analysts to use the query editor, which simulates an adequate amount of SQL features, in order to avoid coding at all. If they got accustomed to using the query builder, then for the 80% of their needs they would have achieved this with no SQL, so the rest of the Self-Service Analysts (the even-less tech savvy) would be inspired to try it out too.
After ~10 months of usage (on the Self-Hosted Open Source version costing zero dollars per user per month, which translates to *calculator clicking * zero dollars total) we have almost 100 Monthly Active Users and over 80 Weekly Active users, and a vibrant community of Self-Service Analysts looking to get more value from the data. The greatest piece of news is that the Self-Service Analysts become more and more sophisticated in their questions. This is solid proof that, within the course of 10 months, they have greatly improved their own Data Analysis skills, and subsequently the effectiveness of their day-to-day tasks.
Within those (on average) 80WAUs, the majority is Product Owners, Business Analysts, Operations Analysts, etc., but there are also around five high level executives, including a member of the BoD.
Conclusion
The BI team and ORFIUM itself have evolved in the past few years. We started from Amazon Athena and Quicksight, and after a part of the journey with python by our side, we have established Snowflake, Airflow, dbt and Tableau as the BI stack, while adding in ORFIUM’s stack Select Star for Data Discovery and Metabase for Self-Sevice Analytics.
More info on these in upcoming posts, but we have more insights to share for the Self-Service Initiative, the Staff BI role, and the Data Culture at ORFIUM.
We are only eager to find out what the future holds for us, but at the moment we feel future-proof.
Foreword
This is the story of Business Intelligence and Analytics in Orfium, from before there was a single team member or a team itself within the company, to now, where we have a BI organization that scales, Data goals and excited plans for future projects.
Our story is a long one to tell, and one that makes us enthusiastic to tell. We’ll go through the timeline of our journey to the present day, but we fully plan to elaborate on the main points discussed today in their own articles.
We hope you’ll enjoy the ride. Buckle up, here we go!
Where we started
Before we formally introduced Business Intelligence to Orfium, there were a few BI-adjacent functions at the company. A number of Data Engineers, Operations Managers, and Finance execs created some initial insights with manual data pipelines.
These employees primarily gathered insights on two main parts of the business.
1. Operations Insights – Department and Employee Performance
To get base-level information on the performance of departments and specific employees, a crew of Data Engineers and Operations Managers with basic scripting skills came together, put together a python script, which didn’t lack bugs. The script pulled data from CSV exports from our internal software, as well as exports from AWS S3 that were provided by the DE teams, and connected them to produce a final table. All the transformations were performed within the script. No automation was initially required, as our needs were mostly for data on a monthly basis. The final table would then be loaded on Excel and analyzed through pivot tables and graphs.
This solution provided some useful insights. However, it certainly couldn’t scale along with the organization. A few problems came up along the way which could not be solved by this solution. Not the least of which, our need to have more frequent updates of daily data, to be able to view historical performance, and to join the data with important data from other sources were all reasons why Orfium seeked a bigger, smarter and more scalable solution to BI.
2. Clients Insights
Data Engineers put together a simple dashboard on Amazon QuickSight to give clients insights into the revenues we were generating for them. The data flowed from AWS S3 tables they had created, and they displayed bar charts of revenues over time, with some basic filtering. This dashboard was maintained for a couple of years but ultimately was replaced in March of 2022 with a more comprehensive solution that the BI team provided (spoiler alert: we created a BI team).
A new BI era
In light of some of the issues mentioned above, a small team of BI Analysts was assembled to help with the increasing needs of the Operations team.
The first decision made by the BI Analysts was related to which tools to use for visualization and the ETL process. Nikos Kastrinakis, Director of Business Intelligence, had worked with Tableau previously, so he did a demo and trial with Tableau and ultimately convinced the team to use this as our visualization tool. We also use Tableau Prep as our ETL tool. The company was now storing all relevant data in AWS S3, and the Data Engineers used AWS Athena to create views that transformed the data provided into usable tables that BI could join in Tableau Prep.
During the Tableau trial, the BI team started working on the first dashboard, set to be used by the Operations department, replacing the aforementioned buggy script. We created one Dashboard to rule them all, with information on overall department performance, employee performance, and client performance. This gave users their first taste of the power of a BI team. Our goal was to answer many of their questions in one concise dashboard, complete with historical breakdowns of different types of data, and bring insights that users hadn’t seen before. The Tableau trial ended right before the Dashboard was set to launch. So of course, we purchased our first Tableau licenses for Orfium and onboarded the initial users with the launch of this Dashboard.
It was a huge success! The Operations team was able to phase out their use of the script, stop wasting time monthly to generate reports for themselves, and were exposed to a new way of gaining insights.
Our work with the Operations team didn’t stop there. Over the next many months we continued to work with this data stack. We created further automation to bring daily data to the Operations team so they could manage departments and employees in real-time. But this introduced some new challenges we had to face.
With the introduction of daily estimated data, the frequency of updates and the size of the views made the extracts unusable and obsolete, so we had to face the tradeoff of data freshness VS dashboard responsiveness. Most of the stakeholders were happy to wait 30 seconds more when they looked at their dashboards, knowing that they had the most up-to-date data possible. Operations needed to be more agile in their decisions and actions, so having fresh data was very important for them. To date, members of the Operations team remain the most active users of Tableau at Orfium and have been active participants in other data initiatives across the company.
The reception of these initial dashboards was amazing. The stakeholders could derive value and make smarter decisions faster, so the BI team gained confidence and trust. However, the BI team was still mainly serving the Operations department (with some requests completed for Clients and Corporate insights) but was starting to get many requests from Finance, Products, and other departments. We began to add additional BI Analysts to serve these needs. However, this was just the beginning of the creation of a larger team that could serve more customers more effectively, as we also began improving internal tech features and utilizing external solutions for ready-made software.
Where we are today
We had many questions to clear up:
Where to store our data, how to transform them, who is responsible for these transformations, who is responsible for the ready and delivered data points, who has access and how do they get it, where do we make our analyses, how do data move around platforms and tools, how do our data customers discover our work?
Months and months of discussions between departments on all these questions lead to a series of decisions and commitments about our strategic data plans.
Where we stand now is still a transition from the previous step, as we decided to take a giant step forward, by embracing the Data Mesh Initiative. We’ll have the chance to talk about some of the terms and combinations of software that we’re about to mention in future blog posts, but we can run through the basics right now.
Our company is growing very quickly and, given the fact that we prefer being Data-(insert cliche buzzword here), the needs and requests for BI and Data analysis are growing at double the speed.
The increase in the number of BI Analysts was inevitable with the increasing requests and addition of new departments in the company that needed answers for their data questions.
By hiring more BI Analysts, we split our workforce between our two main Data customers, and thus created two BI Squads.
One is focused on finance and external client requests. We named it the Corporate squad, and it consists of a BI Manager and 2 BI Analysts. This is the team that prepares the monthly Board meeting presentation materials (P&L, Balance Sheets), and the dashboards shared with our external customers so that we can use data to demonstrate the impact of our work on their revenue and so that they get a better understanding of their performance on YouTube. This squad also undertakes many urgent ad-hoc requests on a monthly basis. This squad has a zero tolerance policy for mistakes and usually works on a monthly revision/request cycle.
The second squad is more focused on analyzing, and evaluating the performance and usage of our internal products, and connecting that info with the performance of our Operations teams, which generate the largest portion of our revenue. This squad, which happens to work from two different time zones, again consists of a BI Manager and 2 BI Analysts, and has more frequent deadlines, as new features come up very fast and need evaluation. The nature of data and continuous evolution of the data model results in less robust data.
In the meantime, we had already realized that we had to bulletproof our infrastructure and technical skills before scale gets to us. We decided to have some team members focused on delivering value by creating useful analyses, as described above, but we also reserved time and people who were more focused on paving the way for the rest of the analysts to be able to create more value, more efficiently.
We researched the community’s thoughts on this and we found the term Analytics Engineer, which seemed very close to what we were looking for. We thought this would be very important for the team and decided to go one step further and create a separate role that would be the equivalent of the Staff Engineer for software engineering teams. This role is more focused on setting the technical direction of the department, researching new technologies, consulting on the way projects should be driven, and enforcing best practices within the Analytics Chapter of the Data Unit. Quality, performance, and repeatability are the three core values that the code produced by this department should have.
We currently have a team of 8 people including the BI Director, two squads, each with one BI Manager and two BI Analysts, plus the Staff BI Analyst.
In terms of skillsets, we left python behind. Instead, we’re focusing more on writing reliable and performant SQL code and collaborating on git efficiently as a team. Our new toolkit is also more or less co-decided by the endorsement of a centralized data mesh, which is currently hosted by Snowflake. Nothing is being hosted/processed locally anymore, we develop data pipelines using SQL, apply them to our dev/staging/production environments through dbt and orchestrate the scheduling and data freshness using Airflow. We are the owners of our own Data Product, which is Orfium’s BI layer. It is a schema in Orfium’s production database where we store our fresh, quality, and documented data resulting from our processes. This set of tables connects data from other teams’ data products (internal products, external reports, data science results) and creates interoperating tables. These tables are the base for all our Tableau Dashboards, and help other teams use curated data without having to reinvent the wheel of the Orfium data model on their own again. Our Data product and our Tableau sites with all of our dashboards are fully documented and enriched with metadata so that our data discovery tool Select Star allows stakeholders to search and find all aspects of our work.
The future
Data Mesh was a big bet for Orfium and we will continue to build on it. The principles are applied and we are in the process of onboarding all departments on the initiative to be able to take advantage of the outcomes to the fullest extent. When this hard process is completed, all teams will enjoy the centralized data and the interoperability that derives from it, the Domain-Driven Data ownership, ensuring the agreed levels of quality on data, and it will help Orfium become more data-powered.
In addition to the obvious outcomes of applying the Data Mesh principles in a company, we believe that we need to follow up with two more major bets.
We decided to initiate, propose and promote Data Culture in Orfium. This is a very big project and is so deep that all employees need to get out of their comfort zones to achieve it. We need to change the way we work, to start planting data seeds very early in all our projects, products, initiatives, and working behavior so that we can eventually enjoy the results later on. This initiative will come with a Manifesto, which is being actively written and soon will be published. It will require commitment and follow-up on the principles proposed so that we achieve our vision.
Self-Service Analytics is also one of the Principles that Data Mesh is based on, and we decided to move forward emphatically with this too. Data will be generally accessible on Snowflake by everyone, but Data Analysis requires data literacy, SQL chops, and infrastructure that can host large amounts of data in an analysis. We decided to use Metabase as the proxy and facilitator for Self-Service Analytics. It provides the infrastructure by analyzing the data server-side, and not locally, and its query builder for creating questions is an excellent tool to create no-code analyses. Surely, it is not as customizable as SQL, but it will cover 85% of business users’ needs with superior usability for non-technical users.
This leaves us with data literacy and consultancy. For this, we have set up a library of best practices, examples, tutorials, and courses explaining how to handle business questions, analyses, limitations of tools, etc. At Orfium we always want to take a step further though, and we have been working to formulate a new role that will provide in-depth consultancy on data issues. This role will act as a general family doctor, who you know personally and trust, and will handle all incoming requests on data problems. Even if the data doctor cannot directly help you, they can direct you to a more suitable “doctor”, a set of more specialized experts, each one in their data sector (Infrastructure, SQL, Data Visualization, Analysis Requirement, you name it).
To infinite data and beyond
What a journey this has been over the last 3-4 years for BI in Orfium! We have gone through a lot, from not having official Business Intelligence to a BI team that has plans, adds value for the organization and inspires all teams to embrace the data-driven lifestyle. We’ve done a great job so far, and we have great plans for the future too.
It’s a long way to the top, if you want to rock and roll, ACDC
