Dataset Labelling For Entity Resolution & Beyond with Dr. Fiona Browne

Dataset Labelling for Entity Resolution

In late 2019 our Head of AI, Dr Fiona Browne, delivered a series of talks to the Enterprise Data Management Council on AI-Enabled Data Quality in the context of AML operations, specifically for resolving differences in dataset labelling for legal entity data. In this blog post, her colleague Dr Raghad Al-Shabandar goes under the hood to explain some of the techniques that underpin Datactics’ extensible AI Framework.

Across the financial sector, Artificial Intelligence (AI) and Machine Learning (ML) have been applied to a number of areas, including the profiling of behaviour for fraud detection and Anti-Money Laundering (AML), through to the use of natural language processing to enrich data in Know-Your-Customer processes (KYC).

An important part of the KYC/AML process is entity resolution, which is the process of identifying and resolving entities from multiple data sources[2]. This is traditionally the space that high-performance matching engines have been deployed, with associated fuzzy-match capabilities used to account for trivial or significant differences (indeed, this is part of Datactics’ existing self-service platform).

In this arena, Machine Learning (ML) techniques have been applied to address the task of entity resolution using different approaches from graphs and network analysis to probabilistic matching[3].

Although ML is a sophisticated approach for democratizing entity resolution, a limitation of applying this approach is the requirement of large volumes of labelled data for the ML model to learn from when supervised ML is used[4].

What is Supervised ML? 

For supervised ML, a classifier is trained using a labelled dataset. This is a dataset that contains example inputs paired with their correct output label. In the case of entity resolution, this includes examples of input matches and non-matches which are correctly labelled. The ML algorithm learns from these examples and identifies patterns that link to specific outcomes. The trained classifier then uses this learning to make a prediction on new unseen cases based on their input values [5].

Dataset Labelling

As we see from above, for supervised ML we need high quality labelled examples for the classifier to learn from. The process of labelling a dataset from scratch can be time-consuming and labour intensive especially if experts are required to provide labels for, in this example, entity resolution outputs. The work is repetitive in nature, and there is a need for consistency in the labelling process to ensure high quality and correct labels are applied [6]. It is also costly in monetary terms, as those involved in processing the entity data require a high level of understanding of the nature of entities and ultimate beneficial owners, and in the context of failure where regulatory sanctions and fines can result.

Approaches for Dataset Labelling

As AI/ML progresses across all sectors, we have seen the rise in industrial level dataset labelling where companies/individuals are able to outsource their labelling tasks. For example, the Amazon Mechanical Turk service, which enables the crowdsourcing of labelling of data [7]. This can reduce labelling tasks from months to hours.  ML can also be harnessed for the labelling task using approaches such as weak and semi-supervised learning along with Human-In-The-Loop Learning (HITL). HITL enables the improvement on ML models through the incorporation of human feedback through stages such as training, testing and evaluation.

ML approaches for Budgeted Learning

We can think of budgeted learning as a balancing act between the expense (in terms of cost, effort and time) of acquiring training data against the predictive performance of the model that you are building. For example, can we label a few hundred examples instead of hundreds of thousands? There are a number of ML approaches that can help with this question and reduce the burden of manually labelling large volumes of training data. These include transfer learning, where you reuse previously gained knowledge. For instance, leveraging existing labelled data from a related sector or similar task. The recent open-source system Snorkel [8] uses a form of weak supervision to label datasets via programmable labelling functions.

Active learning is a semi-supervised ML approach which can be used to reduce the burden of manually labelling datasets. The ‘active learner’ proactively selects the data it needs to learn from. This is based on the concept that an ML model can achieve good predictive performance with fewer training sample instances by prioritising the examples to learn from. During the training process, an active learner poses queries which can be a selection of unlabelled instances from a dataset. These ML selected instances are then presented to an expert to manually label [9].

Vector created by macrovector – www.freepik.com

As it is seen above, there are wide and varied approaches to tackling the task of dataset labelling. What approach to select depends on a number of factors from the prediction task through to expense and budgeted learning. The connecting tenet is ensuring high quality labelled datasets for classifiers to learn from.  

Sources: 

[1]      W. Paper, “Anti-Money Laundering : Challenges and Trends,” 2017.

[2]      Pitney Bowes, “AML of Tomorrow: Business Value of Entity Resolution Solutions for Financial Crimes and Compliance Operations,” 2019.

[3]      B. Li, “Entity Resolution over Graphs,” 2014.

[4]      H. H. Hoos, “A survey on semi-supervised learning,” Mach. Learn., vol. 109, no. 2, pp. 373–440, 2020.

[5]      S. S. Sawant and M. Prabukumar, “A review on graph-based semi-supervised learning methods for hyperspectral image classification,” Egypt. J. Remote Sens. Sp. Sci., no. xxxx, pp. 1–6, 2018.

[6]      M. Lease, “Semi-Supervised Consensus Labeling for Crowdsourcing,” in In SIGIR workshopcrowdsourcing for information retrieval (CIR), 2011, pp. 36–41.

[7]      A. M. Turk, “AWS Mechanical Turk,” 2020. [Online]. Available: https://www.mturk.com/.

[8]      “Snorkel – Programmatically Building and Managing Training Data,” 2020. [Online]. Available: https://www.snorkel.org/.

[9]      P. Bachman, A. Sordoni, and A. Trischler, “Learning Algorithms for Active Learning,” 2017.

Contact us here

Click here for the latest news from Datactics, or find us on LinkedinTwitter or Facebook