Proposing Field Matching Similarity Methods. Implementation and Comparison of Field Similarity Metrics with Duplicate Entities Detection Purpose in Database

      Аннотация: Duplicate records do not have a common key but refer to a unit entity. Databases that include these records have often some errors which cause the matching problem in duplicate records becomes a complex problem. These errors are: typing errors, incomplete information such as abbreviations, ignoring of standard formats or a combination of the above factors. In this book, databases are used in which typing errors are more than other errors. This database contains real estate information that includes 4 fields: name, surname, property address and property area. The goals of this book are: a review on existing algorithms in identifying duplicate data in the fields which are: Edit-distance, Smith-waterman, Jaro, Jaro-Winkler, Lcs and N-gram; description of the proposed algorithms was presented to improve the efficiency and increase the precision of identifying duplication which are the proposed token-based algorithm and the proposed algorithm based on typing error; and comparing these algorithms efficiency in a large Persian database.

Ключевые слова: n-gram, Edit-Distance, Damerau-Leventein, Jaro, Jaro-winkler, longest common string, Token-based-Jaro, Typological-error-baed-Jaro