| Journal: | Polibits |
| Database: | PERIÓDICA |
| System number: | 000374546 |
| ISSN: | 1870-9044 |
| Authors: | Endredy, Istvan1 Novak, Attila1 |
| Institutions: | 1Pazmany Peter Catholic University, Faculty of Information Technology and Bionics, Budapest. Hungría |
| Year: | 2013 |
| Season: | Jul-Dic |
| Number: | 48 |
| Pages: | 79-83 |
| Country: | México |
| Language: | Inglés |
| Document type: | Artículo |
| Approach: | Aplicado, descriptivo |
| English abstract | The ever-increasing web is an important source for building large-scale corpora. However, dynamically generated web pages often contain much irrelevant and duplicated text, which impairs the quality of the corpus. To ensure the high quality of web-based corpora, a good boilerplate removal algorithm is needed to extract only the relevant content from web pages. In this article, we present an automatic text extraction procedure, GoldMiner, which by enhancing a previously published boilerplate removal algorithm, minimizes the occurrence of irrelevant duplicated content in corpora, and keeps the text more coherent than previous tools. The algorithm exploits similarities in the HTML structure of pages coming from the same domain. A new evaluation document set (CleanPortalEval) is also presented, which can demonstrate the power of boilerplate removal algorithms for web portal pages |
| Disciplines: | Ciencias de la computación |
| Keyword: | Procesamiento de datos, Tecnología de la información, Minería de texto, Análisis de textos, Algoritmos, Extracción de contenidos |
| Keyword: | Computer science, Data processing, Information technology, Text mining, Text analysis, Contents extraction, Algorithms |
| Full text: | Texto completo (Ver HTML) |
