The methods used to collect data vary based on the type of application. Some involve the use of technology, while others are manual procedures. In practice, some of the challenges often faced when collecting data, these include:
Data quality issues. Raw data typically includes errors, inconsistencies and other issues. Ideally, data collection measures are designed to avoid or minimize such problems. That isn’t foolproof in most cases, though. As a result, collected data usually needs to be put through data profiling to identify issues and data cleansing to fix them.
Finding relevant data. With a wide range of systems to navigate, gathering data to analyze can be a complicated task for data scientists and other users in an organization. The use of data curation techniques helps make it easier to find and access data. For example, that might include creating a data catalog and searchable indexes.
Deciding what data to collect. This is a fundamental issue both for upfront collection of raw data and when users gather data for analytics applications. Collecting data that isn’t needed adds time, cost and complexity to the process. But leaving out useful data can limit a data set’s business value and affect analytics results.
Dealing with big data. Big data environments typically include a combination of structured, unstructured and semistructured data, in large volumes. That makes the initial data collection and processing stages more complex. In addition, data scientists often need to filter sets of raw data stored in a data lake for specific analytics applications.
Low response and other research issues. In research studies, a lack of responses or willing participants raises questions about the validity of the data that’s collected. Other research challenges include training people to collect the data and creating sufficient quality assurance procedures to ensure that the data is accurate.
More articles on: