In data profiling and monitoring, data from existing sources is checked according to previously defined rules and criteria. Afterwards, the results are documented in the form of statistics and information. By performing such data quality analysis, it is possible to precisely assess the level of quality of existing data, as well as create measurable indicators for continuous monitoring. datenfabrik.profiler collects all relevant information within an Integration Services data flow and saves the output in a central repository.
datenfabrik.profiler is equipped with numerous different rules
, such as knowledge bases, regular expressions or sample column profiles for data analysis and allows more rules to be subsequently installed by a special plug-in concept — even within the SSIS component. You can get more information about the rule types of datenfabrik.profiler on the rules page
The information gathered and statistics compiled with by the rules can be recreated, extended or put under version control with each execution.
This allows data profiling to be implemented for individual updates, incremental updates or continuous monitoring. Data management employees thus have the opportunity to retrieve and analyze the gathered statistics from the repository. Based on the resulting findings, datenfabrik.profiler offers the option of defining events and notifications for the data elements to be monitored.
We put together comprehensive informationen about datenfabrik.profiler in our Whitepaper (PDF, 540 kB). You will find everything about continuous data profiling and monitoring with Microsoft SQL Server Integration Services, the consequences of bad data quality and the rule set in datenfabrik.profiler.
- Analysis of data with extensive rule types
- Storage of data and results in a central repository
- Historiography of executions in the central repository
- Refreshing of statistics in incremental loading processes
- Definition of threshold values for different rules and setting alerts
- Configuration of events and notifications at exceedance or undercutting of threshold values
- Export of identified recognized errors for post-processing through a data steward
- Data profiling for individual updates, incremental updates or within the scope of continuous monitoring
- Extensive plug-in concept for subsequent installation of additional rules