ETL processes design are changing
Using data directly from source systems is not efficient. Many solutions have been created that are designed to remove unnecessary records from data, combine data from various tables or databases, and ultimately prepare them for effective use by business users. It is not always such a simple task. This is especially the case when business users want to have a direct impact on the data processing process. This is often due to dynamic changes in selected business areas.
How to “open” ETL?
Direct access to the data processing source code is usually not possible. On the one hand, this would violate the principles of responsibility for the implementation of processes, on the other hand, business users will not know how to make changes securely.
In this situation, business users outsource changes to qualified IT staff. This significantly increases the costs of making changes and introduces the possibility of errors during communication.
The solution is to implement data transformation code with parameters, develop code validation rules and provide editing capabilities for business users. Users operate in a familiar environment, where they can independently prepare data processing parameters. Automatic validation using rules reduces the risk of errors.
Tools such as Metastudio DRM, allowing validation of SQL and other scripting languages, allow you to automate the process of making changes. Additionally, recording changes allows you to determine when and what parameter sets were used.
The implementation of such a solution allows to introduce changes in ETL processes in a continuous manner. The costs of implementing changes on the IT side are minimal, and the responsibility for configuration lies entirely with the people who ultimately use the data.
Stay in touch! Soon we invite you to the webinar on this topic.
