Make Java to Handle Only Business Logics (4) - How about Using an ETL Process in Place of Spring Boot Batch?

 

The below are the employee records associated to the department table via department number.

The department table comes here. Please pay attention to the record with grey background, which is not valid any more as the department's name was changed effective as of Jan. 1, 2011.

The two source tables will be imported and re-organized as the master table shown as below. Retail Sales becomes the new new name of  department No. 2, instead of Online Sales.

 

To fulfill this purpose, I create a job flow as below with one transformation and a mail step. The transformation takes on the major work needed to be done from extract through transform to load. The mail step sends an email to the support team when the process fails due to an unexpected error. 

The major part is the transformation which extracts data from [Employee] table and [Department] table, filters out invalid records, joins the two data sets and then loads the clean data into [EmployeeDetails] table.

Spoon provides a graphical interface where you can configure log setting. Double click [importEmployeeDetails] transformation on the job design window and the setting dialog pops up. Go to [Logging] tab and specify a log file as well as the preferable log level. 

Here is an example of the log with basic level for the above transformation.

With plenty of in-box functionalities, you can develop an ETL process in a stress-free way to import employee master data into your application. The process is visualized on Spoon and each of steps is visible to you during execution.