Last updated on:
It is seldom that an enterprise app runs without batches, especially processing data between interfaces with other apps. Easily we think of ETL tools. True, they are always an option coming up on the table. At the same time, Spring batch appears to be another good choice: it leverages the features of Spring Framework such as logging, tracing, transaction management, resource management, AOP etc.; it is lightweight while supporting robust batch apps; it enables daily complex and demanding business operations.
We’ll open up the veil by walking through an example which imports a csv file into tables in the database, as illustrated in the below diagram.
Passwords are encrypted and then loaded into the Users table. Roles are transformed by inserting “ROLE_” before them.
Here’s what the work flow looks like.
The source codes are available on GitHub.
Prerequisite
Make sure Spring Tool Suite is installed, and the table are created in the database as well. By the way, “id” column of the Users table gets the value populated by sequence “users_seq”.
create table users (
id number
,username varchar2(64) unique
,password varchar2(255) not null
,enabled number(1) not null
);
create table authorities (
username varchar(64) not null
,authority varchar(64) not null
);
create sequence users_seq
start with 1
increment by 1
nocache
order
nocycle;
One more note, a Spring Batch logs execution status to several batch-dedicated tables, which need to be created beforehand. The DDL file, for example, oracle_ddl.sql, is located in the org.springframework.batch.core jar file under Maven Dependencies.
Create a Spring Boot Project
Spring JDBC along with Oracle will be used for data persistence. Select them on the Dependencies tab. Surely, you can manually add them to pom.xml later, as shown in the Dependencies section.
Dependencies
We’re going to truncate the tables before import through JDBC and load the data to Oracle. So, their dependencies are added into the pom.xml file, not to mention, Spring Boot Batch.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jdbc</artifactId>
</dependency>
<dependency>
<groupId>com.oracle.database.jdbc</groupId>
<artifactId>ojdbc11</artifactId>
<scope>runtime</scope>
</dependency>
Property Setting
We assume the file could be on any directory in the server, so we define a couple of properties in the application property file: use “import.path” to indicate the directory; use “file.users” to specify the file name.
spring.profiles.active=dev
import.path=xxx/shared/
file.users=users.csv
To connect to the database, we provide the connection strings and use the default data source that is configured by Spring Boot.
spring.datasource.driver-class-name=oracle.jdbc.OracleDriver
spring.datasource.url=jdbc:oracle:thin:@localhost:1521/xepdb1
spring.datasource.username=user
spring.datasource.password=password
Configure the job
The job takes two steps:
truncateStep – implements a tasklet truncating the tables beforehand;importFileStep – implements file reading, data transforming and data writing tasks.
They are the implementation of the preceding job flow.
@Configuration
@EnableBatchProcessing
public class FileToDbBatchConfig {
public static final String JOB_NAME = "FileToDbJob";
@Autowired
public JobRepository jobRepository;
@Autowired
public PlatformTransactionManager transactionManager;
/*
* Job flow
*/
@Bean
public Job ftdImportUserJob() {
return new JobBuilder(JOB_NAME, jobRepository)
.incrementer(new RunIdIncrementer())
.validator(ftdJobParamValidator())
.start(truncateStep())
.next(importFileStep())
.build();
}
@Bean
public JobParametersValidator ftdJobParamValidator() {
String[] requiredKeys = new String[]{"filePath"};
String[] optionalKeys = new String[]{"executedTime"};
return new DefaultJobParametersValidator(requiredKeys, optionalKeys);
}
/*
* truncteStep: run a tasklet truncating users and authorities tables
*/
@Autowired
MethodInvokingTaskletAdapter ftdTruncateStepTasklet;
@Bean
public Step truncateStep() {
return new StepBuilder("truncateStep", jobRepository)
.tasklet(ftdTruncateStepTasklet, transactionManager)
.build();
}
/*
* importFileStep: read the csv file and write to the database
*/
@Autowired
FlatFileItemReader<User> ftdImportFileStepReader;
@Autowired
ItemProcessor<User, User> ftdImportFileStepProcessor;
@Autowired
JdbcBatchItemWriter<User> ftdImportFileStepWriter;
@Bean
public Step importFileStep() {
return new StepBuilder("importFileStep", jobRepository)
.<User, User>chunk(10, transactionManager)
.reader(ftdImportFileStepReader)
.processor(ftdImportFileStepProcessor)
.writer(ftdImportFileStepWriter)
.build();
}
}
Truncate table step
In this step, the tasklet calls UsersDao’s truncateUsers() method to delete all data from the tables.
@Configuration
public class FileToDbBatchTasklet {
@Autowired
private UsersDao usersDao;
@Bean
public MethodInvokingTaskletAdapter ftdTruncateStepTasklet() {
MethodInvokingTaskletAdapter adapter = new MethodInvokingTaskletAdapter();
adapter.setTargetObject(usersDao);
adapter.setTargetMethod("truncateUsers");
return adapter;
}
}
@Repository
public class UsersDaoImpl implements UsersDao{
@Autowired
JdbcTemplate jdbcTemplate;
public ExitStatus truncateUsers() {
String truncUsers = "truncate table users";
String truncAuthorities = "truncate table authorities";
jdbcTemplate.execute(truncUsers);
jdbcTemplate.execute(truncAuthorities);
return ExitStatus.COMPLETED;
}
}
Import file step
This step implements a reader, a processor and a writer, which realize file reading, data transformation and data writing functions in the work flow, respectively.
The reader reads records from the csv file and save to User objects.
@Configuration
public class FileToDbBatchReader {
@Bean
@StepScope
@Value("#{jobParameters['filePath']}") // to get job parameter
public FlatFileItemReader<User> ftdImportFileStepReader(String filePath) {
return new FlatFileItemReaderBuilder<User>()
.name("UserItemReader")
.resource(new FileSystemResource(filePath))
.delimited()
.names(new String[]{"username", "password", "role", "enabled"})
.targetType(User.class)
.build();
}
}
public class User {
private String username;
private String password;
private String role;
private Integer enabled;
/*
* Constructors, Getters, Setters etc.
*/
}
The processor encrypts the passwords and inserts “ROLE_” before each role name.
@Configuration
public class FileToDbBatchProcessor {
@Bean
@StepScope
public ItemProcessor<User, User> ftdImportFileStepProcessor() {
return item -> {
/*
* Transforming data: encrypt the password
*/
item.setUsername(item.getUsername());
item.setPassword(
PasswordEncoderFactories
.createDelegatingPasswordEncoder()
.encode(item.getPassword()));
item.setRole("ROLE_"+item.getRole());
item.setEnabled(item.getEnabled());
return item;
};
}
}
The writer loads the transformed data to the tables, users and authorities.
@Configuration
@EnableBatchProcessing(dataSourceRef = "dataSource")
public class FileToDbBatchWriter {
@Autowired
DataSource dataSource;
@Bean
@StepScope
public JdbcBatchItemWriter<User> ftdImportFileStepWriter() {
String sql = "INSERT ALL "
+ "INTO users(id, username, password, enabled) "
+ "VALUES (users_seq.nextval, :username, :password, :enabled) "
+ "INTO authorities(username, authority) "
+ "VALUES (:username, :role) "
+ "SELECT * FROM DUAL";
return new JdbcBatchItemWriterBuilder<User>()
.sql(sql)
.dataSource(dataSource)
.itemSqlParameterSourceProvider(
new BeanPropertyItemSqlParameterSourceProvider<>())
.build();
}
}
Run the batch
Configure a CommandLineRunner bean in the batch application to launch the file import job.
@SpringBootApplication
public class SpringBatchApplication {
public static void main(String[] args) {
SpringApplication.run(SpringBatchApplication.class, args);
}
@Bean
public CommandLineRunner runImportFileJob(
JobLauncher jobLauncher,
Job importUserJob,
@Value("${import.path}") String importPath,
@Value("${file.users}") String fileUsers) {
return args -> {
JobParameters jobParameters = new JobParametersBuilder()
.addString("filePath", importPath + fileUsers)
.addLong("executedTime", System.currentTimeMillis())
.toJobParameters();
jobLauncher.run(importUserJob, jobParameters);
};
}
Click Run button in Eclipse.
After the batch finished, you can view the inserted records in the tables, as shown in the following screenshot.
Recap
A job consists of steps. A step typically is made up of a reader and a processor and a writer, or a tasklet. A tasklet is suitable for a specific, separate task like cleaning up a table. For a Spring batch is able to access all Spring features, it can implement complicated business logics and easily adapt to daily operation, monitor, enhancement in an enterprise environment. Moreover, no extra infrastructure is required to host it.
No comments:
Post a Comment