Export Form Data - Technical Document

Introduction:

The export data feature in ClinPortalh as two parts: Export Data in CSV format and Export Data in SAS format. In both the cases when user clicks on any of the Export Data button, a request is send to the server with the details of what data is to be exported. Along with this a table below, which shows the details of exported data by that user is updated. In case of Admin user, he can see details of all the exported data by all users.

The main reason for the feature is that whenever user used to perform any kind of export data(Form/SAS), he was supposed to wait on that page till the data gets exported and a file is generated which he can download. Also if user just keeps the download popup unattended for some time and tries to download it, an error used to occur. This means all the time put into exporting the data is waste and user has to again start the process. For more details refer to the following Bugs #16624, #16691 and #16692

Overall Architecture:

The architecture model defined for this feature is based on the scheduling architecture which is generally used to perform tasks offline. This kind of architecture helps in sending user response back very fast. In such scenarios user does not have to wait till the task is complete. He can perform others operations and come back later to get the exported data.

When ever a user exports the data, the task of exporting is given to a Quartz Scheduler which schedules the task by calling the execute method of the class implementing Job class. The scheduler does not wait for the job to complete. It returns the control back, thus avoiding the blockage.

Once the task is done the export data details are stored in database by calling appropriate methods in the Job listener class and can be populated on UI by fetching it from database.

Design Principles:

Design principles used is of scheduling the task for background execution. For this purpose we have used the Quartz scheduler which is an open source project. It provides a lot of support for such kind of background execution. It requires a class to implement its Job interface which has n execute method. This method is supposed to execute the task of exporting data.

It also provides hook in forms of API which are called before and after the task is performed. This API is defined in JobListener interface of Quartz Scheduler. If some operations are to be performed before job is to be executed or after job is executed, Job Listener interface should be implemented.

Internal Design:

When user clicks on any of the export data button (FORM/SAS), a request is send to execute the task of exporting data. The corresponding Action class (ExportFormDataAction.java/GenerateSAASPGMAction.java) calls their corresponding RequestProcessor(FormDataRequestProcessor.java/ SASDataRequestProcessor.java) for processing the request object. Request processor populates a POJO(ExportDataDetails) object out of request parameters and sends it for validation. If validation does not return any error messages, request processor then forwards the POJO object for initializing the task of exporting data in background.

The initializing part involves instantiating scheduler object, populating Job Detail object, initializing Trigger and scheduling the task. Once this is done the response is send back to user with message that the task of exporting data has been started in background and the table below gets populated with status “In Progress”.

When scheduler starts the task, the “jobToBeExecuted()” of JobListener is called. In this method the job details in the POJO object created is inserted into database. This is required to show the details on UI. Once completed the task of exporting data is started by calling the “execute()” method of Job class. Once the Job gets successfully executed, the exported data file name, file content are updated in the POJO object which is put inside JobDetail object.

At the end of the task, the “jobWasExecuted()” of JobListener is called. In this method the POJO object is details are updated in database by fetching the row inserted for this Job. When user clicks on the “complete” link in the table populated on the UI, a database call is given to download the file contents and it is then send to user in response object. User can either download the Zip file or open it and see the contents.

The table structure in database for the Job Details is as follows

Changes in Export functionality:

On export data page, user can select or deselect events/forms/attributes to be exported. Events, forms, attributes are shown in tree structure.

* Tree generation logic: Given a study Id, GetdataBizLogic.getEventTree(Long studyId,SessionDataBean sessionDataBean)- method will generate XML for dhtmlx tree.
* As per selection of nodes in tree - XML is formed using GetDataOutUtility.convertToExportDataXML(String selectedTreenodes, String studyId)
* This XML is then stored into DB table as form_id (Clob)
* To read/convert this XML as ExportDataXML object , GetdataBizLogic.convertToExportDataXMLObj(Clob data) - method is used to convert Clob data into ExportDataXML object using JAXB.
* Grid display: Grid gets populated using dhtmlx connector.
ExportDataGridImpl.java - has query to populate grid data.

* Following methods are modified to get data for only selected attributes:
DynamicExportDataHandler.getHeaderForDyanamicDataCSV()

DynamicExportDataHandler.generateDynamicDatamap()

DynamicExportDataHandler.getDataListForDyanamicDataCSV()  

GenerateSAASPGMProcessor.getAbstractAttributeControl()

Files Affected: Following is the list of Files modified/added

• For Export Data
  • edu.wustl.clinportal.action
    
    • ExportFormDataAction.java
  • edu.wustl.clinportal.requestprocessor
    
    • FormDataRequestProcessor.java
  • edu.wustl.clinportal.dataexporter
    
    • GetDataBizLogic.java
  • edu.wustl.clinportal.util
    
    • GetDataOutUtility.java
  • edu.wustl.clinportal.exportformdata
    
    • ExportDataValidator.java
    • ExportDataUtil.java
    • ExportFormDataJob.java
    • ExportFormDataJobListener.java
    • ExportFormDataHelper.java
    • GridUtil.java

• For Export SAS Data
  • edu.wustl.clinportal.action
    
    • GenerateSAASPGMAction.java
  • edu.wustl.clinportal.dataexporter
    
    • GenerateSAASPGMProcessor.java
  • edu.wustl.clinportal.requestprocessor
    
    • SASDataRequestProcessor.java
  • edu.wustl.clinportal.exportsasdata
    
    • ExportSASDataJob.java
    • ExportSASDataJobListener.java
    • ExportSASDataHelper.java

• For Populating Grid on UI -
  • edu.wustl.clinportal.gridImpl
    • ExportDataGridImpl.java Grid is populated using dhtmlx Gridconnector.
• For POJO and its operations
  • edu.wustl.clinportal.exportformdata
    • ExportDataDetails.java
    • ExportDataDetailsImpl.java
    • ExportDataOperations.java

• For Quartz Scheduler
  • QuartzSchedulerFactory.java
  • Also required is quartz-all-1.6.6.jar

Rerun:

To export the latest data of participants for study/events/forms for which data was already exported.

On export page, on click of 'Download file' tab, grid with all related data (like status, study name, form names etc) on downloaded files is displayed. On extreme left of grid, we should have an extra column with an icon to rerun the export data for already downloaded files.
On click rerun icon , pop up will show up to agree on terms and conditions required to export PHI data and then only data for same study/events/forms should get exported and new entry in grid should get updated with latest data (like zip file name, job end time etc).

And should have facility to provide a title for exported job.

Implementation details:

• add activityStatus field in ExportFormDetails pojo but doesn't need to persist in DB.
• On export data page, at extreme left of downloaded file grid add one column with an icon to rerun the export data for already downloaded files.
• last column (icon column) should have identifier of 'ExportDetails' as a hidden value.
• On click of rerun icon -> rerun(exportDetailsId) method should be called which calls ExportFormDataAction

GetDataBizLogic: public void rerun(Long id, String exportType) throws Exception
{
ExportDataDetails details = getExportDataDetailsById(id);
ExportDataUtil.initializeFileDownloadJob(details,exportType);
}

Sequence Diagram