Closure Report

Background

The creation of EDW will provide the data and information needed by decision makers across the University, giving a consistent view of data across multiple sources, adhering to consistent data definitions. This is in line with the University’s BI/MI Strategy, our reference architecture and the growing requirement for evidence-based planning and decision making.

ETL is a key component of the EDW programme, allowing data to efficiently flow from operational systems into EDW and be validated and transformed through the different layers within EDW. The tool selected for this purpose by project DTI004 is the Talend Data Management Platform.  Talend’s implementation partner for this tool is Datalytyx Ltd.  This project will deliver the installation, configuration and implementation of this tool to the University’s technical standards and the EDW project team’s requirements.

In their response to the University’s Invitation To Tender, Talend and Datalytyx proposed that we consider an option to implement the tool in a hybrid-cloud environment (the Invitation specified an on-premise solution).  Talend has subsequently clarified that this option would not involve further costs, either in licensing or implementation.  The project will therefore conduct a short piece of work to evaluate the relative risks and feasibility of these two options, prior to beginning technical design work.  The product of this work will be a short Options Report, produced by the project manager and reviewed internally by IS Applications stakeholders.  It is anticipated that Talend and Datalytyx will provide input to this report. 

Wherever possible, the delivery strategy for this project will aim to make the supplier (i.e. Datalytyx Ltd on behalf of Talend) to carry out as much of the project work as possible, as outlined in their response to Question 18 of the Invitation To Tender ‘Implementation’.  This work will be controlled by means of a statement of work agreed with the University’s Project Manager, and monitored by means of weekly project meetings.  Acceptance of the delivered product will be by sign-off of an agreed set of Acceptance Tests defined by the University project team, who will be responsible for carrying out those acceptance tests. 

Scope

The scope of this project is to implement (‘install and productionise’) the ETL tool selected by project DTI004 (ETL Tool Procurement), integrated with the three EDW environments (Development, Test and Live).

The tool will be used to create and maintain robust live technical processes to extract data from operational systems, to validate and transform the data and then load information into EDW (Enterprise Data Warehouse). The tool will also provide facilities for production mgmt.

This is part of the EDW programme, under the Digital Transformation portfolio.

Project Summary

This completion report has been written by the current project manager, appointed to the project in February 2019. Comments and input have been collated from a review of the project website  documentation and others.

This project has delivered the installation, configuration and implementation of this tool incorporating;

• The required infrastructure to support the Talend Hybrid Cloud solution across all three EDW environments (Development, Test and Live)
• The successful migration of Org Hierarchy into the new environments 
• System documentation supporting 
  • The technical design 
  • Developers support guide 
  • post go-live support recommendations 
• Training to the nominated Applications staff 

This project has seen the introduction of new technology into the University using a hybrid-cloud environment which has resulted in a steep learning curve and the work of main project development team should be recognised.

Objectives 

Deliverables

Analysis of Resource Usage:

Staff Usage Estimate: 82 days

Staff Usage Actual: 146 days

Other Resource Estimate: £xxx

Other Resource Actual: £26,313

Training Credits: 12,136

Training Consultant: £2,307

Implementation Consultancy: £11,870

Other Resource Variance: xx%

Breakdown by Team 

Explanation for variance

Timescales 

• The project  initially scheduled to commence on 04/12/17 but the start was delayed until 02/04/18.  This knock-on effect was due to the delays encountered in the ETL Procurement (DTI004) - ref piccl item 1
• The project commenced on the 02/04/18, with a scheduled delivery date 31/08/18
• The project experienced a number of delays within the planning phase of the project, namely;
  • The initial planning milestone scheduled for 20/04/18 was rescheduled initially to 31/07/18 - due to further delays in the ETL Procurement process (completed w/c 11/06/18) - ref piccl item 3
  • Due to unavailability of the project manager, due to unforeseen circumstances, planning was completed on 09/11/18 - ref piccl item 6
• Subsequent delays were encountered within the design phase of the project due a combination of Technical resource constraints; Delays in issuing Purchase Order and Delays to the training schedule.  As a result, the delivery date was re-scheduled from 28/01/19 to 22/03/19 - ref piccl item 7
• The requirement to re-arrange the in-house training to 4th and 5th March, had a further knock on effect on the project, with the project re-planned to complete the remaining activity and deliver on 24/05/19 - ref piccl item 10
• The project team noted completion of all project related tasks on 21/06/19 - ref project team meeting minutes 

Budget 

Having completed  the in-house training the project budget was increased from 100 days to 135 days to acknowledge the work completed to-date and the outstanding activities.  However, additional time was required to resolve the issues associated with the (Jiras)

• DTI005-06: Talend / Git credential encryption: 
• DTI005-07: Questions on debugging Studio to TRE (finally acknowledged by Talend as a bug)
• DTI005-17: complete Disaster Recovery tests
• DTI005-19: Prepare post go-live support recommendations 
• DTI005-2: DTI005-21: Generic Named Workspace and RE's for TMC - to enable the promotion of systems to all environments 

Key Learning Points

• Whilst much of the initial configuration was carried out by Datalytx, it would have been better if the training for the main project team could have been scheduled earlier in the project
• The introduction of new technology has raised a number of questions of how the products will be subsequently supported. Whilst the project has provided a series of recommendations towards the end of the project, it would have been beneficial to consider this matter, with colleagues from both Production and Service Management from the Design stage onwards
• There should have been consideration of an on-going support budget as part of the project

Outstanding Issues

JIRA DTI005-5: Log level and Redirect logs from Cloud to TRE

• Test out various log levels produced and information recorded in logs Test out redirecting log to TRE (remote server)

• The project team agreed that this could be carried out at a later date.

• To ensure that this requirement is not overlooked, the following Unidesk call has been raised - I190712-0486: Log level and Redirect logs from Cloud to TRE
• There are some outstanding Talend standards to be developed which will be taken forward by the virtual BI/MI team as part of BAU