A review of the past, present and future of Payroll EDI including solutions for organisations who may need support for their legacy Payroll EDI services until new systems have been commissioned.

The era of Payroll EDI (as we know it) is coming to an end in less than eighteen months’ time. The purpose of this paper is to review the past and present and provide solutions for organisations who may need support for their legacy Payroll EDI services until new systems have been commissioned. Based on past experience, this process always takes longer than expected.

We strongly believe that there will be a role for data conversion from legacy formats for some time to come. Furthermore, we expect the need for traditional data transfer services, where Payroll EDI files are delivered and collected from data centres such as Avco’s, to continue to be in demand for some time. The main reason for this is that employers can use data transfer methods of their own choosing while communicating with us. They can then take advantage of our existing procedures, which have been developed over the last decade, in order to enable fully automated and seamless data exchange, including data validation where necessary, with HMRC.

Payroll EDI might not be very exciting but it is very important. We became involved with Payroll EDI when the Ministry of Defence, an existing data exchange customer who had been with us since 1990, asked us to implement it for them in 2004. (This variant of the Avco Payroll EDI Client has been upgraded recently and is still running in 2016.) For SMEs like Avco, any software project with repeat potential has to be exploited and the next logical step was to get in front of the then Inland Revenue’s Payroll EDI team in Shipley. This we did in 2004 and the reaction we got was quite amazing. Inland Revenue’s attitude was, we need a small player, hence we will help you. Soon after, some members of the Payroll EDI team came to visit Avco and agreed that Avco’s Payroll EDI services would be mentioned on the web pages of the Electronic Business Unit. This web presence has generated excellent Payroll EDI leads for us throughout the years. As a result, we are now working with over fifty large organisations and assisting them with their Payroll EDI. Some of these organisations use our Payroll EDI client. Soon after the development of the Client software in 2004, however, we were asked by a large airline to provide a bureau service. We worked together with them to adapt the Payroll EDI Client into a bureau environment and nowadays this represents two thirds of our Payroll EDI income.

Definitions

Payroll EDI (Electronic Data Interchange) is the computer-to-computer exchange of Payroll information in a standard file format between HMRC and employers.

In this paper we shall use both Inland Revenue and HMRC when referring to the UK Tax Authorities. As we are delving into the past, and the changeover from Inland Revenue to HMRC occurred in 2005, we shall refer to HMRC for events beyond 2005 and refer to Inland Revenue otherwise.

One of the first things we did, when we decided to write this article, was to look into the history of Payroll EDI. No information was found on the Internet and therefore here is our take on a potted history of Payroll EDI to put this right.

The Past

Very early days (mid to late 1950s)

At the end of the day, EDI’s main purpose is the elimination of information exchange via paper documents which are used for rekeying data. So, when during the mid to late 1950s Inland Revenue acquired its first computers, the opportunity must have presented itself where large employers with similar data processing resources would resort to submitting Payroll returns on magnetic tape rather than paper documents.

payroll

Furthermore, the first business computers in the UK were locally manufactured LEOs (Lyons Electronic Office) and, as Inland Revenue had its own LEO, there must have been good compatibility in the context of Operating Systems, magnetic media used and agreed data formats in the form of fixed length records with other LEO users. Consequently, Payroll EDI must have been happening as far as sixty years ago, even though in those days this would have been carried out on an ad hoc basis. It is anticipated that, as time went by, the data formats for Payroll EDI became more formal but the primary data exchange medium remained half-inch magnetic tape. From mid 1960s onwards mini computers became more and more popular but, as they also happened to support half-inch magnetic tapes, there must have been considerable Payroll EDI activity originating from mini computer users as well.

The PC becomes universal (30 years later – mid to late 1980s)

DataComms & Internet become ubiquitous (20 years or so later – early to mid-2000s)

2003-2004 was the first time when Payroll EDI was formally mooted. In this context, formal Payroll EDI meant being able to transfer data directly between Inland Revenue’s computers and employers’ Payroll systems and eliminating the need for magnetic media (or indeed paper returns). To this end, Inland Revenue created a format called Generic Flat File (GFF). Having compared the earlier tape format definitions and GFF, we suspect that the latter is based on the former although we were unable to ascertain this point. Employers were allowed to use GFF format for Payroll EDI with Inland Revenue. However, this could only be implemented via direct phone connection to Inland Revenue in the form of Integrated Services Digital Network (ISDN) and using the Odette protocol (a file transfer method created in 1986 used for general EDI). Inland Revenue also supported Payroll EDI data in EDIFACT format as this format was quite significant in EDI circles, especially where invoices and purchase orders were directly sent from one organisation’s computer to another, thus eliminating rekeying of data. Once again we suspect that the EDIFACT implementation was also based on earlier magnetic media formats.

payroll

As well as supporting ISDN, Inland Revenue also encouraged the use of the Internet for Payroll EDI. However, the data exchange process had to go via the Government Gateway and the only format allowed was XML (eXtensible Markup Language - a set of rules for encoding documents in a format that is both human-readable and machine-readable). XML brought in more flexibility but took up double the space of GFF, which meant that in the earlier days the number of employees in a return was limited (initially this was as low as 1000, increasing to 5000 and not a problem nowadays thanks to broadband – this is discussed later in the text).

Towards the end of the decade a method had to be found as an alternative to ISDN and HMRC adopted the AS2 (Applicability Statement 2) standard. This methodology enables secure and reliable data exchange over the Internet by means of digital certificates and encryption. However, AS2 implementations were relatively onerous and expensive.

From 2004 onwards all employers were expected to use Payroll EDI before the end of the decade. The larger employers had to come on board immediately, with smaller ones to follow. In order to encourage smaller organisations (i.e. those who employed fewer people) to use Payroll EDI as quickly as possible, HMRC provided financial incentives for early adopters. Consequently, many smaller organisations started using Payroll EDI early in order to enjoy those incentives.

RTI - Real Time Information (10 years or so later)

In 2013, one of the most significant changes in Payroll reporting took place.

Since its inception in 1944, payroll reporting had been conducted on an annual basis and the returns had largely stayed the same during that seventy-year period. From 2013 onwards, HMRC stipulated that employers had to file a return for each pay period - 12 a year for monthly payrolls and 52+ returns a year for weekly payrolls. This new reporting system was called RTI. It has had a huge impact on Payroll EDI, effectively increasing data volumes by a factor of 12 or more.

RTI was devised in order to handle a workforce where job changes are more frequent and the number of employees with more than one source of employment is higher. HMRC also had other business aims with RTI such as, reducing fraud, issuing correct benefits, improving the handling of under/over payments and providing other departments, such as Work & Pensions, with more up-to-date employment records.

The Present

2016 – Official Announcement of the demise of GFF and EDIFACT support

For many years HMRC has been predicting/promising the demise of Payroll EDI and hence ending support for GFF and EDIFACT formats. 2017-18 tax year looks to be the last – i.e. support should end on 5 April, 2018. Even though this has been on the cards for quite a few years, we believe this time it will actually happen. We have been working with HMRC since 2004 and their software team is very well run and has for the last couple of years done exactly what they said they would do. On top of that there has been very good communication and involvement with employers’ developers.

Data flow diagrams for Payroll EDI (as things are today):

ISDN

AS2

The mandatory monthly returns, such as FPSs and additional forms such as EPSs, NVREQs and EYUs, will therefore have to be sent to HMRC using the only available method left, the Government Gateway. Forms that get returned from HMRC to employers, such as P6s, P9s and NVREPs, will also have to be retrieved via this method. The Government Gateway supports data only in XML format and we anticipate that there will still be a requirement to convert legacy data from GFF, EDIFACT and even CSV (Comma Separated Value, still used by certain payroll systems).

The Future

payroll

We recognised several years ago that the Payroll EDI route would one day close and as we only had solutions to send data via this route we invented a hybrid system that could take GFF files and send them via the Government Gateway.

This has now placed us in a unique (as far as we know) position as we fully understand both ways of sending data and the relationship between the two types of data.

Here is our current hybrid solution:

As we have already developed this system, it is a very small step to be able just to send data in the form that the Government Gateway accepts (XML) or tailor a solution that takes a file in any format and sends it to HMRC.

The Government Gateway uses a SOAP (Simple Object Access Protocol) type interface to exchange data with employers. We have taken our guidance from information provided by HMRC for software developers on the Document Submission Protocol implementation used by Government Gateway for our implementation. This has also been the case for the reverse process. Once again, guidance is provided by HMRC to assist software developers with data retrieval.

A very important point to consider is that if an Employer is using an AS2 based system to exchange data with HMRC and relies on the fact that the HMRC pushes data to the Employer, this option will no longer be available as the Government Gateway expects the Employer to poll for returning data. Our automated solution can address this issue by being used periodically to check for any data HMRC has available.

We can therefore take Payroll EDI files (mainly GFF but also EDIFACT plus other proprietary formats) and translate them to a form the Government Gateways accepts when sending data to HMRC. The reverse is also supported where the XML files received from HMRC are converted to the format supported by the Employer (mainly GFF but also other formats). A dozen or so of our Payroll EDI Client users have already upgraded to this new version supporting XML and the Government Gateway.

payroll

Observations Regarding the Bureau Service post the Demise of Payroll EDI:

As mentioned in the introduction, the Payroll EDI Client software has been adapted in order to work in a bureau environment hosted and managed in our Data Centre. The following subtleties have to be taken into account post the demise of Payroll EDI:

• For customers who produce their own GFF files rather than use a system like Oracle we will carry on accepting the GFF format of files as long as we can; we would just expect the 4-star headers to reflect the correct year (as well as the current tax year and such segments).
• If the Employers’ payroll systems generate GFF or EDIFACT output, then it is important to note that the specifications that describe the data in these files will no longer be updated by HMRC. Therefore, for those Employers who wanted to continue using GFF, should HMRC introduce new data/form specifications these would no longer be supported in GFF. Consequently, our translation software would not support the new data/form. However, we can work around this by inventing our own segments. Being specialists in data exchange and conversion, we have solved such problems many times since 1986 (when we started).
• Sometimes some of our customers’ customers accidentally turn off the receiving of forms from HMRC (e.g. tax code notices). Currently we get notified of this and asked by HMRC if this is correct. We contact our customer to check whether this is correct or not and if not HMRC are informed accordingly and the service continues. With the demise of Payroll EDI this would be a problem and strict measures must be taken in order to ensure that the receiving of forms is not turned off.
• When using Avco’s bureau service Employers originally filled in a form, authorising Avco to send and retrieve PAYE data with HMRC via EDI. This form will no longer be valid and Avco will have to apply for the Employer to be moved to our Government Gateway bureau account. This will mean Avco must trigger HMRC to send the Employer a letter to allow us to operate on their behalf via the new method. This letter will contain a code that the Employer will provide us for confirming our authority.
• Customers already used to our Bureau Service may be familiar with the current acknowledgment reports, “Ack”, or response reports, “Resp” issued by email upon receipt of a file at the HMRC to indicate of any business validation errors. There are no direct parallels for this via the government gateway route. However, AVCO can provide a human friendly conversion of the validation messages returned from the Government Gateway.

Finally, as we are able to communicate with HMRC via the Government Gateway, if the Employer’s payroll system can already produce XML we can simply automate and manage the file transfer process. Employers who use other formats (for example CSV) shouldn’t have any problems as we have automated the data conversion to and from XML prior to initiating data transfers to and from HMRC. Although we believe that we have already seen all the various formats produced by different Payroll systems for Payroll EDI, we are always open to new challenges and would therefore develop new convertors should we be presented with a data format that we have not seen before.

In conclusion, we trust that this paper demonstrates our considerable domain knowledge and that we have indeed solved problems arising from the demise of Payroll EDI.