Data Conversions: How We Clean-Up Data During the Conversion Process
We’ve recently implemented new ways to do data conversions from legacy systems into OrangeLeap.
The first way, for simple data sets, is to use the import option in the UI to bring in Constituents and Gifts.
For more complex source data, previously we had populated database tables directly using scripts, which relied on a number of complex data relationships being constructed correctly, and manual implementation of business rules.
The new process starts with population of a set of intermediate conversion tables. These tables are fairly simple and straightforward, and vary sightly based on the customer since they include separate columns for custom fields. The data is run through a first pass of sanity checking and scrubbing at this point by an analyst or dba. The second step is where the program is used to take the data from the intermediate tables into the program using the same application service layer that the UI uses to process the data. This results in all the business rules being executed and the relationships and internal structures being populated correctly and consistently.
Another advantage the new conversion process provides is the ability to import many-to-one data sets such as multi-line gifts, and to associate the gift lines with pledges, soft gifts, and events. The associations are made using references to the original legacy system’s identifiers, so that the original data links can be brought over in many cases.
We are also able to import many more complex relationships in a two step process, where all the constituents are brought over first using the ‘add’ function, then the complex relationships between the constituents are constructed afterwords using a ‘change’ function.
By Dan Meany
Dan is a Senior Developer at Orange Leap