Data Conversions: How We Clean-Up Data During the Conversion Process

We’ve recently implemented new ways to do data conversions from legacy systems into OrangeLeap.

The first way, for simple data sets, is to use the import option in the UI to bring in Constituents and Gifts.

For more complex source data, previously we had populated database tables directly using scripts, which relied on a number of complex data relationships being constructed correctly, and manual implementation of business rules.

The new process starts with population of a set of intermediate conversion tables.  These tables are fairly simple and straightforward, and vary sightly based on the customer since they include separate columns for custom fields.  The data is run through a first pass of sanity checking and scrubbing at this point by an analyst or dba.  The second step is where the program is used to take the data from the intermediate tables into the program using the same application service layer that the UI uses to process the data.  This results in all the business rules being executed and the relationships and internal structures being populated correctly and consistently.

Another advantage the new conversion process provides is the ability to import many-to-one data sets such as multi-line gifts, and to associate the gift lines with pledges, soft gifts, and events.  The associations are made using references to the original legacy system’s identifiers, so that the original data links can be brought over in many cases.

We are also able to import many more complex relationships in a two step process, where all the constituents are brought over first using the ‘add’ function, then the complex relationships between the constituents are constructed afterwords using a ‘change’ function.

By Dan Meany

Dan is a Senior Developer at Orange Leap

Explore posts in the same categories: Open Source, Orange Leap Developer Network, Orange Leap News and Happenings, Programming

Tags: , , , ,

You can comment below, or link to this permanent URL from your own site.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: