TSG Blog

Frequent readers of the TSG Blog know OpenMigrate is our flexible open source migration framework; that it can be configured or extended to move content and data between different types of repositories; and that it’s been used in numerous successful migrations with Documentum, Alfresco, Qumas, Hummingbird and many others.

But perhaps you’re wondering: “There’s no way it could migrate from an old FileNet IDM system, could it?  I mean, that system was built for old optical disk storage devices!  I heard there was a C API but that people had mixed results trying to use it for high-volume use cases.  It’s practically older than Java and the web; and certainly much older than web services.  Surely you’ll tell me it can’t be done?”

Well, I’m happy to report that TSG has two successful FileNet IDM migrations under our belts, both using OpenMigrate.  For the first, we migrated 1M files and data and made minor customizations to the framework; for the second, we migrated just over 2M, and ran OpenMigrate out of the box.

Migration Features

Some features of both migrations include:

1. Selection of documents and their metadata using specific rules for each FileNet document class
2. For picklists (FileNet menus): inclusion of codes, descriptions or both
3. Translations of FileNet dates
4. Concatenation of multi-page image files into single PDF files
5. Binary concatenation of very large files (e.g., SAP Archive files)
6. Migrations executed in “platter order,” in order to minimize optical disk swapping
7. Two migration phases: a “pre-cutover” bulk migration in the weeks prior to cutover; and a final “delta” migration extracting the remaining documents and metadata
8. Robust error recovery

In both instances, we executed OpenMigrate with 6 threads during business hours and 15 threads overnight in order to minimize the impact to interactive users.

Technical Approach

After studying the underlying FileNet database structure and the system tools available for retrieving content, TSG determined the most cost-effective approach for implementing a migration from FileNet.  Rather than attempting to integrate the Java-based OpenMigrate with the dated C API, we configured OpenMigrate to query the underlying database tables directly.  And to retrieve the content, we used a few key system tools installed on the FileNet server, scripted and controlled by a single OpenMigrate component.

The details….

1. The JDBC Queue Populator builds the migration’s “To-Do List” from FileNet’s underlying DOCTABA table (joining against the menu item table if appropriate for the doc classes being migrated).  It includes built-in and business metadata in the query.
2. Each OpenMigrate Source thread uses the FileNetCsmContentLoader component to retrieve the document from FileNet.  It uses a variety of FileNet system tools to extract the image or images, determines whether to concatenate the images using iText, and translates FileNet dates to Java dates.
3. The Mapping layer performs any metadata translation in order to prepare the document for import into the target system (e.g., set the r_object_type in Documentum based on the FileNet doc class).
4. The Target threads write the content and its metadata to the target repository.

The resulting migration approach ended up being refreshingly straightforward and repeatable; as I noted above, our second migration was able to use the custom component out of the box.