Welcome

In this lesson we're going to finish mapping the fields for the player migration and learn about how to deal with source data that requires some additional massaging before being saved to the destination. We'll learn about the use of field mapping callback functions and the migration's prepareRow method as possible spots to perform additional logic during a migration. Then we'll use these techniques to combine our player first and last name fields together for the node title field, deal with our birth and death date fields by concatenating the three source columns together into a single date string, and finally add some additional information to the notes field during import that will allow us to track imported records in the future.

Note: This lesson was recorded using the 7.x-2.6 version of the date module, however the 7.x-2.7 version is now out which contains some changes to the module's integration with the migrate module. The biggest change being that the date_migrate module is no longer required and has been deprecated. You can read more about the changes here: https://drupal.org/node/2034231

Additional resources

• Migration class documentation
• Migrate Extras project

Although not required when writing your own migration, the Migrate module provides ways for us to decorate our migrations with additional information making it easier to keep track of who is working on what, what needs to get done, and related issues. We've already seen some of this in previous lessons with with the Do Not Migrate option for fields and ability to provide field mapping descriptions. In this lesson we'll take a look at how we can make use of the additional tools to do things like; Show the name, contact info, and roles of individuals working on the migration, pose questions to other team members via the migration UI, and link individual field mappings to related tickets in our bug tracking software. Making it easier for team members who don't want to be involved with the code to help move our migration along.

Additional resources

Field Mappings documentation

In this lesson we'll continue to add field mappings to the basic migration class created in the previous lesson. We'll see how to add more information about the available source fields. We'll map more of the player source data fields to their equivalent destination fields and learn about some of the many ways that fields can be mapped. Finally we'll also cover mapping sub-fields which allows us to import data for things like the text format of a node's body field or the alt column in an image field. Information that's contained within a meta field.

Additional resources

Migration class documentation

Running your own migration requires a bit of setup and boilerplate code, Drupal needs to know where to find the source data, and the Migrate module needs to be provided with some basic information about our custom code. In this lesson we'll look at setting up Drupal to be able to connect to multiple databases, and then create the skeleton of a simple module which will house our custom migration code and an implementation of hook_migrate_api(). Finally we'll create a base class from which we can begin writing our own custom migrations, and talk about why this is a good way to start organizing our migration.

Before we can write our own custom migration we need to construct the site that we're going to import data into and of course we need some source data to import. In this lesson we'll obtain some source data to work with and configure our Drupal site by installing a couple contributed modules, and creating the content types and fields necessary for our information architecture. During this process we'll also be taking a look at the baseball player and team data that we'll be importing and familiarizing ourselves a bit more with the tables and columns in our source database.

Note: The Lahman database structure has changed since this video was recorded, and the latest files provided by Lahman don't match what is used in the video. When following along with these examples you'll either need to use the 2012 source data, or make a few adjustments to field/table names throughout the source code as you follow along so that they match the current structure of the Lahman data.

Additional resources

• Baseball statistics for our source data
• Date module
• Entity Reference module
• Chaos Tools module
• Entity API module

In this lesson we're going to write our first custom data migration and start importing some of the player data in to Drupal. The primary concepts covered in this lesson are the creation of a migration class following the pattern necessary for Migrate module to be able to discover our code and then dealing with defining source and destination objects so that the Migrate module will know where to read data from and where to write data to. Finally we'll add a simple single field mapping where we map the player's name to the node.title field allowing us to run the migration for the first time and import some real data.

Additional resources

• Migration class documentation
• Schema API documentation
• Schema API data types

The Migrate module itself contains some excellent examples of data migrations implementing the various APIs provided by the module and serves as the canonical documentation for how to write a migration. In this lesson we'll take a look at the beer and wine import examples provided with the migrate module as a way to familiarize ourselves with the concepts discussed earlier and to be able to see the code that makes up a basic migration before attempting to write our own. In practice these examples serve as a great starting point and can often times be copy/pasted and adjusted for your own needs.

Additional resources

Migrate module

In this lesson we'll take a look at running migrations via drush rather than via the Migrate module's UI. We'll take a look at the commands provided by the Migrate module and talk about what they do. Then we'll practice running, rolling back, and otherwise interacting with migrations via drush commands. Throughout the lesson we'll learn about some of the functionality you get from running migrations via drush that are not provided by the UI, like the ability to specify a single record to migrate with the --idlist flag. Finally we'll learn about why in most cases drush is a better tool for running large data migrations because of the limitations imposed on the UI. Pay close attention to this lesson since throughout the remainder of this series we'll be running all of our migrations via drush.

Additional resources

Drush project
Introduction to Drush series

In this lesson we'll take a more in-depth look at the migrate module's UI with a focus on being able to identify and execute custom migrations. For now we'll work with the provided example migrations just so that we have something to work with. Throughout the lesson we'll learn how to run a migration to import it's data into Drupal, rollback a migration that was previously run in order to set a clean slate, and other ways we can interact with a migration via the UI. Then we'll discuss some of the challenges inherit in running migrations via the UI and Drupal's Batch API and how to identify them.

Additional resources

• Why you should run migrations in Drush rather than the UI
• Running imports and rollbacks from the UI via drush

This lesson includes a short presentation that explains the basics terminology and architecture of the migrate module and the components that make up a custom data migration. We'll talk about the Extract / Transform / Load process and how it relates to data migrations, the types of data sources that the migration module can read from, and a little bit about how the code in both the migrate module and our own custom migrations will be organized.

Additional resources

Migrate module architecture documentation

In this lesson we'll cover downloading and installing the Migrate module (version 7.x-2.6) and ensuring that our local environment is ready to be able to run migrations via both the UI and drush. Once that's setup we'll take a high level look at the migrate module's UI and drush commands to familiarize ourselves with the tools that we'll be using throughout the rest of the series. This will also help formalize some of concepts introduced in the previous lesson.

Additional resources

Migrate module project page
migrate-7.x-2.6-rc1 download
Migrate module documentation
Introduction to Drush series
Drupalize.Me Migrate module series code on GitHub

This series is focused on using the Migrate module to import data that exists in various different sources into a Drupal 7 website. The Migrate module provides an extremely flexible and robust framework for accessing data from various sources and importing or migrating that data to Drupal. With built in support for creating core Drupal data types likes nodes, users, and taxonomy terms, the Migrate module is one of the best solutions available for importing content into Drupal.

This series kicks off with Joe Shindelar explaining the basic components that make up a data migration, and the terminology and code that is specific to the Migrate module. Then continues with a series of lessons that take you from installing the Migrate module to writing and running your own custom data migration.

Throughout the series Joe teaches us how to run a data migration using both the Migrate module's UI and drush, and some of the plusses and minuses of both methods. Joe also talks about the various different sources, or types of data, from which the Migrate module can read data and how to map the unique fields in a row of source data to their corresponding Drupal content types and fields.

By migrating from a single source into two different Drupal content types we'll also have the opportunity to learn about creating relationships during a migration and mapping the resulting information to an entity reference field. During the course of writing a custom migration Joe will show us how and where we can add code to perform additional runtime data munging during our import process. We'll learn about importing data into multi-value fields, and even providing defaults for fields that don't have information. Then we'll look at some of the tools the migrate module provides for collaborating with team members in order to create a successful migration path.

Finally we'll wrap up the series by looking at a couple of different techniques for debugging our migrations and dealing with pesky source material that just doesn't want to be imported.

Because this series is focused primarily on writing custom data migrations, and since the Migrate module itself requires at least some amount of code to be written to perform a migration, it is suggested that students be familiar with PHP and basic Object Oriented Programming techniques. Although not required to run a migration, Joe uses the drush command line tool extensively in this series. If you need a refresher on using drush take a moment to watch our drush series.

Additional resources

Migrate module — Drupal.org