So far all the data we've been migrating has come from a MySQL database but Migrate supports a number of other data sources which we can access by using a different source migration class. In this lesson we'll look at the source migration classes that are available with the migrate module and talk about what each one could be used for. Then we'll implement a migration that imports data from a CSV file since that's another common way of receiving source data.
For your convenience, we've included a copy of the sample data in the companion files. This is a duplicate of the data from lesson Set Up Migrate Demo Site and Source Data. If you've been following the lessons in order, you should not need to download the sample data set again.
Additional resources
Migrate Source Class documentation
Drupal 8: WYSIWYG and In-Line Editing
Blog postDrupal 8 has come a long way since Drupal 7 to being easier to use out of the box. When I built sites for people using Drupal 7 the biggest complaint was the lack of a WYSIWYG editor. At some point installing WYSIWYG module and adding an editor role just became part of my usual installation of modules. This process wasn't complicated, but it was rather annoying.
First you had to install the module, then download the third party WYSIWYG files from another website. Once you had all the components you needed, you then had to setup text formats to use the editing interface, and also configure the role for your content managers to assign correct permissions. My favorite part of the process was the web page of check boxes of buttons you could add to the toolbar. The page was hideous and not very configurable if you'd added any extra add-ons to your WYSIWYG.
It is that time of the week when we publish a new Drupalize.Me podcast. You're excited. We're excited. Unfortunately we don’t have one for you today. We recorded a great podcast on getting involved with Drupal 8 with Joe Shindelar, Juampy, Alex Bronstein, and Larry Garfield. It was a fun podcast talking about all the new things that excite us about Drupal 8. Somehow the final recorded audio file turned bad about 10 minutes into it. Normally we have a fall-back plan which involves two people recording the podcast.
Very few migrations are simple and straight-forward enough that a simple field mapping gets all of the little bits you need. In this week's Importing Data With Migrate and Drupal 7 lessons, we take a look at some complicated field mappings. We are going to add some new data we need to import, with team information. The team data is related to our player data through a relationship, and a player can have multiple teams they've played for.
This lesson is a short one but it covers an important topic, multi-value fields. Almost any field in Drupal can be configured to support more than one value being entered for a single field. Our teams entityreference field is a good example of this, a player could have played for one or more team over the course of their career. This lesson will look at two different ways to map multiple values for a single field.
First we'll look at doing it in a callback method where we perform an additional query and then use the values returned by that query. And second we'll look at using the field mapping's separator method to take a column in a source row that has multiple values separated by a comma and import them as individual field values.
The infamous causality dilemma of the chicken and the egg examines which of the two came first constantly battling with the fact that you need one to produce the other. It's a vicious circle. In this lesson we're going to explore this dilemma in the context of data migrations. Imagine a scenario where you've got an article node type that has a reference field for similar articles which you need to populate with the node ID of the similar articles. During a migration when the article is being imported the article that is being referenced may or may not exist already. If it doesn't exist already how do we know what ID we need to put into the reference field?
One option would be to solve this problem using multiple passes. A first pass that goes through and creates all the articles, and a second that comes back through and updates the similar articles field. Though what happens if the similar articles field is required? You wouldn't be able to save the article without a value in that field the first time around? So you see how this quickly becomes another example of the chicken or the egg problem?
Lucky for us the migrate module has a solution to this called stub migrations. A process that allows creating a stub or a shell for the referenced but not yet created article so that we can use it's unique ID, then when that article is encountered in the migration it will update the stub rather than create a new article.
Additional resources
In this lesson we're going to take a look at creating relationships between two Drupal nodes during a migration. In our case we've got player and team nodes, and each player node has an entity reference to a team node which we need to populate during our migration. In order for this to work we need to ensure that the team node has already been created so that we know the unique node.nid to use in the entity reference field for the player.
To accomplish this we're going to write a migration for team data and ensure that it is run prior to our player migration being run. Then we're going to make use of the mapping between source and destination rows that the Migrate module is tracking for teams so that during a player migration we can lookup the corresponding team node's nid and make use of it.
Welcome Justin to Our Team!
Blog post
We are super excited to introduce you to our new Interactive Designer, Justin Harrell. Justin started with us right before DrupalCon Prague, and so came over to Europe with the rest of the team for an intense introduction to the Drupal world.
In this lesson we're going to finish mapping the fields for the player migration and learn about how to deal with source data that requires some additional massaging before being saved to the destination. We'll learn about the use of field mapping callback functions and the migration's prepareRow method as possible spots to perform additional logic during a migration. Then we'll use these techniques to combine our player first and last name fields together for the node title field, deal with our birth and death date fields by concatenating the three source columns together into a single date string, and finally add some additional information to the notes field during import that will allow us to track imported records in the future.
Note: This lesson was recorded using the 7.x-2.6 version of the date module, however the 7.x-2.7 version is now out which contains some changes to the module's integration with the migrate module. The biggest change being that the date_migrate module is no longer required and has been deprecated. You can read more about the changes here: https://drupal.org/node/2034231
Additional resources
Although not required when writing your own migration, the Migrate module provides ways for us to decorate our migrations with additional information making it easier to keep track of who is working on what, what needs to get done, and related issues. We've already seen some of this in previous lessons with with the Do Not Migrate option for fields and ability to provide field mapping descriptions. In this lesson we'll take a look at how we can make use of the additional tools to do things like; Show the name, contact info, and roles of individuals working on the migration, pose questions to other team members via the migration UI, and link individual field mappings to related tickets in our bug tracking software. Making it easier for team members who don't want to be involved with the code to help move our migration along.
Additional resources
In this lesson we'll continue to add field mappings to the basic migration class created in the previous lesson. We'll see how to add more information about the available source fields. We'll map more of the player source data fields to their equivalent destination fields and learn about some of the many ways that fields can be mapped. Finally we'll also cover mapping sub-fields which allows us to import data for things like the text format of a node's body field or the alt column in an image field. Information that's contained within a meta field.
Additional resources
Release Day: Migration Field Mapping
Blog postIn this week's lessons for the Importing Data With Migrate and Drupal 7 series we get a good look at our field mappings. This is the time that we make sure our source data is being properly handled and getting to the right destination fields in our Drupal site's database. We walk through basic field mapping, and see how that looks and works in the site UI, as well as with Drush. Then we explore different ways we can transform source data during the migration process, so that it matches what our destination field is expecting.
The entire Drupalize.Me team went to Prague, along with a number of other Lullabots, for DrupalCon Prague. In our latest podcast episode a few of us share what it was like, and the excitement and energy of new things on the horizon.
Getting up to Speed on Drupal 8 Webinar
Blog postWhen is Drupal 8 coming out? What's going to change in the new version? How do I keep track of it, and most importantly, how am I going to learn what I need to know? These are the questions we're going to answer in a Drupal Association webinar on October 22. Joe Shindelar is going to give an update on where things are with Drupal 8, covering a raft of new features and changes that you'll want to know about. He's going to look at it from different perspectives — as a site-builder, themer, and module developer.
Release Day: Writing a Custom Migration
Blog postThis week we're finally ready to dive in and write our first custom migration with the Migrate module. To get things set up, we need to get our source data, and create the content type in our Drupal site so that we have a destination. Joe walks through grabbing some baseball stats in SQL format for the source, and then building out the content types we'll need for teams and players.
Running your own migration requires a bit of setup and boilerplate code, Drupal needs to know where to find the source data, and the Migrate module needs to be provided with some basic information about our custom code. In this lesson we'll look at setting up Drupal to be able to connect to multiple databases, and then create the skeleton of a simple module which will house our custom migration code and an implementation of hook_migrate_api(). Finally we'll create a base class from which we can begin writing our own custom migrations, and talk about why this is a good way to start organizing our migration.
Before we can write our own custom migration we need to construct the site that we're going to import data into and of course we need some source data to import. In this lesson we'll obtain some source data to work with and configure our Drupal site by installing a couple contributed modules, and creating the content types and fields necessary for our information architecture. During this process we'll also be taking a look at the baseball player and team data that we'll be importing and familiarizing ourselves a bit more with the tables and columns in our source database.
Note: The Lahman database structure has changed since this video was recorded, and the latest files provided by Lahman don't match what is used in the video. When following along with these examples you'll either need to use the 2012 source data, or make a few adjustments to field/table names throughout the source code as you follow along so that they match the current structure of the Lahman data.
Additional resources
In this lesson we're going to write our first custom data migration and start importing some of the player data in to Drupal. The primary concepts covered in this lesson are the creation of a migration class following the pattern necessary for Migrate module to be able to discover our code and then dealing with defining source and destination objects so that the Migrate module will know where to read data from and where to write data to. Finally we'll add a simple single field mapping where we map the player's name to the node.title field allowing us to run the migration for the first time and import some real data.
Additional resources
The Migrate module itself contains some excellent examples of data migrations implementing the various APIs provided by the module and serves as the canonical documentation for how to write a migration. In this lesson we'll take a look at the beer and wine import examples provided with the migrate module as a way to familiarize ourselves with the concepts discussed earlier and to be able to see the code that makes up a basic migration before attempting to write our own. In practice these examples serve as a great starting point and can often times be copy/pasted and adjusted for your own needs.
Additional resources
In this lesson we'll take a look at running migrations via drush rather than via the Migrate module's UI. We'll take a look at the commands provided by the Migrate module and talk about what they do. Then we'll practice running, rolling back, and otherwise interacting with migrations via drush commands. Throughout the lesson we'll learn about some of the functionality you get from running migrations via drush that are not provided by the UI, like the ability to specify a single record to migrate with the --idlist flag. Finally we'll learn about why in most cases drush is a better tool for running large data migrations because of the limitations imposed on the UI. Pay close attention to this lesson since throughout the remainder of this series we'll be running all of our migrations via drush.