When running migrations you can use the highwater_mark source plugin configuration option to influence which rows are considered for import on subsequent migration runs. This allows you to do things like only look at new rows added to a large dataset. Or to reimport records that have changed since the last time the migration was run. The term, highwater mark, comes from water line marks found on structures in areas where water level changes are common. In running migrations, you can think of a highwater mark as a line that denotes how far the migration has progressed, and saying, "from now one, we only care about data created after this line".

Another common use case for highwater marks is when you're importing a large dataset and the system runs out of resources. Usually this will look like a migration failing because it timed out, or the process ran out of memory. A highwater mark should allow you to pickup from where you left off.

In this tutorial we'll:

• Define what a highwater mark is, and how you can use them to limit the rows considered for importing each time a migration is executed.
• Demonstrate how highwater marks can be used to reimport source records that have been modified since the previous time the migration was executed.
• Introduce the track_changes option.

By the end of this tutorial you should be able to define what a highwater_mark is and how to use them to speed up the import of large datasets or force the migration to reimport records when the source data is changed.

You should also be aware of the track_changes feature which is a slower, but more dynamic, method of checking for changes in source data and reimporting records when a change is found.

If you need to write a migration that is capable of being executed multiple times and picking up changes to previously imported data, you can use the track_changes configuration option of most source plugins. This will tell the migration to keep a hash of each source row, and on subsequent runs of the same migration it will compare the previous hash to the current one. If they don't match this means the source data has changed, and Drupal will reimport that record and update the existing destination record with new data.

Using track_changes differs from calling drush migrate:import --update in that using --update will force every record to be re-imported regardless of whether the source data has changed or not.

In this tutorial we'll:

• Learn how change tracking works to detect changes in your source data
• Use the track_changes option in a migration

Note that progress of a migration can also be tracked using highwater marks if the source data has something like a last_updated timestamp column. Using highwater marks in this case is likely more efficient than using track_changes. It's a good idea to understand how both features work, and then choose the appropriate one for each migration.

Before you can migrate your Drupal 7 site to the latest version of Drupal you'll need to be able to build the features that make up the current site. Part of this is evaluating all the modules you've got installed, figuring out what you're using them for, and if there's a version that's compatible with the latest version of Drupal along with a migration path.

I usually make a spreadsheet for this. But any list of the modules you’re currently using that allows you to keep track of how you plan to update them will work. You also want to keep track of where you are in the process of figuring that all out. Because it’s likely you’ll have some modules for which the path is clear, and others where it’s pretty murky and requires more in-depth research to find a path forward. Having a list means you can break that up into tasks, and ensure you’re not missing something. It'll also help you define when your migration is done as well as any final quality assurance (QA) tasks.

In this tutorial we'll:

• Start a list of the modules that make up our current site.
• Point to some tools that can help speed up the process of evaluating a module's readiness.
• Provide a set of questions that you can ask about each module you're using as part of your planning process.

By the end of this tutorial you should have a list of all the modules you're currently using, and some tools you can use to help you figure out how to move forward with each one.

If you want to be able to say you're done with your Drupal-to-Drupal migration, first you have to be able to define "done". And part of that is doing a content analysis and inventory. Performing a content analysis and inventory will help you ensure that you don't miss any fields or important records. It also gives you an opportunity to spend some time thinking about the overall information architecture of your site. You're already going to be doing a lot of work to migrate your content, so deciding to shuffle things a bit now might not add any significant extra time. Additionally, the latest Drupal (as of Drupal 10) is a different platform than either Drupal 6 or 7, and as such, there are some new best practices and new ways of doing things that might not have been available before.

In this tutorial we'll:

• Provide a set of questions you can ask yourself about the content of your current site to kick-start your analysis.
• Give an example of how we create a content type inventory in a spreadsheet and use that to help define "done" for our projects.

By the end of this tutorial you should be able to get started analyzing the content and content types of your existing site in order to start planning for your Drupal-to-Drupal migration.

Because current versions of Drupal are so much different from older versions like Drupal 7 upgrading to the latest version requires creating a new site using the latest version of Drupal and then migrating your old site's content and configuration into it.

There is no one right way to tackle a Drupal-to-Drupal migration. Instead, it's like walking down a path and coming to a fork in the trail and then choosing a direction over and over and over. Since every site is different, every path to a finished migration will be different, too. I know nobody wants to hear it, but every migration is its own unique adventure. Every successful migration will require its own custom code, weird shell scripts, and detailed lists of the exact order of things. But that doesn't mean there's no plan to follow.

A successful migration charts a course through the maze, while leveraging existing tools and experience to help find the shortest route and the right path at each fork--with minimal backtracking due to wrong choices.

In this tutorial we'll:

• Look at what makes a Drupal-to-Drupal migration (a major version upgrade) so tricky, and how to think about performing one
• Define 3 high-level approaches to performing a Drupal-to-Drupal migration
• Get a better idea of what the migration work entails, so you go into it with a proper mental model.

By the end of this tutorial you should be able to explain the different approaches to performing a Drupal-to-Drupal migration in broad strokes, and have a better picture of the work that will be involved.

The migration_lookup process plugin allows you to populate entity reference fields when the entity that you want to reference is created by another migration, and you don't yet know what the unique ID will be in the Drupal database. Think of this as being able to answer the question: I have a migration that imports new users from a CSV file, and another that imports articles from a different CSV file. The articles' CSV file has a column that indicates which row from the users' CSV file is the author of this file. But in order to populate the author field for the Drupal node I need to know the ID of the imported user which is an auto-incremented ID that I'm not setting. So, what is the correct ID to use?

Another use case is circular dependencies, like hierarchical taxonomies with parent/child relationships. Or a content type with a field that references other nodes of the same type, like articles that reference other articles.

In this tutorial we'll:

• Learn about the migration_lookup process plugin and its configuration
• Provide examples of using the plugin to populate entity reference fields during a migration
• Look at alternatives

By the end of this tutorial you should be able to use the migration_lookup plugin to effectively manage entity relationships across multiple individual migrations.

When a migration is run the Migrate API creates a mapping table that keeps track of what source record was used to create which destination record. A record is automatically added for each successfully migrated row. This mapping table can be used later to:

• See if a row was previously imported
• Look up the ID of the entity created for the row
• Tracking if a source record has changed and needs to be re-imported.

In this tutorial we'll learn:

• What migration map tables are
• Why they exist, and what the Migrate API does with them

By the end of this tutorial you should be able to explain the use case for map tables and describe the data they contain.

If you want to define a new URL that a user can navigate to, and custom PHP code that will generate the content of the page, you need a route and a controller. Most of the time you'll want to do something more complex than hard code the content of the page. This will require using services in your controller. This can be accomplished in different ways.

In this tutorial we'll:

• Provide the definition for a new route which maps a path to the callback method of a controller class.
• Create a controller that returns a hard coded string.
• Look at examples of using both ControllerBase and dependency injection to access services from a controller, and discuss the benefits of both approaches.

By the end of this tutorial, you should be able to define a new route that maps to a controller and displays content on the page as a result of your custom logic.

The routing system can get dynamic values from the URL and pass them as arguments to the controller. This means a single route with a path of /node/{node} can be used to display any node entity. Route parameters can be validated, and upcast to complex data types via parameter conversion. If you ever want to pass arguments to the controller for a route, you'll use parameters to do so.

In this tutorial we'll:

• Define what parameters (slugs, placeholders) are and what they are used for in a route definition.
• Explain how URL parameters are passed to a controller.
• Define parameter upcasting.

By the end of this tutorial, you should be able to explain how to define a route that uses parameters to pass dynamic values to the route controller, and explain how parameter upcasting works.

Every route should define its access control parameters. When you define routes in a module, you can limit who has access to those routes via different access control options. Route-level access control applies to the path. If your route defines a path like /journey/example, the access control configuration will determine whether to show the current user the page at the path defined by the route, or to have Drupal serve an "HTTP 403 Access Denied" message instead.

In this tutorial we'll look at different ways of adding access control to a route including:

• Access based on the current user's roles and permissions
• Access based on custom logic in a callback method
• Logic in an access checker service

By the end of this tutorial, you should be able to add access control logic to your custom routes that will meet any requirement.

When defining a route and subsequently displaying the page, often we need to calculate the page title based on route parameters or other logic. In these cases, we can't just hard code the value into the _title configuration of the route. To set a dynamic title for a page, we'll use the route's _title_callback option, and point to a PHP callback that contains the logic that computes the title of the page.

In this tutorial we'll:

• Learn how to use the _title_callback route configuration option to dynamically set a page title
• Explain how arguments are provided to the title callback method
• Update the route and controller from a previous tutorial to use a dynamic title callback

By the end of this tutorial, you should be able to configure a route so that its title can be set dynamically using route parameters, instead of hard-coding the title with a static string of text.

Let's write some code that will allow us to see route parameters in action. We'll define a new route with a path like /journey/42/full but where 42 can be any node ID, and full can be any view mode. When a user accesses the path we'll pass the dynamic parameters from the URL to the controller. The controller will then load the corresponding node and render it using the provided view mode, and return that to display on the page.

By the end of this tutorial you should be able to:

• Use dynamic slugs in a route to pass parameters to the route controller.
• See how Drupal will upcast a value like the node ID, 42, to a Node object automatically.
• Explain what happens when you visit a URL that matches a route but the parameters don't pass validation.

By the end of this tutorial, you should be able to pass dynamic values from the URL to a route's controller.

Every web framework, including Drupal, has basically the same job: provide a way for developers to map URLs to the code that builds the corresponding pages. Drupal uses Symfony's HTTPKernel component. Kernel events are dispatched to coordinate the following tasks:

• Process the incoming request
• Figure out what to put on the page
• Create a response
• Deliver that response to the user's browser

Knowing a bit more about how Drupal handles the request-to-response workflow will help you better understand how to use routes and controllers to create your own custom pages or deal with authentication, access checking, and error handling in a Drupal module.

In this tutorial we'll:

• Walk through the process that Drupal uses to convert an incoming request into HTML that a browser can read
• See how the Symfony HTTPKernel helps orchestrate this process
• Learn about how the output from a custom controller gets incorporated into the final page

By the end of this tutorial, you should be able to describe the process that Drupal goes through to convert an incoming request for a URL into an HTML response displayed by the browser.

Sometimes we need the response returned from a controller to be something other than HTML content wrapped with the rest of a Drupal theme. Maybe we need to return plain text, or JSON structured data for an application to consume. Perhaps we need greater control over the HTTP headers sent in the response. This is possible by building on the fact that controllers can return generic Response objects instead of renderable arrays, allowing you to gain complete control over what is sent to the requesting agent.

In this tutorial we'll:

• Look at how to return a plain text response, and JSON data
• Show how to make your responses cacheable by adding cacheability metadata
• Learn about how to use a generic Symfony Response to gain greater control over what gets returned

By the end of this tutorial, you should be able to return responses from a controller in a Drupal module that are not HTML content wrapped in a Drupal theme.

By default, individual forms in Drupal are not output using Twig template files. It's possible to associate a form with a Twig template file by creating a new theme hook, and then referencing that theme hook from the $form array that defines the form. Doing so allows theme developers to customize the layout of the elements in the form using HTML and CSS.

This is useful when you want to change the layout of the entire form. For example, putting the elements into 2 columns. If you want to change individual elements in the form, you can often do so by overriding element specific Twig template files.

In this tutorial, we'll:

• Learn how to create a new theme hook that can be used to theme an element in a render array.
• Associate the $form we want to theme with the new theme hook we created.
• Create a Twig template file for the theme hook that will allow us to lay out the form elements using custom HTML.

By the end of this tutorial, you should be able to associate a Twig template file with any form in Drupal, so that you can customize its layout using HTML and CSS.

Tools like PHP_Codesniffer (phpcs) can be used to help ensure your code adheres to Drupal's coding standards. As a module developer, you should use phpcs and its Drupal-specific rule sets on all custom module code.

In this tutorial, we'll:

• Learn about PHP_Codesniffer (phpcs).
• Install PHP_Codesniffer and the Drupal-specific rules.
• Use phpcs to lint our custom code.

By the end of this tutorial, you should be able to use PHP_Codesniffer to help automate the process of adhering to Drupal's coding standards.

Coding standards are vital in collaborative environments like Drupal. These guidelines ensure consistency, readability, and maintainability of the codebase. Drupal has its own coding standards that outline best practices and formatting guidelines, promote code quality, simplify code reviews, and enhance the project's overall health.

In this tutorial, we'll:

• Explore the use of coding standards in Drupal.
• Highlight basic practices and sources for detailed information.

By the end of this tutorial you should be able to explain why coding standards are important and where to find more info about Drupal's coding standards.

To use a hook in a Drupal module, we need to add a class with a method for each hook we want to implement and a #[Hook] attribute that declares which specific hook we're implementing. Each hook has unique arguments and an expected return value. In this tutorial, we'll walk through the process of implementing a hook by adding end-user help text for the anytown module, which Drupal's administrative UI will display. The process we'll use here applies to any hook implementation.

In this tutorial, we'll:

• Locate the documentation for hook_help().
• Implement the hook in the anytown module.
• Verify our hook implementation.

By the end, you'll have implemented hook_help() to display help text in the Drupal UI.