The Search API module supports a handful of data alterations and processors; additional operations that can be performed on a document before it's indexed or during the display of search results. While Solr actually handles the majority of these for us already, this tutorial will look at the available options, talk about what each one does, and explain which ones are still relevant when using Solr as a backend.
Looking at data alterations in the Search API module also raises an important point about security. By default, Search API doesn't care about your content's access control settings. In order to prevent people from seeing results for their searches that contain data they shouldn't have access to we need to make sure we account for that in our configuration.
Here's a good list of the currently available data alterations and processors, though it's worth noting that not all of them are available for all search backends. Also, as we'll see, not all of them are recommended when using Solr even if they are available. Solr's tokenizer for example is much more full featured than the Search API tokenizer, so when using Solr as a backend it's best to keep the Search API tokenizer turned off and let Solr do its thing.
By the end of this lesson you should be able to use data alterations and processors to filter out specific content types from your Solr index and to highlight keywords found when displaying search results. You'll also be able to explain why some alterations and processors are better left off so that Solr can handle those tasks directly.
Additional resources
Being able to display search results using the Views module provides a huge amount of flexibility with respect to what is listed, what it looks like, and more. In this tutorial we'll look at using the Search API Views module, included in the Search API project, to create a view that allows users to search our Solr index and display the results as a table, or really, in any other way that Views can display content. We'll also cover some special considerations regarding access control and entity relationships that we need to keep in mind when using Views to display search results.
The biggest difference between creating a view that lists a bunch of nodes, and one that displays search results is that you need to use your Search API index as the base table from which you're building your view. Then, by default, the view only has access to the fields that are in the Solr index. This allows you to build the entire view without having to query the database. Or you can use the Views module's ability to define relationships to other buckets of content to query the database and pull in additional information. There's a huge amount of flexibility.
When building views from the Solr index you can optionally expose one or more filters. Essentially creating a form that allows someone to construct a search query. This can be as simple as exposing a keyword text field, or as complex as you would like to get. We'll look at using exposed filters to create a form that users can perform a search with and create a more complete search experience. We'll also look at how you can move those exposed filters into a block that can be displayed on the home page of our site, allowing us to replace the functionality provided to the Drupal core Search module with Views and Solr.
By the end of this tutorial you should be able to create a view that displays search results using the Search API Views module.
Additional resources
Using facets allows users of your search application to further narrow the results returned from a keyword search by selecting one or more attributes of the returned content and saying either show me only these, or show me everything but these. In this tutorial we'll take a look at some examples of faceted searching in practice, and then we'll use the Facet API module to expose facets for our genus and species fields.
One of the most common uses of facets is on e-commerce sites like Zappos.com that have huge collections of products that users can browse through, and narrow down, to focus in on exactly the pair of shoes they are after. In this example facets allow you do to things like narrow the results returned from your initial keyword search to just shoes for men, which are brown, size 10.5, and on sale. You can can also see faceting in action any time you perform a search on our site.
We'll use facets to allow users of the fish finder application to limit the results returned to just those of a specific species or genus. In doing so we'll also look at the options available for determining how facets should be displayed, whether or not we should show a facet that has zero documents in our result set, and how to combine multiple facets together into a single query using either AND or OR logic.
By the end of this tutorial you should be able to use the Facet API module in conjunction with Search API in order to provide facets that your users can use to further narrow and refine their search results.
Additional resources
Depending on the data that is being searched, some shorter general words, like "a", "the", or "is" can adversely effect search result relevancy. Consider the word "the", which in a standard description of a fish in our database could easily appear hundreds of times or more. When a search is performed, part of the algorithm that calculates the relevancy of any document in the index is to count the number of times a word appears in the text being searched. The more often it appears, the more relevant the document. Words like "the" however often have little to no real bearing on a document's actual relevancy. These words should instead be excluded from the ranking algorithm.
Stop words can also serve another purpose. You can filter out words that are so common in a particular set of data that the system can't handle them in a useful way. For example, consider the word "fish" in our dataset. It's probably very common. With only 500 fish being indexed it's not really going to make much difference, but what if we were indexing five million fish, and each one had the word "fish" in the description even just five times? That's 25 million occurrences of the word "fish". Eventually we might start to hit the upper limit of what Solr can handle. The word "fish" in this case is probably also not very useful in a search query. You're browsing a fish database. Are you really likely to search for the query fish and expect any meaningful results? Likely it would instead return every result. It would be like going to Drupal.org and searching for the word "drupal" and expecting to get something useful. Not going to happen.
Solr has the ability to read in a list of stop words, or words that should be ignored during indexing, so that those words do not clutter your index and are removed from influencing result relevancy. In this tutorial we'll take a look at configuring stop words for Solr.
First, we'll use the Solr web UI to see the most common terms in our index for the body field. Then, based on that list, and the list of common stop words provided by the Solr team, we'll configure our stopwords.txt file. Finally, we'll re-index all the content of our site so that it makes use of the new stop words configuration and re-examine the most common terms noting that our stop words no longer appear in the list.
By the end of this tutorial you should be able to use the Solr web UI to get a list of the most common terms in your index, and know how to add terms to Solr's stopwords.txt file to prevent them from showing up in your index.
Additional resources
Solr provides the option to configure synonyms for use during both indexing and querying of textual data. A synonym is a word or phrase that means exactly or nearly the same thing as another word or phrase in the same language. For example, shut is a synonym of close. Synonyms, if not accounted for, can cause a dilution of search result relevancy when searching for a keywords that have lots of variations in your index.
Consider for example the words, "ipod", "i-pod", and "i pod". It's pretty easy to imagine a scenario in which the content of our site could contain all three variations of the word. When someone searches though they are likely just going to search for one, but expect results for all three. In order to not break those expectations we need to make sure we account for this scenario. Another example from the the Drupal world would be the terms "CMI" and "configuration management". Chances are if you search for one you would be happy to see results for the other.
In this tutorial we'll look at using the synonyms.txt file that is part of our Solr configuration in order to account for synonyms in our data. Of course the exact words you use will depend on the content of your site, but we can at least cover how they work and how to configure them.
By the end of this tutorial you should be able to configure Solr to be aware of synonyms in your data in order to improve the quality of your search results.
Additional resources
One of the benefits of building our own search application is that we have ultimate control over the ranking of items. Combined with our superior knowledge of our own content we can use this to ensure that when someone searches for a specific keyword we bubble our best content for that term to the top of the list, regardless of whatever Solr might rank it based on its internal algorithms. This is commonly referred to as promoted, or sponsored, results; the artificial boosting of a particular document to the top of the result list for a specific query.
A similar, but not exactly the same, example would be sponsored results on Google searches, where you can pay to have your page listed at the top of the results for a specific keyword or set of keywords. We are going to be doing all of this except for the part where we let people pay to promote results, though you could certainly build that part on your own if you need that.
Solr uses a configuration file named elevate.xml, in conjunction with a processor, to elevate results at the time a query is performed. We can promote specific documents in our Solr index by figuring out the unique Solr ID for a document and then adding it to the elevate.xml file along with some information about a query, or queries, this document should be promoted for.
In this tutorial we'll learn how to find a Solr document's unique ID, and then configure Solr to use an elevate.xml file that will promote the "How to Use the Fish Finder" page to the top of the results when someone searches for the term "fish". This configuration is all within the Solr application itself and doesn't really rely on Drupal in anyway. As such, the material in this tutorial should be applicable to your Solr search applications even if you're not building them with Drupal.
By the end of this lesson you should be able to configure promoted documents in your own Solr-based search application.
Additional resources
With our interface taken care of, now we can configure multilingual support for our content. That is, we need to be able to identify which language a given piece of content is written in and then create translations. So, we already have articles, and we need to add a knowledge base to the site as well. For both of these types of content, we’ll need to identify the language they are written in, as well as provide related, linked translated versions. In this lesson we'll enable the core Book module for our knowledge base, enable multilingual content, and start translating our knowledge base.
Additional resources
Not everyone wants or needs to install yet another module, and due to the caveats we mentioned about Localization client, it is still a good idea to be familiar with all of your options. In this lesson, we'll review the Locale module translation process, then enable Localization Client and start translating some text!
Additional resources
Localization Client project
Using Drupal, 2nd edition
Using Drupal source code
So far we’ve got a nice start with getting our site translated, but everything is not quite smooth yet. We have translations for some of our content and menu items, but we're seeing all the content at the same time. You will also see some stray interface text still in English. To take our multilingual site further and really make it shine, in this lesson we're going to turn to a package of modules called Internationalization (i18n). There is a central Internationalization module, which comes packaged with a handful of other modules designed to work together to extend core’s multilingual features.
Additional resources
Internationalization (i18n) project
Using Drupal, 2nd edition
Using Drupal source code
Translated content on a Drupal is all shown at the same time by default. The Internationalization module will let you display only content that is relevant to the language currently in use, which is referred to as "content selection." In this lesson we're going to configure our site's content selection and the test it out to make sure it working as expected.
Additional resources
Before we can translate the various site-wide variables, like the site name or slogan, we need to let Drupal know which ones we want to make translatable. To do this, we’ll need to use the Variable translation module from the Internationalization package. In this lesson we'll enable a multilingual variable so that we can see how to translate the site name.
Additional resources
We’ve translated the site name, but we still have other stubborn text on the site that we couldn't select on the Variable translation page. Our content types for the site are still using English for the content type name on the “Create content” page, and for field names when making new content. In this lesson, to fix this up, we'll walk through translating content type names, the Title label on the content type, and also see how to tackle the field labels as well.
Additional resources
When using Drupal’s taxonomy system, we need to find a way to sync the terms that we create to keep the taxonomy selection limited to just the terms for a given language. We don’t want all of the different languages showing up at the same time when someone is looking at a vocabulary or individual terms on the site. We're going to work with taxonomy to create our site's forum. The Forum module’s containers and forums structure is built on Drupal’s core taxonomy, so it is also creating a new vocabulary on our site. The forums on our client’s site need to display the threads that follow the same content selection rule as the rest of the content on the site. That is, we’ll only show the forum posts for the selected language. The site will have preset containers and forums, and then users may post to them using whichever language they choose.
Additional resources
There are a lot of tools available for working with multilingual sites, and we've implemented the main pieces we needed to build the site that Blue Peak Fanatics needs. In this lesson we'll take a look at some more modules we might consider down the road.
- Language icons
- Language switcher dropdown
- Transliteration
- Translation overview
- Translation table
Additional resources
We built a nice, simple, easy-to-use site for our clients that gave them the tools they needed for discussions and a knowledge base. We set up a forum that displays only posts that are in the user’s language and a knowledge base book where all of the site members can create translations for the pages. The major need for this community was being able to use multiple languages and easily extend those languages in the future. Using Drupal’s core internationalization features with a handful of contributed modules, we have given them a very flexible multilingual solution. In this final lesson we'll tour the Blue Peak Fanatics site, discuss our implementation points, and review the modules and resources we used to build this site.
Additional resources
Using Drupal, 2nd edition
Using Drupal source code
Up next: Using Drupal Chapter 9: Online Store
The last little thing we need to do to clean up our site is to get our Home tab in the Main menu translated as well. Everything else in the menu is now translated, but the Home menu item isn't from one of the other features we've created. It's just a regular menu item. In this lesson we'll enable multilingual menus and get that home page tab translated.
Additional resources
In this lesson we will cover all the settings that are new available in the Features module from the Bean module. We will update our current feature with all of the settings we have configured from creating different Bean types. We will discuss what has been added and be reminded that content is not stored in the feature. It is good practice to setup all of your configurations and get those configurations into code, then move those settings to your final environment before you start creating content.
In this short series you will learn how to find and evaluate Drupal modules for your project. This first tutorial will show you where to find modules on Drupal.org, and various ways you can use Drupal.org's search tools.
Additional resources
Drupal.org Modules page
Drupal module categories
Similar modules group
Drupal case studies
Planet Drupal
An open source project’s strength comes from the power of its base of contributors, and a Drupal project is no different. Although every line of code added or changed in Drupal core goes through rigorous peer review, contributed modules are more of a Wild West where anyone who jumps through a few basic hoops can add modules for everyone to download. Whether or not a module is well maintained, its overall code quality, and how well used it is in the overall community are all important factors for you to consider when selecting modules. This tutorial will talk about determining these factors by closely inspecting the tools Drupal.org provides, starting with the central feature of all Drupal modules: the project page.