Milestone reached regarding Search API for Drupal 8

For all of us who care about site search for Drupal, the maintainer Thomas Seidl has written a report about the current status of Search API for Drupal 8. The search crew’s vision is not only to port Search API to Drupal 8, but also to remove all known limitations, making site search for Drupal 8 the most powerful and flexible search integration so far. At Visitors Voice we are really excited about this since our own vision is to empower all Search API sites with actionable search analytics and tools to help more visitors find what they are looking for. This is Thomas’ report.

After eight months of intense development in a sandbox, a few days ago the Drupal 8 version of the Search API module finally reached the first large milestone: I moved it back out of the sandbox into its proper repository and created a first Alpha release right away. A good moment, I think, to go a bit into detail about what has happened so far, what changes you can expect from the Drupal 8 version and maybe whet your appetite a bit for the great features that will soon come to a CMS module near you.

What is a “Search API”?

In case you don’t know it, the Search API is a Drupal 7 module which provides a feature-rich, powerful and flexible search framework for Drupal. Together with its numerous extension modules, it covers a wide range of search functionality and can easily be extended and customized further. Its prime features include support for searches on any kind of entity (as well as extension points for adding other kinds of data sources), a large selection of supported backends which can be switched on the fly, and a powerful Views integration. It is currently the most popular search-related module on drupal.org, with over 50,000 reported installations.

Changes in Drupal 8

If you want to keep up to date about what is planned and what has happened for the module’s D8 version, just follow the meta issue and any of its sub-issues that interest you. Below, I want to highlight a few of the larger improvements to expect in the Drupal 8 version.

Apache Solr

Most of you have probably heard it already, but one of the largest improvements in the Search API in Drupal 8 is not a technical, but a “political” one: after a lot of discussions, the maintainers of the Apache Solr Search module and I eventually agreed to merge our two Apache Solr integration modules into one for Drupal 8, thus hopefully eliminating a lot of confusion and duplicated work, leading to a better solution for everyone. You can read more about it in Nick Veenhof’s announcement blog post.

Searching across multiple item types

The probably biggest technical innovation concerns searching across multiple item types (like nodes, comments, user profiles, …) – especially, creating site-wide searches. Even though this is a very common use case, it is frustratingly difficult to implement in the Drupal 7 version of the Search API. There is the Multi-Index Searches module which tries to solve the problem, but it has to fight against a lot of inherent technical restrictions and consequently only does a rather poor job, lacking support for nearly all add-on features of “normal” searches.

This problem was caused by the initial decision of limiting each search index to a single entity (or other) type – since normal searches are always executed on a single index, a search across multiple types cannot be created this way. While other, maybe better-working workarounds in Drupal 7 would be possible, this is solved completely in Drupal 8 where this restriction doesn’t exist anymore. You can now use any number of item types (or “data sources”, as we now call them) for a search index, and even change them completely at any time later on, which makes searches across entity types, or even over both Drupal-internal and external data, work exactly as smoothly as searches on a single entity type did before. This makes it possible even for a large site, with lots of different types of content and associated searches, to use a single search index for the whole site. When creating a search (view), you can then just filter by the appropriate types to get a type-specific search.

Indexing only specific content types/bundles

Still, sometimes you want to be very specific about what you index – e.g., indexing only nodes of a certain content type. Though this was already solved early on in the module’s Drupal 7 version, people soon complained that this solution was very limited: it excluded the items with wrong types from being indexed on the server, but the Search API still had to look at each item when indexing, and also afterwards every time the item was edited. (Especially very problematic on large sites with over a million nodes, where you want to only index a few hundred.) Also, the list of fields that could be indexed wasn’t affected by this setting, offering fields that weren’t present in the selected content types. While at least the former issue was finally (more or less) fixed pretty recently, the Drupal 8 version took care of this functionality right from the start, thus achieving a much cleaner integration than possible in Drupal 7. Selecting bundles to include/exclude in/from the index is done right at the beginning, when also selecting the data sources to use, and this is then also reflected when selecting the fields to index.

Usability

If you’re like me, then reading the words “feature-rich, powerful and flexible” (as above) you immediately think “Usability nightmare” – and, unfortunately, that was really one major problem that the Search API always had. It always took new users a while to figure out how things are supposed to work there, and there a countless issues in its issue queue bearing witness to that. An initiative to provide a helpful handbook also didn’t change that much – as is usually the case. No, the problem had to be tackled at its root, and in the Drupal 8 version that’s exactly what we are planning to do: providing a UI that makes intuitive sense to users, that works much more along the lines of how someone building a search is thinking – not along the lines of how it works internally. One major plan to this end is providing a “simplified UI” mode (or module), which hides all the more complex configuration behind a wizard-like frontend and aims to provide the correct defaults for those settings that inexperienced users generally don’t care about. Currently, only a few minor UI tweaks have been implemented, since finishing the underlying framework makes sense before overhauling the UI. If you still want to chime in already, or keep up to date with this work (once it starts), this issue is the right place to go. Once the framework is stable enough and the other larger changes implemented, we’ll re-open that and start to work there.

And various others …

The list is really too long to include in a single blog post, but in general we also looked at all the other complaints the Search API received in the past years that were hard to resolve in Drupal 7, or the features that were hard to add. So also expect a lot of smaller improvements, drawing from over four years of experience with the module, of the use cases it should solve, the restrictions it had and the possibilities it has for evolving. If you have a pet peeve in the Search API, just take a look – maybe it was now finally resolved (or we’re planning to)! Otherwise, try to push or open the issue for the Drupal 8 version – we are still far from finished and very open to any suggestions you have!

Try it out!

As said, we are now officially in “Alpha” stage with this module. While this means that a lot of the planned improvements still need to be implemented, it has already reached a stage where it can be tested by normal users to get a glimpse of what’s coming. So, if you are interested, please do that, give it a try and then go to the issue queue to provide feedback: Were there any bugs, did you get any error messages? Did you think of some additional feature which would be great to have – or another way to present existing functionality? In all these cases, please either find an existing issue or, if none exists, create a new one and provide input, and we will try to incorporate as much of that as possible as we move forward. However, there is yet no upgrade path from D7, and also no support for D8-to-D8 updates planned at least until Beta, so it is definitely not recommended to use the module in production yet. Also, a lot of the Views integration is currently still missing (you can show rendered entities and add a fulltext search filter, but that’s about it), so you are also very limited in the range of searches you can create.

Other ways to help

Testing out the module and providing feedback is a great way to help, as is helping to reproduce bugs and test patches – and both don’t require any programming skills. We can also always use help keeping the issue queue organized and up-to-date. Then, for developers, there are of course countless programming tasks to help with, for every level of experience (with both the Search API and Drupal 8), so please contact me if you are interested in contributing. If you want to help in any of these ways, try to attend our strategy meeting next Tuesday (12/16), where we’ll discuss how to move forward and coordinate the corresponding tasks. Or join #drupal-search-api on Freenode IRC and ask around there. Finally, for companies planning to use the Search API in Drupal 8 (especially early on), please consider donating to support this project. Nearly all of the work on it is currently unpaid, and some funding could go a long way in ensuring people are able to work on it in a reliable way. Contact me if you want to help with that.

Coda

I hope this post was interesting to you and I could make you as excited about the new Search API version as I am. I also want to seize this opportunity to thank everyone who has helped me with this project so far! An incomplete list can be found here. If you have any questions left, please just comment here, create an issue or ping me in IRC.

Thanks!

Thomas Seidl