Flexible elasticseach builder to run complex queries with an easier way

07 January 2016 on indexing. 14 minutes

Build Status Latest Stable Version Total Downloads License

Laravel, Lumen and Native php elasticseach query builder to build complex queries using an elegant syntax

  • Keeps you away from wasting your time by replacing array queries with a simple and elegant syntax you will love.
  • Feeling free to create, drop, mapping and reindexing throw easy artisan console commands.
  • Lumen framework support.
  • Native php and composer based applications support.
  • Can be used as a laravel scout driver.
  • Dealing with multiple elasticsearch connections at the same time.
  • Awesome pagination based on LengthAwarePagination.
  • Caching queries using a caching layer over query builder built on laravel cache.

Requirements

  • php >= 5.6.6

    See Travis CI Builds.

  • laravel/laravel >= 5.* or laravel/lumen >= 5.* or composer application

Installation

Laravel Installation

1) Install package using composer.
$ composer require basemkhirat/elasticsearch
2) Add package service provider.
Basemkhirat\Elasticsearch\ElasticsearchServiceProvider::class
3) Add package alias.
'ES' => Basemkhirat\Elasticsearch\Facades\ES::class
4) Publishing.
$ php artisan vendor:publish --provider="Basemkhirat\Elasticsearch\ElasticsearchServiceProvider"

Lumen Installation

1) Install package using composer.
$ composer require basemkhirat/elasticsearch
2) Add package service provider in bootstrap/app.php.
$app->register(Basemkhirat\Elasticsearch\ElasticsearchServiceProvider::class);
3) Copy package config directory vendor/basemkhirat/elasticsearch/src/config to root folder alongside with app directory.
4) Making Lumen work with facades by uncommenting this line in bootstrap/app.php.
$app->withFacades();

If you don’t want to enable working with Lumen facades you can access the query builder using app("es").

app("es")->index("my_index")->type("my_type")->get();
# is similar to 
ES::index("my_index")->type("my_type")->get();

Composer Installation

You can install package with any composer-based applications

1) Install package using composer.
$ composer require basemkhirat/elasticsearch
2) Creating a connection.
require "vendor/autoload.php";

use Basemkhirat\Elasticsearch\Connection;

$connection = Connection::create([
    'servers' => [
        [
            "host" => '127.0.0.1',
            "port" => 9200,
            'user' => '',
            'pass' => '',
            'scheme' => 'http',
        ]
    ],
    'index' => 'my_index',
]);


# access the query builder using created connection

$documents = $connection->search("hello")->get();

Configuration (Laravel & Lumen)

After publishing, two configuration files will be created.

  • config/es.php where you can add more than one elasticsearch server.
# Here you can define the default connection name.

'default' => env('ELASTIC_CONNECTION', 'default'),

# Here you can define your connections.

'connections' => [
	'default' => [
	    'servers' => [
	        [
	            "host" => env("ELASTIC_HOST", "127.0.0.1"),
	            "port" => env("ELASTIC_PORT", 9200),
	            'user' => env('ELASTIC_USER', ''),
	            'pass' => env('ELASTIC_PASS', ''),
	            'scheme' => env('ELASTIC_SCHEME', 'http'),
	        ]
	    ],
	    'index' => env('ELASTIC_INDEX', 'my_index')
	]
],
 
# Here you can define your indices.
 
'indices' => [
	'my_index_1' => [
	    "aliases" => [
	        "my_index"
	    ],
	    'settings' => [
	        "number_of_shards" => 1,
	        "number_of_replicas" => 0,
	    ],
	    'mappings' => [
	        'posts' => [
	            'title' => [
	                'type' => 'string'
	            ]
	        ]
	    ]
	]
]

  • config/scout.php where you can use package as a laravel scout driver.

Working with console environment (Laravel & Lumen)

With some artisan commands you can do some tasks such as creating or updating settings, mappings and aliases.

Note that all commands are running with --connection=default option, you can change it throw the command.

These are all available commands:

List All indices on server

$ php artisan es:indices:list

+----------------------+--------+--------+----------+------------------------+-----+-----+------------+--------------+------------+----------------+
| configured (es.php)  | health | status | index    | uuid                   | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
+----------------------+--------+--------+----------+------------------------+-----+-----+------------+--------------+------------+----------------+
| yes                  | green  | open   | my_index | 5URW60KJQNionAJgL6Q2TQ | 1   | 0   | 0          | 0            | 260b       | 260b           |
+----------------------+--------+--------+----------+------------------------+-----+-----+------------+--------------+------------+----------------+

Create indices defined in es.php config file

Note that creating operation skips the index if exists.

# Create all indices in config file.

$ php artisan es:indices:create

# Create only 'my_index' index in config file

$ php artisan es:indices:create my_index 

Update indices defined in es.php config file

Note that updating operation updates indices setting, aliases and mapping and doesn’t delete the indexed data.

# Update all indices in config file.

$ php artisan es:indices:update

# Update only 'my_index' index in config file

$ php artisan es:indices:update my_index 

Drop index

Be careful when using this command, you will lose your index data!

Running drop command with --force option will skip all confirmation messages.

# Drop all indices in config file.

$ php artisan es:indices:drop

# Drop specific index on sever. Not matter for index to be exist in config file or not.

$ php artisan es:indices:drop my_index 

Reindexing data (with zero downtime)

First, why reindexing?

Changing index mapping doesn’t reflect without data reindexing, otherwise your search results will not work on the right way.

To avoid down time, your application should work with index alias not index name.

The index alias is a constant name that application should work with to avoid change index names.

Assume that we want to change mapping for my_index, this is how to do that:

1) Add alias as example my_index_alias to my_index configuration and make sure that application is working with.

"aliases" => [
    "my_index_alias"
]       

2) Update index with command:

$ php artisan es:indices:update my_index

3) Create a new index as example my_new_index with your new mapping in configuration file.

$ php artisan es:indices:create my_new_index

4) Reindex data from my_index into my_new_index with command:

$ php artisan es:indices:reindex my_index my_new_index

# Control bulk size. Adjust it with your server.

$ php artisan es:indices:reindex my_index my_new_index --bulk-size=2000

# Control query scroll value.

$ php artisan es:indices:reindex my_index my_new_index --bulk-size=2000 --scroll=2m

# Skip reindexing errors such as mapper parsing exceptions.

$ php artisan es:indices:reindex my_index my_new_index --bulk-size=2000 --skip-errors

# Hide all reindexing errors and show the progres bar only.

$ php artisan es:indices:reindex my_index my_new_index --bulk-size=2000 --hide-errors

5) Remove my_index_alias alias from my_index and add it to my_new_index in configuration file and update with command:

$ php artisan es:indices:update

Usage as a Laravel Scout driver

First, follow Laravel Scout installation.

All you have to do is updating these lines in config/scout.php configuration file.

# change the default driver to 'es'
	
'driver' => env('SCOUT_DRIVER', 'es'),
	
# link `es` driver with default elasticsearch connection in config/es.php
	
'es' => [
    'connection' => env('ELASTIC_CONNECTION', 'default'),
],

Have a look at laravel Scout documentation.

Usage as a query builder

Creating a new index

ES::create("my_index");
    
# or 
    
ES::index("my_index")->create();
Creating index with custom options (optional)
ES::index("my_index")->create(function($index){
        
    $index->shards(5)->replicas(1)->mapping([
        'my_type' => [
            'properties' => [
                'first_name' => [
                    'type' => 'string',
                ],
                'age' => [
                    'type' => 'integer'
                ]
            ]
        ]
    ])
    
});
    
# or
    
ES::create("my_index", function($index){
  
      $index->shards(5)->replicas(1)->mapping([
          'my_type' => [
              'properties' => [
                  'first_name' => [
                      'type' => 'string',
                  ],
                  'age' => [
                      'type' => 'integer'
                  ]
              ]
          ]
      ])
  
});

Dropping index

ES::drop("my_index");
    
# or
    
ES::index("my_index")->drop();

Running queries

$documents = ES::connection("default")
                ->index("my_index")
                ->type("my_type")
                ->get();    # return a collection of results

You can rewrite the above query to

$documents = ES::type("my_type")->get();    # return a collection of results

The query builder will use the default connection, index name in configuration file es.php.

Connection and index names in query overrides connection and index names in configuration file es.php.

Getting document by id
ES::type("my_type")->id(3)->first();
    
# or
    
ES::type("my_type")->_id(3)->first();
Sorting
ES::type("my_type")->orderBy("created_at", "desc")->get();
    
# Sorting with text search score
    
ES::type("my_type")->orderBy("_score")->get();
Limit and offset
ES::type("my_type")->take(10)->skip(5)->get();
Select only specific fields
ES::type("my_type")->select("title", "content")->take(10)->skip(5)->get();
Where clause
ES::type("my_type")->where("status", "published")->get();

# or

ES::type("my_type")->where("status", "=", "published")->get();
Where greater than
ES::type("my_type")->where("views", ">", 150)->get();
Where greater than or equal
ES::type("my_type")->where("views", ">=", 150)->get();
Where less than
ES::type("my_type")->where("views", "<", 150)->get();
Where greater than or equal
ES::type("my_type")->where("views", "<=", 150)->get();
Where like
ES::type("my_type")->where("title", "like", "foo")->get();
Where field exists
ES::type("my_type")->where("hobbies", "exists", true)->get(); 

# or 

ES::type("my_type")->whereExists("hobbies", true)->get();
Where in clause
ES::type("my_type")->whereIn("id", [100, 150])->get();
Where between clause
ES::type("my_type")->whereBetween("id", 100, 150)->get();

# or 

ES::type("my_type")->whereBetween("id", [100, 150])->get();
Where not clause
ES::type("my_type")->whereNot("status", "published")->get(); 

# or

ES::type("my_type")->whereNot("status", "=", "published")->get();
Where not greater than
ES::type("my_type")->whereNot("views", ">", 150)->get();
Where not greater than or equal
ES::type("my_type")->whereNot("views", ">=", 150)->get();
Where not less than
ES::type("my_type")->whereNot("views", "<", 150)->get();
Where not less than or equal
ES::type("my_type")->whereNot("views", "<=", 150)->get();
Where not like
ES::type("my_type")->whereNot("title", "like", "foo")->get();
Where not field exists
ES::type("my_type")->whereNot("hobbies", "exists", true)->get(); 

# or

ES::type("my_type")->whereExists("hobbies", true)->get();
Where not in clause
ES::type("my_type")->whereNotIn("id", [100, 150])->get();
Where not between clause
ES::type("my_type")->whereNotBetween("id", 100, 150)->get();

# or

ES::type("my_type")->whereNotBetween("id", [100, 150])->get();
Search by a distance from a geo point
ES::type("my_type")->distance("location", ["lat" => -33.8688197, "lon" => 151.20929550000005], "10km")->get();

# or

ES::type("my_type")->distance("location", "-33.8688197,151.20929550000005", "10km")->get();

# or

ES::type("my_type")->distance("location", [151.20929550000005, -33.8688197], "10km")->get();  
Search the entire document
ES::type("my_type")->search("bar")->get();
    
# search with Boost = 2
    
ES::type("my_type")->search("bar", 2)->get();
Return only first record
ES::type("my_type")->search("bar")->first();
Return only count
ES::type("my_type")->search("bar")->count();
Scan-and-Scroll queries

These queries are suitable for large amount of data. A scrolled search allows you to do an initial search and to keep pulling batches of results from Elasticsearch until there are no more results left. It’s a bit like a cursor in a traditional database

$documents = ES::type("my_type")->search("foo")
                 ->scroll("2m")
                 ->take(1000)
                 ->get();

Response will contain a hashed code scroll_id will be used to get the next result by running

$documents = ES::type("my_type")->search("foo")
                 ->scroll("2m")
                 ->scrollID("DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAFMFlJQOEtTdnJIUklhcU1FX2VqS0EwZncAAAAAAAABSxZSUDhLU3ZySFJJYXFNRV9laktBMGZ3AAAAAAAAAU4WUlA4S1N2ckhSSWFxTUVfZWpLQTBmdwAAAAAAAAFPFlJQOEtTdnJIUklhcU1FX2VqS0EwZncAAAAAAAABTRZSUDhLU3ZySFJJYXFNRV9laktBMGZ3")
                 ->get();

And so on …

Note that you don’t need to write the query parameters in every scroll. All you need the scroll_id and query scroll time.

To clear scroll_id

ES::type("my_type")->scrollID("DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAFMFlJQOEtTdnJIUklhcU1FX2VqS0EwZncAAAAAAAABSxZSUDhLU3ZySFJJYXFNRV9laktBMGZ3AAAAAAAAAU4WUlA4S1N2ckhSSWFxTUVfZWpLQTBmdwAAAAAAAAFPFlJQOEtTdnJIUklhcU1FX2VqS0EwZncAAAAAAAABTRZSUDhLU3ZySFJJYXFNRV9laktBMGZ3")
        ->clear();
Paginate results with per_page = 5
$documents = ES::type("my_type")->search("bar")->paginate(5);
    
# Getting pagination links
    
$documents->links();

# Bootstrap 4 pagination

$documents->links("bootstrap-4");

# Simple bootstrap 4 pagination

$documents->links("simple-bootstrap-4");

# Simple pagination

$documents->links("simple-default");

These are all pagination methods you may use:

$documents->count()
$documents->currentPage()
$documents->firstItem()
$documents->hasMorePages()
$documents->lastItem()
$documents->lastPage()
$documents->nextPageUrl()
$documents->perPage()
$documents->previousPageUrl()
$documents->total()
$documents->url($page)
Getting the query array without execution
ES::type("my_type")->search("foo")->where("views", ">", 150)->query();
Getting the original elasticsearch response
ES::type("my_type")->search("foo")->where("views", ">", 150)->response();
Ignoring bad HTTP response
ES::type("my_type")->ignore(404, 500)->id(5)->first();
Query Caching (Laravel & Lumen)

Package comes with a built-in caching layer based on laravel cache.

ES::type("my_type")->search("foo")->remember(10)->get();
	
# Specify a custom cache key

ES::type("my_type")->search("foo")->remember(10, "last_documents")->get();
	
# Caching using other available driver
	
ES::type("my_type")->search("foo")->cacheDriver("redis")->remember(10, "last_documents")->get();
	
# Caching with cache key prefix
	
ES::type("my_type")->search("foo")->cacheDriver("redis")->cachePrefix("docs")->remember(10, "last_documents")->get();
Executing elasticsearch raw queries
ES::raw()->search([
    "index" => "my_index",
    "type"  => "my_type",
    "body" => [
        "query" => [
            "bool" => [
                "must" => [
                    [ "match" => [ "address" => "mill" ] ],
                    [ "match" => [ "address" => "lane" ] ]
                ]
            ]
        ]
    ]
]);
Insert a new document
ES::type("my_type")->id(3)->insert([
    "title" => "Test document",
    "content" => "Sample content"
]);
     
# A new document will be inserted with _id = 3.
  
# [id is optional] if not specified, a unique hash key will be generated.
Bulk insert a multiple of documents at once.
# Main query

ES::index("my_index")->type("my_type")->bulk(function ($bulk){

    # Sub queries

	$bulk->index("my_index_1")->type("my_type_1")->id(10)->insert(["title" => "Test document 1","content" => "Sample content 1"]);
	$bulk->index("my_index_2")->id(11)->insert(["title" => "Test document 2","content" => "Sample content 2"]);
	$bulk->id(12)->insert(["title" => "Test document 3", "content" => "Sample content 3"]);
	
});

# Notes from the above query:

# As index and type names are required for insertion, Index and type names are extendable. This means that: 

# If index() is not specified in subquery:
# -- The builder will get index name from the main query.
# -- if index is not specified in main query, the builder will get index name from configuration file.

# And

# If type() is not specified in subquery:
# -- The builder will get type name from the main query.

# you can use old bulk code style using multidimensional array of [id => data] pairs
 
ES::type("my_type")->bulk([
 
	10 => [
		"title" => "Test document 1",
		"content" => "Sample content 1"
	],
	 
	11 => [
		"title" => "Test document 2",
		"content" => "Sample content 2"
	]
 
]);
 
# The two given documents will be inserted with its associated ids
Update an existing document
ES::type("my_type")->id(3)->update([
   "title" => "Test document",
   "content" => "sample content"
]);
    
# Document has _id = 3 will be updated.
    
# [id is required]
Incrementing field
ES::type("my_type")->id(3)->increment("views");
    
# Document has _id = 3 will be incremented by 1.
    
ES::type("my_type")->id(3)->increment("views", 3);
    
# Document has _id = 3 will be incremented by 3.

# [id is required]
Decrementing field
ES::type("my_type")->id(3)->decrement("views");
    
# Document has _id = 3 will be decremented by 1.
    
ES::type("my_type")->id(3)->decrement("views", 3);
    
# Document has _id = 3 will be decremented by 3.

# [id is required]
Update using script
# increment field by script
    
ES::type("my_type")->id(3)->script(
    "ctx._source.$field += params.count",
    ["count" => 1]
);
    
# add php tag to tags array list
    
ES::type("my_type")->id(3)->script(
    "ctx._source.tags.add(params.tag)",
    ["tag" => "php"]
);
    
# delete the doc if the tags field contain mongodb, otherwise it does nothing (noop)
    
ES::type("my_type")->id(3)->script(
    "if (ctx._source.tags.contains(params.tag)) { ctx.op = 'delete' } else { ctx.op = 'none' }",
    ["tag" => "mongodb"]
);
Delete a document
ES::type("my_type")->id(3)->delete();
    
# Document has _id = 3 will be deleted.
    
# [id is required]

Releases

See Change Log.

Author

Basem Khirat - basemkhirat@gmail.com - @basemkhirat

Bugs, Suggestions and Contributions

Thanks to everyone who has contributed to this project!

Please use Github for reporting bugs, and making comments or suggestions.

License

MIT

Have a happy searching..