NAME

    Catmandu::Store::ElasticSearch - A searchable store backed by
    Elasticsearch

SYNOPSIS

        # From the command line
    
        # Import data into ElasticSearch
        $ catmandu import JSON to ElasticSearch --index-name 'catmandu' < data.json
    
        # Export data from ElasticSearch
        $ catmandu export ElasticSearch --index-name 'catmandu' to JSON > data.json
    
        # Export only one record
        $ catmandu export ElasticSearch --index-name 'catmandu' --id 1234
    
        # Export using an ElasticSearch query
        $ catmandu export ElasticSearch --index-name 'catmandu' --query "name:Recruitment OR name:college"
    
        # Export using a CQL query (needs a CQL mapping)
        $ catmandu export ElasticSearch --index-name 'catmandu' --q "name any college"
    
        # From Perl
    
        use Catmandu;
    
        my $store = Catmandu->store('ElasticSearch', index_name => 'catmandu');
    
        my $obj1 = $store->bag->add({ name => 'Patrick' });
    
        printf "obj1 stored as %s\n" , $obj1->{_id};
    
        # Force an id in the store
        my $obj2 = $store->bag->add({ _id => 'test123' , name => 'Nicolas' });
    
        # Commit all changes
        $store->bag->commit;
    
        $store->bag->delete('test123');
    
        $store->bag->delete_all;
    
        # All bags are iterators
        $store->bag->each(sub { ... });
        $store->bag->take(10)->each(sub { ... });
    
        # Query the store using a simple ElasticSearch query
        my $hits = $store->bag->search(query => '(content:this OR name:this) AND (content:that OR name:that)');
    
        # Native queries are also supported by providing a hash of terms
        # See the ElasticSearch manual for more examples
        my $hits = $store->bag->search(
            query => {
                # All name.exact fields that start with 'test'
                prefix => {
                    'name.exact' => 'test'
                }
            } ,
            limit => 1000);
    
        # Catmandu::Store::ElasticSearch supports CQL...
        my $hits = $store->bag->search(cql_query => 'name any "Patrick"');

METHODS

 new(index_name => $name, [...])

 new(index_name => $name , index_mapping => \%map, [...])

 new(index_name => $name , ... , bags => { data => { cql_mapping => \%map }
 })

    Create a new Catmandu::Store::ElasticSearch store connected to index
    $name. Optional extra ElasticSearch connection parameters will be
    passed on to the backend database.

    Optionally provide an index_mapping which contains a ElasticSearch
    schema for each field in the index (See below).

    Optionally provide for each bag a cql_mapping to map fields to CQL
    indexes.

 drop

    Deletes the Elasticsearch index backing this store. Calling functions
    after this may fail until this class is reinstantiated, creating a new
    index.

INHERITED METHODS

    This Catmandu::Store implements:

    Catmandu::Store

    Catmandu::Droppable

    Each Catmandu::Bag in this Catmandu::Store implements:

    Catmandu::Bag

    Catmandu::Droppable

    Catmandu::Searchable

    Catmandu::CQLSearchable

INDEX MAP

    The index_mapping contains a Elasticsearch schema mappings for each bag
    defined in the index. E.g.

        {
            data => {
                properties => {
                    _id => {
                        type           => 'string',
                        include_in_all => 'true',
                        index          => 'not_analyzed'
                    } ,
                    title => {
                        type           => 'string'
                    }
                }
            }
        }

    In the example above the default 'data' bag of the ElasticSearch
    contains an '_id' field of type 'string' which is stored automatically
    also in the '_all' search field. The '_id' is not analyzed. The bag
    also contains a 'title' field of type string.

    See
    https://www.elastic.co/guide/en/elasticsearch/reference/2.2/mapping.html
    for more information on mappings.

    These mappings can be passed inside a Perl program, or be written into
    a Catmandu 'catmandu.yml' configuration file. E.g.

       # catmandu.yml
       store:
           search:
              package: ElasticSearch
              options:
                index_name: catmandu
                index_mappings
                  data:
                    properties:
                        _id:
                            type: string
                            include_in_all: true
                            index: not_analyzed
                        title:
                            type: string

    Via the command line these configuration parameters can be read in by
    using the name of the store, search in this case:

       $ catmandu import JSON to search < data.json
       $ catmandu export search to JSON > data.json

CQL MAP

    Catmandu::Store::ElasticSearch supports CQL searches when a cql_mapping
    is provided for each bag. This hash contains a translation of CQL
    fields into Elasticsearch searchable fields.

     # Example mapping
      {
        indexes => {
          title => {
            op => {
              'any'   => 1 ,
              'all'   => 1 ,
              '='     => 1 ,
              '<>'    => 1 ,
              'exact' => {field => [qw(mytitle.exact myalttitle.exact)]}
            } ,
            field => 'mytitle',
            sort  => 1,
            cb    => ['Biblio::Search', 'normalize_title']
          }
        }
     }

    The CQL mapping above will support for the 'title' field the CQL
    operators: any, all, =, <> and exact.

    The 'title' field will be mapping into the Elasticsearch field
    'mytitle', except for the 'exact' operator. In case of 'exact' we will
    search both the 'mytitle.exact' and 'myalttitle.exact' fields.

    The CQL mapping allows for sorting on the 'title' field. If, for
    instance, we would like to use a special ElasticSearch field for
    sorting we could have written "sort => { field => 'mytitle.sort' }".

    The callback field cb contains a reference to subroutines to rewrite or
    augment a search query. In this case, the Biblio::Search package
    contains a normalize_title subroutine which returns a string or an
    ARRAY of strings with augmented title(s). E.g.

        package Biblio::Search;
    
        sub normalize_title {
           my ($self,$title) = @_;
           my $new_title =~ s{[^A-Z0-9]+}{}g;
           $new_title;
        }
    
        1;

    Also this configuration can be added to a catmandu.yml configuration
    file like:

        # catmandu.yml
        store:
            search:
               package: ElasticSearch
               options:
                 index_name: catmandu
                 index_mappings
                   data:
                     properties:
                         _id:
                             type: string
                             include_in_all: true
                             index: not_analyzed
                         title:
                             type: string
                 bags:
                   data:
                      cql_mapping:
                        indexes:
                            title:
                                op:
                                    'any': true
                                    'all': true
                                    '=':   true
                                    '<>':  true
                                    'exact':
                                        field: [ 'mytitle.exact' , 'myalttitle.exact' ]
                                field: mytitle
                                sort: true
                                cb: [ 'Biblio::Search' , 'normalize_title' ]
                        }

    Via the command line these configuration parameters can be read in by
    using the name of the store, search in this case:

       $ catmandu export search -q 'title any blablabla' to JSON > data.json

COMPATIBILITY

    This store expects version 1.0 or higher of the Elasticsearch server.

    To talk to older versions of Elasticsearch the appropriate client
    should be installed.

        # Elasticsearch 2.x
        cpanm Search::Elasticsearch::Client::2_0::Direct
        # Elasticsearch 1.x
        cpanm Search::Elasticsearch::Client::1_0::Direct

    And the client version should be specified in the options:

        Catmandu::Store::ElasticSearch->new(index_name => 'myindex', client => '1_0::Direct')

    Note that Elasticsearch >= 2.0 doesn't allow keys that start with an
    underscore such as _id. You can use the key_prefix option at store
    level or id_key at bag level to handle this.

        # in your catmandu.yml
        store:
          yourstore:
            package: ElasticSearch
            options:
              # use my_id instead of _id
              key_prefix: my_

    If you want to use the delete_by_query method with Elasticsearch >= 2.0
    you need have to install the delete by query plugin
    <https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugins-delete-by-query.html>.

MIGRATING A STORE FROM ELASTICSEARCH 1.0 TO 2.0 OR HIGHER

    1. backup your data as JSON

        catmandu export yourstore --bag yourbag to --file /path/to/yourbag.json -v

    2. drop the store

        catmandu drop yourstore

    3. upgrade the Elasticsearch server

    4. update your catmandu.yml with a key_prefix or id_prefix (see
    COMPATIBILITY)

    5. import your data using the new keys specified in your catmandu.yml

        catmandu import --file /path/to/yourbag.json --fix 'move_field(_id, my_id)' \
        to yourstore --bag yourbag -v

ERROR HANDLING

    Error handling can be activated by specifying an error handling
    callback for index when creating a store. E.g. to create an error
    handler for the bag 'data' index use:

        my $error_handler = sub {
            my ($action, $response, $i) = @_;
            do_something_with_error($response);
        };
    
        my $store = Catmandu::Store::ElasticSearch->new(
            index_name => 'catmandu'
            bags       => { data => { on_error => $error_handler } }
        });

    Instead of a callback, the following shortcuts are also accepted for
    on_error:

    log: log the response

    throw: throw the response as an error

    ignore: do nothing

        my $store = Catmandu::Store::ElasticSearch->new(
            index_name => 'catmandu'
            bags       => { data => { on_error => 'log' } }
        });

SEE ALSO

    Catmandu::Store

AUTHOR

    Nicolas Steenlant, <nicolas.steenlant at ugent.be>

CONTRIBUTORS

    Dave Sherohman, dave.sherohman at ub.lu.se

    Robin Sheat, robin at kallisti.net.nz

    Patrick Hochstenbach, patrick.hochstenbach at ugent.be

LICENSE AND COPYRIGHT

    This program is free software; you can redistribute it and/or modify it
    under the terms of either: the GNU General Public License as published
    by the Free Software Foundation; or the Artistic License.

    See http://dev.perl.org/licenses/ for more information.