As of ES 5.x , they have given this feature out of the box with logstash plugin. This will periodically import data from database and push to ES server. One has to create a simple import file given below (which is also described here) and use logstash to run the script. Logstash supports running this … Read more
Just create a template. run curl -XPUT localhost:9200/_template/template_1 -d ‘{ “template”: “*”, “settings”: { “index.refresh_interval”: “5s” }, “mappings”: { “_default_”: { “_all”: { “enabled”: true }, “dynamic_templates”: [ { “string_fields”: { “match”: “*”, “match_mapping_type”: “string”, “mapping”: { “index”: “not_analyzed”, “omit_norms”: true, “type”: “string” } } } ], “properties”: { “@version”: { “type”: “string”, “index”: “not_analyzed” … Read more
Hi I added a grok filter to do just this. I only wanted to have the filename not the path, but you can change this to your needs. filter { grok { match => [“path”,”%{GREEDYDATA}/%{GREEDYDATA:filename}\.log”] } }
For Kibana 4 go to this answer This is easy to do with a terms panel: If you want to select the count of distinct IP that are in your logs, you should specify in the field clientip, you should put a big enough number in length (otherwise, it will join different IP under the … Read more
It depends what you intend when you say JOIN. Elasticsearch is not like regular database that supports JOIN between tables. It is a text search engine that manages documents within indexes. On the other hand you can search within the same index over multiple types using a fields that are common to every type. For … Read more
This is what you’re looking for: https://github.com/ForgeRock/es-change-feed-plugin Using this plugin, you can register to a websocket channel to receive indexation/deletion events as they happen. It has some limitations, though. Back in the days, it was possible to install river plugins to stream documents to ES. The river feature has been removed, but this plugin above … Read more
You can achieve this with the Logstash aggregate filter only, however, you’d have to substantially re-implement what the elapsed filter already does, so that’d be a shame, right? Let’s then use a mix of the Logstash aggregate filter and the elapsed filter. The latter is used to measure the time of each stage and the … Read more
That’s a great start! I would definitely flatten it all out (i.e. denormalize) and come up with product documents that look like the one below. That way you get rid of the N:M relationship between products and flags by simply creating a flags array for each product. It will thus be easier to query those … Read more
By default logstash writes the position is last was on to a logfile which usually resides in $HOME/.sincedb. Logstash can be fooled into believing it never parsed the logfile by specifying /dev/null as sincedb_path. Here the part of the documentation Input File. Where to write the since database (keeps track of the current position of … Read more