Skip to main content

Auto-upload Elastisearch template mapping with Apache Camel

When feeding data into Elastisearch, one important step is to configure the correct template for the index/type so that, for instance, numeric fields are stored as numbers to ensure that they can be sorted by and/or confronted correctly.

The Elasticsearch Logstash plugin has a handy option just for this purpose. If you are not using Logstash you have to do it yourself, eithr through configuration mgmt, startup scripts or simply manaully launching the appropriate curl command.

If you have followed my previous post on using Apache Camel to feed sql data into Elasticsearch then it might come natural to attempt to use Camel also for the purpose of uploading the template mapping.
How hard can it be? Turns out it's prety simple, so let me present you with the solution right away and leave the nitty gritty details for later:

This route will run exactly only once at Camel startup and then fetch the file elmapping.json and PUT it into elastisearch. A sprinkle of groovy makes populating the body of the request a piece of cake (the route requires the camel-groovy and camel-script components).

I have then added an initialDelay to the other routes to allow enough time for elasticsearch to process and acknowledge the mapping.

Happy hacking!


Popular posts from this blog

Indexing Apache access logs with ELK (Elasticsearch+Logstash+Kibana)

Who said that grepping Apache logs has to be boring?

The truth is that, as Enteprise applications move to the browser too, Apache access logs are a gold mine, it does not matter what your role is: developer, support or sysadmin. If you are not mining them you are most likely missing out a ton of information and, probably, making the wrong decisions.
ELK (Elasticsearch, Logstash, Kibana) is a terrific, Open Source stack for visually analyzing Apache (or nginx) logs (but also any other timestamped data).

From 0 to ZFS replication in 5m with syncoid

The ZFS filesystem has many features that once you try them you can never go back. One of the lesser known is probably the support for replicating a zfs filesystem by sending the changes over the network with zfs send/receive.
Technically the filesystem changes don't even need to be sent over a network: you could as well dump them on a removable disk, then receive  from the same removable disk.

RUNDECK job maintenance

Learn more about Rundeck.

Now that I have a fair number of jobs scheduled by Rundeck, how do I periodically prune the job execution history and keep only the last, say, 30 executions for each job?