Skip to content

ingestor_cloudfoundry-firehose job from logsearch-for-cloudfoundry/211.1.0

This job ingests log messages from Cloud Foundry via Firehose interface

Github source: 758c1e9 or master branch

Properties

cloudfoundry

api_endpoint

URI of the CF cloud controller

app_name_suffixes_to_strip

List of suffixes that should be stripped from application names

cc_polling_request

Cloud Controller polling requests by second

Default
50

doppler_endpoint

URI of the loggregator endpoint

extra_fields

(Optional) Additional fields to add to syslog output

Example
extra_fields:
  env: dev
  foo: bar

firehose_cc_pull_interval

full app metadata update interval; defaults to 60s

Default
60s

firehose_client_id

CF UAA client ID with ‘doppler.firehose’ permissions

Default
firehose-to-syslog

firehose_client_secret

CF UAA client secret with ‘doppler.firehose’ permissions

firehose_events

List of events you would like to get. Valid options are ContainerMetric, CounterEvent, Error, HttpStartStop, LogMessage, ValueMetric.

Default
- LogMessage

firehose_subscription_id

ID for the firehose-to-syslog subscription

Default
firehose

ignore_missing_apps

Enable throttling on cache lookup for missing apps

Default
false

log_formatter_type

Log formatter type to use. Valid options are text, json, json-cee

Default
json

logs_buffer_size

Firehose message are bufferized before being send to syslog

Default
10000

max_retry_count

Doppler Cloud Foundry Doppler max. retry Count duration

Default
1000

max_retry_delay

Doppler Cloud Foundry Doppler max. retry delay duration

Default
1m

min_retry_delay

Doppler Cloud Foundry Doppler min. retry delay duration

Default
500ms

orgs

List of CF orgs to forward logs from

skip_ssl_validation

Set to true to skip SSL validation when connecting to CF (defaults to false)

Default
false

create-uaa-client

cloudfoundry

system_domain

The Elastic Runtime system domain ( eg: system.10.244.0.34.xip.io )

uaa_admin_client_id

The UAA admin client id (required scope is uaa.admin). The admin client is used to manage the UAA Kibana oauth2 client.

Default
admin
uaa_admin_client_secret

The UAA admin client’s secret

ingestor_cloudfoundry-firehose

debug

Debug level logging. This disables forwarding to syslog

Default
false

enable_stats_server

Will enable firehose-to-syslog stats server on port 8080

Default
false

logstash

metadata_level

Whether to include additional metadata throughout the event lifecycle. NONE = disabled, DEBUG = fully enabled

Default
NONE

syslog

cert_pem

Certificate PEM.

host

IP or hostname of the syslog drain

Default
127.0.0.1

port

port of the syslog drain

Default
5514

protocol

Syslog protocol (tcp/udp).

Default
tcp

skip_ssl_validation

Skip SSL validation for syslog

Default
false

Templates

Templates are rendered and placed onto corresponding instances during the deployment process. This job's templates will be placed into /var/vcap/jobs/ingestor_cloudfoundry-firehose/ directory (learn more).

  • bin/ingestor_cloudfoundry.sh (from bin/ingestor_cloudfoundry)
  • bin/pre-start (from bin/pre-start.erb)
  • config/bpm.yml (from config/bpm.yml.erb)
  • config/cert.pem (from config/cert.pem.erb)

Packages

Packages are compiled and placed onto corresponding instances during the deployment process. Packages will be placed into /var/vcap/packages/ directory.