The Subscribe to a Kafka Topic for GeoJSON input connector can be used to retrieve and adapt event data records, formatted as GeoJSON, from a Kafka Topic. For more information about getting started with Kafka, see Apache Kafka Introduction. For more information on examples of GeoJSON data record formatting, see the GeoJSON specification.
Usage notes
Keep the following in mind when using the Subscribe to a Kafka Topic for GeoJSON input connector:
- Use this input connector to consume GeoJSON data from a Kafka Topic. This input connector is a consumer of Kafka.
- This input connector pairs the GeoJSON inbound adapter with the Kafka inbound transport.
- Like feature JSON, GeoJSON is a subtype of generic JSON. Different GeoEvent Server adapters are used to process these specific formats.
- The Kafka inbound transport supports TLS 1.2 and SASL security protocols for authenticating with a Kafka cluster or broker.
Parameters
The following are the parameters for the Subscribe to a Kafka Topic for GeoJSON input connector:
| Parameter | Description |
|---|---|
Name | A descriptive name for the input connector used for reference in GeoEvent Manager. |
Override with Custom Kafka Properties | Specify whether to override the default GeoEvent Server Kafkaclient properties. The default is No.
|
Kafka Bootstrap Servers (Conditional) | A list of hostname:port pairs used to establish the initial connection to the Kafka cluster. Hostname:port pairs must be comma separated, for example, broker0.example.com:9092,broker1.example.com:9092,broker2.example.com:9092. The parameter is shown when Override with Custom Kafka Properties is set to No. |
Topic Name(s) | The name of a Kafka topic, or list of Kafka topics, to consume data of interest from. Multiple topics must be separated by a semicolon.
Note:The ability to specify multiple Kafka topics is supported in ArcGIS GeoEvent Server 10.8 and later. |
Number of Consumers | Specifies the number of consumers for each consumer group. The default number of consumers is 1. Note:The number of consumers is limited by the number of partitions on the Kafka topic. See Apache Kafka Introduction for more information on consumer instances. |
Consumer Group ID (Conditional) | An optional string that uniquely identifies the consumer group for a set of consumers. This is also known as the consumer group name. If a Consumer Group ID is unspecified, GeoEvent Server will assign a static consumer group ID called geoevent-consumer. This static consumer group ID is shared across all instances of the Kafka connector where the Consumer Group ID is left unspecified. It is highly recommended that you specify a custom Consumer Group ID. Refer to Apache Kafka Introduction for more information about consumer groups. The parameter is shown when Override with Custom Kafka Properties is set to No. |
Registered Folder for the Kafka Properties File (Conditional) | The folder registered with GeoEvent Server that contains the Kafka .properties file. The Kafka .properties file defines the custom Kafka properties when Override with Custom Kafka Properties is set to Yes. Ensure that the folder registered with GeoEvent Server is the full path to where the Kafka .properties file is located. The parameter is shown when Override with Custom Kafka Properties is set to Yes. |
Kafka Properties File Name (Conditional) | The name of the Kafka .properties file that contains the custom Kafka properties for client configuration. The name of the file should be specified without the .properties extension.
The parameter is shown when Override with Custom Kafka Properties is set to Yes. |
Start from Beginning | Specifies whether records are consumed from the topic starting at the beginning offset or from the last offset committed for the consumer. The default is Yes.
Note:For more information about offsets, see Apache Kafka Configuration. |
Create GeoEvent Definition | Specifies whether a new or existing GeoEvent Definition should be used for the inbound event data. A GeoEvent Definition is required for GeoEvent Server to understand the inbound event data attribute fields and data types.
|
GeoEvent Definition Name (New) (Conditional) | The name assigned to a new GeoEvent Definition. If a GeoEvent Definition with the specified name already exists, the existing GeoEvent Definition will be used. The first data record received will be used to determine the expected schema of subsequent data records, a new GeoEvent Definition will be created based on that first data record's schema. The parameter is shown when Create GeoEvent Definition is set to Yes and is hidden when set to No. |
|
GeoEvent Definition Name (Existing) (Conditional) | The name of an existing GeoEvent Definition to use when adapting received data to create event data for processing by a GeoEvent Service. The parameter is shown when Create GeoEvent Definition is set to No and is hidden when set to Yes. |
Default Spatial Reference | The well-known ID (WKID) of a spatial reference to be used when a geometry is constructed from attribute field values whose coordinates are not latitude and longitude values for an assumed WGS84 geographic coordinate system, or geometry strings are received that do not include a spatial reference. A well-known text (WKT) value or the name of an attribute field containing the WKID or WKT may also be specified. |
Expected Date Format | The pattern used to match expected string representations of date/time values and convert them to Java Date values. The pattern's format follows the Java SimpleDateFormat class convention. While GeoEvent Server prefers date/time values to be expressed in the ISO 8601 standard, several string representations of date/time values commonly recognized as date values can be converted to Java Date values without specifying an Expected Date Format pattern. These include the following:
If the date/time values received are expressed using a convention other than one of the five shown above, you will have to specify an expected date format pattern so GeoEvent Server knows how the date/time values should be adapted. |
Authentication Required | Specifies whether the connection to the Kafka cluster, or Kafka broker, requires authentication. The default is No.
|
Authenticate Using (Conditional) | Specifies the security protocol that is used to secure the Kafka cluster. Available security protocols include TLS 1.2 and SASL.
Note:When using Kerberos, ensure the operating system user account running ArcGIS GeoEvent Server has read access to the keytab file in the Kerberos setup and configuration. The parameter is shown when Authentication Required is set to Yes. |
Registered Folder for Credential File (Conditional) | The folder registered with GeoEvent Server that contains the Kafka cluster's PKI file (x509 certificate). Ensure that the folder registered with GeoEvent Server is the full path to where the Kafka cluster's certificate is located. The parameter is shown when Authentication Required is set to Yes. It is applicable to TLS 1.2 only. |
Credential Configuration File (Conditional) | The name of the Kafka cluster's PKI file (x509 certificate). The certificate and its associated private key must be stored in the PKCS#12 format, which is represented by a file with either the .p12 or .pfx extension. Provide the name of the file in addition to the extension.
Note:Only the certificate file name and extension are supported for this parameter. Relative paths to the certificate should not be specified in this parameter. Register the full path to the certificate file using the Registered Folder for Credential File parameter. The parameter is shown when Authentication Required is set to Yes. It is applicable to TLS 1.2 only. |
Keystore Password (Conditional) | The password for the Kafka cluster's PKI file (x509 certificate). This is also known as the certificate's private key. The parameter is shown when Authentication Required is set to Yes. The parameter is applicable to TLS 1.2 only. |
SASL Authentication Type (Conditional) | Specifies the type of SASL authentication mechanism supported by the Kafka cluster. Available SASL authentication types include SASL GSSAPI (Kerberos) and SASL PLAIN.
The parameter is shown when Authentication Required is set to Yes. It is applicable to SASL only. |
Kerberos Principal (Conditional) | The Kerberos principal for the specific user, for example, GeoEventKafkaClient1@example.com. The parameter is shown when Authentication Required is set to Yes. The parameter is applicable to SASL/GSSAPI (Kerberos) only. |
Use Key Tab (Conditional) | Indicates whether to use the keytab in the Kerberos settings. The default is Yes.
The parameter is shown when Authentication Required is set to Yes. It is applicable to SASL/GSSAPI (Kerberos) only. |
Store Key (Conditional) | Indicates whether to store the key in the Kerberos settings. The default is Yes.
The parameter is shown when Authentication Required is set to Yes. It is applicable to SASL/GSSAPI (Kerberos) only. |
Username (Conditional) | Specifies the username used to authenticate with the Kafka cluster. This is also known as a connection string with certain cloud providers. Refer to the documentation of the chosen cloud provider for correct syntax. The parameter is shown when Authentication Required is set to Yes. It is applicable to SASL/PLAIN only. |
Password (Conditional) | Specifies the password used to authenticate with the Kafka cluster. Refer to the documentation of the chosen cloud provider for the correct syntax. The parameter is shown when Authentication Required is set to Yes. The parameter is applicable to SASL/PLAIN only. |
Considerations and limitations
There are several considerations to keep in mind when using the Subscribe to a Kafka Topic for GeoJSON input connector:
- Insufficiently managing and optimizing consumers will result in certain instances of the Subscribe to a Kafka Topic for GeoJSON input connector not retrieving data. The number of consumers, in a consumer group, is limited by the number of partitions on a Kafka Topic. If the number of consumers in a consumer group exceeds the number of partitions available on a Kafka Topic, the excess consumers will not consume data. Consider optimizing the number of consumers to best align with the number of partitions on the Kafka Topic or implement different consumer groups for each connector to avoid this. For more information on consumers and consumer groups, see the Kafka Documentation.
- The Subscribe to a Kafka Topic for GeoJSON input connector is a client consumer of Kafka. Apply the same considerations to this input connector as would be required for any other client consumer of Kafka. For example, if this input connector is not receiving data from a Kafka Topic but a separate client consumer of Kafka is, consider the factors that are involved with having two client consumers. This includes but is not limited to the configured consumer group ID, number of partitions available on the Topic, and the number of existing consumers. Alternatively, if the input connector is stopped and started in quick succession, consider the implications for Kafka from a consumer point of view. A rebalancing of the Kafka Topic's partitions may prevent the input connector from immediately rejoining as a consumer under the same consumer group.