Client Design

openGeminiJuly 1, 2024About 7 minAbout 1959 words

Background

Due to the InfluxDB 1.X client largely being in maintenance mode, and openGemini continuously evolving, the community has decided to develop its own client SDK for openGemini to better support it. This includes functionalities like supporting multiple server addresses and the Apache Arrow Flight protocol.

Client SDK Planned Features

  • Support for connecting to multiple server addresses
  • Support for the Apache Arrow Flight protocol
  • Capabilities for SQL queries, structured queries, writing, and batch writing are detailed in the UML diagram below
  • Default timeouts, with a connection timeout of 10 seconds and read/write timeout of 30 seconds

The methods described in this document assume that the programming language does not support overloading. If overloading is supported by the programming language, some optimizations and adjustments can be made to the method names.

Client constructor params design

contains
1
many
contains
contains
OpenGeminiClient
+ List<Address> addresses
+ AuthConfig authConfig // nullable, if null, means no auth
+ BatchConfig batchConfig // nullable, if null, means batch is disabled
+ timeout
+ connectTimeout
+ enum contentType // json, csv, msgpack
+ enum compressMethod // gzip, zstd, br
+ TlsConfig tlsConfig // nullable, language specific
+ GrpcConfig grpcConfig // if null, call WriteByGrpc will nothing to do, otherwise send write request by gRPC
+void close()
Address
+ String host
+ int Port // in rust, it is u16
AuthConfig
+ AuthType authType // enum Password, Token. The server currently does not support the Token type, and the SDK implementation for Token is incomplete.
+ String username
+ String password
+ String token
BatchConfig
+ Duration batchInterval // must be greater than 0
+ int batchSize // must be greater than 0, if set too large, may cause client overflow or server-side rejected the request.
GrpcConfig
+ List<Address> addresses
+ AuthConfig authConfig
+ BatchConfig batchConfig
+ enum compressMethod // gzip, zstd, br
+ TlsConfig tlsConfig
+ timeout

Database & RetentionPolicy management design

OpenGeminiClient
+void CreateDatabase(String database)
+void CreateDatabaseWithRp(String database, rpConfig RpConfig)
+String[] ShowDatabases()
+void DropDatabase(String database)
+void CreateRetentionPolicy(String database, RpConfig rpConfig, bool isDefault)
+void UpdateRetentionPolicy(String database, RpConfig rpConfig, bool isDefault)
+RetentionPolicy[] ShowRetentionPolicies(String database)
+void DropRetentionPolicy(String database, String retentionPolicy)
+void CreateMeasurement(CreateMeasurementBuilder builder)
+String[] ShowMeasurements(ShowMeasurementBuilder builder)
+void DropMeasurement(String database, String retentionPolicy, String measurement)
+Map[String]String[] ShowTagKeys(ShowTagKeysBuilder builder)
+String[] ShowTagValues(ShowTagValuesBuilder builder)
+Map[String]Map[String]String ShowFieldKeys(String database, Option measurement)
+String[] ShowSeries(ShowSeriesBuilder builder)
RpConfig
+ String Name // non-null
+ String Duration // non-null
+ String ShardGroupDuration // nullable
+ String IndexDuration // nullable
CreateMeasurementBuilder
+CreateMeasurementBuilder Tags(String[] tags)
+CreateMeasurementBuilder FieldMap(map[String]FieldType fields)
+CreateMeasurementBuilder ShardType(ShardType shardType)
+CreateMeasurementBuilder ShardKeys(String[] shardKeys)
+CreateMeasurementBuilder FullTextIndex()
+CreateMeasurementBuilder IndexList(String[] indexes)
+CreateMeasurementBuilder EngineType(EngineType engineType)
+CreateMeasurementBuilder PrimaryKey(String[] primaryKeys)
+CreateMeasurementBuilder SortKeys(String[] sortKeys)
+String build()
ShowMeasurementBuilder
+ShowMeasurementBuilder Filter(ComparisonOperator operator, String regex)
+String build()
«enum»
FieldType
Bool // BOOL
Int64 // INT64
Float64 // FLOAT64
String // STRING
«enum»
ShardType
Hash // HASH
Range // RANGE
«enum»
EngineType
ColumnStore // columnstore
ShowTagKeysBuilder
ShowTagKeysBuilder Database(String database)
ShowTagKeysBuilder Measurement(String measurement)
ShowTagKeysBuilder RetentionPolicy(String rp)
ShowTagKeysBuilder Limit(int limit)
ShowTagKeysBuilder Offset(int offset)
ShowTagValuesBuilder
ShowTagValuesBuilder Database(String database)
ShowTagValuesBuilder Measurement(String measurement)
ShowTagValuesBuilder RetentionPolicy(String rp)
ShowTagValuesBuilder Limit(int limit)
ShowTagValuesBuilder Offset(int offset)
ShowTagValuesBuilder With(String[] keys)
ShowTagValuesBuilder Where(String key, ComparisonOperator operator, String value)
ShowSeriesBuilder
ShowSeriesBuilder Database(String database)
ShowSeriesBuilder Measurement(String measurement)
ShowSeriesBuilder RetentionPolicy(String rp)
ShowSeriesBuilder Limit(int limit)
ShowSeriesBuilder Offset(int offset)
ShowTagValuesBuilder Where(String key, ComparisonOperator operator, String value)

Write point design

Execute interface design

The Execute interface provides a unified SQL execution interface that automatically routes different types of statements to appropriate underlying methods. This design supports SQL-like statements including INSERT, SELECT, CREATE, DROP, and other database operations with parameter support and type safety.

Statement routing logic

SELECT, SHOW, EXPLAIN, DESCRIBE, WITH
CREATE, DROP, ALTER, UPDATE, DELETE
INSERT
Other
Execute Statement
Parse Statement Type
StatementTypeQuery
StatementTypeCommand
StatementTypeInsert
StatementTypeUnknown
Route to Query method
Route to Write methods
Return error
Return QueryResult + AffectedRows=0/1
Return AffectedRows=point count
Return error result

Parameter support

The Execute interface supports parameterized statements with automatic type conversion:

uses
converts
«enum»
ParameterTypes
String // "value" → value
Integer // 42 → 42i
UInteger // 42 → 42u
Float // 3.14 → 3.14
Boolean // true → true
ParameterProcessor
+replaceParams(command String, params Map) : String
+convertParamValue(value Object) : String
+validateParams(command String, params Map) : Error
Statement

Usage examples

Basic usage

result, err := client.Execute(opengemini.Statement{
    Database: "mydb",
    Command:  "SELECT * FROM weather LIMIT 10",
})

Parameterized query

result, err := client.Execute(opengemini.Statement{
    Database: "mydb", 
    Command:  "SELECT * FROM weather WHERE location=$loc AND temp>$temp",
    Params: map[string]any{
        "loc":  "beijing",
        "temp": 25.0,
    },
})

Parameterized insert

result, err := client.Execute(opengemini.Statement{
    Database: "mydb",
    Command:  "INSERT weather,location=$location temperature=$temp,humidity=$hum",
    Params: map[string]any{
        "location": "shanghai", 
        "temp":     30.2,
        "hum":      70,
    },
})

Query design

Query
+ String database
+ String retentionPolicy
+ String command
+ TimeFormat timeFormat
+ Precision precision
«enum»
Precision
PRECISIONMICROSECOND("u")
PRECISIONNANOSECOND("ns")
PRECISIONMILLISECOND("ms")
PRECISIONSECOND("s")
PRECISIONMINUTE("m")
PRECISIONHOUR("h")
«enum»
TimeFormat
RFC3339("rfc3339")
TIMESTAMP("timestamp")

OpenTelemetry integration design

To enhance the observability of the OpenGemini client and facilitate tracking of performance metrics, errors, and other information related to query and write operations, this solution adopts the interceptor pattern to integrate OpenTelemetry, enabling full-link tracing. The design supports non-intrusive extensions, allowing coexistence with other interceptors (such as logging and authentication interceptors) while minimizing modifications to the original client.

Interceptor design

The interceptor pattern defines a standardized interface to hook into client operations (query/write) and inject telemetry logic.

«interface»
Interceptor
+Query(ctx context.Context, query *InterceptorQuery) : InterceptorClosure
+Write(ctx context.Context, write *InterceptorWrite) : InterceptorClosure

Define the base client class,associated with the Interceptor interface

The base  Client  class manages a collection of interceptors, allowing dynamic registration and execution of interceptor logic during client operations.

Client
- interceptors: List<Interceptor>

Define the interceptor implementation class integrating OpenTelemetry,implementing the Interceptor interface

The OtelClient class implements the Interceptor interface, embedding OpenTelemetry logic to capture traces, metrics, and logs for client operations.

implements
«implements Interceptor»
OtelClient
Interceptor

Tracing system core module

Usage Example(Go language examples)

func main() {
    var ctx = context.Background()
    shutdown, err := setupOtelSDK(ctx)
    if err != nil {
        return
    }
    defer func() {
        err = errors.Join(err, shutdown(ctx))
    }()

    config := &opengemini.Config{
        Addresses: []opengemini.Address{{
            Host: "127.0.0.1",
            Port: 8086,
        }},
    }
    client, err := opengemini.NewClient(config)
    if err != nil {
        fmt.Println(err)
        return
    }

    client.Interceptors(opengemini.NewOtelInterceptor())

    err = client.CreateDatabase("db0")
    if err != nil {
    }
}

QueryBuilder design

QueryBuilder
+static Create() : QueryBuilder
+Select(Expression[] selectExprs) : QueryBuilder
+From(String[] from) : QueryBuilder
+Where(Condition where) : QueryBuilder
+GroupBy(Expression[] groupByExpressions) : QueryBuilder
+OrderBy(order: SortOrder) : QueryBuilder
+Limit(limit: int64) : QueryBuilder
+Offset(offset: int64) : QueryBuilder
+Timezone(timezone: *time.Location) : QueryBuilder
+Build() : Query
«interface»
Expression
ConstantExpression
- Object value
StarExpression
FunctionExpression
- FunctionEnum function
- Expression[] arguments
AsExpression
- String alias
- Expression expression
ArithmeticExpression
- Expression Left
- Expression Right
- Operator ArithmeticOperator
«interface»
Condition
ComparisonCondition
- String column
- ComparisonOperator operator
- Object value
CompositeCondition
- LogicalOperator logicalOperator
- Condition[] conditions
«enum»
SortOrder
Asc
Desc
«enum»
ComparisonOperator
Equals
NotEquals
GreaterThan
LessThan
GreaterThanOrEquals
LessThanOrEquals
«enum»
LogicalOperator
And
Or
«enum»
FunctionEnum
Mean
Count
Sum
Min
Max
Time
FieldExpression
Query

Ping design

OpenGeminiClient
+Pong ping(int index) : // index selects one from multiple servers
Pong
+ String version

Inner Http client design

Using a design similar to InnerHttpClient, encapsulate authentication, load balancing, retry logic, and more within the internal structure, providing a simple interface to the client. This enhances modularity and code clarity.

InnerHttpClient
+void executeHttpGetByIdx(int idx, ...) : // specify server index
+void executeHttpRequestByIdx(int idx, String method, ...) : // specify server index
+void executeHttpGet(String method, ...) : // load balance
+void executeHttpRequest(String method, ...) : // load balance
-void executeHttpRequestInner(String url, String method, ...) : // inner method
executeHttpGetByIdx
executeHttpRequestByIdx
executeHttpRequestInner
executeHttpGet
executeHttpRequest

Error handling

Error message

Scene1 http request failed

$operation request failed, error: $error_details

Scene2 http response code is not 200~300

$operation error resp, code: $code, body: $body

Scene3 other error

$operation failed, error: $error_details
# example:
writePoint failed, unmarshall response body error: json: cannot unmarshal number ...
OpenGeminiClient
+WritePoint(String database, Point point)
+WritePointWithRp(String database, String rp, Point point)
+WriteBatchPoints(String database, BatchPoints batchPoints)
+WriteBatchPointsWithRp(String database, String rp, BatchPoints batchPoints)
+WriteByGrpc(req WriteRequest) : // WriteRequest build from RecordBuilder
BatchPoints
+ List<Point> points
+AddPoint(Point)
Point
+ String measurement
+ Precision precision // enum, second, millisecond, microsecond, nanosecond, default is nanosecond
+ Time time // language specific
Syntax error in graphmermaid version 10.0.2
OpenGeminiClient
+ExecuteResult Execute(Statement statement)
+ExecuteResult ExecuteContext(Context ctx, Statement statement)
Statement
+ String database
+ String command
Syntax error in graphmermaid version 10.0.2
QueryResult
+ List<SeriesResult> results
+ String error
SeriesResult
+ List<Series> series // Series is an uncountable noun.
+ String error
Series
+ String name
Syntax error in graphmermaid version 10.0.2
TraceContext
+ traceId: String
+ parentTraceId: String
+ operationName: String
+ timestamp: Long
Syntax error in graphmermaid version 10.0.2