Show HN: C++ AWS MSK IAM Auth Implementation – Goodbye Kafka Passwords

39 points by jovezhong 3 days ago

In 2023, AWS announced[1] IAM authentication for MSK Kafka clusters with support for "all programming languages"… except C++. While Java[2], Python[3], Go[4], and others got official SDKs, C++ developers/vendors were stuck hardcoding SCRAM-SHA credentials in code/configs or relying on heavier Java-based tools like Kafka Connect or Apache Flink.

Later, community projects added Rust[5] and Ruby[6] support. Why no C++? Rust might be the new favorite, but C++ is still king for high-performance data systems: minimal dependencies, lean resource use, and raw speed.

At Timeplus, we needed IAM auth for our C++ streaming engine, Proton, so we built it ourselves. Today, we’re open-sourcing our code for AWS MSK IAM authentication. It’s live in Timeplus Proton 1.6.12

Just attach an IAM role to your EC2 instance or EKS pod, then put the Timeplus Proton single binary inside, start the server, then run the following SQL to read or write MSK:

CREATE EXTERNAL STREAM msk_stream(column_defs) SETTINGS type='kafka',topic='topic2', brokers='prefix.kafka.us-west-2.amazonaws.com:9098', security_protocol='SASL_SSL', sasl_mechanism='AWS_MSK_IAM';

The core logic is just two files under 200 lines and you can reuse the code anywhere. https://github.com/timeplus-io/proton/blob/develop/src/IO/Ka... https://github.com/timeplus-io/proton/blob/develop/src/IO/Ka...

We’d love to get your feedback and work together to make this a standalone library—or even get it into ClickHouse or AWS SDK for C++.

For those curious about Timeplus Proton: it’s an open-source streaming engine we built in C++ (think “FlinkSQL in C++” meets ClickHouse’s columnar storage). Later this month, we will also open-source our C++ code for Apache Iceberg read&write. Stay tuned.

Links:

[1] https://aws.amazon.com/blogs/big-data/amazon-msk-iam-authent... [2] https://github.com/aws/aws-msk-iam-auth [3] https://github.com/aws/aws-msk-iam-sasl-signer-python [4] https://github.com/aws/aws-msk-iam-sasl-signer-go [5] https://docs.rs/aws-msk-iam-sasl-signer [6] https://rubygems.org/gems/aws-msk-iam-sasl-signer/