PART — 2— Dockerized Ubuntu 20.04 — SPARK 3.2.1 with KAFKA 2.13–2.8.1 integration

Firstly due credits to the BOS the BOSS!

Structured Streaming in Spark 3.0 Using Kafka

I took the docker image from his github

GitHub - Wesley-Bos/spark3.0-examples: Basic Spark examples.

But it did not work for 3.0, got the usually java errors, so as per my part 1, upgraded to 3.2.1 and added the missing jars

Here is the customized docker image, which is working now.

Docker customization done

Docker file

GitHub - anbuanand/spark_kafka_integration: Due acknowledgment to Wesley Bos at…

So guys please follow Wesley Bos’s link above, docker file is form my side.

Other good references that I used, credits to Shuyi Yang — https://towardsdatascience.com/kafka-docker-python-408baf0e1088

Happy Sparking and Kafkaying in an Docker !!