Data+AI Summit 2021 starts on Monday, May 24 till Friday, May 28. The training will be held on May 24-25 and will cater to a large set of practitioners, definitely more extensive than previous times: Data Analyst, Data Engineer, Data Scientist, ML Engineer, Partner Data Engineer, Platform Engineer, Technical. The wide range of roles makes me curious about the various technical personas in the Data and AI space. It’s not only Data Engineers and Data Scientists. There is a wide range of people who can benefit from attending the summit.

Interestingly, Monday training starts at 6 am PT, allowing European timezone folks to join at their after noon time.

The pass for the summit is free upon registration.

The agenda is somewhat missing from the main page, but trying out multiple URLs brought me to the whole agenda on the site.

There are 3 main focus areas:

• Productionizing Machine Learning
• Databricks Experience
• Technical Deep Dives

As you know by now, I think all the sessions are great! Sessions were carefully picked out of hundreds to provide the maximum value to the summit participants.

But, let’s assume we can attend only three. Which 3 will you choose?

Here are my top picks:

1st: From the machine learning space - Strategies for Debugging Machine Learning Systems

This session is interesting since there are many ways to build machine learning models, from centralized distributed machine learning to decentralized distributed like Federated Learning. Of course, many places where things can go wrong when training on one machine or multiple machines. The space of security and adversaries in machine learning training is a magical one. Like a rare diamond, you don’t know its worth until carefully examined. For example, in federated learning, each device is training a model locally on its data and shares the model summary. Assuming one of the devices is an adversary, an attacker who wants to change the model’s overall result can share a false summary with the rest of the group and impact the model’s overall behavior. In some cases, that means smaller revenue, but when we use machine learning to save lives like healthcare usabilities, that’s more complicated.

2nd From the Databricks Experience space - Video Analytics At Scale: DL, CV, ML On Databricks Platform

The future of data is complex data. And complex data for is sound and video. Let me bottom-line this for you; video is here to stay, may it be social media, Netflix, or Autonomous drowns/cars/… The ability to process complex data at scale and enabling machine learning at scale is the future.

And lastly, from the Technical Deep Dives:

3rd Becoming a Data-Driven Organization with Modern Lakehouse

Understanding the Modern lakehouse is one thing, but knowing how to rally all the stakeholders and build a robust one is a different challenge.

I’m going to let you in on a secret, all sessions are going to be recorded and shared after the summit. But there is some magic in attending, asking questions and participating in Twitter and Chat with like minded who came to learn, exchange ideas and network.

Links:

• Registration
• Agenda
• Learn about Azure Databricks

Here is a list of concepts for you to get started:

ML Algorithm

ML algorithm is a procedure that runs on data and produces a machine learning model. Some of the popular ones are Decision trees, Naive Bayes, and Linear Regression.

ML Model

ML model is the ML algorithm process outcome; It often contains a statistical representation of the data ingested into the algorithm. ML model input is data, and the output is either a prediction, decision, or classification.

Training set

The training set is the data ingested into the machine learning algorithm; it trains the ML model.

Testing set

The testing set is the dataset we test the ml model with. To test the ML model’s accuracy, we ingest the data into the model and measure the accuracy level of the outcome. It helps us reason about the quality of the machine learning model.

Machine Learning pipeline

The machine learning pipeline is an automation process of the machine learning workflow. It includes data transformation and correlation to fit the ML algorithm, running the algorithm to produce a model, and testing it with a test set.

Model interpretability

ML Model interpretability is the degree to which a human can reason the machine learning model’s output. The higher the degree, the easier it is for a human to understand the model’s decision or prediction.

Data Quality

Data quality measures the data’s condition based on accuracy, precision, legitimacy, validity, reliability, consistency, completeness, and more. In machine learning, data quality is important for producing high-quality, non-bias machine learning models.

Data drifts

Data drift is unexpected and undocumented changes to the data structure, semantics. Data drift can result in corrupted data and data low quality. Lack of awareness of data drift can result in a lesser quality of ML models.

Concept drift

Concept drift refers to the changes in target variables. Target variables are the outcomes of the prediction process you do with machine learning models. You can detect concept drift by measuring the statistical properties of the target variables. Machine learning’s actual target variable can change over time in unforeseen ways and presents a challenge since the predictions become less accurate as time passes.

I hope it was helpful for you and gave you more clarity about the concepts.

💡 Curious to learn more?

Read here about how to create machine learning models with python.

There are many great features that you can leverage in delta lake, from the ACID transaction, Schema Enforcement, Time Traveling, Exactly One semantic, and more.

Let’s discuss two common data pipelines patterns and solutions:

Spark Structured Streaming ETL with DeltaLake that serves multiple Users

Spark Structured Streaming- Apache Spark structured steaming are essentially unbounded tables of information. There is a continuous stream of data ingested into the system. As developers, we write the code to process the data continuously. ETL stands for Extract, Transform and Load.

Scenario - Ingest data from Kafka topic, process the information using Spark structured streaming, and save it to DeltaLake for multiple users on-the-fly queries.
Note! The output of the solution/pipeline is used by real users and not a machine, which means that there is no need to refresh the data every couple of seconds as the person taking actions on the data won’t be able to use it.

System Requirements

Let’s assume we have these system requirements:

Input: any unstructured one input stream for example Kafka topic
Output: structured tabular data for users to query
Latency: 5 minutes
Constraints: multiple users query the table at the same time

High-level Pipeline Architecture

Advantages

In our scenario, we have multiple users that query the data on the fly and should see the same data - single source of truth. In distributed steaming, when we query the data, it might be that from two identical queries that ran at the same time, we will get different results. This is why we introduce DeltaLake into the pipeline. We save the streaming tabular data in DeltaLake, which in practice means that the user read operations take place on the DeltaTable snapshot, which guarantees consistency of the data. At the same time, the table is continuously being written.
Simultaneously, the user can also run updates/deletes and fixes on the data when necessary, this is important for controlling incoming data that are bounded to GDPR or other compliances. The conflicts are resolved using Delta conflict resolution mechanism (Discussed in Part 2 - the DeltaLog).

If you are using Databricks services, you will get the Auto Optimize out of the box, which coalesces small files into larger files using Auto Compaction.

When to exclude Delta Lake

As much as it’s important to know what are the advantages and when to use DeltaLake, it’s important to understand when to exclude it. For example, when you want to have a latency of seconds to update a Key-Value output for lookup tables, you should probably avoid DeltaLake since it introduce the overhead of the optimistic concurrency and commits to the DeltaLog itself. But if your system can handle a couple minutes of latency, you should consider using it for enforcing data - single source of truth for your users.

What to use instead of DeltaLake for updating lookup tables with seconds latency

A lookup table is an array that replaces runtime computation with a simpler array indexing operation, which means that the data is being stored in memory. This makes read queries significantly faster than loading data from disk, which involves I/O operations. Hence for stateful streaming operations, we would prefer to use in-memory databases such as Redis, Cassandra or Amazon DynamoDB. This solution is more expensive since it requires dedicated servers/services to be used in the system, vs. using DeltaLake which is a storage layer, but this is the price to be paid for lookup table that is being updated and accurate to all users with a latency of seconds.

Join Multiple Data Streams based on a common key on Azure Databricks

Scenario - Ingesting data into the system from multiple different data streams that need to be joined based on a common key written to a shared table for future analytics/ML workloads/lookup tables.

System Requirements

Let’s assume we have these system requirements:

Input: multiple unstructured input streams from various sources
Output: tabular data combining the streams inputs
Latency: 2 minutes
Constraints: support join on one fast and one slow data streams with dimension changes

Slowly changing dimensions is a data management problem where the data warehouse contains relatively static data and schema linked to a dimension table that can change its schema and data as time passes. To learn more about it, read here. Imagine a user bank account information that needs to be joined by a user bank transaction.

Suppose you are familiar with broadcast join mechanism. In that case, the solution might look simple, broadcast the small static data table and use Spark Structured Streaming to stream the fast, ever-changing stream for the join operation.
But, what if you are attempting to join two big tables that constantly change? This is when you need to understand how to leverage Delta Lake versioning capabilities.

High-level Pipeline Architecture

In this high-level architecture diagram, we have 2 Spark workloads; the first one is a batch, reading data from MongoDB, processing it, and saving it to DetlaLake. The second one is the fast data, ingesting data from a Kafka topic directly into Spark Streaming and joining the fast data with the slow data saved in DeltaLake. After the join and further logic, the data is being kept in a tabular store for future use.

Advantages

Without DetlaTable, Structured streaming will hold a static view of the first “slow” data, and it won’t be updated until you restart the streaming query. But we already know that this data can be updated; think about the bank customer who changed their home address, and it needs to be updated in the system. Using DeltaLake helps introduce table versioning and allows you to control the changes and read/use the latest/most updated version inside the streaming query without restarting it.

I hope you enjoyed reading about Delta Lake, the two practical scenarios, and the breakdown of the open-source;
I will continue to share scenarios, insights, and code samples throughout the blog. As always, if you have questions, suggestions, ideas, please don’t hesitate to DM me on Adi Polak 🐦.

If you would like to get monthly updates, consider subscribing.

💡 Learn more!

• Watch this video on how to architect structured streaming.
• Read here about Azure Databricks and Streaming.

If you didn’t get a chance to read the previous posts, read here:

1. Delta Lake essential Fundamentals: Part 1 - ACID
2. Delta Lake essential Fundamentals: Part 2 - The DeltaLog
3. Delta Lake essential Fundamentals: Part 3 - Compaction and Checkpoint
4. Delta Lake essential Fundamentals: Part 4 - Practical Scenarios (You are here)

Checkpoint

There are two known checkpoints mechanism in Apache Spark that can confuse us with DeltaLake checkpoint, so let’s understand them and how they differ from each other:

Spark RDD Checkpoint

Checkpoint in Spark RDD is a mechanism to persist current RDD to a file in a dedicated checkpoint directory while all references to its parent RDDs are removed. This operation, by default, breaks data lineage when used without auditing.

Structured Streaming Checkpoint

Structured Streaming is a scalable and fault-tolerant stream processing built on Spark SQL engine. The queries are processed using a micro-batch processing engine as a series of small-batch jobs. Structured Streaming enables exactly once fault-tolerant guarantees through checkpoint and writing ahead logs. The streaming engine record the offset range of the data that is being processed in each trigger. Hence if a trigger failed, we have the exact range of processed data there and can recover from it.

DeltaLake checkpoint

On each Delta Table state compute, Delta reads the JSON files discussed in Delta Lake essential Fundamentals: Part 2 - The DeltaLog. To avoid reading all the files and executing a long compute, every 10 commit files are being aggregated to a checkpoint file of type parquet. These checkpoint files save the entire state of the table at a point in time. It allows the Spark engine to avoid reprocessing thousands of tiny JSON files. This mechanism ensures that for computing table state, Spark only needs to read the latest parquet checkpoint file with up to 10 JSON files, it makes the computation faster and efficient. Checkout the visualization of Delta Checkpoint file from Databricks site:

checkpoint files can be a one file for a specific table version or multiple files, it depends on what it contains.

In one part, table version(n) 10 the file name will be of the structure n.checkpoint.parquet:

00000000000000000010.checkpoint.parquet

In multi-part, table version(n) 10 the files name will be of a structure that Fragment o of p: n.checkpoint.o.p.parquet:

00000000000000000010.checkpoint.0000000001.0000000003.parquet
00000000000000000010.checkpoint.0000000002.0000000003.parquet
00000000000000000010.checkpoint.0000000003.0000000003.parquet

Snapshot of the function that is in charge of the writing the checkpoint files, the modulo operation is in charge of the checkpointInterval which can be updated in DeltaConfig.

Delta Lake configuration can be set as a Spark Configuration property, or Hadoop configuration depends on the LogStore, the cloud used, etc.

I hope this provides more clarity into the differences between the three checkpoint mechanisms and their usage.

Next, let’s examine the compact files mechanism Delta recommends as part of its best practices:

Delta Lake Compact files

The same way Delta Lake handles its own small JSON DeltaLog files is creating, we as developers need to take care of the small files we might introduce to the system when adding data in small batches. Small batches can happen when we have Streaming workloads or continuous small batches of data ingesting without compacting it.

Small files can hurt the efficiency of table reads, and it can also affect the performance of the file system itself. Ideally, a large number of small files should be rewritten into a smaller number of larger files regularly. This is known as compaction.

We can compact a table by repartitioning it to a smaller number of files.

Delta Lake also introduces the ability to set the dataChange field false; this indicates that the operation did not change the data, only rearranges the data layout. But be careful with it, since if you are introducing a data change that is not only a layout, it can corrupt the data in the table.

For exploring and learning about Delta, you are invited in joining me by watching the videos. Let me know if that is useful for you, and we can schedule twitch as well.

What’s next?

Next, scenarios and use cases for DeltaLake!

As always, I would love to get your comments and feedback on Adi Polak 🐦.

If you would like to get monthly updates, consider subscribing.

Transaction Log

A transaction log is a history of actions executed by a (TaDa 💡) database management system with the goal to guarantee ACID properties over a crash.

DeltaLake transaction log - DetlaLog

DeltaLog is a transaction log directory that holds an ordered record of every transaction committed on a Delta Lake table since it was created. The goal of DeltaLog is to be the single source of truth for readers who read from the same table at the same time. That means, parallel readers read the exact same data. This is achieved by tracking all the changes that users do: read, delete, update, etc. in the DeltaLog.

DeltaLog can also contain statistics on the data; depending on the type of the data/field/column, each column can have min/max values. Having this extra metadata can help with faster querying. DeltaTable read mechanism uses a simplified push down predict.

Here is a simplification of DeltaLog on the file systems from Databricks site:

The DeltaLog itself is a folder that consists of multiple JSON files. When it reaches 10 files, DeltaTable does a checkpoint and compaction operations (we will dive into it in the next chapter).

Here is an example of a DeltaLog JSON file from the code source test resources, each entry in the file is on JSON:

{"remove":{"path":"part-00001-f1cb1cf9-7a73-439c-b0ea-dcba5c2280a6-c000.snappy.parquet","dataChange":true}}
{"remove":{"path":"part-00000-f4aeebd0-a689-4e1b-bc7a-bbb0ec59dce5-c000.snappy.parquet","dataChange":true}}

There was a total of two commits captured in this file: remove -it can be a delete operation on a whole column or only specific values in it. In this operation the metadata field dataChange is set to true.

Here is a more complex JSON file example, each entry in the file is on JSON:

{"metaData":{"id":"2edf2c02-bb63-44e9-a84c-517fad0db296","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["id"],"configuration":{}}}
{"remove":{"path":"part-00001-6d252218-2632-416e-9e46-f32316ec314a-c000.snappy.parquet","dataChange":true}}
{"remove":{"path":"part-00000-348d7f43-38f6-4778-88c7-45f379471c49-c000.snappy.parquet","dataChange":true}}
{"add":{"path":"id=5/part-00000-f1e0b560-ca00-409e-a274-f1ab264bc412.c000.snappy.parquet","partitionValues":{"id":"5"},"size":362,"modificationTime":1501109076000,"dataChange":true}}
{"add":{"path":"id=6/part-00000-adb59f54-6b8f-4bfd-9915-ae26bd0f0e2c.c000.snappy.parquet","partitionValues":{"id":"6"},"size":362,"modificationTime":1501109076000,"dataChange":true}}
{"add":{"path":"id=4/part-00001-36c738bf-7836-479b-9cc1-7a4934207856.c000.snappy.parquet","partitionValues":{"id":"4"},"size":362,"modificationTime":1501109076000,"dataChange":true}}

In this example, there is the metadata object entry - it represents a change in the table columns either an update to the table schema or that a new table was created. Later we see two remove operations, followed by three add operations. These operation objects can have a stat field, which contains statistical information, such as the number of records, minValues, maxValues, and more.

These JSON files might also contain operation objects with fields such as - “STREAMING UPDATE”, “NOTEBOOK” if the operation took place from a notebook, isolationLevel, etc.

This information is valuable for managing the table and avoiding redundant full scan on the storage.

To simplify the connection between DeltaTable and DeltaLog, it’s easier to think about DeltaTable as a direct result of a set of actions audited by the DeltaLog.

DeltaLog and Atomicity

From part one, you already know that atomicity means that a transaction, either happened or not. The DeltaLog itself consists of atomic operations; each line in the log (like the ones you saw above) represents an action, which is an atomic unit; These are called commits. The transactions that took place on the data can be broken into multiple components in which each one individually represents a commit in the DeltaLog. These breaking complex operations into small transactions help with ensuring atomicity.

DeltaLog and Isolation

Operations such as Update, Delete, Add can harm isolation; Hence, since we want to guarantee isolation with DeltaTable, readers only get access to the table snapshot. This guarantees all parallel readers read the exact data. For handling deletion operations, Delta postpones the actual delete operation on the files; it first tags the files as deleted and later, remove them when considered safe (similar to Cassandra, and ElasticSearch delete operations with a tombstone).

In DeltaLake 0.8.1 source code, there is a comment saying that it’s recommended to have the delete retention set to at least 2 weeks or longer than a duration of a job.
Note: This will impact streaming workload as well, because there will be a need to delete the actual files at some point, which might result in blocking the stream.

DeltaLog and Consistency

Delta Lake solves the problem of consistency by solving conflicts with an optimistic concurrency algorithm. The class in charge of this algorithm is the OptimisticTransaction class. It achieves it by using Java 8 ReentrantLock that is controlled from a DeltaLog instance.
Here is the code snippet:

A DeltaTable instance actively uses the ReentrantLock in the OptimisticTransaction under the doCommitRetryIteratively function. The optimistic approach was chosen here because in the big data world there is a tendency to add more data than to update existing records. It’s rare to find and update a specific record, it is usually done when there was some data corruption on necessary data.

Here is the code snippet for the optimistic algorithm:

Notice that in line 572, the program records the attempted version as the commitVersion instance which is of type var. var in Scala represents a mutable object instance, which means we should expect its value to change.

In line 575, we start the algorithm: it starts the while(true) loop and maintains an attemptNumber counter; if it’s ==0, it will try to commit; if it fails here, that means that a file with this commitVersion was already written/committed into the table and it will throw an exception. That exception is being caught in lines 592+593. From there, with each failure, the algorithm is increasing the attemptNumber by 1. After the first failure, the program won’t go into the first if statement on line 577; it will go straights into the else if on line 579. If the program reached the state where attemptNumber is bigger than the maximum allowed/configured, it will throw a DeltaErrors.maxCommitRetriesExceededException exception. maxCommitRetriesExceededException exception will provide information about the commit version, the first commit version attempt, the number of attempted commits, and total time spent attempting this commit in ms. Otherwise, it will try to record this update with checkForConflict functionality in line 588. Multiple scenarios can bring us to this state.

High-level pseudo-code:

while(tryCommit)
    if first attempt:
        do commit
    else if: attempt number > max retries
            throw an exception - exit loop
        else:
            record retry operation
            try fixing logical conflicts - return valid commit version or throw an exception
            do commit
    retry on exceptions and attempt version +1
    if no exception - end loop
end     

To support the users, DeltaLake introduces a set of conflict exceptions that provide more information about the data and the conflicts:

Let’s look at some of the conflict scenarios.

Two Writers:

This is the case of two writers who appends data to the same table simultaneously, without reading anything. In this scenario, one writer will commit, and the second writer will read the first one’s updates before adding their own updates. Suppose it was only an append operation, like a counter which both are incrementing. In that case, there is no need to redo all computations, and it will automatically commit; if that’s not the case, writer number two will need to redo the computation given the new information from writer one.

Delete and Read:

In a more complex scenario like this one, there is no automated solution. For concurrent Delete-Read, there is a dedicated ConcurentDeleteReadException. That means that if there is a request to delete a file that at the same time is being used for a read, the program throws an exception.

Delete and Delete:

When two operations delete the same file, it might be due to a compaction mechanism or other operation, here too an exception will occur.

DeltaLog and Durability

Since all transactions made on a DeltaTable are being stored directly to the disk/file system, durability is a given. All commits are being persisted to disk. In case of a system failure, they can be restored from the disk. (Unless there is a true disaster like fire etc and damage to the actual disks holding the information).

For exploring and learning about Delta, I did a deep dive into the code source itself. If you are interested in joining me, I captured it through videos, let me know if that is useful for you.

What’s next?

Next, we will see more examples, scenarios and use cases for DeltaLake! We will learn about the compaction mechanism, schema enforcement and how it can enforce exactly once operation.

As always, I would love to get your comments and feedback on Adi Polak 🐦.

If you would like to get monthly updates, consider subscribing.

What is Delta Lake ?

Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads.

DeltaLake open source consists of 3 projects:

1. detla - Delta Lake core, written in Scala.
2. delta-rs - Rust library for binding with Python and Ruby.
3. connectors - Connectors to popular big data engines outside Spark, written mostly in Scala.

Delta provides us the ability to “travel back in time” into previous versions of our data, scalable metadata - that means if we have a large set of raw data stored in a data lake, having metadata provides us with the flexibility needed for analytics and exploration of the data. It also provides a mechanism to unify streaming and batch data.
Schema enforcement - handle schema variations to prevent insertion of bad/non-compliant records, and ACID transactions to ensure that the users/readers never see inconsistent data.

Part 1 focuses on ACID Fundamentals:

ACID Fundamentals in Delta Lake:

Let’s break it down to understand what each means and how it translates in Delta:

Atomicity

The transaction succeeded or not, all changes, updates, deletes, and other operations either happened as a single unit or not. Think Binary, there is only yes or no - 1 or 0. In Delta, it means that a commit of a transaction happened, and a new transaction log file was written. Transaction log file name example - 000001.json, the number represents the commit number.

Consistency

A transaction can only bring the DB from one state to another; data is valid according to all the rules, constraints, triggers, etc. The transaction itself can be consistent but incorrect. To achieve consistency, DeltaLake relay on the commit timestamp that comes from the storage system modification timestamps. If you are using cloud provider storage such as Azure blob or AWS S3, the timestamp will come from the storage server.

Isolation

Transactions taking place concurrently result in an equals state as if transactions would have been executed sequentially. This is the primary goal of Concurrency control strategies. In Delta, after 10 commits, there is a merging mechanism that merges these commits into a checkpoint file. The checkpoint file has a timestamp. 1 second is being added to the modification timestamp to avoid flakiness. This is how it looks in the code base of Delta:

Durability

Once a transaction has been committed, it will remain committed even if the system fails. Think about writing to disk vs. writing to Ram memory. A machine can fail, but if the commit data was written to disk, it could be restored. Delta writes all the commits in a JSON file directly to the storage; it is not left floating in RAM space for too long.

What’s next?

After understanding ACID basics and a bit about the Transaction Log (aka DeltaLog), you are ready to take the next chapter!
In Diving deeper into to DeltaLog, how it looks like on disk, and the open-source code you need to be familiar with.

As always, I would love to get your comments and feedback on Adi Polak 🐦.

If you would like to get monthly updates, consider subscribing.

Let’s take a look at three exciting open sources:

Ray:

Ray is an open source, python based framework for building distributed applications. Their main audience is ML developers and Data Scientists who would like to accelerate their machine learning workloads using distributed computing. Ray was open sourced by UC Berkly RISELab, the same lab who created the AMPLab project, where Apache Spark was created. BTW, if you are curious, their next big 5 years project is all about Real-time Intelligence with Secure Explainable decision.

RayOnSpark is a feature that was recently added to Analytic Zoo, end to end data analytics + AI open sourced platform, that helps you unified multiple analytics workload like recommendation, time series, computer vision, nlp and more into one platform running on Spark, Yarn or K8S.

“RayOnSpark allows users to directly run Ray programs on Apache Hadoop*/YARN, so that users can easily try various emerging AI applications on their existing Big Data clusters in a distributed fashion. Instead of running big data applications and AI applications on two separate systems, which often introduces expensive data transfer and long end-to-end learning latency, RayOnSpark allows Ray applications to seamlessly integrate into Apache Spark* data processing pipeline and directly run on in-memory Spark RDDs or DataFrames.” Jason Dai.

To learn more about Ray and RayOnSpark, checkout Jason Dai article from RISELab publication.

Koalas:

From the Pandas docs: “pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.”

From the Koalas docs: “The Koalas project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.”

If you are familiar with exploring and running analytics on data with panads,
Koalas provides a similar API for running the same analytics on Apache Spark DataFrames.
Which makes it easier for Pandas user to run their workloads at scale.
When using it, notice the different versions of Koalas, many new versions are NOT available with Spark 2.4 and require Spark 3.0 cluster.

Koalas is built with an internal frame to hold indexes and information on top of Spark DataFrame.

To learn more about it, checkout Tim Hunter talk on Koalas from Spark Summit 2019.

Delta Lake:

Delta Lake is nothing new with the Spark ecosystem, but still many confuse Delta Lake to be a … DataBase! (DB) well.. delta lake is NOT a database. Detla Lake is an open source storage layer that brings ACID (atomicity, consistency, isolation, and durability) transactions to Apache Spark and Big data workloads but is not a DB! Just like Azure Blog storage and AWS S3 are not acting as databases, they are defined as storage.
Delta helps with ACID that is hard to achieve and a great pain point with distributed storage. It provides scalable metadata handling on the data itself. When combined with Spark this is highly useful due to the nature of Spark SQL engine the catalyst which uses this metadata to better plan and executed big data queries.

There is also data versioning through snapshot of the storage named Time Travel feature. I recommend being mindful with using this feature as saving snapshots and later using them might create an overhead to the size and compute of your data.

If you are curious to learn more about it, read here.

That’s it.

I hope you enjoyed reading this short recap on open sources for January 2021.
If you are interested in learning more and getting updates, follow Adi Polak on Twitter..

Today, you will learn how to take a web app (it can be any programming language,
we used Java & Kotlin) and distribute it using Kubernetes (K8s) and Virtual Kubelet (VK).

Well, if you don’t know yet why you should consider distributing your web app - read my post here.

So, you are probably asking yourself
“what is Kubernetes and what can I use it for?”
Just keep reading

Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. It is used to build distributed, scalable microservices.

It brings many new concepts and terminology we need to familiarize ourselves with, these are the very basics:

Basic Glossary:

Node - Hardware component. Often a VM hosted on a cloud and provide CPU and RAM resources to be used by the Kubernetes cluster.
Kubernetes Master - A node or nodes that are in charge of managing the Kubernetes cluster state.
Kubelet - Primary “node agents” that runs on each node. It manages the containers that were created by Kubernetes and runs on the node it manages.
It communicates with the K8S master.
Pod - hold one or more containers. Containers that share the same pod also share resources and network.
Pod can be in charge of containers on different nodes- different physical machines or virtual machine(VM).
It serves as unit of deployment, horizontal scaling, and replication.
PodSpec - Yaml or JSON file that describes the pod spec. It is used by kubelet to make sure that the containers are healthy and running according to expectations.
Cluster - Series of nodes connected together.
There are many more concepts and terminology but this is the basic that we need to understand virtual kubelet and to start using K8S.
Kubernetes API - Server (REST) that runs on the master node and speaks directly with the kubelets running on the nodes.\

In the chart from Kubernetes.io we can see the nodes and master:

Hey, where are the pods? Well, the pods can be part of the Deployment or the ReplicaSet.
The ReplicaSet/Deployment defines the replicas that are distributed among multiple nodes.

Here is another chart that shows the pods work from the new stack website:

Another diagram shows how ReplicaSet work with Deployment,
where Deployment can be view as a template for ReplicaSet with replicas default of 3.
Diagram from Nir Mata site:

How Kubernetes works?

Kubernetes manages N number of nodes and within those nodes, there are these kubelets.
Kubelets manage everything related to the node and the pods running on it. Pods are just a collection of containers.

When we take an app put it in a container, upload it to container registry and deploy it into Kubernetes.
It then deployed onto a VM somewhere that is managed by Kubernetes cluster in our case Azure Kubernetes Service (AKS).
We can see that VM and track it from the CLI and the UI - at that point,
there is no per-second or pay-as-you-go billing since it is the classic scenario of
managed K8S service where we pay for the machines in use even if we end up not using them.

What about Virtual Kubelet (VK)?

With Virtual Kubelet we don’t see the actual node only one virtual node for each service used.
It acts as an abstraction for us and can spin as many pods as needed.
Behind the scene, we can have multiple VMs but we will see only one for the specific service that we are using.
We are not exposed to the VMs running in the managed service that
we are using from the Virtual Kubelet.
Virtual Kubelet acts as a stand-in that help us proxy to other managed services with higher abstraction.

Virtual Kubelet is an open-source implementation of Kubernetes kubelet
with the purpose of connecting Kubernetes to other APIs.
It registers itself as a node and allows us to deploy unlimited amounts of pods and containers.
It gives us the ability to connect with serverless containers platforms as well.
Meaning we can take any stateless app, containerize it and provision it through
the pods and the Virtual Kubelet will manage it for us and will shift it to the
managed service. We don’t need to manage the infrastructure.
It can scales up or down - all managed by the service.
According to the managed service in use, we can benefit from Pay-as-you-Go accounts, flexible auto-scaling and many more.

When combining AKS with Azure Container Instances(ACI) you benefit from a fast orchestration of containers.
We combine the two using virtual nodes. Results in the automation of containers scheduling.
Scheduling in container context refers to the ability of the administrator to load a
service onto a host system that defined how to run a specific container.
Using ACI with virtual nodes results in faster provisioning of pods.

Virtual nodes can be used with AKS and are powered by the open-source Virtual Kubelet.

Pros:

✅ Fully managed solution of top of Kubernetes Allow us to connect to many managed solutions from various cloud providers in various regions.

✅ Pay exactly for what you use Managed solutions like ACI or AWS Fargate help us
scale up or down according to our needs without intervention from our side.

✅Portability Everywhere K8S runs, you can run your Virtual Kubelet and connect it with your managed service.

✅ Regions and other clusters From Virtual Kubelet you can leverage services that run on other regions and even other cloud providers.

Cons:

❗️ Security In general, you should always think about security.
Remember, Security is everyone job! The overall security aspect of using Kubernetes is pretty complex to begin with.
When adding Virtual Kubelet, one should be aware of security issues that can
arise from communicating with other services outside of the Kubernetes cluster and outside of the region/cloud provider.
If we decide to work with ACI or other internal services, we can establish an internal virtual
network from K8s cluster to ACI. This way we can eliminate this security concern.

Let’s get practical with a tutorial

In JVM world there are many frameworks that can help us create a web app fast. One that includes the server and UI.
Our app uses Spring boot. Spring Boot has many embedded features like server and more.
For the server options, we can pick from Tomcat, Jetty or Undertow.

So you are probably asking yourself, how to get started with Spring Boot?
go to Spring initializer site and download a template or download the demo app from this github repository.

In this tutorial, we will deploy a kotlin-spring app to a Virtual Node on K8s cluster. We will use the next services: AKS, ACR and ACI.

For the tutorial you will need:

1. Demo app
2. Azure free subscription
3. Azure CLI
4. AKS cluster

At this point we have an AKS cluster , an app to deploy to our cluster and CLI tools installed.
For the second phase, we will need an ACI account, Docker registry to store our app image (we will use Azure container registry - ACR

Our demo app comes with a docker file that defines the app already, so we can push it to ACR.
Navigate in the terminal or CMD into your app directory and run:

set ACR_NAME={acr name}
az login
az acr login --name $ACR_NAME
docker build --no-cache -t demo .
docker tag demo $ACR_NAME.azurecr.io/samples/demo
docker push $ACR_NAME.azurecr.io/samples/demo

This is the push process:

\

To test yourself - run your local docker with remote image

docker run -it --rm -p 8080:80 $ACR_NAME.azurecr.io/samples/demo

The docker container will start running locally and you will see something like this: you can stop it with ctrl+C.

Now let’s connect to our AKS cluster, for that we will need our resource group name and our AKS cluster name:

set RES_GROUP={resource group name}
set AKS_NAME={AKS name}
az aks get-credentials --resource-group $RES_GROUP --name $AKS_NAME

Verify the connection to the cluster

kubectl get nodes

we will get the list of our nodes, version, status and more.

Next we will create the authentication between the container registry (ACR) and AKS,
this is an important step, without it, AKS cluster will not be able to pull the image from the registry. We will do it using secret - follow this

in the tutorial you are running this - remember to take a note of them both!

# Output used when creating Kubernetes secret.
echo "Service principal ID: $CLIENT_ID"
echo "Service principal password: $SP_PASSWD"

Validate your connection and secret with logging into docker -

docker login $ACR_LOGIN_SERVER --username $CLIENT_ID --password $SP_PASSWD

If this is failing, AKS will not be able to pull the image, and later in the tutorial you will get this error got HTTP response status code 400 error code “InaccessibleImage”. Make sure to follow the tutorial in the link step by step.

Install connector:

For installing the connector and the ability to use virtual nodes, we will create a subnet in our network and will install an AKS cluster there with add-ons for virtual node. This is a more secure way since we create an internal network that is isolated from our bigger K8s cluster.
Follow the step-by-step here but don’t deploy the app - we will deploy our app instead.

For deploying the app run:

kubectl apply -f kotlin-spring-virtual-kublet-linux.yaml

This YAML file describes to K8s, pods and kubelet how we want our app to run,
what is the deployments and the services in use. Each deploy component in our file
starts with apiVersion followed by kind, metadata and spec In our file we have one service named- azure-spring-kotlin-front-virtual-service and one deployment named: azure-spring-kotlin-front-virtual. Under deployment and under spec -> template -> spec we have the configuration
for the node selector, we might have many nodes in our cluster, and we would like this app
to be deployed to our virtual node one and not to the rest.
For achieving this, under nodeSelector we describe the type by giving it the value of virtual-kubelet.
This specifies the pods and kubelet that we will deploy this app only on this specific type of node and no other.

Our second component is of kind service , it’s spec type is loadBalancer and
it will have an External API for the app so we can load it in our browser.
For doing it, we need to expose it first - notice that we are exposing the app and not the LoadBalancer itself since we can expose a deployment:

kubectl expose deployment azure-spring-kotlin-front-virtual --type=LoadBalancer --port 80 --target-port 8080

To find the External API run:

kubectl get services

And search for External API at azure-spring-kotlin-front-virtual entry.

How to debug:

Use the next commands to debug and get a hold of what is happening in the cluster:

kubectl get services
kubectl get pods
kubectl get deployment

From the commands above we will get the data and first statuses of the various component, after figuring out what failed we can run:

kubectl describe {pod/service/node} {name of pod/service/node}

This will give us a JSON back with information like events, under event we will see what failed, it can be - FailedSynch app status Terminated - which usually reflects that the app crashed and we should check the node logs using

kubectl logs {name of node}

There are many more commands to debug K8s cluster and this was just the tip of the iceberg. Feel free to play and investigate the API.

Have something to add that I forgot to mention? want to discuss more options? write in comments or send a DM on twitter.

Learn more 💡

👉🏼 Watch this video on Virtual Kubelet by Ria Bhatia and Scott Hanselman

👉🏼 Quickstart: Develop with Java on Kubernetes using Azure Dev Spaces

👉🏼 Java and Azure

👉🏼 Kubernetes and Apache Spark on Azure tutorial

This article originally appeared in Adi Polak’s Dev.to blog https://dev.to/adipolak/kubernetes-and-virtual-kubelet-in-a-nutshell-gn4.