From generating DNS logs to end-to-end implementation of structured streaming — DNS A quick note on DNS before we start describing DNS tunneling and how to detect it using Spark Structured Streaming. DNS stands for Domain Name System and even though you may not have explicitly used it though your machine certainly used it before getting to this page. When you clicked…

How to wrap a PySpark DataFrame? — In one of my projects, I needed to enhance the existing DataFrame functionality. One way would be to implement utility methods that can take the DataFrame and implement additional functionality as needed. …

How to broadcast tokenizer and use it with UDFs — In this short post, I would show how to use Hugging Face Tokenizer. Before going into details on how to apply tokenizer on a DataFrame using PySpark, let us first look at some simple tokenizer code. The above code simply loads a pre-trained tokenizer roberta-base. It prints out a…

in python i trust — verse 4 — list comprehension is some snazzy s**t make ma code smaller it’s called pying it in python i trust in python i trust List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each…

in python i trust — verse 1 — if you got a list or gotta dictionary add them up now that I call some witchery in python i trust in python i trust This is about a simple list and dictionary addition. Above prints out [1, 2, 3, 4, 5], which is merging two lists together. Starting…

Using python decorator to do random sampling — This post is more educational in terms of learning a few concepts together. I was wondering how can I use python decorators with a specific Data Science concept and the idea for this post was born. Let us first look into the random sampling on how it is done. Random Sampling The…

Why are they not the same? — “The DecoratorPattern is a pattern described in the DesignPatternsBook. It is a way of apparently modifying an object’s behavior, by enclosing it inside a decorating object with a similar interface. This is not to be confused with PythonDecorators, which is a language feature for dynamically modifying a function or class.”…