Applying neural networks on domain name analysis to detect phishing — Domain Name Analysis StreamingPhish is one of the implementations of domain name analysis. The idea behind domain name analysis is to train curated data of known benign and phishing domains. In StreamingPhish a feature vector of 348 dimensions is derived from the domain name and then trained using LogisticRegression. …

From generating DNS logs to end-to-end implementation of structured streaming — DNS A quick note on DNS before we start describing DNS tunneling and how to detect it using Spark Structured Streaming. DNS stands for Domain Name System and even though you may not have explicitly used it though your machine certainly used it before getting to this page. When you clicked…

How to wrap a PySpark DataFrame? — In one of my projects, I needed to enhance the existing DataFrame functionality. One way would be to implement utility methods that can take the DataFrame and implement additional functionality as needed. …

How to broadcast tokenizer and use it with UDFs — In this short post, I would show how to use Hugging Face Tokenizer. Before going into details on how to apply tokenizer on a DataFrame using PySpark, let us first look at some simple tokenizer code. The above code simply loads a pre-trained tokenizer roberta-base. It prints out a…

in python i trust — verse 4 — list comprehension is some snazzy s**t make ma code smaller it’s called pying it in python i trust in python i trust List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each…

in python i trust — verse 3 —

in python i trust — verse 2

in python i trust — verse 1 — if you got a list or gotta dictionary add them up now that I call some witchery in python i trust in python i trust This is about a simple list and dictionary addition. Above prints out [1, 2, 3, 4, 5], which is merging two lists together. Starting…

Using python decorator to do random sampling — This post is more educational in terms of learning a few concepts together. I was wondering how can I use python decorators with a specific Data Science concept and the idea for this post was born. Let us first look into the random sampling on how it is done. Random Sampling The…