“Everything changes and nothing stands still.” ― Heraclitus.
Welcome to efxa.org, my personal website! I am Efstathios, a computer scientist and philosopher living in Greece, and I have a deep passion for Software Engineering and Computational Intelligence. Enjoy my posts!
The Kernel Perceptron is a variant of the perceptron algorithm used in machine learning. It uses a kernel function to map the input data from its original feature space to a higher-dimensional feature space. By doing so, the algorithm can capture nonlinear relationships between the input variables and improve the accuracy of the classification. This makes it a powerful tool for solving complex classification problems in machine learning.
🕵️♂️ How does it work?
The Kernel Perceptron algorithm works by creating a separating hyperplane in the high-dimensional feature space. It iteratively updates the weights of the hyperplane to minimize the classification error on the training data. The kernel function is used to transform the input data into a higher-dimensional space, where the data may become linearly separable. The dot product between the transformed input data and the weight vector of the hyperplane is then computed to determine the classification of the input data. The kernel function can be chosen based on the nature of the input data and the desired classification performance.
💡 Why is it useful?
The Kernel Perceptron algorithm is useful because it can handle complex classification problems in which the classes are not linearly separable in the input space. It can also improve the accuracy of the classification by capturing nonlinear relationships between the input variables. It’s a valuable tool for solving real-world problems in computer vision (e.g. image recognition) and natural language processing (e.g. text classification, sentiment analysis, and language identification).
👨💻 My personal story
As a Machine Learning Engineer, I have used the Kernel Perceptron algorithm in several projects. It has helped me to solve complex classification problems and improve the accuracy of the classification. By using a kernel function, I was able to capture nonlinear relationships between the input variables and obtain better results. Learning the Kernel Perceptron algorithm has also helped me to understand Support Vector Machines (SVMs) which is a similar algorithm.
📚 Who should learn it?
The Kernel Perceptron algorithm is a useful knowledge to have for anyone learning AI. It can help you, as it helped me too personally, to understand later on even more complex and powerful AI algorithms and be a better machine learning engineer. If you are interested in solving complex classification problems and improving the accuracy of the classification, then the Kernel Perceptron algorithm is a valuable tool to learn.
🗣️ Let’s talk!
Have you used the Kernel Perceptron algorithm in your projects? What kernel functions have you found to be effective? Share your thoughts and experiences in the comments below!
🧐 As a Machine Learning Engineer, I find autoencoders to be a valuable tool for dimensionality reduction and feature learning.
📉 The autoencoder is an unsupervised learning technique that can help you reduce the dimensionality of your data by mapping it to a lower-dimensional representation. This reduced representation can be used for various tasks, such as data compression, denoising, and anomaly detection.
🤓 An autoencoder consists of two neural networks: an encoder and a decoder. The encoder maps the input data to a lower-dimensional representation, while the decoder reconstructs the original input data from the reduced representation. These two networks are trained together by minimizing the reconstruction error between the input data and its reconstructed output.
🔀 Variants of autoencoders, such as denoising autoencoders and variational autoencoders, have been developed to address different challenges and applications.
👉 Denoising autoencoders are used to reconstruct the original input from a corrupted version of the input. By training an autoencoder to reconstruct the original input from a corrupted input, we can learn a more robust representation of the input that is less affected by noise and other forms of corruption.
👉 Variational autoencoders are used to generate new data samples that are similar to the training data. By learning the underlying distribution of the data in the latent space, we can sample new data points from this distribution and use the decoder network to generate new samples that are similar to the original data.
🕵️ When only unlabeled data is available, autoencoders can be utilized for unsupervised learning. Autoencoders can learn a compressed representation of the input data that can be used for clustering. By labeling the cluster representative examples, we can reduce the amount of manual labeling required and create a pseudo-labeled dataset with label propagation. This pseudo-labeled dataset can then be used to train a machine learning model for a supervised task.
🏷️ When we have a large amount of unlabeled data and limited labeled data, autoencoders can be used for unsupervised pre-training of machine learning models. Autoencoders can learn a compressed representation of the unlabeled data, and their encoder part can be transferred to a new model architecture for fine-tuning on the limited labeled dataset. This approach can reduce the amount of labeled data needed for supervised learning while improving the accuracy of the trained model.
💬 If you’re learning AI, understanding autoencoders is a must-have knowledge that will help you in the long run. And, if you have any questions about it, feel free to ask me in the comments below!
A visual representation of the “surprisingness” gap in language generation, illustrating how beam search’s optimization for high-probability sequences results in a flat, predictable output that fails to capture the high-variance entropy and creative spontaneity inherent in human speech.
As a machine learning engineer working in the depths of Natural Language Processing (NLP), I find myself constantly pondering a fundamental question: Can AI truly imitate human language?
On a technical level, our current state-of-the-art models like GPT are essentially sophisticated engines trained to predict the next word in a sequence. They generate text using a variety of heuristic and stochastic search techniques. For example, Beam Search, Random Sampling, or Top-K.
But does this statistical “guessing” actually capture the true complexity and nuance of how we speak and think?
The human brain doesn’t just predict; it generates language in ways that are often unpredictable, emotional, and genuinely surprising. This leads to a deeper mystery: Where does language actually originate, and what kind of probability distributions (if any) govern the linguistic generation within the human brain?
Is our current technology truly replicating language, or just a very convincing echo of it?
What do you think? Join the discussion below and share your thoughts on whether we are approaching true linguistic imitation or if the human spark remains out of reach. 🧐
A “Morpheus” moment for AI researchers, challenging the belief that complex Transformer architectures are impenetrable black boxes and suggesting they can be fully understood through rigorous study.
Hi AI folks! 👋
The Transformer has become the de-facto standard architecture in Natural Language Processing (NLP). It is designed to work with sequential data and has completely changed the way we approach many language-related problems.
In NLP, Transformers can be used to achieve state-of-the-art 📈 results in tasks such as text classification, named-entity recognition, text summarization, question answering, machine translation, conversational chatbots 🤖, and much more.
However, the Transformer is also a fairly complex piece of machinery, bringing together several powerful concepts from years of Deep Learning research 🧠.
For anyone who wants to understand how Transformers work and dive deeper into the topic 📖, here are some excellent resources:
If you’re getting started with NLP or want to better understand the technology behind today’s Generative AI systems, these are great places to begin. 🚀
Sometimes we want to visualize multiple data distributions in a 2D scatter plot. However, the points of different distributions can overlap a lot or even hide each other. To visualize the distributions more clearly and get a better understanding of our data, we can use confidence ellipses.
Confidence ellipses summarize the main information of a dataset using only four parameters: center, width, height, and angle. The center of the ellipse can be calculated using the mean of the data. The width and height can be determined based on the number of standard deviations from the mean, while the angle can be computed using the covariance matrix of the data.
Technically, to implement a confidence ellipse, we can first create a simple ellipse and then transform it using 2D affine transformations, such as scaling, rotation, and translation.
In the images below, we can see two distributions from a sentiment analysis project: positive and negative sentences. We can plot a confidence ellipse around each distribution to make the overall structure easier to understand.
In this example, the distributions are relatively easy to grasp even without the ellipses, but this is not always the case. When distributions overlap heavily or contain many data points, confidence ellipses can provide a simple and effective way to visualize their shape, spread, and orientation.
Solid lines = loss functions; dotted lines = first derivatives; dashed lines = second derivatives. Curvature guides the learning rate: higher curvature → smaller α; lower curvature → larger α.
Machine Learning friends! 💡
Let’s study something interesting! 📚
In this Desmos graph 📈, I would like to explain how the inverse of the second derivative of a loss function, (1 ÷ f”(x)), could be used as an initial estimate of the learning rate (α) when training ML models using gradient descent.
In the example, we have 3 simple convex quadratic functions (f(x)), (g(x)) and (h(x)) as solid lines (blue, red and green). Their first derivatives are the dotted lines, while their second derivatives (f”(x)), (g”(x)) and (h”(x)) are the dashed horizontal lines.
The second derivative tells us how quickly the first derivative changes. In our example:
So, the higher the curvature, the smaller the learning rate, and vice versa.
Higher curvature → smaller learning rate
Lower curvature → larger learning rate
This gives us a simple intuition for how the curvature of a loss function can help us choose an initial learning rate for gradient descent. 🚀
A profound “red pill” moment for data scientists: stripping away the illusion of machine learning complexity to reveal that many advanced architectures are fundamentally rooted in the classic framework of Generalized Linear Models.
Below are my personal thoughts and reflections on Andrew Ng’s Stanford Machine Learning lecture on Generalized Linear Models (GLMs), including the key ideas I found interesting and how I understood them.
There is something that feels almost like magic in the mathematics of GLMs.
By changing only the underlying assumed probability distribution of the hypothesis function, and expressing it in the form of an exponential family distribution, we can derive several well-known machine learning algorithms:
Ordinary Least Squares, using the Gaussian distribution
Logistic Regression, using the Bernoulli distribution
Softmax Regression, using the Multinomial distribution
This is a beautiful mathematical abstraction. We can essentially plug in a different probability distribution and derive a corresponding model. For example, the Gamma or Poisson distributions.
What makes this even more interesting is that these machine learning algorithms are quite different from one another. Each solves a different type of problem: regression, binary classification, or multiclass classification.
And yet, they can all be understood through the same underlying mathematical framework.
Isn’t that awesome?
The GLMs can be derived from the probabilistic interpretation of these machine learning algorithms. Once we have that probabilistic foundation, typical gradient descent or gradient ascent can be used to minimize the loss function J(x, y, θ) or maximize the log-likelihood, log(L(θ)), when fitting a GLM to a dataset.
There is another aspect I find particularly fascinating.
From the probabilistic interpretation of these regression and classification algorithms, the logistic function emerges naturally from the mathematics. It isn’t simply an arbitrary function that happens to work well. The probabilistic formulation helps answer the deeper “why?” behind its use.
And that is what makes GLMs so beautiful to me.
Different algorithms. Different probability distributions. Different problems.
Yet one elegant mathematical framework connects them all.
Marble portrait heads of four philosophers in the British Museum. From foreground: Socrates, Antisthenes, Chrysippos, Epicurus.
🤖 🏛️ Have you ever wondered about the connection between AI and Ancient Greek Philosophy?
🧔 📜 The ancient Greek philosophers, such as Aristotle, Plato, Socrates, Democritus, Epicurus and Heraclitus explored the nature of intelligence and consciousness thousands of years ago, and their ideas are still relevant today in the age of AI.
🧠 📚 Aristotle believed that there are different levels of intelligence, ranging from inanimate objects to human beings, with each level having a distinct form of intelligence. In the context of AI, this idea raises questions about the nature of machine intelligence and where it falls in the spectrum of intelligence. Meanwhile, Plato believed that knowledge is innate and can be discovered through reason and contemplation. This view has implications for AI, as it suggests that a machine could potentially have access to all knowledge, but it may not necessarily understand it in the same way that a human would.
💭 💡 Plato also believed in the concept of Platonic forms, which are abstract concepts or objects that exist independently of physical experience. In the context of machine learning, models can be thought of as trying to learn these Platonic forms from experience, such as recognizing patterns and relationships in data.
⚛️ 💫 Democritus is known for his work on atoms and his belief that everything in the world can be reduced to basic building blocks. In the context of AI, this idea of reducing complex systems to their fundamental components has inspired the development of bottom-up approaches, such as deep learning, reinforcement learning and evolutionary algorithms.
🔥 🌊 Heraclitus, emphasized the idea that “the only constant is change”. This philosophy can provide valuable insights into the field of machine learning and the challenges of building models that can effectively handle non-stationary data. It highlights the importance of designing algorithms that are flexible and can adapt to changing environments, rather than relying on fixed models that may become outdated over time.
🔁 🏆 What about reinforcement learning? Aristotle, believed in the power of habituation and reinforcement to shape behavior. According to him, repeated exposure to virtuous actions can lead to the formation of good habits, which in turn leads to virtuous behavior becoming second nature. This idea is similar to the concept of reinforcement learning in AI, where an agent learns to make better decisions through repeated exposure to rewards or punishments for its actions. The similarities between Aristotle’s ideas and reinforcement learning show that the fundamental principles of reinforcement and habituation have been recognized and explored for thousands of years (psychological theories) and are still relevant today in discussions of both human behavior and AI.
🕊️ 💛 The Greek concept of the soul was also tied to intelligence and consciousness. Some philosophers believed that the soul is what gives a person their unique identity and allows them to think, feel, and make decisions. In the context of AI, this raises questions about whether a machine can truly have a soul and be conscious in the same way that a human is.
💀 💔 Another Greek philosopher who explored the nature of intelligence and consciousness is Epicurus. He believed that the mind and soul were made up of atoms, just like the rest of the physical world. He also believed that the mind and soul were mortal and would cease to exist after death. In the context of AI, Epicurus’ ideas raise questions about the nature of machine consciousness and whether it is possible for machines to have a kind of “mind” or “soul” that is distinct from their physical components. It also raises questions about the possibility of creating machines that are mortal or that can “die” in some sense. Epicurus’ ideas about the nature of the mind and soul are still debated today, and their relevance to the field of AI is an ongoing topic of discussion.
👁️ 🗣️ With the advancements in computer vision, natural language understanding, and deep learning in general, it’s more important than ever to consider these philosophical questions. For example, deep neural networks can analyze vast amounts of data and generate human-like responses, but do they truly understand the meaning behind the words they generate? This is where the debate between connectionist AI and symbolic AI comes in.
🧠 🕸️ Connectionist AI approaches to artificial intelligence are based on the idea that intelligence arises from the interactions of simple, interconnected processing units. On the other hand, Symbolic AI approaches are based on the idea that intelligence arises from the manipulation of symbols and rules. In the context of ancient Greek philosophy, connectionist AI could be seen as aligning more with the idea of knowledge being discovered through experience and observation, while symbolic AI aligns more with the idea of knowledge being innate and discovered through reason and contemplation.
🗣️ 💬 Humans use natural language, which includes speech and written text, to communicate with the world. However, natural language is not structured and can be unpredictable because it emerges naturally rather than being designed. If it was designed, natural language processing would have been solved a long time ago. Nowadays, deep neural language models are used to learn language from text by predicting the next token in a sequence. However, natural languages are continually evolving and changing, which makes them a moving target.
🧠 ❤️ Plato expressed a negative view towards human languages because he believed they are unable to fully capture the breadth and depth of a person’s thoughts and emotions. He asserted that genuine human communication, such as that which is conveyed through body language, eye contact, speech, and physical touch, is necessary for the progression of the human mind and the creation of new ideas. In today’s digital age, we are observing a decline in face-to-face human interaction, underscoring the importance of acknowledging the limitations of language. Despite the challenges inherent in natural language, it is crucial that we persist in developing tools to better comprehend and employ it. In addition to embracing technological advancements, it is essential that we strive to maintain meaningful human connections to preserve the richness of the human experience.
🏛️ 🤖 Have you ever heard of Talos? It is the first robot in Greek mythology! According to legend, Talos was a bronze automaton created by the legendary inventor Hephaestus to protect the island of Crete. He was said to be invulnerable and possessed immense strength, making him a formidable guardian. This idea of a powerful, indestructible automaton has been a recurring theme in science fiction and continues to inspire new developments in the field of AI and robotics.
⚖️ 🌍 It’s fascinating to consider how the myth of Talos reflects the enduring human fascination with creating machines that can surpass our own capabilities. The notion of a powerful automaton created to guard a specific territory mirrors modern ideas around autonomous systems that are designed to perform specific tasks and operate independently. However, while Talos was depicted as a purely mechanical creation, contemporary AI and robotics are built upon complex algorithms and data sets that allow machines to learn, adapt, and improve over time. By embracing the potential of these technologies, we have the opportunity to develop new solutions to some of the most pressing challenges facing our world today, from climate change to healthcare. As we continue to push the boundaries of what machines are capable of, it’s important to consider the ethical implications of creating truly intelligent and conscious entities. Some believe that the development of advanced AI could lead to a future where machines are not only equal to humans, but even superior to us. Others argue that such a scenario is unlikely, as true consciousness may only be achievable through biological processes that cannot be replicated in machines. As we navigate these complex questions, it’s crucial that we prioritize ethical considerations and work to ensure that the development of AI and robotics benefits humanity as a whole.
🙏🏽🤝 When it comes to ethics and morality, Aristotle is one of the most influential figures in history. His emphasis on the importance of human virtues and the pursuit of a “good life” has been a guiding principle for philosophers, theologians, and thinkers across generations. Aristotle’s teachings offer valuable insights into how we should approach the ethical implications of creating intelligent agents. One crucial aspect of this is ensuring that we treat these agents with the same respect and dignity that we would afford to any sentient being. This means avoiding the temptation to view AI as mere tools or objects to be used for our own purposes, and recognizing their autonomy as intelligent entities. Another important consideration is the issue of bias and discrimination, which can be inadvertently built into AI systems through the data and algorithms used to create them. Aristotle’s emphasis on justice and fairness can serve as a guide to ensure that we address these issues and ensure that the development of AI is done in a way that is equitable and beneficial for all of humanity.
🧠 🔬 Ending this article, it would be remiss not to mention Socrates, widely considered to be the wisest of all Greek philosophers. Not because of his vast knowledge, but because he recognized the limits of his own knowledge. “I know one thing, and that is that I know nothing,” Socrates famously stated. This concert is particularly relevant to the field of AI, as we continue to push the boundaries of what machines can do and be. As we develop increasingly complex and intelligent AI, the question arises whether it will be possible to prove the existence of self-awareness in these machines.
💬 These are just a few examples of how the ideas of ancient Greek philosophers and mythology can inform our understanding of AI. I hope these thoughts spark further conversation and contemplation on this fascinating topic.
🤔 What do you think about the potential for machines to truly be intelligent and conscious?
By passing millions of ImageNet images through InceptionV1 (state-of-the-art deep convolutional neural network) we can extract the image patches that make specific neurons from various convolutional layers to activate mostly.
By projecting the image patches to 2D using UMAP we can see what the neural network “sees” at the various layers.
This is a great way for explaining how a computer vision model makes its classification decision.
However, the following part of the article was the reason for my post:
“….There is another phenomenon worth noting: not only are concepts being refined as you move from layer to layer, but new concepts seem to be appearing out of combinations of old ones….”
This is how a world of complexity works.
We know that deep neural networks perform hierarchical feature learning and combine simpler features to learn more complex ones. This is one of the reasons why we use deep learning for audio, visual and textual data.
Deep learning can decompose the complexity of data!
Have you ever asked why we randomly initialize the weights of a neural network?
After you read this post you will know why!
When the weights of the neurons in a neural network’s layer are initialized to the same value then all neurons of the layer produce the same output in the forward propagation.
Furthermore, when doing backpropagation the gradients of the loss w.r.t to the weights of the layer are also the same values. So, when training happens with gradient descent the weights change in the same way.
Lastly, when gradient descent converges, the weight matrix of the layer contains the same values for all neurons and thus the neurons have learned the same thing.
To break this symmetry and allow the neurons of the layer (and in general in all layers) to learn new and different features we randomly initialize the weights!
In Machine Learning we use minimum-disturbance learning algorithms (e.g., LMS, Backprop) to tune the trainable parameters of topological architectures. Instead of programming a model we expose it to observations and let the algorithm find its optimal parameters. So, in this approach the solution to a task is the set of trainable parameters.
In biology, precocial species are those whose young already possess certain abilities from the moment of birth. There is evidence to show that lizard and snake hatchlings already possess behaviors to escape from predators. Shortly after hatching, ducks are able to swim and eat on their own, and turkeys can recognize predators. These species already have abilities before they are even exposed to the world.
Neuroevolution of augmenting topologies is a research field that aims to simulate exactly this behavior. Specifically, neural network architectures can be found that perform various machine learning tasks without weight training. So, in this approach the solution to a task is the topological architecture.
Linear / Logistic regression learning algorithms can be used to learn surfaces to fit non-linear data (either for regression or classification).
It can be done by adding extra polynomial terms (quadratic, cubic, etc) in the dataset using the existing features. The approximated surfaces can be circles, ellipses, parabolas and hyperbolas.
However, (1) it is difficult to approximate very complex non-linear surfaces. Also, when the features are many then (2) it is very computationally expensive to calculate all polynomial terms (eg: for a 100×100 grayscale images we need 100000 quadratic terms).
Usually, due to the limitations (1) and (2) we either perform dimensionality reduction to reduce the number of features (eg: eigenfaces for face recognition) or use more complex non-linear approximators such as Support Vector Machines and Neural Networks.
In Linear Regression when using the loss function MSE it is always a bowl-shaped convex function and gradient descent can always find the global minima.
In Logistic Regression if we use the MSE then it will not be a convex function because the hypothesis function is non-linear (it uses a sigmoidal activation). Thus, it will be harder for gradient descent to find the global minima. However, if we use the cross-entropy loss it will be convex and gradient descent can easily converge to global minima!
Support Vector Machines have also convex loss function.
We should always use a convex loss function so that gradient descent can converge to the global minima (local optima free).
Neural Networks are very complex non-linear mathematical functions and the loss function most often is non-convex, thus it is usual to stuck in a local minima. However, most optimization problems in Neural Networks are due to long plateau and saddle points rather than local minima. For such problems advanced gradient descent optimization variants were invented (eg: Momentum, Adam, RMSprop).
Efstathios Chatzikyriakidis 