In this article, you'll learn about six of the most critical and "hottest" elements that make up or are associated with AI today. In addition, you'll learn the basics of the tools that will shape the future of trading. First in the biggest and wealthiest funds and then in smaller ones as well.
I invite you to take a journey into this near and far future of trading.
NLP
Natural Language Processing is the common name for many tools to analyze written and spoken text: company documents, press articles, news, analysis, web pages, social media posts, company’s product reviews.
Advanced NLP software recognizes context up to about a thousand words. That's a lot, and soon, there will be more.
NLP allows you to analyze many features of text, such as:
- whether the text about a particular company is positive or negative,
- whether it is clear and transparent or obscure and convoluted,
- whether the authors express themselves positively or negatively about the future.
When we analyze texts of reports and press statements, it turns out that all of the above elements can be a good indicator of future financial performance.
Social media texts
Already now, the analysis of posts in social media and online shops allows determining the sentiment - the opinion about the company and its products, which usually precedes the financial results.
Sometimes it is also possible to find and analyze the sentiment of different investor groups about the company and its future which also affects the share price.
"The stripper that will change our World"
The next gigantic step in the evolution of NLP come from extracting written knowledge from millions of books, academic articles and other texts and help create coherent theories of how the economy, or supply chains, works.
This development will give theoretical and practical insights into the factors that affect the financial performance of companies, industries and all relevant economic processes. Already today, we see the first signs of the creation of such tools.
Spoken text
NLP also covers spoken word analysis: statements from TV news, films, other video material, or telephone conversations are automatically transcribed and subject to the same analysis as written content.
Thanks to this, we now have access to knowledge about the level of Forex volume. The systems analyze the volume at major banks and brokers and traders' volume over the phone. Until last year, data on this was not available in real-time. I will find out if anything has changed yet and write about it in future issues. I know that there were plans to provide this volume in real-time as well.
Machine Learning
Machine learning is dozens of tools for machine problem-solving.
But today is different than you might think. We are in the first phase of the evolution of these tools. To understand this and to understand their potential, I will give a practical example: how does solving a problem using machine learning tools look like...
In simple terms:
1. the process consists of problem formulation and preparation of a mathematical model (specialist),
2. further collection and preparation of data (specialist),
3. selection of one of the ML solutions (specialist),
4. feeding the software with data (specialist),
5. data processing (software),
6. finally, we have the interpretation of the obtained result (specialist) - someone has to explain the result in non-mathematical terms.
Only one element of this sequence is automated - the fifth. All the rest requires the use of specialists' knowledge and experience.
Today, the real driving engine of AI is... the specialists. And it will remain so for a long time to come. "Real AI" is still very, very scarce.
Over time, each step will be done automatically. Only then will we see the true power of machine learning and AI. We are at the beginning of this journey, the first stage of evolution (and I believe there will be five).
I didn't want to start this thematic series by describing ML tools. I preferred to show their current place in general. In the future, I will describe some Machine Learning solutions and how they are used to create trading systems. I will also give examples of such systems so that you can form your own opinion about them.
What is worth knowing is that despite the impressive achievements of AI-related technology, this is just the beginning of this revolution.
It will change everything we know.
We are only at the beginning of the AI revolution. It will change everything we know.
XAI or Explainable AI
It is currently probably the hottest topic in AI.
Some ML tools are so complex that we don't know how the machine got the result, how it made the decision or the recommendation.
We call them "black box" for short - it's dark inside, and we don't know what's happening there. Nevertheless, the math behind it is excellent, and the results are often astounding.
So, we have a result, but we don't know how it was achieved. We don't know because the process leading to the development is very complex and has many steps. And if we don't understand "how it works", then several problems arise. I will describe them for the case where we have a black-box that gives input and output signals:
- beyond simply allocating a small % of capital to the position, risk management becomes problematic;
- we have little or no control over the position (except for the exit);
- we don't get the most out of a tool we don't trust. And this is a problem when we have spent several million in its creation;
- we don't know if a given series of losses is temporary because the market has changed, or maybe the system has stopped working for a given market. So it will only lose from now on.
And since the results are good, we will try to explain how it works in one way or another. The problem of finding an explanation and education for exploiting the potential of AI will also run through the following issues.
For a fund that employs traders, this problem is as practical as it gets.
A trader in his seventies would like to know how much money can be made on a "black box."
For example, let's take a trader who is 75 years old, active and, on top of that, a co-owner of a fund. And he would like to find out how "this new thing" works because it may be worth increasing the capital that this "new toy" has to use.
But how, without knowing what's going on inside, define the trading framework? What risks to assume, a reasonable capital commitment, when difficulties arise, and what to do when they do?
Moreover, after all, we have to adapt to the boss's scope of knowledge and experience. Thus, for example, we cannot start the lecture with the geometry of differentiable manifolds and Kullback - Leibler divergence for probability distributions (such mathematics can be used there) if he has no idea about it.
It is a fascinating problem. Important enough that we are preparing for publication a broader article on this topic: how to explain and help traders understand new tools, in particular black-boxes. How to estimate risk, build confidence, define a framework in which they will feel safe with the new device.
Someone who has an easygoing boss already thinks they can relax and not bother explaining the operating principle of their wonderful black box. But, unfortunately, this is not the case because other people on the horizon would like to know how it works.
There are several groups of such people.
Those who want or need to know what's going on inside
The first would be the law regulators and the courts. The Financial Supervisory Commission may want to know if, by any chance, the recent large positions, as claimed by us placed by a "black box", are not an instance of insider trading.
If you have a similar idea of defending yourself in court (from being accused of insider trading), then know that it makes no sense. We may not know what is inside, but the signal must appear again after simulating the conditions created by the signal.
Then we have the risk management department, which would also like to know how it works or at least what it resembles. All they have left is to allocate a small amount of capital to the signal.
A black box position is like a plane without windows. We take off on command, fly with no way to tell where we are and land on command. The only assurance of safety is the statistic that, for example, a position is profitable six times out of ten. This value means that we have four hard landings per every ten take-offs. It is a moderately comfortable situation, although in some cases, it will suffice.
Has the system already stopped working?
Now, we have something even less pleasant: if we do not know the rules of decision making, we cannot be sure that a given series of losses is not the end of the system because the market has changed and the previous rules no longer work. Exaggerated? Maybe, but only a little.
The use of various AI tools will only grow, including black-boxes, and this has to be dealt with one way or another. The major funds already have some prescriptions for what to do. In future articles, I will describe them.
The topic is even more important because, for the vast majority of non-mathematicians, i.e. traders, portfolio managers, C-level managers, practically every AI/ML tool is a black box. For some reason, explanations such as dealing with a multidimensional, differentiable manifold immersed in a vector space do not help.
Explainable AI fits into a broader trend - most people and traders have no idea what new tools do.
There is a great need to explain to users how AI/ML tools work, what they provide, their limits of use and when they stop working. Education is vital because a fund's competitive advantage will soon be created at the interface between the team and the tools, AI systems.
Competitive advantage in the future will depend on the so-called structural intelligence of the company. The largest funds are already working in this direction, although they can not name it so cleverly. We will also devote quite a few articles to this in the future, which is one of this magazine's goals.
If you liked this post, give it a boost 🚀 and drop a comment so we know to publish more for you. Cheers!
I invite you to take a journey into this near and far future of trading.
NLP
Natural Language Processing is the common name for many tools to analyze written and spoken text: company documents, press articles, news, analysis, web pages, social media posts, company’s product reviews.
Advanced NLP software recognizes context up to about a thousand words. That's a lot, and soon, there will be more.
NLP allows you to analyze many features of text, such as:
- whether the text about a particular company is positive or negative,
- whether it is clear and transparent or obscure and convoluted,
- whether the authors express themselves positively or negatively about the future.
When we analyze texts of reports and press statements, it turns out that all of the above elements can be a good indicator of future financial performance.
Social media texts
Already now, the analysis of posts in social media and online shops allows determining the sentiment - the opinion about the company and its products, which usually precedes the financial results.
Sometimes it is also possible to find and analyze the sentiment of different investor groups about the company and its future which also affects the share price.
"The stripper that will change our World"
The next gigantic step in the evolution of NLP come from extracting written knowledge from millions of books, academic articles and other texts and help create coherent theories of how the economy, or supply chains, works.
This development will give theoretical and practical insights into the factors that affect the financial performance of companies, industries and all relevant economic processes. Already today, we see the first signs of the creation of such tools.
Spoken text
NLP also covers spoken word analysis: statements from TV news, films, other video material, or telephone conversations are automatically transcribed and subject to the same analysis as written content.
Thanks to this, we now have access to knowledge about the level of Forex volume. The systems analyze the volume at major banks and brokers and traders' volume over the phone. Until last year, data on this was not available in real-time. I will find out if anything has changed yet and write about it in future issues. I know that there were plans to provide this volume in real-time as well.
Machine Learning
Machine learning is dozens of tools for machine problem-solving.
But today is different than you might think. We are in the first phase of the evolution of these tools. To understand this and to understand their potential, I will give a practical example: how does solving a problem using machine learning tools look like...
In simple terms:
1. the process consists of problem formulation and preparation of a mathematical model (specialist),
2. further collection and preparation of data (specialist),
3. selection of one of the ML solutions (specialist),
4. feeding the software with data (specialist),
5. data processing (software),
6. finally, we have the interpretation of the obtained result (specialist) - someone has to explain the result in non-mathematical terms.
Only one element of this sequence is automated - the fifth. All the rest requires the use of specialists' knowledge and experience.
Today, the real driving engine of AI is... the specialists. And it will remain so for a long time to come. "Real AI" is still very, very scarce.
Over time, each step will be done automatically. Only then will we see the true power of machine learning and AI. We are at the beginning of this journey, the first stage of evolution (and I believe there will be five).
I didn't want to start this thematic series by describing ML tools. I preferred to show their current place in general. In the future, I will describe some Machine Learning solutions and how they are used to create trading systems. I will also give examples of such systems so that you can form your own opinion about them.
What is worth knowing is that despite the impressive achievements of AI-related technology, this is just the beginning of this revolution.
It will change everything we know.
We are only at the beginning of the AI revolution. It will change everything we know.
XAI or Explainable AI
It is currently probably the hottest topic in AI.
Some ML tools are so complex that we don't know how the machine got the result, how it made the decision or the recommendation.
We call them "black box" for short - it's dark inside, and we don't know what's happening there. Nevertheless, the math behind it is excellent, and the results are often astounding.
So, we have a result, but we don't know how it was achieved. We don't know because the process leading to the development is very complex and has many steps. And if we don't understand "how it works", then several problems arise. I will describe them for the case where we have a black-box that gives input and output signals:
- beyond simply allocating a small % of capital to the position, risk management becomes problematic;
- we have little or no control over the position (except for the exit);
- we don't get the most out of a tool we don't trust. And this is a problem when we have spent several million in its creation;
- we don't know if a given series of losses is temporary because the market has changed, or maybe the system has stopped working for a given market. So it will only lose from now on.
And since the results are good, we will try to explain how it works in one way or another. The problem of finding an explanation and education for exploiting the potential of AI will also run through the following issues.
For a fund that employs traders, this problem is as practical as it gets.
A trader in his seventies would like to know how much money can be made on a "black box."
For example, let's take a trader who is 75 years old, active and, on top of that, a co-owner of a fund. And he would like to find out how "this new thing" works because it may be worth increasing the capital that this "new toy" has to use.
But how, without knowing what's going on inside, define the trading framework? What risks to assume, a reasonable capital commitment, when difficulties arise, and what to do when they do?
Moreover, after all, we have to adapt to the boss's scope of knowledge and experience. Thus, for example, we cannot start the lecture with the geometry of differentiable manifolds and Kullback - Leibler divergence for probability distributions (such mathematics can be used there) if he has no idea about it.
It is a fascinating problem. Important enough that we are preparing for publication a broader article on this topic: how to explain and help traders understand new tools, in particular black-boxes. How to estimate risk, build confidence, define a framework in which they will feel safe with the new device.
Someone who has an easygoing boss already thinks they can relax and not bother explaining the operating principle of their wonderful black box. But, unfortunately, this is not the case because other people on the horizon would like to know how it works.
There are several groups of such people.
Those who want or need to know what's going on inside
The first would be the law regulators and the courts. The Financial Supervisory Commission may want to know if, by any chance, the recent large positions, as claimed by us placed by a "black box", are not an instance of insider trading.
If you have a similar idea of defending yourself in court (from being accused of insider trading), then know that it makes no sense. We may not know what is inside, but the signal must appear again after simulating the conditions created by the signal.
Then we have the risk management department, which would also like to know how it works or at least what it resembles. All they have left is to allocate a small amount of capital to the signal.
A black box position is like a plane without windows. We take off on command, fly with no way to tell where we are and land on command. The only assurance of safety is the statistic that, for example, a position is profitable six times out of ten. This value means that we have four hard landings per every ten take-offs. It is a moderately comfortable situation, although in some cases, it will suffice.
Has the system already stopped working?
Now, we have something even less pleasant: if we do not know the rules of decision making, we cannot be sure that a given series of losses is not the end of the system because the market has changed and the previous rules no longer work. Exaggerated? Maybe, but only a little.
The use of various AI tools will only grow, including black-boxes, and this has to be dealt with one way or another. The major funds already have some prescriptions for what to do. In future articles, I will describe them.
The topic is even more important because, for the vast majority of non-mathematicians, i.e. traders, portfolio managers, C-level managers, practically every AI/ML tool is a black box. For some reason, explanations such as dealing with a multidimensional, differentiable manifold immersed in a vector space do not help.
Explainable AI fits into a broader trend - most people and traders have no idea what new tools do.
There is a great need to explain to users how AI/ML tools work, what they provide, their limits of use and when they stop working. Education is vital because a fund's competitive advantage will soon be created at the interface between the team and the tools, AI systems.
Competitive advantage in the future will depend on the so-called structural intelligence of the company. The largest funds are already working in this direction, although they can not name it so cleverly. We will also devote quite a few articles to this in the future, which is one of this magazine's goals.
If you liked this post, give it a boost 🚀 and drop a comment so we know to publish more for you. Cheers!
