Visualising the difference.
Let's take a look at the security function, the differences between version 3 and version 2, and hopefully help give people a better understanding of how these work. As many will know there are differences in how version 2 and version 3 handle the "security()" function. Confusion around the mechanics of this function can lead to headaches for people testing scripts and trying to intuit how it works. So we’re just going to run through some examples to give a visual along with some explanations.
First, let’s look at 2 instances of the following code, one in version 2, one in version 3.
//@version=2
study("My Script", overlay=false)
num = security(tickerid, "60", n)
bgcolor(num%2==0?orange:na)
plot(num)
This will use the built in variable “n” on the 60 minute timeframe using the security function. The variable “n” assigns each bar a sequential number. We will use a 5 minute resolution for our chart and plot the 60minute bar number and change the background color to orange if the bar number is an even number. Let’s see how that looks.
So there’s 2 important points to mention here.
The first is that the highlighted areas are almost exact opposites to each other. This is because in version 3 the security function uses the value of the previous bar when looking back at historical data. So in our example, at the time that version 2 painted the 14454 bar, that was the correct bar number. In version 3, it was using the previous bar of 14453. So with historical data version 3 essentially has 1 bar of lag. This is to avoid issues of the bar using “future data”, which we will show an example of shortly.
The second thing to note is the time at which the bar begins. Using the above example again, notice that the beginning of the new 60min bar has a 1 bar difference on the current timeframe. In version 2 the new bar starts at exactly 09:00, whereas in version 3 it starts at 08:55. Note that this is because we are using the 5 minute chart, if we were using a 3 minute chart the version 3 bar would begin at 08:57, or a 15 minute chart would begin at 08:45.
Why? Well, perhaps the following chart will help explain. We will use the following simple bit of code that again use a 5min chart and plot the 60min high value. The purple line is the code in version 2, the green line is the code in version 3.
h = security(tickerid, "60", high)
plot(h)
There’s a lot going on here but we’ll go through it step by step. The first thing to notice to make sense of this picture is that the green bar is exactly the same as the purple bar, but just shifted to the right. This is the lag we mentioned, because the green bar (version 3) is using the value from the previous 60 min period.
Now, notice the red circled areas. These areas show the beginning of the new bar for version 3 and the end value of the bar for version 2. **In version 3, the new bar begins as soon as we know the final value of the previous bar.** So using our first chart example, the final closing value of the 08:00 – 09:00 period is the closing value of the 08:55 bar (on 5min chart), so the 08:55 value is where the new version 3 bar begins.
The version 2 bar uses future data as default. You can see examples of this where the orange ellipses are on the chart above. Remember, the purple line is charting the high of the current period, but with version 2 the high of that hour is painted on the chart before that highest value is reached, as highlighted in orange. Using the high value from our security function at any of those highlighted points would give us inaccurate back testing results because our indicator would essentially be looking into the future.
So what are the practical implications of this? Well, it means if you’re using version 2 you’re going to get inaccurate back test results because of the future data issue, which is the main reason this was changed for version 3. If you’re using version 3, however, that’s not a magical fix either. For instance if you’re using an hourly chart and pulling the daily data with the security function, the “daily” value last Wednesday will actually be using the values from last Tuesday. So with our examples of using the period “high”, it will be possible for the current hourly high to be above the security data’s daily high, because it’s using the previous day high.
Difference between real time and historical data
Everything we’ve talked about so far has been dealing with historical data.
Real time data for both version 2 and version 3 work the same way and work correctly. That is to say if you’re using a 5min chart and pull the data from the hourly with security, then the real time data from the security function will match the real time data from an hourly chart.
But how does that work if the version 3 data uses the value from the previous period? Well, as soon as you plot security values in a chart you’ll notice that the newest bar value will change. As soon as new data starts getting painted it will switch from the previous period’s value to the current value.
For example, this is a minute chart plotting the 3 minute high via the security function. Purple values are version 2, green are version 3, and you’ll see as soon as we hit real time data (when we clicked “add to chart”, signified by the pink dotted line) the 2 are identical. The version 3 data switches from lagging by 1 period to real time, and the version 2 line stops using future data.
But this is where repainting comes into play. This is the same chart moments later, after refreshing the indicators. Now they will again show both indicators in their historical form. The version 2 purple line is using future data again, and the version 3 green line is lagging again. Simply using version 3 is not enough to avoid repainting, if that’s what you’re trying to do.
So what do we do?
Well, how you deal with this depends on what you’re trying to do. What I’ve tried to do is explain exactly how it’s behaving and why. How you then use that is up to you. There’s nothing “wrong” about any of this data or behaviour as long as you understand what’s happening.
For those of you trying to match up automation with back testing or just current values with back testing, bear in mind that these discrepancies are due to the differences between how real time and historical data are handled in the security function. If you’re comfortable with only ever using the most recently closed bar from the higher timeframe, use version 3 and you can simply add to the end of your expression within the security function, and all these problems go away; no repainting, no future data used, real time data matches historical data. You just have to embrace the lag.
Let's take a look at the security function, the differences between version 3 and version 2, and hopefully help give people a better understanding of how these work. As many will know there are differences in how version 2 and version 3 handle the "security()" function. Confusion around the mechanics of this function can lead to headaches for people testing scripts and trying to intuit how it works. So we’re just going to run through some examples to give a visual along with some explanations.
First, let’s look at 2 instances of the following code, one in version 2, one in version 3.
//@version=2
study("My Script", overlay=false)
num = security(tickerid, "60", n)
bgcolor(num%2==0?orange:na)
plot(num)
This will use the built in variable “n” on the 60 minute timeframe using the security function. The variable “n” assigns each bar a sequential number. We will use a 5 minute resolution for our chart and plot the 60minute bar number and change the background color to orange if the bar number is an even number. Let’s see how that looks.
So there’s 2 important points to mention here.
The first is that the highlighted areas are almost exact opposites to each other. This is because in version 3 the security function uses the value of the previous bar when looking back at historical data. So in our example, at the time that version 2 painted the 14454 bar, that was the correct bar number. In version 3, it was using the previous bar of 14453. So with historical data version 3 essentially has 1 bar of lag. This is to avoid issues of the bar using “future data”, which we will show an example of shortly.
The second thing to note is the time at which the bar begins. Using the above example again, notice that the beginning of the new 60min bar has a 1 bar difference on the current timeframe. In version 2 the new bar starts at exactly 09:00, whereas in version 3 it starts at 08:55. Note that this is because we are using the 5 minute chart, if we were using a 3 minute chart the version 3 bar would begin at 08:57, or a 15 minute chart would begin at 08:45.
Why? Well, perhaps the following chart will help explain. We will use the following simple bit of code that again use a 5min chart and plot the 60min high value. The purple line is the code in version 2, the green line is the code in version 3.
h = security(tickerid, "60", high)
plot(h)
There’s a lot going on here but we’ll go through it step by step. The first thing to notice to make sense of this picture is that the green bar is exactly the same as the purple bar, but just shifted to the right. This is the lag we mentioned, because the green bar (version 3) is using the value from the previous 60 min period.
Now, notice the red circled areas. These areas show the beginning of the new bar for version 3 and the end value of the bar for version 2. **In version 3, the new bar begins as soon as we know the final value of the previous bar.** So using our first chart example, the final closing value of the 08:00 – 09:00 period is the closing value of the 08:55 bar (on 5min chart), so the 08:55 value is where the new version 3 bar begins.
The version 2 bar uses future data as default. You can see examples of this where the orange ellipses are on the chart above. Remember, the purple line is charting the high of the current period, but with version 2 the high of that hour is painted on the chart before that highest value is reached, as highlighted in orange. Using the high value from our security function at any of those highlighted points would give us inaccurate back testing results because our indicator would essentially be looking into the future.
So what are the practical implications of this? Well, it means if you’re using version 2 you’re going to get inaccurate back test results because of the future data issue, which is the main reason this was changed for version 3. If you’re using version 3, however, that’s not a magical fix either. For instance if you’re using an hourly chart and pulling the daily data with the security function, the “daily” value last Wednesday will actually be using the values from last Tuesday. So with our examples of using the period “high”, it will be possible for the current hourly high to be above the security data’s daily high, because it’s using the previous day high.
Difference between real time and historical data
Everything we’ve talked about so far has been dealing with historical data.
Real time data for both version 2 and version 3 work the same way and work correctly. That is to say if you’re using a 5min chart and pull the data from the hourly with security, then the real time data from the security function will match the real time data from an hourly chart.
But how does that work if the version 3 data uses the value from the previous period? Well, as soon as you plot security values in a chart you’ll notice that the newest bar value will change. As soon as new data starts getting painted it will switch from the previous period’s value to the current value.
For example, this is a minute chart plotting the 3 minute high via the security function. Purple values are version 2, green are version 3, and you’ll see as soon as we hit real time data (when we clicked “add to chart”, signified by the pink dotted line) the 2 are identical. The version 3 data switches from lagging by 1 period to real time, and the version 2 line stops using future data.
But this is where repainting comes into play. This is the same chart moments later, after refreshing the indicators. Now they will again show both indicators in their historical form. The version 2 purple line is using future data again, and the version 3 green line is lagging again. Simply using version 3 is not enough to avoid repainting, if that’s what you’re trying to do.
So what do we do?
Well, how you deal with this depends on what you’re trying to do. What I’ve tried to do is explain exactly how it’s behaving and why. How you then use that is up to you. There’s nothing “wrong” about any of this data or behaviour as long as you understand what’s happening.
For those of you trying to match up automation with back testing or just current values with back testing, bear in mind that these discrepancies are due to the differences between how real time and historical data are handled in the security function. If you’re comfortable with only ever using the most recently closed bar from the higher timeframe, use version 3 and you can simply add to the end of your expression within the security function, and all these problems go away; no repainting, no future data used, real time data matches historical data. You just have to embrace the lag.
discord.gg/mFYvr2bGtu - FREE Trading Chat Rooms