I have been running some comparisons between Fei’s current TWAP ETH/USDC oracle and Chainlink’s ETH/USD oracle.
TLDR: A variety of data quality metrics suggest Chainlink’s oracle is superior to Fei’s current TWAP oracle
The Data
I am looking at the range of blocks starting shortly after Genesis and ending shortly before the present day. This is about 365,000 blocks.
(The blue line is wider just so that both lines are visible. The blue line is not actually more variable over time. It’s just that other plotting schemes like dotted lines don’t work well with 365k datapoints)
This graph shows the two oracles overlaid, and there’s a clear similarity between them. How do we know which one is better? There’s not a well-defined “ground truth” to measure which oracle is more accurate, but we can run some summary statistics that measure other aspects of oracle quality.
Mean number of blocks between updates:
- FEI TWAP: 69.14 blocks
- Chainlink: 46.44 blocks
Mean time between updates:
- FEI TWAP: 922.76 seconds
- Chainlink: 619.49 seconds
Mean update size:
- FEI TWAP: 0.44%
- Chainlink 0.35%
The Chainlink oracle updates more frequently and also has smaller value gaps between updates, both of which are desirable qualities in an oracle.
Deviation Analysis
Fully zoomed out, the oracles appear to have nearly identical values. But how close are they? This is a plot of the deviation between these two oracles over time, both subtractively and divisively:
And this is the distribution of deviations (note the log scale on the y axis):
Most of the time, the deviations between the oracles are small. But occasionally they are very large. Let’s see what the the raw data looks like during some periods of maximum deviation:
Example 1:
Example 2:
Example 3:
These plots are very concerning because Fei’s TWAP oracle is consistently outdated in time compared to the Chainlink oracle. This is a separate issue from merely having a lower sampling rate. The TWAP is also delayed in time, “shifted to the right” in the graphs above. When the price of ETH goes up, Fei’s TWAP takes longer to go up. When the price of ETH goes down, Fei’s TWAP takes longer to go down.
This means that Fei’s protocol mechanisms are always trading with stale information, allowing traders to make advantaged trades against the protocol that are more similar to front-running than arbitrage.
Cross-correlation Analysis
To quantify the delay between the oracles we can use a statistical metric called cross-correlation.
Cross-correlation is a measure of the correlation between two signals taken at different time delays. If two signals are measuring the same underlying thing and neither signal is delayed in time, then the cross-correlation will show a peak at 0. If one signal is a delayed version of the other, the cross-correlation will show a peak offset some distance from 0 corresponding to the size of the delay.
Instead of peaking at 0, the cross-correlation of the TWAP and Chainlink oracles shows a peak around 45 blocks. This indicates something like “the TWAP oracle is 45 blocks delayed compared to Chainlink oracle”. 45 blocks is roughly 10 minutes, which is even higher than would be expected for a 10 minute TWAP oracle compared to an instantaneous source of ground truth. This is further evidence that the TWAP is consistently delayed compared to Chainlink.
Conclusion
Taken together these results paint a much prettier picture of the Chainlink oracle compared to the TWAP. Upgrading Fei’s oracle to Chainlink should allow Fei’s various mechanisms to operate more effectively by using more up-to-date information about current market conditions.