21 April 2021 — Venus Vaults Post-Mortem

This post was originally published on this site

21 April 2021 — Venus Vaults Post-Mortem

When things go south, it is easy to point fingers and blame other projects for causing the issue. Autofarm does not believe in such behaviour, and we always encourage our team and moderators alike to be neutral, courteous and respectful to other projects at all times. In the following segments, we would like to present the most objective and transparent post-mortem with details on the how, why, recovery plans, what Autofarm has taken away from this incident and how we plan to move forward.

On 21 April 2021, Venus vaults on Autofarm experienced several complications that temporarily affected the withdrawal and compounding of assets. Only assets in Venus vaults were affected.

First of all, we would like to sincerely apologise for the whirlwind of events. We acknowledge that user experience was not the best and as a project that prides itself in user experience, we admit that we could have done better; in the areas of communication and providing details on the how and why Autofarm Venus vaults were no longer working and there was a loss of funds for users. Once again, we apologise.

We wish to reemphasise that protocol security and protecting user funds remains our top priority, and on this issue, users will be compensated accordingly from our recovery plan.

Below are the details of the post-mortem of the events that ensued regarding the Venus vaults on Autofarm.

Post-Mortem

20th April 2021

biswap

8:20pm GMT+8: Unbeknownst to the Autofarm team, Venus Protocol’s VIP-12 part 1 was executed.

21st April 2021

12:45am GMT+8: First alerts were raised by several users who were facing withdrawal issues on Autofarm’s Venus vaults. Users also noted that Venus vaults did not seem to be earning interest at this time, specifically for USDT and USDC vaults.

12:45am — 1:19am GMT+8: Autofarm developers were alerted by yprivacymatter who was at that time still in the AMA in BSCFrance.

1:20am — 1:26am GMT+8: Autofarm developers began internal checks and had the same findings as those raised from the community. Emergency withdrawals were also found to be affected.

The Venus VIP-12 V2 upgrade proposal that was executed on 20 April was identified as a potential cause of the issue.

1:26am — 2:23am GMT+8: Further internal testings were done by flipping the withdrawal fee switch and deploying another venus contract for testing. The team was able to withdraw funds with the test contract without our implementation of additional withdrawal fees.

No error message was shown on bscscan and it was around this time when the BSC RPC network was heavily congested, making it hard to determine the root cause of the issue.

2:23am — 2:59am GMT+8: The Autofarm team reached out to Venus to check on the implementation details for Venus Protocol’s VIP-12. The Venus team confirmed that USDT and USDC pools were affected by the execution of the VIP-12 proposal which supported the team’s initial hypothesis of VIP-12 being the root cause.

yprivacmatter spoke to the Community Admin of Venus Protocol directly to request for assistance, and to connect the Autofarm team with developers from Venus. He was informed that to reach Venus developers, it was necessary to go through proper channels — SwipeWallet’s Telegram channel. yprivacymatter then proceeded to seek assistance in the chat over here.

3:21am GMT+8: SwipeWallet’s Telegram admin responded but yprivacymatter was unavailable during that moment.

4:00am GMT+8: mildgiraffe deleveraged the Venus vaults, allowing users to withdraw their assets. Users also reported that withdrawals were functioning on Venus vaults.

4:12am GMT+8: Beefy Finance pulled out USDC and USDT deposits from Autofarm’s Venus vaults. (Link)

4:29am GMT+8: Vaults were paused on the front-end and deposits to Venus vaults on Autofarm were disabled during this time.

4:36am GMT+8: Autofarm developers continued internal testing.

4:49am — 5.40am GMT+8: Autofarm users raised issues USDC and USDT deposits on Venus vaults were still no accruing interest and those who withdrew faced some losses.

5.34am GMT+8: The team confirmed internally that though only USDT and USDC deposits were affected, all Venus vaults will be retired effective immediately upon the announcement. (This was decided in part due to VIP-12 Part 2 and 3 implementations which were to be in ~12 hours).

5:47am GMT+8: Beefy Finance pulled out all other deposits from Autofarm’s Venus vaults. (Link)

5.58am GMT+8: mildgiraffe informed users on Autofarm’s main Telegram channel that due to Venus protocol’s introduction of additional fees (VIP-12), all Venus vaults will be stopped (retired) and users were advised to withdraw from all Venus vaults.

6:21am GMT+8: As Beefy has several Autofarm Venus vaults on their platform, the Autofarm team reached out to inform them of the situation and to ensure withdrawals were made from Venus vaults.

6:27am GMT+8: Official announcement by Autofarm was made on Telegram to alert users of the Venus fee upgrade which impacted vault strategies on their platform. Users were advised to withdraw their funds from all Venus vaults on Autofarm while the team worked on finding a resolution (https://t.me/autofarm_network_ann/356).

6:28am — 10:36am GMT+8: Autofarm moderators were supporting the community and getting users to withdraw from all Venus vaults on Autofarm during this time.

10:37am GMT+8: Binance Chain team (Jeff Zhang & Xiao Zhang) helped to coordinate and reach out to the core Venus team.

10:49am GMT+8: Autofarm checked in with Belt to see if they too were affected. They acknowledged it and quickly made an official statement over here at 11:18am GMT+8.

10:58am GMT+8: An official announcement was released on Autofarm’s Twitter account.

Due to the time difference during the sequence of events, the only 2 persons who had access to Autofarm’s Twitter account (yprivacymatter & the marketing executive) were not available to announce on Twitter until 10:58am.

11:31am GMT+8: Joselito (CEO of Swipe & Venus) provided suggestions on how to reduce the loss of funds experienced by users. (Suggestions were not effective)

11:47am GMT+8: The Autofarm team began developing a script to determine the amount of funds that users of Venus v2 vaults lost (Total deposits – Total amount upon withdrawals)

11:53am GMT+8: The team tweeted that we were working on a post-mortem & a recovery plan for this incident. (Link)

12:27pm GMT+8: The team requested for the block number from the Venus team of when Venus’ VIP-12 Part 1 was executed (Block 6586814)

12:30pm GMT+8: Following Belt Finance’s announcement at 11:18am GMT+8, Beefy Finance made an announcement, urging their users to withdraw funds from Beefy (Autofarm was unaffected because we do not have Belt Stablecoin LP vaults) (Link)

1:00pm GMT+8: Similarly, ValueDeFi also made an announcement, urging their users to withdraw from ValueDeFi. (Link)

2:21pm GMT+8: Team clarifies with Venus team on the execution details and timings of Part 2 & 3 of VIP-12.

2:54pm — 3:18pm GMT+8: The team made further clarifications on the execution block number of VIP-12 Part 1, this was done to assist in the creation of the script to backtrack when users started experiencing a loss of funds.

3:14pm GMT+8: The team continued to urge users who still had deposits in the retired Venus v1 vaults to head over to legacy.autofarm.network to make their withdrawals (Link).

3:20pm GMT+8: yprivacymatter tweeted from his personal account that a compensation proposal was being drafted out, of which funds from AutoSAFU will be tapped on (Link)

3:53pm — 5:35pm GMT+8: Team started the discussion with the Venus team to work on a compensation plan together.

5.01pm GMT+8: Venus Protocol’s VIP-12 Part 2 was executed by the Venus community

5.09pm GMT+8: Venus Protocol’s VIP-12 Part 3 was executed by the Venus community

7:27pm GMT+8: Venus released their medium article here, addressing the Venus & BSC community & also all other farm networks (assumed to be yield aggregator projects that have Venus vaults). (English, Chinese)

7:56pm GMT+8: Autofarm follows up on the discussion about the compensation plan and asks about The Ecological Co-construction Alliance as mentioned in Venus’ article (this alliance was not mentioned to Autofarm beforehand)

22nd April 2021

2:25am GMT+8: Joselito (CEO of Swipe & Venus) responded ~7 hours later that the Venus team will conduct a discussion internally and with the community and advise on the next steps in the following days.

Why did Venus v2 vaults stop working?

Dealing with leverage is always generally riskier, hence the increased risk of our only leveraged vault product — Autofarm Venus Vaults. Venus Protocol introduced a new fee structure (redemption fee increase to 0.01% of all vTokens) in their Venus Improvement Proposal 12 (VIP-12). Although steps were taken to prepare for this event, our vaults were still unable to fully cater for this fee. Luckily, Autofarm vaults had sufficient fail-safe measures to manually deleverage vaults, which allowed for user withdrawals. However, withdrawals were non-optimal when our vaults were fully deleveraged. This caused a significant amount of extra transactions (lending and borrowing to the Venus Protocol) which racked up a considerable amount of fees to the Autofarm Vault that benefited the Venus Protocol Treasury directly at the expense of the pool users.

When Autofarm vaults were first created, we specifically chose to not allow (even for ourselves) the option of vault migration — the ability to withdraw user funds and move it elsewhere, such as another vault.

This security design was chosen because vault migration has been the culprit to many loss of fund issues faced by projects. As such, this function was forgoed to increase security levels for users. Today, however, this proved detrimental to our ability in making responsive changes within the DeFi space. When the issue was first brought to our attention, we were unable to migrate user funds out of Venus vaults to an upgraded vault strategy or allow for delayed withdrawals. The only option was to ask users to withdraw funds themselves. We will relook this principle and assess the pros and cons of this feature. In the future, we might launch vaults with vault migration functions. If we do this, users can be assured that it will be displayed clearly in the vault description. For now, however, the security of funds is our top priority. With the introduction of flash loans to BSC, we will definitely err on the safe side and be slow to adopt vault migration practices for the sake of security.

Recovery Plan

All users who had deposits in Autofarm’s Venus vaults and experienced a loss of funds due to the change in redemption fee structure by Venus Protocol in their recent VIP-12 proposal implementation will be duly compensated. We are currently still in discussion with Venus team on how to move forward.

As previously mentioned, we have set up an automated system to trace and identify all users and their corresponding losses.

In addition, going forward, all single token asset vaults which we release on Binance Smart Chain (BSC) over the next two weeks will have their deposit fees waived for 3 days upon their release.

We appreciate your understanding on this matter and greatly thank you for your continued support for Autofarm!

Moving Forward

We acknowledge that the team could have responded faster with regards to the issue on our Venus v2 vaults when concerns were first raised by the community at 12.45am. Though we do seek your understanding that most of the team lives in the GMT+8 timezone, hence the incident was very much untimely for us.

Moving forward, below are a few plans we will be enacting following our takeaways from this incident:

  • AutoSAFU protocol insurance has shown its strength in trying times like this and hence, we will continue pledging the rebates which we receive monthly from Binance’s BUIDL reward program to the AutoSAFU.
  • We will scale on the number of moderators and volunteers to have a more holistic coverage across the communities and different time zones. This way, better customer support can be provided as required by the community.
  • We will onboard more developers for more coverage on the different time zones so that the protocol has more bases covered even during the team’s wee hours.
  • We will set up an internal community that focuses on the different updates of the projects which we interact with on our platform (vaults which we deploy, DEXes which we integrate into AutoSwap and tokens which are whitelisted on our platform)
  • We will work more closely with these projects in a BSC Alliance to update the various projects who are interdependent to contribute to a thriving BSC community.

Conclusions

We believe that no single party (project) is at fault and fully to be blamed for this incident; there were many instances where the entire BSC ecosystem could have coordinated things better, projects could have in better communication with one another and on our end, Autofarm could have been more alert of changes to projects which we have vaults for, been more responsive to the situation at hand and made better clarifications on our channels for the community to understand what was going on.

BSC ecosystem has shown its strength in the last few weeks with its low fees, a high number of transactions and users. We believe BSC still has much more room to grow and Autofarm will continue to support and build in this ecosystem and we believe greater collaboration and partnerships within the ecosystem will only serve to benefit the ecosystem further.

Together we can be stronger, together we can BUIDL a better BSC ecosystem!


21 April 2021 — Venus Vaults Post-Mortem was originally published in autofarm.network on Medium, where people are continuing the conversation by highlighting and responding to this story.

Leave a Comment