Smart Contracts

This release process is a work in progress. Many infrastructure components required to execute it are not in place, and the process itself is subject to change.

Versioning

Each deployed Celo core smart contract is versioned independently, according to semantic versioning, as described at semver.org, with the following modifications:

  • STORAGE version when you make incompatible storage layout changes

  • MAJOR version when you make incompatible ABI changes

  • MINOR version when you add functionality in a backwards compatible manner, and

  • PATCH version when you make backwards compatible bug fixes.

Changes to core smart contracts are made via on-chain Governance, approximately four times a year. When a release is made, all smart contracts from the release branch that differ from the deployed smart contracts are released, and included in the same governance proposal. Each release is identified by a unique monotonically increasing version number N, with 1 being the first release.

Core Contracts

Every deployed Celo core contract has its current version number as a constant which is publicly accessible via the getVersion() function, which returns the storage, major, minor, and patch version. Version number is encoded in the Solidity source and updated as part of code changes.

Celo Core Contracts deployed to a live network without the getVersion() function, such as the original set of core contracts, are to be considered version 1.1.0.0.

Mixins and libraries

Mixin contracts and libraries are considered part of the contracts that consume them. When a mixin or library has changed, all contracts that consume them should be considered to have changed as well, and thus the contracts should have their version numbers incremented and should be re-deployed as part of the next smart contract release.

Release management in Git/Github

Github branches/tags and Github releases are used to coordinate past and ongoing releases. Ongoing smart contract development is done on the master branch (even after release branches are cut). Every smart contract release has a designated release branch, e.g. release/celo-core-contracts/${N} in the celo-monorepo.

When a new release branch is cut:

  1. A new release branch is created release/celo-core-contracts/${N} with the contracts to be audited.

  2. The latest commit on the release branch is tagged with celo-core-contracts-v${N}.pre-audit.

  3. On Github, a pre-release Github release should be created pointing at the latest tag on the release branch.

  4. On master branch, .circleci/config.yml should be edited so that the variable RELEASE_TAG points to the tag celo-core-contracts-v${N}.pre-audit so that all future changes to master are versioned against the new release.

  5. Ongoing audit responses/fixes should continue to go into release/celo-core-contracts/${N}.

During the release proposal stage:

  1. Whenever release candidates are available they should be tagged celo-core-contracts-v${N}.rc1, ...rc2, etc.

  2. When a release is getting proposed on a Baklava/Alfajores/Mainnet, the commit that was used (which should be the latest release candidate) should be tagged with celo-core-contracts-v${N}.(baklava | alfajores | mainnet).

After a completed release process:

  1. The release branch should be merged into master with a merge commit (instead of the usual squash merge strategy).

  2. On master branch, .circleci/config.yml should be edited so that the variable RELEASE_TAG points to the tag celo-core-contracts-v${N}.mainnet

Release Process

There are several scripts provided (under packages/protocol in celo-org/celo-monorepo and via celocli) for use in the release process and with contract upgrade governance proposals to give participating stakeholders increased confidence.

​ For these to run, you may need to follow the setup instructions. These steps include installing Node and setting nvm to use the correct version of Node. Successful yarn install and yarn build in the protocol package signal a completed setup.

Using these tools, a contract release candidate can be built, deployed, and proposed for upgrade automatically on a specified network. Subsequently, stakeholders can verify the release candidate against a governance upgrade proposal's contents on the network.

Verify Release on Network

Use the following script to compile contracts at a release tag and verify that the deployed network bytecode matches the compiled bytecode.

NETWORK=${"baklava"|"alfajores"|"mainnet"}
RELEASE="celo-core-contracts-v${N-1}.${NETWORK}"
# A -f boolean flag can be provided to use a forno full node to connect to the provided network
# A -r boolean flag should be provided if this is the first release (before linked libraries were proxied)
yarn verify-deployed -n $NETWORK -b $RELEASE -r -f

Check Backward Compatibility

Use the following script to compile candidate and current release contracts to check storage layout and ABI backwards compatibility, subject to the following exceptions: 1. If the STORAGE version has changed, does not perform backwards compatibility checks 2. If the MAJOR version has changed, checks storage layout compatibility but not ABI compatibility

For changed contracts, checks that new version number is strictly greater than previous release. For unchanged contracts, checks that version number exactly matches previous release. Outputs a JSON backwards compatibility report.

NETWORK=${"baklava"|"alfajores"|"mainnet"}
PREVIOUS_RELEASE="celo-core-contracts-v${N-1}.${NETWORK}"
RELEASE_CANDIDATE="celo-core-contracts-v${N}.rc${X}"
yarn check-versions -a $PREVIOUS_RELEASE -b $RELEASE_CANDIDATE -r "report.json"

This should be used in tandem with verify-deployed -b $PREVIOUS_RELEASE -n $NETWORK to ensure the compatibility checks compare the release candidate to what is actually active on the network.

Deploy New Contracts

Use the following script to build a candidate release and, using the corresponding backwards compatibility report, deploy changed contracts to the specified network. (Use -d to dry-run the deploy). STORAGE updates are adopted by deploying a new proxy/implementation pair. These new contracts may need to be initialized, for which values can be provided. Outputs a JSON contract upgrade governance proposal.

NETWORK=${"baklava"|"alfajores"|"mainnet"}
RELEASE="celo-core-contracts-v${N}.rc${X}"
yarn make-release -b $RELEASE -n $NETWORK -r "report.json" -i "initialize_data.json" -p "proposal.json"

The proposal encodes STORAGE updates by repointing the Registry to the new proxy. Storage compatible upgrades are encoded by repointing the existing proxy's implementation.

Submit Upgrade Proposal

Submit the autogenerated upgrade proposal to the Governance contract for review by voters, outputting a unique identifier.

# resultant proposal ID should be communicated publicly
celocli governance:propose <...> --jsonTransactions "proposal.json"

Fetch Upgrade Proposal

Fetch the upgrade proposal and output the JSON encoded proposal contents.

celocli governance:show <...> --proposalID <proposalId> --jsonTransactions "upgrade_proposal.json"

Verify Proposed Release

Verify that the proposed upgrade activates contract addresses which match compiled bytecode from the tagged release candidate exactly.

RELEASE="celo-core-contracts-v${N}.rc${X}"
NETWORK=${"baklava"|"alfajores"|"mainnet"}
# A -f boolean flag can be provided to use a forno full node to connect to the provided network
yarn verify-release -p "upgrade_proposal.json" -b $RELEASE -n $NETWORK -f

Testing

All releases should be evaluated according to the following tests.

Unit tests

All changes since the last release should be covered by unit tests. Unit test coverage should be enforced by automated checks run on every commit.

Performance

A ceiling on the gas consumption for all common operations should be defined and enforced by automated checks run on every commit.

Backwards compatibility

Automated checks should ensure that any new commit to master does not introduce a breaking change to storage layout, ABI, or other common backwards compatibility issues unless the STORAGE or MAJOR version numbers are incremented.

Backwards compatibility tests will also be run before every release to confirm that no breaking changes exist between the pending release and deployed smart contracts.

Audits

All changes since the last release should be audited by a reputable third party auditor.

Emergency patches

If patches need to be applied before the next scheduled smart contract release, they should be cherry picked to a new release branch, branched from the latest deployed release branch.

Promotion process

Deploying a new contract release should occur with the following process. On-chain governance proposals should be submitted on Tuesdays for consistency and predictability.

Date

Action

T

  1. Create a Github issue tracking all these checklist items as an audit log

  2. Implement the git management steps for when a new release branch is cut.

  3. Submit release branch to a reputable third party auditor for review.

  4. Begin drafting release notes.

T+1w

  1. Receive report from auditors.

  2. Add audit summary to final draft of the release notes.

  3. If all issues in the audit report have straightforward fixes:

    1. Submit a governance proposal draft using this format: https://github.com/celo-org/celo-proposals/blob/master/CGPs/template.md

    2. Announce forthcoming smart contract release on: https://forum.celo.org/c/governance

  4. Commit audit fixes to the release branch

  5. Submit audit fixes to auditors for review.

  6. Tag the first release candidate commit according to the git release management instructions.

  7. Let the community know about the upcoming release proposal by posting details to the Governance category on https://forum.celo.org and cross post in the Discord #governance channel. See the 'Communication guidelines' section below for information on what your post should contain.

T+2w

  1. On Tuesday: Run the smart contract release script in order to to deploy the contracts to Baklava as well as submit a governance proposal.

    • Transition proposal through Baklava governance process.

    • Tag the commit that is being proposed with celo-core-contracts-v${N}.baklava

  2. Update your forum post with the Baklava PROPOSAL_ID, updated timings (if any changes), and notify the community in the Discord #governance channel.

T+3w

  1. Confirm all contracts working as intended on Baklava.

  2. Run the smart contract release script in order to to deploy the contracts to Alfajores as well as submit a governance proposal.

  3. Tag the commit that is being proposed with celo-core-contracts-v${N}.alfajores

  4. Update your forum post with the Alfajores PROPOSAL_ID, updated timings (if any changes), and notify the community in the Discord #governance channel.

T+4w

  1. Confirm all contracts working as intended on Alfajores.

  2. Confirm audit is complete and make the release notes and forum post contain a link to it.

  3. On Tuesday: Run the smart contract release script in order to to deploy the contracts to Mainnet as well as submit a governance proposal.

  4. Tag the commit that is being proposed with celo-core-contracts-v${N}.mainnet

  5. Update the corresponding governance proposal with the updated on-chain PROPOSAL_ID and mark CGP status as "PROPOSED".

  6. Update your forum post with the Mainnet PROPOSAL_ID, updated timings (if any changes), and notify the community in the Discord #governance channel.

  7. At this point all stakeholders are encouraged to verify the proposed contracts deployed match the contracts from the release branch.

  8. Monitor the progress of the proposal through the governance process.

    • Currently the governance process should take approximately 1 week: 24 hours for the dequeue process, 24 hours for the approval process, and 5 days for the referendum process. After which, the proposal is either declined or is ready to be executed within 3 days.

    • For updated timeframes, use the celocli: celocli network:parameters

T+5w

  1. If the proposal passed:

    1. Confirm all contracts working as intended on Mainnet.

    2. Update your forum post with the Mainnet governance outcome (Passed or Rejected) and notify the community in the Discord #governance channel.

    3. Change corresponding CGP status to EXCECUTED.

    4. Merge the release branch into master with a merge commit

  2. If the proposal failed:

    1. Change corresponding CGP status to EXPIRED.

If the contents of the release (i.e. source Git commit) change at any point after the release has been tagged in Git, the process should increment the release identifier, and process should start again from the beginning. If the changes are small or do not introduce new code (e.g. reverting a contract to a previous version) the audit step may be accelerated.

Communication guidelines

Communicating the upcoming governance proposal to the community is critical and may help getting it approved.

Each smart contract release governance proposal should be accompanied by a Governance category forum post that contains the following information:

  • Name of proposer (individual contributor or organization).

  • Background information.

  • Link to the release on Github.

  • Link to the audit report(s).

  • Anticipated timings for the Baklava and Alfajores testnets and Mainnet.

It's recommended to post as early as possible and at minimum one week before the anticipated Baklava testnet governance proposal date.

Make sure to keep the post up to date. All updates (excluding fixing typos) should be communicated to the community in the Discord #governance channel.

Emergency patches

Work in progress

Vulnerability Disclosure

Vulnerabilities in smart contract releases should be disclosed according to the security policy.

Dependencies

None

Dependents

Work in progress