Skip to main content
Details of the release process for updating smart contracts on the Celo platform.
This release process is a work in progress. Many infrastructure components required to execute it are not in place, and the process itself is subject to change.

​
Versioning

Each deployed Celo core smart contract is versioned independently, according to semantic versioning, as described at semver.org, with the following modifications:
  • STORAGE version when you make incompatible storage layout changes
  • MAJOR version when you make incompatible ABI changes
  • MINOR version when you add functionality in a backwards compatible manner, and
  • PATCH version when you make backwards compatible bug fixes.
Changes to core smart contracts are made via on-chain Governance, approximately four times a year. When a release is made, all smart contracts from the release branch that differ from the deployed smart contracts are released, and included in the same governance proposal. Each release is identified by a unique monotonically increasing version number N, with 1 being the first release.

​
Core Contracts

Every deployed Celo core contract has its current version number as a constant which is publicly accessible via the getVersionNumber() function, which returns the storage, major, minor, and patch versions. The version number is encoded in the Solidity source and updated as part of code changes. Celo Core Contracts deployed to a live network without the getVersionNumber() function, such as the original set of core contracts, are to be considered version 1.1.0.0.

​
Mixins and libraries

Mixin contracts and libraries are considered part of the contracts that consume them. When a mixin or library has changed, all contracts that consume them should be considered to have changed as well, and thus the contracts should have their version numbers incremented and should be re-deployed as part of the next smart contract release.

​
Initialize Data

Whenever Celo Core Contracts need to be re-initialized, their initialization arguments should be checked into version control under packages/protocol/releaseData/initializationData/release${N}.json.

​
Release management in Git/Github

Github branches/tags and Github releases are used to coordinate past and ongoing releases. Ongoing smart contract development is done on the master branch (even after release branches are cut). Every smart contract release has a designated release branch, e.g. release/core-contracts/${N} in the celo-monorepo.

​
When a new release branch is cut:

  1. A new release branch is created release/core-contracts/${N} with the contracts to be audited.
  2. The latest commit on the release branch is tagged with core-contracts.v${N}.pre-audit.
  3. On Github, a pre-release Github release should be created pointing at the latest tag on the release branch.
  4. On master branch, .github/workflows/celo-monorepo.yml should be edited so that the variable RELEASE_TAG points to the tag core-contracts.v${N}.pre-audit so that all future changes to master are versioned against the new release.
  5. Ongoing audit responses/fixes should continue to go into release/celo-core-contracts/${N}.

​
After a completed release process:

  1. The release branch should be merged into master with a merge commit (instead of the usual squash merge strategy).
  2. On master branch, .github/workflows/celo-monorepo.yml should be edited so that the variable RELEASE_TAG points to the tag core-contracts.v${N}

​
Release Process

There are several scripts provided (under packages/protocol in celo-org/celo-monorepo and via celocli). The release owners’ own reference is RELEASE_PROCESS_FOUNDRY.md, which maps each retired Truffle command to its Foundry replacement. These are for use in the release process and with contract upgrade governance proposals to give participating stakeholders increased confidence.
For these to run, you may need to set up the celo-monorepo: follow the Getting Started steps and install the Node version pinned in .nvmrc (managed with nvm). A successful yarn install and yarn build in the protocol package signal a completed setup. The release scripts also shell out to Foundry (forge, cast) and jq, so install those too.
Using these tools, a contract release candidate can be built, deployed, and proposed for upgrade automatically on a specified network. Subsequently, stakeholders can verify the release candidate against a governance upgrade proposal’s contents on the network. Typical script options:
  • By default, the scripts expect a celo-blockchain RPC at port 8545 locally. With -f you can specify the scripts to use a hosted forno node
  • By default, scripts will output verbose logs under /tmp/celo-${script-name}.log. You can change the location of the log output with -l file.log

​
View the tagged releases for each network

​
Verify the previous Release on the Network

release:verify-deployed:foundry is a script that allows you to assess whether the bytecode on the given network matches the source code of a particular commit. It will run through the Celo Core Contracts and verify that the contracts’ bytecodes as specified in the Registry match. Here, we will want to sanity-check that our network is running the previous release’s audited commit.
A $NETWORK-$PREVIOUS_RELEASE-libraries.json file is written to disk describing linked library addresses. release:make:foundry requires it.

​
Check Backward Compatibility

This script performs some automatic checks to ensure that the smart contract versions in the source code have been set correctly with respect to the latest release. It is run as part of CI and helps ensure that backwards incompatibilities are not accidentally introduced by requiring that devs manually update version numbers whenever smart contract changes are made. Specifically, it compiles the latest and candidate releases and compares smart contracts:
  1. Storage layout, to detect storage version changes
  2. ABI, to detect major and minor version changes
  3. Bytecode, to detect patch version changes
Finally, it checks release candidate smart contract version numbers and requires that they have been updated appropriately since the latest release by following semantic versioning as defined in the Versioning section above. The following exceptions apply:
  • If the STORAGE version has changed, it does not perform backward compatibility checks
  • If the MAJOR version has changed, it checks storage layout compatibility but not ABI compatibility
Critically, this ensures that proxied contracts do not experience storage collisions between implementation versions. See this article by OpenZeppelin for a good overview of this problem and why it’s important to check for it. The script generates a detailed report on version changes in JSON format, written to report-$PREVIOUS_RELEASE-$RELEASE_CANDIDATE.json.
This should be used in tandem with release:verify-deployed:foundry -b $PREVIOUS_RELEASE -n $NETWORK to ensure the compatibility checks compare the release candidate to what is actually active on the network.

​
Deploy the release candidate

Use the following script to build and deploy a candidate release. This takes as input the corresponding backward compatibility report and canonical library address mapping to deploy changed contracts to the specified network. The Foundry script has no dry-run; pass -u with a local anvil fork’s RPC URL to rehearse it. STORAGE updates are adopted by deploying a new proxy/implementation pair. This script writes a JSON contract upgrade governance proposal to proposal-$NETWORK-$RELEASE_CANDIDATE.json.
-k "$PRIVATE_KEY" puts the deployer key on the command line, where it is readable by other processes on the machine (for example via /proc on Linux) and is written to your shell history. Run it from a dedicated deploy machine, and clear the key from the environment and history afterwards.
The proposal encodes STORAGE updates by repointing the Registry to the new proxy. Storage compatible upgrades are encoded by repointing the existing proxy’s implementation.

​
Submit Upgrade Proposal

Submit the autogenerated upgrade proposal to the Governance contract for review by voters, outputting a unique identifier.

​
Fetch Upgrade Proposal

Fetch the upgrade proposal and output the JSON encoded proposal contents.

​
Verify Proposed Release Candidate

This script serves the same purpose as release:verify-deployed:foundry but for a not-yet accepted contract upgrade (in the form of the upgrade_proposal.json you fetched in the step prior). It gives you the confidence that the branch specified in the -b flag in (same as release:check-versions:foundry) will be the resulting network state of the proposal if executed. It does so by going over all Celo Core Contracts and determining updates to the Registry pointers, proxy or implementation contracts and verifying their implied bytecode against the compiled source code. Additionally, include initialization_data.json from the CGP if any of the contracts have to be initialized.

​
Verify Executed Release

After a release executes via Governance, you can use release:verify-deployed:foundry again to check that the resulting network state does indeed reflect the tagged release candidate:

​
Testing

All releases should be evaluated according to the following tests.

​
Unit tests

All changes since the last release should be covered by unit tests. Unit test coverage should be enforced by automated checks run on every commit.

​
Manual Checklist

After a successful release execution on a testnet, the resulting network state should be spot-checked to ensure that no regressions have been caused by the release. Flows to test include:
  • Do a USDm and CELO transfer
  • Register a Celo account
  • Report an Oracle rate
  • Do a CP-DOTO exchange
  • Complete a round of attestation
  • Redeem from Escrow
  • Register a Validator
  • Vote for a Validator
  • Run a mock election
  • Get a validator slashed for downtime and ejected from the validator set
  • Propose a governance proposal and get it executed

​
Automated environment tests

Stakeholders can use the env-tests package in celo-monorepo to run an automated test suite against the network

​
Verify smart contracts

Verification of smart contracts should be done both on https://celoscan.io/ and https://celo.blockscout.com/.
  1. Update your Smart Contract on celoscan
  2. Update your Smart Contract on Blockscout

​
Performance

A ceiling on the gas consumption for all common operations should be defined and enforced by automated checks run on every commit. For troubleshooting please see Readme.md of protocol package.

​
Backwards compatibility

Automated checks should ensure that any new commit to master does not introduce a breaking change to storage layout, ABI, or other common backward compatibility issues unless the STORAGE or MAJOR version numbers are incremented. Backwards compatibility tests will also be run before every release to confirm that no breaking changes exist between the pending release and deployed smart contracts.

​
Audits

All changes since the last release should be audited by a reputable third party auditor.

​
Emergency patches

If patches need to be applied before the next scheduled smart contract release, they should be cherry-picked to a new release branch, branched from the latest deployed release branch.

​
Promotion process

Deploying a new contract release should occur with the following process. On-chain governance proposals should be submitted on Tuesdays for consistency and predictability.
DateAction
T
  1. Create a Github issue tracking all these checklist items as an audit log
  2. Implement the git management steps for when a new release branch is cut.
  3. Submit the release branch to a reputable third party auditor for review.
  4. Begin drafting release notes.
T+1w
  1. Receive report from auditors.
  2. Add audit summary to the final draft of the release notes.
  3. If all issues in the audit report have straightforward fixes:
    1. Submit a governance proposal draft using this format:
    2. Add any initialization data to the CGP that should be included as part of the proposal
    3. Announce forthcoming smart contract release on: https://forum.celo.org/c/governance
  4. Commit audit fixes to the release branch
  5. Submit audit fixes to auditors for review.
  6. Tag the first release candidate commit according to the git release management instructions.
  7. Let the community know about the upcoming release proposal by posting details to the Governance category on https://forum.celo.org and cross post in the Discord #governance channel. See the ‘Communication guidelines’ section below for information on what your post should contain.
T+2w
  1. Confirm all contracts working as intended on an Anvil testnet.
  2. Run the smart contract release script in order to to deploy the contracts to Celo Sepolia as well as submit a governance proposal.
  3. Update your forum post with the Celo Sepolia PROPOSAL_ID, updated timings (if any changes), and notify the community in the Discord #governance channel.
T+3w
  1. Confirm all contracts working as intended on Celo Sepolia.
  2. Confirm audit is complete and make the release notes and forum post contain a link to it.
  3. On Tuesday: Run the smart contract release script in order to to deploy the contracts to Mainnet as well as submit a governance proposal.
  4. Update the corresponding governance proposal with the updated on-chain PROPOSAL_ID and mark CGP status as “PROPOSED”.
  5. Update your forum post with the Mainnet PROPOSAL_ID, updated timings (if any changes), and notify the community in the Discord #governance channel.
  6. At this point all stakeholders are encouraged to verify the proposed contracts deployed match the contracts from the release branch.
  7. Monitor the progress of the proposal through the governance process.
    • Currently the governance process should take approximately 1 week: 24 hours for the dequeue process, 24 hours for the approval process, and 5 days for the referendum process. After which, the proposal is either declined or is ready to be executed within 3 days.
    • For updated timeframes, use the celocli: celocli network:parameters
T+5w
  1. If the proposal passed:
    1. Confirm all contracts working as intended on Mainnet.
    2. Update your forum post with the Mainnet governance outcome ( Passed or Rejected) and notify the community in the Discord #governance channel.
    3. Change corresponding CGP status to EXECUTED.
    4. Merge the release branch into master with a merge commit
  2. If the proposal failed:
    1. Change corresponding CGP status to EXPIRED.
If the contents of the release (i.e. source Git commit) change at any point after the release has been tagged in Git, the process should increment the release identifier, and process should start again from the beginning. If the changes are small or do not introduce new code (e.g. reverting a contract to a previous version) the audit step may be accelerated.

​
Communication guidelines

Communicating the upcoming governance proposal to the community is critical and may help getting it approved. Each smart contract release governance proposal should be accompanied by a Governance category forum post that contains the following information:
  • Name of proposer (individual contributor or organization).
  • Background information.
  • Link to the release on Github.
  • Link to the audit report(s).
  • Anticipated timings for the Celo Sepolia testnet and Mainnet.
Make sure to keep the post up to date. All updates (excluding fixing typos) should be communicated to the community in the Discord #governance channel.

​
Emergency patches

Work in progress

​
Vulnerability Disclosure

Vulnerabilities in smart contract releases should be disclosed according to the security policy.

​
Dependencies

None

​
Dependents

Work in progress