Change Release Protocol

Abhishek Srivastava

Published: 2023-02-20 DOI: 10.17504/protocols.io.5jyl8jk57g2w/v1

Abstract

This is a set of protocols for releasing the changes or new feature to staging and production. This in being introduced so as the developer does not skip any critical step required during the whole release process. Any critical step if is missed could lead to incidents in production environment.

Steps

Merge checklist Staging

Merge checklist Staging

1.1.

Regen pb.go from shared-proto master branch

1.2.

Create Gitlab PR for config change on Staging

1.3.

Merge Gitlab PR for Staging

1.4.

Run Migration Scripts if any on Staging

1.5.

Test in local

1.6.

Test in feature env:

1.7.

Test Party

Merge checklist Production

2.1.

Run Migration Scripts if any on Production

2.2.

Create Gitlab PR for config change on Staging

2.3.

Merge Gitlab PR for Staging

2.4.

Merge shared-proto change (if applicable)

Starting the Jenkins Pipeline

Starting the Jenkins Pipeline. Cutoff 3pm unless absolutely necessary/hot-fix

3.1.

Announce on #backend-release channel

3.2.

Check the result of QA test suite run on #qa-backend-releases slack channel

Citation

Zero Test Case Failures

3.3.

If any failures then check with the QA engineers

3.4.

Get a go ahead from QA engineers to proceed in case of failures

3.5.

Make sure config changes done

3.6.

Make sure migration done if required

3.7.

Proceed to Canary

Validate Changes on Canary

4.1.

Check logs on Canary Server

cat /tmp/carousell-stdout.log|grep -o -E -i ".*panic.*"|wc -l

4.10.

Update the PR Template

4.2.

Check for errors related to your grpc method

tail -100f /tmp/carousell-stdout.log|grep -E -i "\"method\":\".*<your_grpc_method_name>.*

4.3.

Verify Changes on Canary if possible using port-forwarding

4.4.

Check Grafana Dashboards for Hystrix and GRPC for error rates on New Canary pod

4.5.

Wait for another 1 hour at least for baking the changes in Canary [Mandatory]

4.6.

Check Grafana Dashboards for Hystrix and GRPC for error rates on New Canary Pod

4.7.

If all good, proceed for all server roll-out

4.8.

Verify Changes by completing the whole new flow on app/web to see all good with new feature/ old feature

4.9.

Attach Screenshots from test on production

Author Correction: Efficient and strand-specific profiling of replicating chromatin with enrichment and sequencing of protein-associated nascent DNA in mammalian cells

Practical and concise synthesis of nucleoside analogs

Preparation and quantitative analysis of multicenter luminescence materials for sensing function

Inferring cellular and molecular processes in single-cell data with non-negative matrix factorization using Python, R and GenePattern Notebook implementations of CoGAPS