Skip to content

Run migrations in order across repos - #4466

Merged
zoldar merged 21 commits into
masterfrom
order-migrations
Sep 9, 2024
Merged

Run migrations in order across repos#4466
zoldar merged 21 commits into
masterfrom
order-migrations

Conversation

@ruslandoga

@ruslandoga ruslandoga commented Aug 24, 2024

Copy link
Copy Markdown
Contributor

Changes

This PR makes /entrypoint.sh db migrate and /entrypoint.sh db pending-migrations order all pending migrations and run them in a single queue instead of per-repo.

Related card: https://3.basecamp.com/5308029/buckets/29267832/card_tables/cards/7508896723

Contents of that card

Some migrations in PostgreSQL implicitly depend on migrations in ClickHouse (there is at least one such migration), and because PostgreSQL migrations run first, they fail because from their point of view ClickHouse schemas are now "outdated":

How migrations were added (example):

  • add table to PostgreSQL db
  • add column to ClickHouse table
  • run PostgreSQL data migration that reads from new column in ClickHouse

How migrations are run:

  • run all Plausible.Repo migrations
  • run all Plausible.IngestRepo migrations

Which results in this broken order:

  • add table to PostgreSQL db
  • run PostgreSQL data migration that reads from new column in ClickHouse <- fails
  • add column to ClickHouse table <- never runs

It might be possible to run them in a single queue / order in db migrate command:

  • collect all migrations to be run, order them by version / date
  • start all repos (Repo, IngestRepo, ClickhouseRepo)
  • execute them one by one

Tests

  • Automated tests have been added (only for pending migrations listings, I'll try to add an actual db migrate test later)

Changelog

  • Entry has been added to changelog

Documentation

  • This change does not need a documentation update

Dark mode

  • This PR does not change the UI

@ruslandoga ruslandoga changed the title Order migrations in CE Order migrations Aug 24, 2024
@ruslandoga ruslandoga changed the title Order migrations Run migrations in order across repos Aug 26, 2024
@ruslandoga ruslandoga added the self-hosting Anything self-hosted label Aug 27, 2024
Comment thread lib/plausible_release.ex Outdated
This was referenced Aug 29, 2024
@ruslandoga
ruslandoga marked this pull request as ready for review August 30, 2024 04:12
@ruslandoga
ruslandoga requested review from a team August 30, 2024 04:12
Comment thread lib/plausible_release.ex Outdated
@zoldar zoldar added the preview label Sep 6, 2024
@github-actions

github-actions Bot commented Sep 6, 2024

Copy link
Copy Markdown
Preview environment👷🏼‍♀️🏗️
PR-4466

@zoldar
zoldar merged commit 8ee4827 into master Sep 9, 2024
@zoldar
zoldar deleted the order-migrations branch September 9, 2024 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

preview self-hosting Anything self-hosted

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants