Migrate to Xormigrate

[qwerty287]

Migrate the migration handling to xormigrate instead of the native handling.

The main difference between the methods: gitea has a version-based system, xormigrate uses id-based. This means: In theory, each migration could be executed standalone (xormigrate will however execute them in order).

This allows backporting migrations, and xormigrate is already in use in projects like woodpecker ci (and is working well there).

[techknowlogick]

Thanks @qwerty287! I need to reply to your email, but thanks for getting started on this.

[qwerty287]

I think this is ready for a first review round. Tests pass both in CI and also in my manual tests, everything seems to work.

@techknowlogick

[techknowlogick]

With backports, we could do more than modify the database; we could add doctor commands to be run, so we still meet the requirement of being changeable between minor versions.

[lunny]

With backports, we could do more than modify the database; we could add doctor commands to be run, so we still meet the requirement of being changeable between minor versions.

Since we have a web UI for administration, I think it's better to let the user do a self-check there and do a fix there. But not fix them automatically because some doctor commands may take a long time.

In summary, I couldn't find the necessary and obvious benefits to migrate to Xormigrate. It cannot resolve the backports in minor versions.

[qwerty287]

It cannot resolve the backports in minor versions.

It cannot do that alone obviously, because if there's a dependency of a migration defined, it must be backported as well. Xormigrate is not an automation tool that does this, the devs still have to do this.
Implementing strict dependency checking in xormigrate is not a big problem and could help here - if I should follow this approach let me know.

[lunny]

It cannot resolve the backports in minor versions.

It cannot do that alone obviously, because if there's a dependency of a migration defined, it must be backported as well. Xormigrate is not an automation tool that does this, the devs still have to do this. Implementing strict dependency checking in xormigrate is not a big problem and could help here - if I should follow this approach let me know.

I think my concerns are still here which ask the migrations system support both dependency and rollback. And I can't guarantee the rollback can always work because some migrations are unrevertable. So that will still break the compatibility.

Does this backported migration have dependency migrations that haven't been backported? It's better to have clear >dependencies on the code but not according to our brain which can reduce possible backport problems.
Once 1.xx.2 merged a backported migration, how user downgrades to 1.xx.1 like @wxiaoguang said? It's very important >once 1.xx.2 have some big bugs for users. It's also a compatible promise. So maybe a rollback migration needs to be >supported?

[qwerty287]

Does this backported migration have dependency migrations that haven't been backported? It's better to have clear dependencies on the code but not according to our brain which can reduce possible backport problems.

That's what I said above. Currently xormigrate does not support this, but I can implement it there.

Once 1.xx.2 merged a backported migration, how user downgrades to 1.xx.1 like @wxiaoguang said? It's very important once 1.xx.2 have some big bugs for users. It's also a compatible promise. So maybe a rollback migration needs to be supported?

Xormigrate supports rollbacks. You would probably need some cli command to perform this.

[qwerty287]

@lunny what about my suggestion above? #31523 (comment)

[lunny]

Does this backported migration have dependency migrations that haven't been backported? It's better to have clear dependencies on the code but not according to our brain which can reduce possible backport problems.

That's what I said above. Currently xormigrate does not support this, but I can implement it there.

Once 1.xx.2 merged a backported migration, how user downgrades to 1.xx.1 like @wxiaoguang said? It's very important once 1.xx.2 have some big bugs for users. It's also a compatible promise. So maybe a rollback migration needs to be supported?

Xormigrate supports rollbacks. You would probably need some cli command to perform this.

Thanks for the effort to push the pull request. I think even if both of these two features are implemented. Backporting migrations to a stable version is still dangerous. Most of the time, rollback cannot work well. Assume we delete a column in v1.22.1, I don't know how it can be rolled back when we are v1.22.2. The column data has been missed, it cannot be rolled back in real. If this pull request cannot bring real benefit but more risk, I don't know how should I agree to merge this.

Taking a look at Golang which Gitea has a similar versioning, I don't think they will have a traditional LTS version plan.

[qwerty287]

OK @techknowlogick what do you say about this?

[techknowlogick]

@lunny we don't need to add rollback support if this library is included, it is not mandatory. We would also need to ensure that if any migrations are backported, they still meet our upgrade/downgrade on minor version compatibility promises.

One of the biggest benefits is that our migrations can align with migrations approaches in many other software offerings (rails, laravel, and many more) where instead of relying on a single int, all of the migrations that have been run are saved into the DB. We also don't need to no-op migrations, as we only had to do that because we relied on an int to be the same (allows for code cleanliness). We could also have doctor tasks run on prior versions without having to require a user to manually run something in the CLI.

Assume we delete a column in v1.22.1, I don't know how it can be rolled back when we are v1.22.2. The column data has been missed, it cannot be rolled back in real. If this pull request cannot bring real benefit but more risk, I don't know how should I agree to merge this.

Deleting a column is already possible in the existing code on minor versions, the only thing that is stopping that is those PRs would be viewed by a maintainer and blocked. So we would keep those same policies, and if a migration that doesn't meet the upgrade/downgrade policy on minor versions it would be blocked.

[qwerty287]

@lunny I'd like to get that done. Could you check out @techknowlogick's comment?