I think there are multiple ways to go about this, which will also shape how we go about updating GCC's master with our frontend later down the line.

1. Add a changelog entry for each of those ~310 commits

I don't mind doing this, but it's a herculean task. I'm hoping that ./contrib/mklog.py will help with that, and that I can reuse some of the commits messages/PR descriptions

2. Add a changelog entry for each "bors merge"

A bors merge will contain one or more PRs, and can be used as a single point of reference in the frontend. We also haven't been the best at making sure each commit builds and passes the test (I know I'm at fault for this), but we do make sure that each bors merge passes our CI.

For this, we'd need to squash multiple commits together into a multitude of "batch commits" which will each contain one long Changelog entry, so one Changelog message per set of PRs merged together via bors.

Going forward, I also think this might be the best way to generate changelogs without hindering contributors or newcomers with Changelogs.

git log 36a9255..HEAD --author bors --oneline shows only 90 lines, which seems really low, but maybe?

I think there are two parts to this issue:

Sync the two git tree's

I think there are a few options like you have pointed out here:

1. Script a way to add changelog entries to each commit and upstream them
2. Do one mega commit just to copy and paste to make sure no changes are missed (loses author into and history)

I think if we work on scripting this will be useful regardless but we end up with git sha's that don't match in both tree's so it makes me think that option 2 might just be the simpliest.

On going upstream of changes

Question: do we want to enforce changelog entries on new patches?

I don't think so this seems really tedious to me and not helpful since git already provides this info

Bors script?

If we squash the commits and generate the changelog this is not good for multi commit PR's

Scripts

If we script it so we upstream each patch with a changelog that's done with a script we end up with git sha's that no longer match in each tree.

Possible solution moving forward

I think on going we need some mechanism to auto-generate changelogs for people, this will be possible with a git pre-commit hook to check clang-format and generate a changelog for us when we are making commit's from now on.

We can also turn on the magic that GCC uses to stop commits which don't have a changelog so that we don't miss this on merging as it should fail. Though this might make bors a bit of a pain because it always wants to do merge commits. Maybe bors is just not going to work long term.

I think if we work on scripting this will be useful regardless but we end up with git sha's that don't match in both tree's so it makes me think that option 2 might just be the simpliest.

Agreed, but also if we go through the trouble of updating those 310 commits it might be worth considering rewriting our history with these commits + Changelog

I think @dkm mentioned before an alternative to bors with a github pipeline thing that allowed to rebase changes etc maybe we should look at that moving forward.

https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue maybe this is it seems pretty much a bor's setup is possible with just github magic

https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue

Looks very similar to bors. It also groups pending changes together.

The way to use ./contrib/mklog.py is to first generate a patch for a commit, and then run the script on it.

For 56f503b88a240f1d77c8d6564656bc22269b4842, this produces the following log:

gcc/rust/ChangeLog:

	* resolve/rust-ast-resolve-path.cc (ResolvePath::resolve_path):
	* resolve/rust-ast-resolve-type.cc (ResolveTypeToCanonicalPath::visit):
	* resolve/rust-ast-resolve-type.h:
	* typecheck/rust-hir-type-check-path.cc (TypeCheckExpr::visit):

gcc/testsuite/ChangeLog:

	* rust/execute/torture/issue-1249.rs: New test.

No surprise to anyone here, I do have opinions on this topic. ;-) (..., but not time right now to discuss in detail.)

[email protected] ~/G/r/gccrs (master)> git log 36a9255..HEAD --format=%H -- gcc/rust/ > all_commits.old
[email protected] ~/G/r/gccrs (master)> git log 36a9255..HEAD --format=%H -- gcc/testsuite/rust >> all_commits.old
[email protected] ~/G/r/gccrs (master)> cat all_commits.old | uniq > all_commits

Just use git log 36a9255..HEAD --format=%H -- gcc/rust/ gcc/testsuite/rust/ > all_commits.

And, given git remote upstream git+ssh://gcc.gnu.org/git/gcc.git, further restrict to 36a9255..HEAD ^upstream/master to filter out all GCC upstream master branch commits that we're not interested in tracking here.

I agree that we should separate the two items: (1) getting the backlog upstream, and (2) how to deal with new work/commits.

For (1) I think we need some proper spreadsheet (or whatever; editable by all interested parties), so that we can cross out, for example, patch plus later reversion, and that we can mark up individual commits that belong together. (These will often correspond to one GitHub Pull Request, but may also span over several.) The bors merges sometimes combine several unrelated changes, so I'd prefer to not work based on these.

It's a bit more effort, but I suggest to not do bulk merges/mega commits, because that loses author information as well as comprehensibility of a changeset. Generally it's better to handle conceptually different items individually. (..., but, of course, merge in any partial reversions, fix-up commits, etc.)

For (1) I think we need some proper spreadsheet (or whatever; editable by all interested parties), so that we can cross out, for example, patch plus later reversion

Good idea, even if I'm not a fan of a spreadsheet :D

, and that we can mark up individual commits that belong together. (These will often correspond to one GitHub Pull Request, but may also span over several.)

Do you mean that these should be squashed together in one "upstream commit"?

The bors merges sometimes combine several unrelated changes, so I'd prefer to not work based on these.

:+1:

It's a bit more effort, but I suggest to not do bulk merges/mega commits, because that loses author information as well as comprehensibility of a changeset. Generally it's better to handle conceptually different items individually. (..., but, of course, merge in any partial reversions, fix-up commits, etc.)

Yeah, I'd rather do Changelog entries for all 300 commits than lose author information. I think this needs to be a focal point of this whole thing

For (1) I think we need some proper spreadsheet (or whatever; editable by all interested parties), so that we can cross out, for example, patch plus later reversion

Good idea, even if I'm not a fan of a spreadsheet :D

I do sympathize! ;-)

But it works for that case. ;-) (Doing the same at work, too, to get changes from our development branches into GCC upstream.) Could also use Git notes, or a lot of other things, but in that case, a simple spreadsheet does get the job done.

If you'd like, I can set this up, so that you can see what I mean.

, and that we can mark up individual commits that belong together. (These will often correspond to one GitHub Pull Request, but may also span over several.)

Do you mean that these should be squashed together in one "upstream commit"?

Yes. For example, things where we currently have individual pull requests with commits like:

PR1:

• fix issue in file 1
• revert, do it differently
• fix same issue in file 2

PR2:

• fix same issue in files 3, 4
• add test cases

... that should become one "upstream commit".

I think most of our PRs are done in ways where each commit makes sense as a standalone. I know that apart from a few straggler PRs I've taken the time to squash commits like the ones you mention, so for your examples:

PR1:

• fix issue in file 1
• do it differently
• fix same issue in file 2

gets rebased before merging and becomes

• fix issue in file 1
• fix same issue in file 2

but yes, that leaves joining PR1 and PR2. Which seem like a lot of work for some things...

FYI, I've been playing a bit with this and first tried to apply all missing commits onto current gcc master. The command used to get the commit list

 git rev-list --reverse --no-merges  '^upstream-gcc/master' '^36a9255' upstream-gccrs/master   -- gcc/rust/ gcc/testsuite/rust/ 

I had to skip several commits already applied, and in the end, the result is mostly in good shape... But I'm wondering if the cutoff date is the correct one, because the "mostly" above hints that something is not quite right.

For example, 36a9255 (last change from gccrs from github to have been part of the merge in GCC) is a parent of f5f32fb4a36c742f0b6b5cac994550b3003c9f3a (the inline visitor)... but the corresponding files are nowhere to be found in what has been merged in GCC.

Files gccrs-master/gcc/rust/ChangeLog and gcc-master/gcc/rust/ChangeLog differ
Only in gcc-master/gcc/rust: CONTRIBUTING.md
Only in gccrs-master/gcc/rust: gource-gccrs.sh
Files gccrs-master/gcc/rust/lang.opt and gcc-master/gcc/rust/lang.opt differ ## empty line diff
Files gccrs-master/gcc/rust/lex/rust-token.h and gcc-master/gcc/rust/lex/rust-token.h differ ## empty line diff
Only in gcc-master/gcc/rust: logo.png
Only in gccrs-master/gcc/rust: monthly-diff.py
Files gccrs-master/gcc/rust/parse/rust-parse-impl.h and gcc-master/gcc/rust/parse/rust-parse-impl.h differ ## Real diff, also hints something is not correct
Only in gcc-master/gcc/rust: README.md
Files gccrs-master/gcc/rust/rust-lang.cc and gcc-master/gcc/rust/rust-lang.cc differ ## some include changes
Only in gccrs-master/gcc/rust/util: rust-inline-visitor.h  ## the inline visitor missing
Files gccrs-master/gcc/rust/util/rust-make-unique.h and gcc-master/gcc/rust/util/rust-make-unique.h differ ## indent change

I may have more time this weekend to have a look, maybe this is all wrong... The wip branch https://github.com/Rust-GCC/gccrs/tree/dkm/gcc-master-updated-to-0152926ab36

I've messed up a bit some commits around bootstrap flags, so some commits should have been skipped but instead caused conflict and still appear in the branch. I've listed all commits that caused conflict or have been skipped.

FYI, some commits are only in gccrs, but applied to generic code. For example: 05f1f87274a4ed13c0ab84de77d4253776b46637:

commit 05f1f87274a4ed13c0ab84de77d4253776b46637
Author: David Malcolm <[email protected]>
Date:   Thu Jul 21 18:35:01 2022 -0400
 
    diagnostics: add error_meta
 
 gcc/diagnostic-core.h |  3 +++
 gcc/diagnostic.cc     | 15 +++++++++++++++

So filtering rust/ and testsuite/rust may miss some commits.

FWIW, I've pushed https://github.com/Rust-GCC/gccrs/tree/dkm/gcc-master-rebase-on-pr that adds few missing commits. There are some minor diff I'm trying to track down, but bootstrap and test are looking good so far.

FYI, some commits are only in gccrs, but applied to generic code. For example: 05f1f87:

commit 05f1f87274a4ed13c0ab84de77d4253776b46637
Author: David Malcolm <[email protected]>
Date:   Thu Jul 21 18:35:01 2022 -0400
 
    diagnostics: add error_meta
 
 gcc/diagnostic-core.h |  3 +++
 gcc/diagnostic.cc     | 15 +++++++++++++++

So filtering rust/ and testsuite/rust may miss some commits.

Ah, I think we should ask @davidmalcolm about that one. @dkm do you know if there is a patch associated with them yet?

AFAIK, there's no existing patch pending for review for @davidmalcolm 's patch (same as for @ibuclaw target related patches).

You can see the current diff between the branch merging GCC's master in gccrs's master and the sequence of gccrs's commits being rebased onto the same GCC's master : https://github.com/Rust-GCC/gccrs/compare/dkm/merge-master..dkm/gcc-master-rebase-on-pr

• files are moved from the root directory to gcc/rust (all the CONTRIBUTING.md, logo, etc)
• .github stuff
• .gitignore
• gource config removed
• gcc/configure: minor diff, caused by 066b6b8df960796362dbdcff62851ebea201b8e9
• gcc/rust/lang.opt: blank space
• gcc/rust/lex/rust-token.h: blank space
• gcc/rust/parse/rust-parse-impl.h: Jan's patch a bit different than Arthur's patch for the same issue.
• gcc/rust/rust-lang.cc: include diff
• gcc/rust/util/rust-make-unique.h: blank space
• gcc/testsuite/rust/execute/torture/macro-issue1426.rs file moved and changed (not sure which one is to be used).

Commits since 2022-08-23 need Changelog entries

All my changes should have had a changelog in the commit message, unless something happened during the merge process.

Yes, they have! Do you want your commits to be sent along with the other ones for review, or do you want to handle them yourself @ibuclaw ?

I believe we've now reached a point where the only work remaining is to do the actual work :) @dkm put together a list of the commits we need to check before integrating them into GCC's upstream. Since we want to preserve authorship, this concerns 224 commits. Again, thanks to Marc we have a nice script to perform all of the checks :)

In order to keep the GCC history nice, clean and bisectable, we need to make sure that each of these commits is in a proper state. To be in a proper state, a commit should be able to bootstrap and then pass the testsuite.

We are thus running bootstrap builds for each of those 224 commits and logging whether or not each of these builds. If a commit builds, then we'll write a Changelog for it and open a PR to gcc-patch-dev. Then, I'll submit it to the mailing list for review (if needed) or just as [committed] for those commits that only affect gcc/rust/ files.

If a commit does not build, we'll integrate it to a valid commit and squash them together. Since all of our pull-requests end up valid, I'm hoping that at maximum we'll only have to squash together a couple of commits and that it should be easy.

@dkm and I have started that painful process and you should see PRs popping up :)

I think we should soon open up a new "discussion" issue to discuss how to move forward with contributors and Changelogs :) As @philberty pointed out in Zulip, we have git gcc-commit-mklog and the associated ./contrib/mklog.py script to generate Changelog skeletons for us. I'm thinking that it would be good to integrate this with a Github bot, whose job would be to check that each commit in a PR contains a valid Changelog. For new contributors, I think it would also be useful if that bot was able to generate that Changelog skeleton (so simply calling ./contrib/mklog.py on each commit) and post it on Github. This will help new users figure out the format and not have to deal with too much of the annoyances.

The most important thing will be to figure out a name for the bot, obviously :D

I think we should keep this issue open for a bit longer, as I'm sure more issues will prop up and this will be a good place to discuss them.

Thanks everyone for your help!!!

I've opened a PR https://github.com/Rust-GCC/gccrs/pull/1738 to enforce gnu changelog on our PR's this is just for discussion I wont merge this and force it though but since finding the tools in contrib I think it makes sense.

I've pushed a branch (first-half-update, name is garbage but I don't expect it to live for very long) containing the first half of the ~224 commits we need to push to GCC's master to update the frontend. All of these commits pass a bootstrap build and tests on my machine. If you fetch this remote, you'll have access to the all of these commits, for which we need to write Changelog entries. I've added the list to the original message of this issue.

I'm not sure on how to go about splitting them between us in order to write the CLs. I know @dkm wanted to contribute to some of those, but we do need to apply them in order to gcc-patch-dev, so I don't really know how to go about that. I also don't mind doing most of those as it's a long and boring task :)

I've opened a PR #1738 to enforce gnu changelog on our PR's this is just for discussion I wont merge this and force it though but since finding the tools in contrib I think it makes sense.

@philberty I think this is also going to come in very handy for this process. The PR is good imo and I want it merged.

for this specific workflow, there are a few things I think we should add to that script:

1. That all commits start with gccrs:
2. That all changelog lines end with a period.

I'm not sure about number 2, but I think it's a "requirement" as all commits in GCC have Changelogs finishing with periods. This was also one of the review comments we got one the mailing list.

❯❯❯ git log --oneline --grep 'Merge pull ' $(git merge-base upstream-gcc/master HEAD)..upstream-gccrs/gcc-patch-dev                                  
d9f4a1849a3 (upstream-gccrs/gcc-patch-dev, upstream-gccrs-rw/gcc-patch-dev) Merge pull request #1748 from CohenArthur/cl-cleanup-macro-expansion-and-parsing
8e8bf50c1be Merge pull request #1749 from CohenArthur/cl-do-not-lint-public-items
3af69068516 Merge pull request #1743 from Rust-GCC/cls-double-borrows-builtin-overflow-refactor
42d081e810a Merge pull request #1745 from Rust-GCC/dkm/gcc-patch-dev-constexpr
300fd6d7dce Merge pull request #1721 from CohenArthur/first-update-changelog-tryout
92a9a32895e Merge pull request #1746 from CohenArthur/cl-automation

See the cleaned branch: https://github.com/Rust-GCC/gccrs/pull/new/dkm/gcc-patch-dev/remove_merge

Discovered that git rebase automatically removes empty merge:

git rebase -i $(git merge-base upstream-gcc/master HEAD)

the todo only contains real commits without the empty merges.

I've since force-pushed @dkm's branch on gcc-patch-dev so that the Merge commits are no longer present :) going forward I'll make sure to hit the rebase and merge button instead haha