Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bodhi does not expire buildroot overrides #616

Closed
mbooth101 opened this issue Sep 30, 2015 · 13 comments
Closed

Bodhi does not expire buildroot overrides #616

mbooth101 opened this issue Sep 30, 2015 · 13 comments
Assignees
@lmacken

Comments

@mbooth101
Copy link

See: https://bodhi.fedoraproject.org/overrides/eclipse-remote-2.0.0-0.1.git76ac23a.fc22

Expiration date is in the past, yet this override is still "active" and the build is tagged in koji: https://koji.fedoraproject.org/koji/buildinfo?buildID=680588

Clicking the "Expire" button in bodhi results in an error "Expiration date in the past"

Performing "koji untag-build" results in an error "ActionNotAllowed: tag requires fedora-override permission"

How do I untag this build?
@mbooth101
Copy link
Author

The polluted build root is preventing some other builds, so this is blocking other work.

@tyll
Copy link
Contributor

tyll commented Sep 30, 2015

I untagged it manually for you.

@mbooth101
Copy link
Author

Thanks -- I appear to have two other buildroot overrides with the same problem, for three in total:

https://bodhi.fedoraproject.org/overrides/eclipse-remote-2.0.0-0.1.git76ac23a.fc22
https://bodhi.fedoraproject.org/overrides/eclipse-launchbar-1.0.0-0.3.git01bfa62.fc22
https://bodhi.fedoraproject.org/overrides/eclipse-tm-terminal-4.0.0-3.fc22

These all show "active" in bodhi but the expiry date is long in the past.

@mbooth101
Copy link
Author

I am also having problems "re-tagging" these builds.... See https://fedorahosted.org/rel-eng/ticket/6270

@tyll
Copy link
Contributor

tyll commented Oct 7, 2015

afaics there is something missing (a cronjob?) that runs /usr/bin/bodhi-expire-overrides but it might also be broken - when I try to run it manually there are several tracebacks and the build is not untagged.

@tyll tyll added the ansible label Oct 7, 2015
@tyll
Copy link
Contributor

tyll commented Oct 7, 2015

Errors:

2015-10-07 19:25:46,677 INFO  [bodhi.scripts.expire_overrides][MainThread] Expiring 183 buildroot overrides...
2015-10-07 19:25:46,810 WARNI [bodhi][MainThread] No buildsystem configured; assuming testing
2015-10-07 19:25:46,810 DEBUG [bodhi][MainThread] untagBuild(f23-override, fedmsg-0.15.0-1.fc23)
2015-10-07 19:25:47,926 INFO  [bodhi][MainThread] fedmsg initialized
2015-10-07 19:25:47,926 DEBUG [bodhi][MainThread] fedmsg enqueueing 'buildroot_override.untag'
2015-10-07 19:25:47,926 INFO  [bodhi.scripts.expire_overrides][MainThread] Expired fedmsg-0.15.0-1.fc23
2015-10-07 19:25:47,959 DEBUG [bodhi][MainThread] fedmsg sending 'buildroot_override.untag'
2015-10-07 19:25:47,960 CRITI [txn.139633908451136][MainThread] A storage error occurred during the second phase of the two-phase commit.  Resources may be in an inconsistent state.
2015-10-07 19:25:47,960 ERROR [txn.139633908451136][MainThread] Error in tpc_abort() on manager <zope.sqlalchemy.datamanager.SessionDataManager object at 0x3a8d650>
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/transaction/_transaction.py", line 432, in _cleanup
    rm.tpc_abort(self)
  File "/usr/lib/python2.7/site-packages/zope/sqlalchemy/datamanager.py", line 110, in tpc_abort
    assert self.state is not 'committed'
AssertionError
Traceback (most recent call last):
  File "/usr/bin/bodhi-expire-overrides", line 9, in <module>
    load_entry_point('bodhi==2.0', 'console_scripts', 'bodhi-expire-overrides')()
  File "/usr/lib/python2.7/site-packages/bodhi/scripts/expire_overrides.py", line 69, in main
    log.info("Expired %s" % override.build.nvr)
  File "/usr/lib/python2.7/site-packages/transaction/_manager.py", line 120, in __exit__
    self.commit()
  File "/usr/lib/python2.7/site-packages/transaction/_manager.py", line 111, in commit
    return self.get().commit()
  File "/usr/lib/python2.7/site-packages/transaction/_transaction.py", line 280, in commit
    reraise(t, v, tb)
  File "/usr/lib/python2.7/site-packages/transaction/_transaction.py", line 271, in commit
    self._commitResources()
  File "/usr/lib/python2.7/site-packages/transaction/_transaction.py", line 417, in _commitResources
    reraise(t, v, tb)
  File "/usr/lib/python2.7/site-packages/transaction/_transaction.py", line 399, in _commitResources
    rm.tpc_finish(self)
  File "/usr/lib/python2.7/site-packages/bodhi/notifications.py", line 178, in tpc_finish
    fedmsg.publish(topic=topic, msg=msg)
  File "/usr/lib/python2.7/site-packages/fedmsg/__init__.py", line 79, in _wrapper
    return func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/__init__.py", line 94, in publish
    return __local.__context.publish(topic, msg, **kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/core.py", line 289, in publish
    self.c['certname'] = self.c['certnames'][cert_index]
KeyError: 'bodhi.bodhi-backend02'

@ralphbean
Copy link
Contributor

  • That No buildsystem configured error means that it isn't talking to koji. It's just talking to an in-memory shim that will end up doing nothing.
  • That KeyError I can fix! The cert just seems to be missing?

@ralphbean
Copy link
Contributor

OK, cert should be in place (fixing the second half).

@tyll
Copy link
Contributor

tyll commented Oct 7, 2015

I found the koji bug, the buildsystem needs to be setup from the settings afaics, will prepare a PR.

@ralphbean ralphbean mentioned this issue Oct 7, 2015
Merged
@tyll
Copy link
Contributor

tyll commented Oct 7, 2015

There are more issues:

@lmacken
Copy link
Contributor

lmacken commented Oct 8, 2015

In my local testing the expire_overrides.py job does correctly untag and expire them in the db. I'm going to setup the cron job on bodhi-backend02 after fixing the 'build is already untagged' exception.

lmacken added a commit that referenced this issue Oct 8, 2015
@lmacken
overrides: Catch exceptions when trying to untag overrides (#616)
68e4017
@lmacken
Copy link
Contributor

lmacken commented Oct 8, 2015

Building a new bodhi release now, and I have this change queued up in ansible.

--- a/roles/bodhi2/backend/tasks/main.yml
+++ b/roles/bodhi2/backend/tasks/main.yml
@@ -275,6 +275,15 @@
   - bodhi
   - cron

+- name: bodhi-expire-overrides cron job.
+  cron: name="bodhi-expire-overrides" hour="*" minute=0 user="masher"
+        job="/usr/bin/bodhi-expire-overrides /etc/bodhi/production.ini"
+        cron_file=bodhi-expire-overrides-job
+  when: inventory_hostname.startswith('bodhi-backend02') and env == "production"
+  tags:
+  - config
+  - bodhi
+  - cron

 - name: have the masher own the bodhi config
   file: >

@lmacken
Copy link
Contributor

lmacken commented Oct 8, 2015

This has been deployed and run. A ton of overrides have just been expired. Please reopen this issue if you notice anything fishy.

@lmacken lmacken closed this as completed Oct 8, 2015
@lmacken lmacken mentioned this issue Oct 8, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lmacken lmacken

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lmacken @tyll @ralphbean @mbooth101