Bamboo Server fails to Upgrade Plans
Symptoms
Bamboo Server fails to run the upgrade task and shows the following error:2012-12-11 15:45:51,160 INFO [6-UpgradeTaskBackgroundThread:pool-13-thread-1] [AbstractUpgradeManager] Running upgrade task 2701 : Set change set IDs for commits
2012-12-11 15:45:51,160 INFO [6-UpgradeTaskBackgroundThread:pool-13-thread-1] [AbstractUpgradeManager] ----------------------------------------------------------
2012-12-11 15:45:57,871 INFO [6-UpgradeTaskBackgroundThread:pool-13-thread-1] [UpgradeTask2701SetCommitId] Updating commit information for plan PROJ-PLAN (1 of 494)
2012-12-11 15:45:57,954 WARN [6-UpgradeTaskBackgroundThread:pool-13-thread-1] [IgnoreMissingFieldXStream] Unable to resolve class for element name myEnableRemoteApi. Null being returned.
2012-12-11 15:45:57,986 ERROR [6-UpgradeTaskBackgroundThread:pool-13-thread-1] [AbstractUpgradeManager] org.springframework.orm.hibernate.HibernateSystemException: Exception occurred inside getter of com.atlassian.bamboo.chains.ChainStageImpl.index; nested exception is net.sf.hibernate.PropertyAccessException: Exception occurred inside getter of com.atlassian.bamboo.chains.ChainStageImpl.index
org.springframework.orm.hibernate.HibernateSystemException: Exception occurred inside getter of com.atlassian.bamboo.chains.ChainStageImpl.index; nested exception is net.sf.hibernate.PropertyAccessException: Exception occurred inside getter of com.atlassian.bamboo.chains.ChainStageImpl.index
Caused by: net.sf.hibernate.PropertyAccessException: Exception occurred inside getter of com.atlassian.bamboo.chains.ChainStageImpl.index
at net.sf.hibernate.property.BasicPropertyAccessor$BasicGetter.get(BasicPropertyAccessor.java:99)
at net.sf.hibernate.persister.AbstractEntityPersister.getPropertyValues(AbstractEntityPersister.java:253)
at net.sf.hibernate.impl.SessionImpl.flushEntity(SessionImpl.java:2504)
at net.sf.hibernate.impl.SessionImpl.flushEntities(SessionImpl.java:2488)
at net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2283)
at net.sf.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1817)
at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:3654)
at net.sf.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:238)
at com.atlassian.bamboo.repository.PlanRepositoryLinkHibernateDao$1.doInHibernate(PlanRepositoryLinkHibernateDao.java:43)
....
at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$2.run(NamedThreadFactory.java:52)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor200.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at net.sf.hibernate.property.BasicPropertyAccessor$BasicGetter.get(BasicPropertyAccessor.java:96)
... 56 more
Caused by: java.lang.IllegalStateException: Unable to find Stage Test in Chain com.atlassian.bamboo.chains.DefaultChain@c03aa7e3
at com.atlassian.bamboo.chains.ChainStageImpl.getIndex(ChainStageImpl.java:158)
... 60 more
2012-12-11 15:45:57,988 INFO [6-UpgradeTaskBackgroundThread:pool-13-thread-1] [AbstractUpgradeManager] Completed upgrade task 2701 with errors.
2012-12-11 15:45:57,989 FATAL [main] [UpgradeLauncher] Upgrade task error: Upgrade task for build 2701 failed with exception: Exception occurred inside getter of com.atlassian.bamboo.chains.ChainStageImpl.index; nested exception is net.sf.hibernate.PropertyAccessException: Exception occurred inside getter of com.atlassian.bamboo.chains.ChainStageImpl.index
2012-12-11 15:45:57,990 INFO [main] [ContextHandler] started o.e.j.w.WebAppContext{/,file:/bamboo/ibamboo432/webapp/},./webapp
2012-12-11 15:45:58,046 INFO [main] [/] [CompressingFilter/1.4] CompressingFilter has initialized
2012-12-11 15:45:58,073 INFO [main] [/] org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: loaded (conf ok)
2012-12-11 15:45:58,074 INFO [main] [AccessLogFilter] AccessLogFilter initialized. Format is: <user> <url> <starting memory free (kb)> +- <difference in free mem (kb)> <query time (ms)>
2012-12-11 15:45:58,126 INFO [main] [AbstractConnector] Started SelectChannelConnector@0.0.0.0:8080
Cause
This is caused due to corruption in the Bamboo database; the LIST_POSITION column of CHAIN_STAGE table, which stores the list order of Bamboo jobs, is corrupted and has the same value for the same BUILD_ID.
Resolution
To check if you're affected by this issue, execute the following command:
select build_id, list_position, count(build_id) from chain_stage group by build_id,list_position having count(build_id)>1;
If there is any row in the result that has a bigger than 1 for count(build_id), then you are facing a DB corruption problem. Get the build_id and list_position from those rows. Let's say the build_id=105906195. Now run the following SQL:
select stage_id, list_position from chain_stage where build_id=105906195;
STAGE_ID LIST_POSITION
---------- -------------
106004493 0
106004492 0
106004494 0
The LIST_POSITION values are all 0s for different stage_IDs with the same build_ID. Instead the LIST_position should be fixed to look be 0,1,2 for different STAGE_IDs with the same BUILD_ID.
For every returned row, make sure that there is only one result when you run queries like these:
select stage_id, build_id, list_position from chain_stage where build_id=105906195 and stage_id=106004492;
select stage_id, build_id, list_position from chain_stage where build_id=105906195 and stage_id=106004493;
select stage_id, build_id, list_position from chain_stage where build_id=105906195 and stage_id=106004494;
update chain_stage set LIST_POSITION=1 where build_id=105906195 and stage_id=106004493;update chain_stage set LIST_POSITION=2 where build_id=105906195 and stage_id=106004494;
As an alternative especially if there are many rows to be fixed you can renumerate all of them at once. This query works only in Postgresql though
SELECT stage_id, row_number() over (partition by build_id order by list_position, stage_id) - 1 as new_list_position INTO TEMP fix_chain_stage_table FROM chain_stage ORDER BY build_id, list_position, stage_id;
UPDATE chain_stage SET list_position = f.new_list_position FROM fix_chain_stage_table f WHERE chain_stage.stage_id = f.stage_id;
DROP TABLE fix_chain_stage_table;
Start your Bamboo server, and you should be good to go. By the way, if there are more than one plans involved, make sure to do the above mentioned steps for all of them.