Hibernate lost my update

Events that won’t go through

First task at a new job: “find out why the events in the queue can’t be processed”. A great first task to get to know the company’s tools – and completely unrelated to Hibernate, whose source code is where I ended up. The code in short: in one transaction we mark a parent record and its records as deleted (soft delete) + write a transactional outbox (opens in a new tab) with an event for every deleted record.

OK, so I check the logs and monitoring. A later event, one that required all that data to be deleted already, kept landing on the DLT. Looks like there’s some inconsistency in the database. First suspect – a bug in the code, or a race condition thanks to Read Committed isolation. Nope: the code looked 100% fine and hadn’t changed since that event showed up.

The pod’s last logs

I noticed that the logs from that operation were among the last ones on the pod, because autoscaling was just stopping it. But that wasn’t the most interesting part. Another service had read the event about a record being deleted, and on our side that record wasn’t deleted at all… Interestingly, the last event of that transaction was about exactly the record whose change disappeared.

Since the code looked fine, I asked the database. Deleting is just a deleted flag for us, so the rows were still there. I learned that every row in Postgres has a hidden xmin column: the id of the transaction that wrote its current version. age(xmin) tells you how many transactions ago that was.

1
SELECT id, deleted, age(xmin) AS transactions_ago
2
FROM record
3
WHERE parent_id = 42;

The parent record and the deleted records – a fresh xmin, from the incident’s transaction. The record that wasn’t deleted – an xmin from hundreds of millions of transactions ago. So nobody had touched that row in a very long time. Which means the UPDATE wasn’t overwritten by another transaction – Postgres simply never got it. But in that same transaction it did get, and commit, the outbox row with the event about deleting that record.

“Connecting the dots” and ruling out a bug in the code or in the database, I started to suspect Hibernate. With some help from AI I found an interesting bit in beforeTransactionCompletion(). But one thing at a time – first: when does Hibernate send changes to the database at all?

How Hibernate writes your changes

If you know how the persistence context works and when Hibernate flushes, you can skip this one – next up is section 1.4.

In a nutshell – Hibernate (when it can) keeps changes to entities in the “persistence context” and sends them to the database only at a flush. A flush runs before a query that might read those changes, and always right before the commit (or so I thought).

Not everything waits for the flush, though. These go to the database right away1:

• the INSERT of an entity with an IDENTITY id, because the database assigns the id
• hand-written update/delete in JPQL or native SQL
• anything that goes through plain JDBC
• changes already sent by an earlier flush

Everything else – changed entity fields, remove(), new entities with a sequence id – waits in memory for the flush. So if the flush at commit didn’t run, the database would end up with only what had been sent before.

That last point explains why it was the last record that got lost. Hibernate flushes not only at commit, but also before a query, if that query reads a table with something waiting to be written. Our code looked roughly like this:

1
for (Long recordId : recordIds) {
2
    Record record = recordRepository.findActiveById(recordId); // query on the record table
3
    record.setDeleted(true);                                     // waits in memory
4
    outbox.save(new RecordDeletedEvent(recordId));               // JDBC, sent right away
5
}

Now step by step, for three records:

1. Query for record 1 – nothing is waiting yet, so no flush. Record 1 gets deleted = true in memory, the event goes to the outbox.
2. The query for record 2 reads the record table, and the change to record 1 is waiting there – so Hibernate flushes first, and the UPDATE of record 1 goes to the database. Then the same for record 2 and its event.
3. The query for record 3 pushes out the change to record 2 the same way. Record 3 is marked in memory, the event goes to the outbox.
4. End of the loop. No more queries for records, so the change to record 3 waits for the flush at commit…
5. …which doesn’t run. The database gets the events for all three records, but the UPDATE for only the first two.

Exactly what xmin showed.

Which leaves the question: why didn’t the flush at commit run?

When Hibernate skips the flush at commit

This is where beforeTransactionCompletion() comes back. At commit it calls managedFlush(), which checks whether the session is open. Here’s what that looks like in Hibernate 7.4.5:

1
private void managedFlush() {
2
    if ( !isOpenOrWaitingForAutoClose() ) {
3
        SESSION_LOGGER.skippingAutoFlushSessionClosed(); // TRACE
4
    }
5
    else {
6
        SESSION_LOGGER.automaticallyFlushingSession();
7
        fireFlush();
8
    }
9
}
10

11
// inherited from AbstractSharedSessionContract
12
public boolean isOpenOrWaitingForAutoClose() {
13
    return !closed && factory.isOpen()
14
        || waitingForAutoClose;
15
}

When the factory is closed, every one of its sessions counts as closed, so managedFlush() logs Skipping auto-flush since the session is closed at TRACE2 and returns, and Connection.commit() goes to Postgres, which commits everything it got. Even worse – nothing throws, the only trace is that one log line at TRACE level (!).

You don’t need Spring to reproduce it, or even a second thread. In the test below Payment has an IDENTITY id, so its INSERT goes out at persist(), while the order’s new status waits for the flush. We close the factory, commit and… the payment is there, and the order is still NEW.

1
@Test
2
void closeAndCommitOnTheSameThreadAlsoSaveOnlyPartOfTheTransaction() throws Exception {
3
    Session session = sessionFactory.openSession();
4
    Transaction transaction = session.beginTransaction();
5
    Payment payment = new Payment();
6
    payment.orderId = 1L;
7
    session.persist(payment);
8
    session.find(Order.class, 1L).status = "PAID";
9

10
    sessionFactory.close();
11
    transaction.commit();
12
    session.close();
13

14
    assertEquals(1, count("select count(*) from payments where orderId = 1"), "the INSERT was committed");
15
    assertEquals("NEW", status(), "the UPDATE was never executed, yet the transaction was committed");
16
}

I reproduced it on Hibernate 6.6.53, 6.6.59, 7.4.5 and 7.4.13 with PostgreSQL, and on 7.4.13 and 8.0.0.Beta3 with H2 (on Hibernate’s test case template). I didn’t check versions older than 6.6.

Which leaves the question: who closes the factory in the middle of a transaction?

What happens on SIGTERM

Remember that autoscaling was just stopping the pod? On SIGTERM, Spring Boot shuts down in two stages. First (the SmartLifecycle phase) it stops whatever can create new work: HTTP endpoints, listeners/consumers etc. + gives them a limited time to finish (by default: the Kafka container waits 10 s, RabbitMQ 5 s, and each shutdown phase in Spring Boot at most 30 s). Then it destroys beans, our EntityManagerFactory among them, and finally the DataSource.

The problem is that the second stage assumes the first one finished everything, and a handler waiting for a slow external API may not make it in time. Spring moves on to the next phase and starts destroying beans even though the thread is still running. The thread finally gets its response, leaves the @Transactional method, and Hibernate commits without a flush, because the EntityManagerFactory is already closed.

I’ll show it on a demo application (GitHub, opens in a new tab): a small shop where a single @Transactional method loads an order, saves a payment, marks the order as paid and calls a slow fraud check (yes, normally we’d split this into several messages, but it’s a good example to show the problem with Hibernate). Below is that transaction step by step – in step 4, while the thread waits, Spring closes the factory. Leave the guard switch (more on it later) off for now.

And here’s how it plays out in time – the worker thread and Spring shutting down, side by side:

Funnily enough, with SIGKILL the connection drops and Postgres rolls back. A partial commit needs exactly the “polite” (SIGTERM) shutdown.

Is it just our app?

At first I thought it was a configuration issue in our service, so I built a clean Spring Boot app (GitHub, opens in a new tab) + Spring JPA + Spring Kafka/AMQP. The result – surprising. I reproduced the bug there too, which means it can hit any application (with a bit of [bad] luck).

The repo is the same shop as in the demos above: Spring Boot 4 + Hibernate 7 + Postgres, with Spring MVC, Kafka and RabbitMQ. Each test sends a payment in through a different entry point, stops the pod in the middle of the fraud check, and finally looks into the database to verify the result.

One thing is deliberately “rigged”: a MetricsExporter bean takes 0.5 s to close and is wired so that Spring destroys it between Hibernate and the pool. That stretches the window (more on it in the next section) from ~1.2 ms to half a second, so the tests hit it every time and are deterministic. Everything else – Spring, Hibernate, HikariCP, the listener containers – runs unchanged. The test that measures the window itself works without this trick – we have that one too :)

Results (the “With the guard” column will make sense two sections from now):

To run the tests you need Docker and Java 21+:

1
$ git clone https://github.com/Mixonn/hibernate-partial-commit-on-shutdown.git
2
$ cd hibernate-partial-commit-on-shutdown
3
$ mvn -B test -Dtest=KafkaPaymentShutdownTest

The 1.2 ms window

There’s one condition. The commit has to land after Hibernate is closed, but before the connection pool is. Once the pool has closed the connection, the commit fails and Postgres rolls back.

In a plain Spring Boot 4.1 app with HikariCP that gap is about 1.2 ms (see for yourself (GitHub, opens in a new tab)), measured on a laptop with a local database3. If the COMMIT goes out in the first ~0.5 ms of the window, it ends as a silent partial commit. Later on, the COMMIT still reaches Postgres, but HikariCP closes the socket before the response arrives, and the app gets Socket closed. So: data saved halfway + an error that invites you to retry something that’s already half done. You can drag the commit in the demo below and see what happens where.

Doesn’t sound like much, but pods get stopped all the time, and the window can get longer with limited CPU, a slow network etc.

Why does nobody notice? First, with short transactions it happens extremely rarely. Second, even when it does happen, the business code sees a successful commit, and in the second part of the window – a connection error while the pod shuts down (i.e. something “normal”). The only log that tells the truth is at TRACE level. In other words, there is none.

What’s left is inconsistent data. We only noticed ours thanks to that event on the DLT. Without it the row would still be sitting there, and any discrepancies would have been written off as “cannot reproduce” or “the code probably had a bug once, now everything looks fine”.

Our case wasn’t critical, but what about when money is involved, or critical business rules for important customers? To me this is unacceptable behaviour from Hibernate.

How to protect yourself

The workaround is a guard that refuses the commit when the factory is already closed. In every transaction it registers a Synchronization, which Hibernate calls after the flush (or after skipping it) and before Connection.commit(). If the factory is closed by then, the guard throws and Hibernate rolls back.

In Spring the easiest way is to make it a TransactionExecutionListener – a single @Component is enough: Spring Boot attaches such beans to the transaction manager on its own, and afterBegin runs at the start of every transaction. The full code is in listing 1.4.

1
@Component
2
public class PartialCommitGuard implements TransactionExecutionListener {
3

4
    private static final Logger log = LoggerFactory.getLogger(PartialCommitGuard.class);
5

6
    /** Called at the start of every transaction. */
7
    @Override
8
    public void afterBegin(TransactionExecution transaction, Throwable beginFailure) {
9
        if (beginFailure != null || !transaction.isNewTransaction()) {
10
            return;
11
        }
12
        try {
13
            for (Object resource : TransactionSynchronizationManager.getResourceMap().values()) {
14
                if (resource instanceof EntityManagerHolder holder) {
15
                    Session session = holder.getEntityManager().unwrap(Session.class);
16
                    session.getTransaction().registerSynchronization(new Guard(session));
17
                }
18
            }
19
        } catch (RuntimeException e) {
20
            // Never throw from here: it would break the start of the transaction.
21
            log.warn("Could not add the guard to the transaction", e);
22
        }
23
    }
24

25
    /** The guard. Hibernate calls beforeCompletion() right before committing the transaction. */
26
    static final class Guard implements Synchronization {
27

28
        private final Session session;
29

30
        Guard(Session session) {
31
            this.session = session;
32
        }
33

34
        @Override
35
        public void beforeCompletion() {
36
            if (session.getSessionFactory().isClosed() && !readOnly()) {
37
                throw new HibernateClosedBeforeCommitException();
38
            }
39
        }
40

41
        @Override
42
        public void afterCompletion(int status) {
43
        }
44

45
        /** In read-only transactions Hibernate writes nothing anyway, so there is nothing to guard. */
46
        private boolean readOnly() {
47
            try {
48
                return session.getHibernateFlushMode() == FlushMode.MANUAL;
49
            } catch (RuntimeException e) {
50
                return false;
51
            }
52
        }
53
    }
54

55
    public static final class HibernateClosedBeforeCommitException extends IllegalStateException {
56

57
        HibernateClosedBeforeCommitException() {
58
            super("Hibernate was closed before the transaction was committed: rolling it back instead of saving only part of the data");
59
        }
60
    }
61
}

The diagram below shows exactly where the guard steps into the commit.

I also ran a test that closed the factory at different moments around the commit – 1000 runs. Without the guard 2–4% of transactions were saved partially, with the guard none.

A closed factory never opens again, and the guard runs after the flush decision has been made, so a skipped flush always hits it. It can be wrong the other way round: if the factory closes right after a full flush, the guard rolls back a transaction that was fine.

It’s your call4 – would you rather have a partially saved transaction, or a rollback of one that could still have made it right before the pod went down?

You can also go back to the demo in the SIGTERM section and turn the guard on.

Where it came from, and what’s next

In 2013 there was an issue, HHH-8586 (opens in a new tab): a JTA commit after EntityManagerFactory.close() triggered a flush, which threw. The fix (fa6a222 (GitHub, opens in a new tab), version 4.3.0.Beta5) changed isClosed() to closed || factory.isClosed(), and ever since then a flush after the factory is closed is silently (yes, TRACE log) skipped instead of ending with an exception and a rollback. The tests for that fix had nothing to flush, so there was nothing to “lose” either, and nobody mentions it in the issue. So my bet is that the data loss is an oversight, not anyone’s decision.

Interestingly, almost every other call on the session after the factory is closed goes through checkOpen(). If there had been one more persist() on the way to the commit, you’d get IllegalStateException: Session/EntityManager is closed and a clean rollback. Only the flush at commit silently skips its job and carries on.

And what does JPA say? The spec (Jakarta Persistence 3.2) doesn’t cover this case directly: EntityManagerFactory.close() says that once the factory is closed, all its entity managers are closed, and EntityTransaction.commit() has “writing any unflushed changes to the database” and a RollbackException for when the commit fails. To me, a commit that can’t write the changes is a failed commit. Hibernate fills that gap – just on the side where data gets lost.

I reported it as HHH-21038 (opens in a new tab) (Improvement, October 22, 2026); at the time of writing it’s still New. I proposed that managedFlush() throw if the session may still have changes (queued actions or managed entities), and keep the silent commit only when there’s nothing to flush – i.e. the HHH-8586 case. If that doesn’t fly, then at least log at WARN/ERROR instead of TRACE.

Both reproducers are public: with H2 on Hibernate’s test case template (GitHub, opens in a new tab) (commit 96be59a) and with PostgreSQL (GitHub, opens in a new tab).

What should I do now?

Don’t panic. This only happens when some thread runs longer than SmartLifecycle allows, and the time window in a plain app is about 1.2 ms (in production services probably longer).

In general it’s worth keeping transactions short. Slow external APIs, HTTP calls to other services, long computations – better before the transaction or after it, with only the database writes inside. Plus timeouts on those calls shorter than the time Spring gives you to shut down.

Except people make mistakes. Someone adds a call to another service “just for a moment”, someone doesn’t notice that a method further up is @Transactional, a timeout stays at its default (which is often infinite5), or a completely different bug – a deadlock, a slow query, a hung client – holds the thread longer than we assumed. And then we run into this problem anyway. That’s why the guard is still the best protection.

So what to do?

1. Check in your metrics how long your transactions take – every long one is a potential candidate for this bug.
2. Move external API calls, HTTP calls to other services and long operations out of your transactions. Where you can’t, set timeouts shorter than the shutdown time.
3. Add the guard (GitHub, opens in a new tab). If your transactions run long – I wouldn’t think twice, I’d add it. If they’re short – no rush, but it’s still worth adding rather than risking losing part of a transaction.
4. Watch HHH-21038 (opens in a new tab) – if Hibernate fixes it, you can remove the guard.

A partial commit, and a silent one too, with no error in the logs, is one of the worst problems that can happen to us. We won’t find the bug in the code, the data will be inconsistent, and on top of that nobody will notice.