The tests were green and the connector was broken: 32 defects the suite could not see

If your Odoo tests pass and the connector fails in production, the reason is almost always the same: the suite does not test the API, it tests your mock of the API. You wrote that mock against the documentation of the day, and it has not changed since. The API has. A .get("shippingDate") whose field is now called something else raises nothing: it returns None forever, and the test covering it stays green because the mock says shippingDate too.

The case: 236 of 236 green on Odoo 19, 18 and 17

We audited one of our own marketplace connectors before putting it in front of a real seller. The suite was 236 of 236 green, and not locally: on real Odoo.sh builds across the three series we support.

It did not work. The audit found 32 real defects underneath the green: 20 of contract and 12 of version. None of the 236 tests touched them, because none of them looked outside the module.

What was actually broken: 20 contract defects and 12 version defects

A contract defect is a mismatch between what the module sends or reads and what the API declares. They came in three flavours:

• A required query parameter that was never sent. The API answers 400 and the failure is loud — in production.
• A parameter no operation in the spec declares. That does not error: it is ignored. The answer comes back unfiltered and looks like it worked. This is the dangerous one.
• A response field read under the wrong name. .get() returns None and nothing ever breaks.

Four of those twenty did not come from reading the spec: they came from calling. The pagination Link header arrived relative, and as-is it was not a valid URL: past page one, not a single order was imported. And a 403 for going over quota did not use the documented error format, so it read as a permission failure: the connector marked the account unauthorised and stopped every sync.

A version defect is something else: the same code behaves differently depending on the Odoo series. These came out of the module itself:

• models.Constraint(...) exists on Odoo 19. On 18 and 17 it is not a constraint at all: the class loads, nothing errors, and the uniqueness guarantee is simply gone.
• res.groups spells its members users before Odoo 19 and user_ids from 19 on. Get the side wrong and the module will not install.
• An ir.cron with no numbercall on Odoo 17 takes the default of 1: the cron runs once and switches itself off. Nobody sees an error; it just stops syncing on Tuesday.
• Odoo 17 spells <tree> where 18 and 19 spell <list>, and it refuses a view whose invisible reads a field the view does not render: 18 and 19 tolerate it.

Notice the pattern: half that list does not error. It goes silent. And silence is caught by no assertion that is not looking for it.

Why the suite missed it: the anatomy of a test that tests itself

A typical connector test has three pieces: a mock returning some JSON, the call to the method, and an assertion on the result. If you wrote that JSON by looking at your own parser instead of the provider's spec, the test is a mirror: it always passes, including the day the API changes.

There is a second kind of empty green, and it is Odoo-specific: the suite never ran. On the Odoo.sh runner, post_install tests only execute on a real install: a -u, or an -i on an already-installed module, runs zero tests and exits 0. And --test-tags "/module:Class.method" selectors match nothing there; only /module works. A badly written pipeline reports “no failures” because there were no tests.

We learned it the uncomfortable way, and it was ours: one of our own conformance checkers reported zero findings because of a bug in itself. It resolved only components/schemas, while that API's 200 responses point into components/responses, so it skipped nearly every endpoint. “Zero findings” and “zero endpoints checked” print identically.

The second case, worse: a double refund with the suite green

Second audit, a different connector, for a marketplace platform. 163 of 163 green, again on all three series and on real builds. Underneath: the call log had never written a single row, the incremental sync asked for the wrong date window, and a refund could go out twice.

The money one is what a mock cannot reproduce. The HTTP client retried on 5xx and on timeout, for any method — including the PUT that sends a refund. A timeout does not tell you the request never arrived; it tells you that you do not know. Retrying a non-idempotent write there is refunding twice. And redirects were followed with method rewriting: a refund PUT could end up as a credential-less GET that answered 200 and was logged as a success. No test saw it, and not out of carelessness: a mock answers first time, every time, with no timeouts, no 502s and no redirects. The branch that does the damage is the one the mock never runs.

The date one fits in a word: the incremental syncs asked for what had been created since the last pass instead of what had changed. An order created in the morning and shipped in the afternoon never re-entered the window, and its state in Odoo froze.

The log one is the quietest of the three. The helper that returns the log model returns a recordset, and when everything is fine that recordset is empty — and in Odoo an empty recordset is falsy. The check read if not log_model: return False, so it hit every time and every “we log this” path was a no-op: no exception, no traceback, no row. The proof that it had been that way from day one came from no test, but from looking at that table's sequence counter in production and finding it at NULL after months of crons.

The three kinds of test that would have caught it

1. Contract against the real spec, not against your mock

You do not need a broker or any infrastructure: keep the provider's OpenAPI in the repository and compare it, statically, against your client. Three gates that take seconds and exit 1 on any finding:

• Query parameters: what each operation declares versus what each wrapper sends. Catches the missing required one and the name no operation declares.
• Write bodies: the payload's top-level keys against the request schema.
• Response parsers: every field name the code reads against the response schema.

The run on that connector: 87 wrappers mapped against 88 spec operations, 20 write paths with a JSON body, 38 parsers over 38 distinct operations (5 multi-endpoint functions not analysable), 0 findings. That zero means something because it carries its denominator printed next to it.

And when the provider offers a sandbox, one real call per endpoint family: it is the only test that separates “the spec says this” from “the server does this”.

2. A real install, not an import (and then, exercise it)

That a module imports says nothing; that it compiles, nothing either. The test is installing it: -i on a clean database and -i on a database that already had the previous version. Two different tests, and the second is the one that blows up on noupdate data and migrations. We take it apart in a module that compiles on Odoo 19 and still will not install.

And after installing, exercise it. We ran 47 apps from our catalogue through clean Odoo 17 and 18 Community installs — install, then call get_views, search_read and onchange: 61 of 94 version runs came out clean. Of the remaining 33, most never got as far as installing, four were artefacts of testing each app without its sibling module, and two installed perfectly and blew up on exercise: an Expected singleton in a product.template compute, and a model assuming Enterprise on a Community database.

3. Verification on all three series, one by one

A module shipping to 19, 18 and 17 is three modules that look alike. We keep the list of landmines above as a static check that runs in one second over the generated port, before spending a build: every entry on it was a real defect and half of them did not fail loudly. It is also the underlying argument when someone compares a native connector against a middle layer: the contract surface does not disappear, it only changes owner. We cover it in Odoo 19 versus middleware connectors.

What to change on Monday

• Keep the provider's spec in the repository, versioned next to the code that consumes it. Its diff is your alarm.
• Put a date and a contract version on every fixture. A mock that does not say what it was written against has already expired, and you do not know it.
• Write a static comparator between your client and that spec. Two hundred lines of Python that exit non-zero.
• Make the pipeline fail if the number of tests executed is zero, or drops against the previous run. “0 failures” means nothing; “0 failures of 236” does.
• Install for real in CI — clean database and previous-version database — and exercise afterwards: open the views and read records through the ORM. And if you support several Odoo series, run the suite on all of them.

And the change that genuinely costs something, which is not technical: “the suite is green” does not answer “does it work?”, it answers “is my code still doing what I thought?”. Two different questions, and confusing them cost us 32 defects in one connector and a latent double refund in another.

If you maintain connectors for clients and want a second reading of what your suite is not testing, this is how we work with consultancies and technical teams and with other Odoo partners.

Useful links within FlexigoTech

Frequently asked questions